Lesson 5: Under the Hood
OpenAI's Tokeniser
OpenAI (the makers of ChatGPT) have an online tokeniser. You can use this to see how their GPT models tokenise words and sentences. You can find this site here: platform.openai.com/tokenizer
The tokeniser used by GPT 5 models splits up the sentence "The uncharacteristically over-enthusiastic scientist waved hello." into these tokens: