Lesson 5: Under the Hood
Token IDs
All the tokens in the tokenisation step become numbers. For example, IDs for the sentence "The uncharacteristically over-enthusiastic scientist waved hello." are shown in the table below.
| Token | ID |
|---|---|
| The | 976 |
| un | 537 |
| character | 38245 |
| istically | 67862 |
| over | 1072 |
| - | 12 |
| enth | 33774 |
| usi | 9955 |
| astic | 5620 |
| scientist | 57204 |
| waved | 135473 |
| hello | 40167 |
| . | 13 |
These IDs have been generated using the OpenAI tokeniser.
There are 200,000+ tokens in the tokeniser used in OpenAI's GPT 5 models (this is their vocabulary).
You will learn more about how these Token IDs are used later in the lesson.