Vocabularies

Our vocabularies are the words that we know and understand.

For example, you probably know the words "apple", "sentence" and "swim" and what they mean.

You can learn new words in lots of ways, for example, by reading books, talking with friends and in school.

LLMs' vocabularies are learned during their training. Although they learn tokens and not words.

LLMs store the tokens as numbers and use them in their calculations, e.g. "the" could be stored as the number 121, like in the picture below.