Tokenisation

The first step in training is taking our training data and "tokenising" it. Tokenisation is breaking down sentences, words, letters, numbers and punctuation marks into tokens.

There are different ways to tokenise sentences, such as separating words by spaces in a sentence. For example, the sentence: "The uncharacteristically over-enthusiastic scientist waved hello." could be split up by spaces:

Splitting up words by spaces and making these tokens is not the best approach. Especially when a LLM needs to support multiple languages. When writing in Chinese language, there are no spaces between words, for example: 新年快乐 translates to "happy new year" in Chinese.

When training LLMs, words often get broken into sub-words or - in some cases - just one letter, number or punctuation mark (like I, 2 or a full stop).

There are different ways of tokenising and different LLMs will do this differently.