Training data in tokens

LLM developers do not usually share how big their training data is in number of words.

But we can estimate the number of words, when we understand how tokenising works. Each token is roughly ¾ an English word (according to OpenAI).

Meta (who own Instagram and Facebook) have their own LLM called LLama. Meta shared that their LLama 3.1 LLMs were trained on 15 trillion tokens, which is roughly 11 trillion words.

The animation below, created by Dan Taylor-Watt, compares the number of words humans read in their lifetime (300 million) to the words LLama LLMs read.