Gathering training data

The training data for LLMs is all in text form. Data comes from a variety of different sources: books, online articles, discussion posts on online forums and more.

In some cases, the data is digitised, like scanning printed books into digital form.

Some data can be non-text data that is turned into text, like a transcript from a YouTube video or podcast.