Context Windows

The Context window is everything the model can see at once: your messages, its replies and anything else in the conversation.

Metadata is extra background information like the date, your location (from your IP address) or instructions the developer has set in advance.

Every model has a limit for its context window in terms of number of tokens (this includes the input and output responses). For example GPT-4 can hold around 128,000 tokens at once, roughly the length of a novel.

The longer you chat with an AI, the worse it tends to perform. This is called Context rot. This happens because as the context window fills up the model starts losing track of things.

Context window limits have increased in size as LLMs have become more advanced and more people have started using them.

In very large projects, like really big apps, it might be possible for the context window to go over the limit. Developers can reduce the context window by doing things like only including parts of documents at a time rather than sending the whole document.