Temperature

The 'temperature' setting of an LLM controls how creative or random the generated output of the LLM is. It varies between 0 and 2.

At low temperature, the LLM almost always picks the highest probability word. Every time you ask the same question you get very similar answers.

In the previous example:

with the probabilities of the next token as:

At low temperature, the model would almost always pick "cat" as the next token, because it has the highest probability.

"Higher" temperatures will increase the range of tokens and might give:

Higher temperatures give the lower probability words a much better chance of being chosen. The results get more surprising, more creative and sometimes more nonsensical.

Use High temperature when you want to be creative:

• Writing a short story or poem
• Brainstorming ideas for a product name
• Generating jokes or creative slogans
• Coming up with unusual solutions to a problem

Use Low temperature when you need accuracy, consistency, or a single correct answer:

• Answering a factual question
• Summarising a document
• Writing code
• Giving step-by-step instructions