Lesson 5: Under the Hood
Reasoning Models
A standard LLM generates one token at a time, in a single pass through the transformer.
It has no ability to pause, reconsider or check its own work.
A reasoning model adds a reasoning phase before the final response. It's like a scratchpad where the model works through the problem step by step before committing to an answer.
They are usually used for tasks that require careful step-by-step thinking like complex coding or multi-step logic problems.
ChatGPT5.5 in Thinking Mode is an example of a reasoning model and when given this simple prompt, it replied with the steps it used to calculate the answer.
"A shop sells apples for $2 each. You buy 3 apples. How much do you pay? Think step by step and explain your thinking."
Extra Training for Reasoning Models
On top of the standard LLM pre-training, reasoning models go through additional training steps:
- Learning from examples: the model is shown extra examples which include the step-by-step reasoning involved.
- Reinforcement Learning from Human Feedback: Human raters score the quality of the reasoning steps provided as well as the output.
- Reward-based training: For tasks like maths and coding, the model's answers can be automatically checked; either the answer is right or it isn't. The model gets rewarded for correct answers and penalised for wrong ones, training it to develop better reasoning strategies over millions of attempts. This is a form of reinforcement learning.