How do we know if an AI is getting better?

When a new AI model comes out, companies need a fair way to compare it against older models and against other companies' models. This is where benchmarks come in.

A benchmark is a big set of questions or tasks that every AI is given so you can compare scores fairly, sort of like NAPLAN for AI.

MMLU (Massive Multitask Language Understanding) is one of the most common benchmarks that all the big AI companies use. It is a test with 57 subjects (like maths, science, history, medicine) from elementary school to university level.