Naru News日本語

Home › Glossary › AI

What is a benchmark in AI (a shared test for comparing AIs)?

Also called: AI benchmark, LLM benchmark

Term explainer · Checked November 16, 2026 · 15 min read

In one minute

In AI, a benchmark is a shared test that pairs a fixed set of questions with a fixed way of scoring the answers; many different AIs are given the same questions, and their scores are used to compare what they can do.

For example, MMLU, built in 2020, is a test of multiple-choice questions (pick one of four answers) in 57 subjects such as math, history and law, and the score is the share answered correctly (accuracy). On the same test, AIs made by different companies can be lined up by score. When companies release a new AI, they often publish a table of benchmark scores.

But a score shows only part of what an AI can do. Once many AIs score near full marks (saturation), the test no longer tells them apart, and if a test’s questions slip into the data an AI learns from (its training data), its score can come out higher than its real ability.

A common mix-up

“The higher an AI’s benchmark scores, the better it is at any job.”

Only partly right. On the same test, it is fair to think that the higher-scoring AI does better on the kind of questions that test measures. What is wrong is “any job.” Most benchmarks measure questions whose answers can be checked in a fixed way, such as multiple-choice questions, math problems with a set numerical answer, or program fixes where running checking programs shows whether the fix really worked. Some researchers point out that fixed questions struggle to measure how useful an AI is with open-ended requests (questions or instructions given to an AI) that have no single right answer. And when a test’s questions slip into the data an AI learns from, its score comes out higher than its real ability. In a 2024 study, when people wrote new math word problems about as hard as those in an existing test, some AIs’ accuracy dropped by up to about 8 points on them (points here are the difference between two percentages: going from 50% to 42% is a drop of 8 points). A score is a guide to the ability that test measures, not the whole of what an AI can do.

Explained in pictures

A diagram of how a benchmark works (the scores are made up). Title: “Same questions, same scoring: compare scores.” On the left, inside a dotted orange frame labeled “Benchmark,” a stack of sheets marked “Q1,” “Q2,” “Q3,” labeled “Fixed questions,” and a sheet with ○ and ✕ marks and a pen, labeled “Fixed scoring.” Three arrows run from the stack to three robots in the middle, under “Same questions,” with A, B and C on their bodies. Each robot holds an answer sheet, and an arrow marked “Score” leads to the “Score table” on the right. The table has horizontal bars: AI A 75% (navy bar), AI B 88% (orange striped bar), AI C 62% (white dashed bar). Below: “Score = share answered correctly.” Band at the bottom: Same test, so AIs from different makers can be compared.
This diagram shows how a benchmark works. The dotted frame on the left is the benchmark (the shared test): it bundles a fixed stack of questions with a fixed way of scoring, shown by the sheet with ○ and ✕ marks (the right answers and how points are given). The arrows show the same questions going out to three AIs (the robots A, B and C), and each one’s answers being scored the same way. On the right is a table of the resulting scores, where the score is the share of questions answered correctly (accuracy). Because the questions and the scoring are the same, AIs from different makers can be lined up and compared by score. But what can be compared is only the kind of ability this test measures.
A chart of scores on a test nearing the top, and a harder test being made. Title: “As scores near the top, harder tests take over.” Subtitle: Points from papers and a company report (lines just join them). Up is “Accuracy (share answered correctly),” from 0% to 100%; across is 2020 to 2025. A striped band from 90% to 100%: “Near the top: hard to tell AIs apart (saturation).” A dotted line at 25%: “Guessing on 4 choices: 25%.” A solid navy line with dots, “MMLU (57 subjects)”: GPT-3 (2020) 43.9%, GPT-4 (2023) 86.4%, and an upward arrow from the early-2025 dot with “Top AIs over 90%.” At bottom right, a short orange striped bar with caps at the top and bottom: “HLE (2025) 2.7–13.4%.” A dotted orange curved arrow, “On to a harder test,” runs from the over-90% dot to HLE. Band at the bottom: Near the top, gaps vanish, so harder tests are made.
This chart shows how tests become too easy as AIs improve. Up is accuracy and across is the year; the points are values from papers and a company report. The solid navy line with dots is MMLU, a test of multiple-choice questions in 57 subjects. GPT-3 (the largest of its several size versions) scored 43.9% in 2020 and GPT-4 86.4% in 2023 (the company’s report), and by early 2025 the top AIs were over 90%, inside the band at the top where AIs are hard to tell apart. The dotted line is the 25% you get by guessing one of four. The orange bar at bottom right is HLE, a test built to be much harder, on which even the top AIs of the same period scored 2.7% to 13.4%. The curved arrow shows the move from a test whose scores are near the top to a new, harder one.
A figure of test questions slipping into training data, with numbers from a 2024 study. Title: “When test questions slip into the training data.” Subtitle: Left: diagram. Right: numbers from a 2024 study (math word problems). Left: in a pile of overlapping sheets, “Training data (huge amounts of web text, etc.),” one orange sheet marked “Q” and “A” is mixed in, and a tag, “Test questions + answers,” points to it. An arrow, “Learns,” runs from the pile to a robot, whose thought bubble says “Memorized!” Under the robot, a box: “Scores higher on the old test.” Right, past a dotted line: a bar chart of “Accuracy (share answered correctly),” with navy bars, “Old questions (GSM8K),” and orange striped bars, “Newly written questions (1,205).” One AI: 43.7% and 35.7%, with “−8.0 points” below. GPT-4: 91.1% and 92.3%, with “About the same” below. Band at the bottom: New questions of equal difficulty can reveal inflated scores.
This figure shows data contamination, when a test’s questions slip into the data an AI learns from. The left is a diagram: test questions and answers (the orange sheet) are mixed into the huge pile of text an AI learns from, and the AI learns them too and memorizes the answers. Such an AI scores higher on the old test than its real ability. The right shows numbers from a 2024 study: navy bars are accuracy on GSM8K, an existing test of math word problems, and striped bars are accuracy on 1,205 problems people newly wrote at about the same difficulty. One AI went from 43.7% to 35.7%, a drop of 8.0 points (the difference between the two percentages), while GPT-4 scored about the same. The researchers think part of such drops likely comes from having learned the old problems. They write that other things, such as collecting training problems similar to the test, may also be causes.

You’re free to use these figures in lessons, slides and posts if you credit Naru News (naru.news). CC BY 4.0.

Think of it like this

A nationwide practice exam with the same questions for everyone

Imagine a practice exam that junior high school students across the country take, with the same questions and the same time limit. Because the questions and the marking are the same, students from different schools can be lined up by their scores. A benchmark is like this practice exam, but for AIs.

But if the questions and answers had been passed around beforehand, the score alone couldn’t tell you whether a high scorer really had the skill or had just memorized the answers. And once everyone scores close to full marks, the exam no longer separates them, and a harder one is needed.

But the comparison has limits. A practice exam for people is written fresh each time, while AI benchmarks often keep the same questions for years so that new AIs can be compared with older ones, which gives the questions more chance to spread. And because AIs learn from huge amounts of web text, if the questions appear somewhere online, an AI can learn them without even its makers noticing. Unlike a student, it hasn’t necessarily memorized the answers on purpose.

Key points

Why is it called a “benchmark”?

The English word benchmark started out as a surveying term. When land heights are measured, a benchmark is a mark made on something that doesn’t move, at a spot whose height is already known, and surveyors use it as the reference for working out the heights of other places. In Japan, marks that do the same job are called suijunten (“level points”); about 16,000 of them have been set nationwide, mainly along major national roads, to serve as the standard for heights.

Here, bench means a small support. An American English dictionary explains that the name comes from fitting a metal piece into the mark to make a little bench, on which the tall measuring staff for heights was rested.

Today benchmark is also used in a broader sense: a standard by which something can be measured or judged. In investing in stocks and the like, a yardstick for comparing results (such as Japan’s Nikkei Stock Average) is called a benchmark, and in computing, a fixed test for measuring and comparing performance under the same conditions is called a benchmark too. AI benchmarks, likewise, are a shared standard for holding every AI up to the same yardstick.

How is a benchmark built?

A benchmark is built by putting together a set of questions, the right answers (or a way to check whether an answer is right), and a way of scoring. The scoring is fixed in advance so that whoever tests whichever AI, the score comes from the same yardstick. The score is usually the share of questions answered correctly (accuracy).

MMLU, from 2020, gathered 15,908 multiple-choice questions in 57 subjects, ranging from elementary to advanced professional level. With four choices, even guessing gets 25% right. Its creators tested GPT-3 (a text-writing AI that OpenAI announced in 2020). GPT-3 came in four versions of different size (size here means how many numbers, or parameters, the AI has inside), and the three smaller ones scored around 25%, about the same as guessing. The largest version scored 43.9%, about 20 points better than guessing, but far short of experts. Expert accuracy was estimated at about 89.8%, from the accuracy of the top 5% of people taking the real exams the questions came from (with educated guesses where that wasn’t known).

Not every benchmark uses multiple choice. GSM8K, from 2021, is a test of about 8,500 grade-school math word problems, each solvable by doing 2 to 8 calculations in a row, and an answer is scored only on whether the final number is right. SWE-bench, from 2023, is a test of 2,294 tasks that ask an AI to fix real problems reported in 12 widely used programs (with code anyone can read). The program with the AI’s fix is run through checking programs (tests) prepared in advance, and the task counts as solved only if every one of them shows it working correctly.

A shared test lets researchers compare their ideas with the same yardstick. In a contest to name the object in a photo out of 1,000 kinds (the ImageNet contest), research teams competed on the same test year after year. The error rate (the share of photos for which none of an AI’s up to five guesses was right; lower is better) fell from 28.2% in 2010 to 6.7% in 2014. The test answers were not shown to the teams; they sent in their AIs’ answers, and the organizers did the scoring.

Why do new tests keep being made?

Because as AIs improve, tests become too easy to tell them apart. When many AIs score close to full marks, the test is said to be saturated. On MMLU, the 43.9% the largest version of GPT-3 scored in 2020 became 86.4% for GPT-4 in 2023 (the company’s report), and by early 2025 the top AIs were scoring over 90%. Older tests such as MMLU and GSM8K, the math word-problem test, now hardly separate the top AIs.

So harder tests are made. Humanity’s Last Exam (HLE), released in 2025, is a test of 2,500 questions written by experts around the world, in many subjects including math, the humanities and the natural sciences (the first released version had 3,000). While collecting questions, its makers tried them on the leading AIs of the time and left out any they answered correctly. In the paper’s table, eight leading AIs of early 2025 scored between 2.7% and 13.4%.

New tests don’t stay hard forever, either. On SWE-bench, created in 2023, the share of tasks the best AI could solve rose from 4.4% at the end of 2023 to 71.7% at the end of 2024. The 71.7%, though, was on a subset of problems checked and selected by people (SWE-bench Verified), and it was reported by the company that made the AI.

HLE’s creators also write that benchmarks saturate quickly. They add that HLE may be the last school-style exam, with set answers, that AIs need to take, but that this is far from the end of tests for measuring AI.

How far can the scores be trusted?

If a test’s questions and answers slip into the data an AI learns from (its training data), the AI can memorize the answers instead of learning how to solve the problems, and its score comes out higher than its real ability. This is called data contamination. Today’s large language models (text-writing AIs like those behind ChatGPT) learn from huge amounts of web text, so if a test’s questions appear somewhere online, they can get in without the makers noticing. In its 2023 GPT-4 report, OpenAI writes that parts of one test had been mixed into the training data by accident, so it left those results out (the company’s report).

One way to check for contamination is to write new questions of the same difficulty and compare. In a 2024 study, people wrote 1,205 new math word problems modeled on GSM8K, and many AIs were given both the old and the new problems. Some AIs scored up to about 8 points lower on the new problems (8 out of every 100), and the researchers see this as a sign that those AIs had been fitted too closely to the old test (overfitting). They also measured how smoothly each AI could write out GSM8K’s problems (the AI was made to predict each problem’s text from the start, word by word; the more often it matched the original, the more likely it had read that text before). The more smoothly an AI could do so, the bigger its drop tended to be. But that didn’t explain every case, and the researchers write that other things, such as collecting training problems similar to the test, may also be causes. Meanwhile, most of the most advanced AIs of the time scored about the same on both tests. GPT-4, for example, scored 91.1% on the old problems and 92.3% on the new ones, a difference small enough to count as about the same.

Test makers take steps too. The photo contest (the ImageNet contest) scored entries without publishing the test answers. HLE keeps part of its questions private, to check whether AIs have been tuned too closely to the public questions, and the 2024 math study did not publish its new problems right away, so that they wouldn’t slip into training data. SWE-bench is designed so that new tasks can be added from problems reported after an AI finished training.

An AI can also notice that it is being tested. In a 2025 study, AIs were shown exchanges from when an AI was solving test questions and exchanges from when people were really using an AI, and leading AIs told which ones came from tests far better than guessing. If an AI behaves differently only when tested, its score doesn’t show how it acts in real use (see the “Evaluation awareness” page).

How should a table of scores be read?

In short, a score is “the result on that test, measured that way.” Compare only scores from the same test under the same conditions, and also check who measured them (the company that made the AI, or someone else) and whether the test is close to how you would use the AI.

First, look at which tests are listed. The side releasing an AI chooses which tests to include, so a table could end up showing mostly tests that AI is good at. A 2022 study writes that, until then, some pairs of well-known AIs had not a single test score in common (so their scores could not be lined up side by side).

Next, check whether the scores were measured under the same conditions. Even the same AI scores differently depending on how it is asked (how the questions or instructions to it are written) and on the details of the scoring. In the 2024 math study, just changing the worked examples included in the request and the way the answer was phrased lowered one AI’s accuracy on the same math word-problem test by about 5 points. When lining up scores measured in different places, you need to check that the conditions match.

When raising the score becomes the goal in itself, the score and the real ability can drift apart. In AI research too, it has been pointed out that leaning too hard on metrics leads to gaming them and to chasing short-term numbers. This is a view known as “Goodhart’s law,” often quoted as “When a measure becomes a target, it ceases to be a good measure.” It goes back to an idea the economist Charles Goodhart wrote in 1975 about British monetary policy; the short wording is said to have been written later by another researcher (see the “Reward hacking” page).

Finally, scores don’t always match how an AI feels to use. Tests with fixed questions struggle to measure open-ended requests with no single right answer, or whether answers suit people’s preferences. So Chatbot Arena was built with no fixed questions: users ask anything they like, compare the answers of two AIs whose names are hidden, and vote for the better one (it started in 2023 and was described in a 2024 paper). By January 2024 it had gathered over 240,000 votes. When two experts also judged a sample of the matchups, they picked the same answer as the users’ votes about 70 to 80 percent of the time. The two experts agreed with each other about as often, so the researchers see the users’ votes and the experts’ judgments as agreeing well.

How is it different from red teaming?

A benchmark is a shared test that scores AIs on fixed questions so they can be compared. A red team doesn’t use fixed questions; it takes an attacker’s point of view and looks on purpose for an AI’s weak spots and dangerous behavior (see the “Red teaming” page). Benchmarks suit comparing how much different AIs can do, while red teaming suits finding dangers no one expected.

The two are connected. OpenAI writes that it uses requests tried by its red teamers as the basis for automated tests that can be run the same way again and again (the company’s account). Weak spots found by a red team can later become a fixed-question test.

There are also yardsticks that express ability in a form other than a score. The “time horizon” expresses how long a task an AI agent can complete, in terms of how long it would take a person. For example, if an AI succeeds half the time at tasks that take a human expert one hour, its time horizon is “one hour” (see the “Time horizon” page).

Check yourself

  1. Q1MMLU, a test built in 2020, is made of questions in 57 subjects such as math, history and law, each asking you to pick one of four answers. Its creators tested four versions of GPT-3, a text-writing AI, that differ in size (how many numbers, or parameters, the AI has inside). Roughly what accuracy (share answered correctly) did the largest version get?

    Answer and explanation

    Answer: A About 44%

    The largest version of GPT-3 scored 43.9%, about 20 points better than guessing (25% for picking one of four). The three smaller versions scored around 25%, about the same as guessing. Even so, it was far below expert accuracy (estimated at about 89.8%). Later, the top AIs came to score over 90%, and the test hardly separates them anymore.

    More on this → Key points: How is a benchmark built?

  2. Q2In a 2024 study, people wrote new math word problems about as hard as those in GSM8K, an existing test of grade-school math word problems. The most advanced AIs of the time (such as GPT-4) and many other AIs were given both the old and the new problems. How did their scores on the new problems compare with the old ones?

    Answer and explanation

    Answer: C The most advanced AIs stayed about the same, and some of the other AIs clearly dropped

    Most of the most advanced AIs of the time scored about the same on both tests. GPT-4, for example, scored 91.1% on the old problems and 92.3% on the new ones. Some of the other AIs, though, dropped by up to about 8 points on the new problems, and the more smoothly an AI could write out the old problems, the bigger its drop tended to be. A high-scoring AI is not always one that memorized the answers.

    More on this → Key points: How far can the scores be trusted?

  3. Q3SWE-bench, a test created in 2023, asks AIs to fix problems reported in real programs. Comparing the best AI of each year, the best AI at the end of 2023 could solve 4.4% of the tasks. What was the best result at the end of 2024 (reported by the company that made the AI, on a subset of problems selected by people)?

    Answer and explanation

    Answer: B It rose to over 70%

    By the end of 2024, the company that made one AI reported that it solved 71.7% of that human-checked subset. Even a new test built to be hard can see scores jump in a short time as AIs improve. Because tests quickly become too easy like this, harder ones keep being made.

    More on this → Key points: Why do new tests keep being made?

Sources and how we checked

AI wrote this page from the sources above, and a different AI checked it against the text of those sources (November 16, 2026). A person with a master’s degree in science and a Japanese high-school science teaching license also read and checked it before publishing. Numbers are as of that date. If anything was hard to follow, you spot a mistake, or there’s a term you’d like explained, please write to [email protected].

AI terms

All 58 AI terms →