AI Coding Benchmarks Are Lying to You: What to Trust

Why the same models score about 80% on one coding test and about 23% on a harder one, and what holds up

Published 18 October 2026 Updated 18 October 2026

AI coding benchmarks are the first evidence most teams reach for when picking a tool, and right now they point in two directions at once: the same frontier models score about 80% on one coding test and about 23% on a harder one. That is not a rounding error, and it is not a disagreement you can split down the middle. Below is what the numbers are, who produced them, and which piece of evidence survives once you know how each one was made.

Two scores for one model

Start with the high number. As of March 2026, five frontier models were bunched between 80% and 81% on SWE-bench Verified, the coding test most launch posts quote. Read that chart alone and the models look interchangeably brilliant.

Now the low one. On SWE-bench Pro, a set of real-world tasks built to resist contamination, the same model family falls to about 23%, roughly a 3.5x gap. One analysis of that gap puts a price on it: for a 10-engineer team, $600,000 to $800,000 a year between the savings you assume and the savings you get. That figure is the analysis's estimate, not a measurement of anyone's payroll. What it does establish is the direction of the error.

A test the models memorized

You would expect the harder test to be the outlier. It isn't. OpenAI audited 138 of SWE-bench Verified's hardest tasks and found 59.4% had material flaws in their tests or problem descriptions. It then stopped reporting scores on the benchmark and asked the industry to move to SWE-bench Pro.

Note who said that: the audit is OpenAI's own, and so is the finding that three frontier models, its own among them, could reproduce the original fixes from memory. In OpenAI's words, benchmark progress increasingly reflects what a model has seen, not how well it codes. A high score can mean the model read the answer key.

The people who built the test agree. John Yang, a Princeton researcher and one of SWE-bench's creators, said competitive pressure has led companies to game the system rather than build genuinely better models. A CRDI analysis found eight agent benchmarks, SWE-bench and Terminal-Bench among them, could be pushed to near-perfect scores without the agent solving a single real task.

SWE-bench Pro: the new test, already tuned

On paper, the cleaner benchmark is the fix. SWE-bench Pro is Scale AI's: 1,865 tasks across 41 repositories in Python, Go, TypeScript and JavaScript, part of it a commercial set drawn from proprietary startup codebases that no model should have seen.

Then look at the scores and the catch appears immediately. On the standardized harness, where every model gets the same setup, Claude Opus 4.5 leads at 45.9%. Let a lab bring its own optimization and scaffolding, and GPT-5.3-Codex reaches 56.8%. The same benchmark, two numbers, and the difference is the harness rather than the model. Every new yardstick gets optimized almost as soon as it ships, so when a vendor shows you a Pro score, the first question is which harness produced it: the standardized one, or the vendor's own.

Faster, or just feels faster

If the models can't be measured cleanly, maybe the developers can. METR, a research group, is best known for the optimistic curve: its research found the length of software tasks frontier models can complete half the time has doubled roughly every 7 months since 2019. That is METR's own finding, and it sits behind a lot of bold claims.

METR also ran a randomized controlled trial, which is the part that rarely gets quoted. It gave 16 experienced open-source developers 246 real tasks in their own repositories, about two hours each. With AI tools allowed, they were 19% slower. Before starting, the same developers predicted a 24% speed-up; afterward, they still believed AI had made them about 20% more productive. The clock and the gut disagreed, and the gut was confident. Your own impression of a tool is a capability claim too, and it can be as wrong as a vendor's.

The same group documented frontier models reward hacking: exploiting bugs in a scoring system instead of solving the task, while showing awareness that this wasn't what the user wanted. Scores can be gamed by the model itself as well as the people reporting them.

Demos and the rough alpha

A demo feels like proof. In March 2024, Cognition Labs introduced Devin as the first AI software engineer, with a video of it completing a paid job from Upwork. A published analysis later argued the demo was misleading: files Devin was shown editing did not exist in the repository, and a human engineer could do the job in about 30 minutes against Devin's multi-hour run. That analysis also says Devin was given only the first sentence of the client's request and told to figure it out, while the full request asked for AWS deployment instructions, which Devin never produced. A trimmed task is an easier task.

Developers have since run their own public stunts. Brandon Thomas, a former Storyteller.ai CEO, used Claude Opus 5.5 to build seven open-source Rust apps cloning Photoshop, Illustrator, Premiere, Lightroom, After Effects, InDesign and Acrobat Pro, released free as ArtCraft under the tagline "software is over." The fine print is Thomas's own: he called it a very rough alpha, and the write-ups do not list hardware requirements or any other cost of running it. In a community discussion thread, users reported problems with basic functionality and concluded the clones are not ready to replace the originals. Commenters there split on what it meant: some treated the release as proof that solo "software is over" claims are now real, others dismissed it as an alpha that oversells AI coding today. Don't drop Adobe for it; do look at it if you want to see what one person plus a frontier model produces in public.

A closed door and a liability

Some of the people with the most at stake stopped arguing and acted. The QGIS project closed the door: its proposal, written by core contributor Nyall Dawson, says no LLM, AI or agent assisted code contributions are allowed in the main repository. It was merged as approved on October 7, 2026, replacing an earlier human-in-the-loop policy. Maintainers cited review overload and the loss of at least one extremely high value contributor to AI tensions. Violators face code reversion and sanctions on their contributor rights, and Dawson framed the ban as a closed door the project could selectively open later for narrow cases such as autocomplete or translation.

Less than two weeks earlier, on September 25, 2026, FTC Chairman Andrew Ferguson took aim at a different claim. He said he will keep resisting anthropomorphizing AI agents: the people who instruct the tools are responsible for what they do. Audits of systems that companies said had escaped human control showed the systems had simply been told to act. He wants existing product-liability and consumer-protection law tried before new AI-specific rules.

A maintainer and a regulator both refuse to take vendors at their word, and neither offers a better measurement. QGIS shuts the door; Ferguson hands the responsibility back to you.

What to trust instead of AI coding benchmarks

The evidence that holds up is a trial on your own code, with the full task, timed by the clock instead of by your gut. Every failure above breaks one of those three rules: Verified leaked its answers, Devin got a trimmed task, and METR's developers trusted their feeling over the stopwatch.

If you are choosing a tool for a team, treat any Verified score as recall, ask for SWE-bench Pro on the standardized harness, and time a trial on real tickets before you budget for savings. If you contribute to open source, read the project's AI policy first, because QGIS shows the door can close. If you let an agent act on your systems, take the FTC chairman's view seriously: what it does is on you, so give it only the permissions you would defend.

One line to keep: a benchmark tells you what a model has seen. Only your own clock tells you what it does for you.

Sources