Does AI Coding Save Time? The 19% Slower Study

A walk through the trial, the rerun, the vendor benchmark and the paper behind the 19% argument

Published 30 October 2026 Updated 30 October 2026

Does AI coding save time, or does it only feel that way? In one controlled trial, experienced developers were 19% slower while using AI, yet finished believing the tools had made them about 20% faster. Below is the evidence behind that gap, chapter by chapter: the randomized trial, the rerun its authors could not run the same way, the vendor benchmark that says the opposite, a survey about trust, and a paper on what ships. Every number here is somebody else's measurement, and the text says whose each time. This channel does not run the studies itself.

Sixteen developers, 246 tasks

The people who got slower were not beginners. METR ran a randomized controlled trial with 16 experienced open-source developers working on large codebases they already contributed to. Together they completed 246 real tasks, about 2 hours each. That design matters for a working reader: it looks like an actual job, not a toy exercise.

The tooling was the early-2025 kind, mostly Cursor Pro running Claude Sonnet models of that time, and the result came out in July 2025. Allowed to use AI, these developers took 19% longer to finish, not shorter. This is METR's own measurement from its own trial, not a vendor's claim and not a benchmark score.

Look at how the time was spent. The participants accepted fewer than 44% of the code suggestions the AI gave them. Most of what the model wrote, they read and then discarded, and reading is time.

The study's main catch is its size and its newcomers. Simon Willison, in his analysis, notes that 56% of the participants had never used Cursor before the trial. Critics lean on exactly that. YouTuber Matt Palmer argues in his review that the study should be largely disregarded, pointing to the small sample and the short Cursor onboarding. On Hacker News, some commenters call the sample too small, while others say their own productivity rose sharply after months of adapting their workflow.

There is nothing to install to follow along: the only address you need is METR's write-up. The study does not publish what the tooling cost the participants, so treat the price of a setup like theirs as unknown here rather than assumed.

Those are fair objections. But the strangest part of the study is not the speed at all.

Feeling fast, being slow

The harder question is whether you would notice being slower. Before starting, these developers predicted AI would cut their task time by 24%. After finishing, they still believed it had sped them up by about 20%. They were in fact 19% slower.

That is the finding worth sitting with. Not that the tool was slow, but that the people using it could not feel it. If experienced developers misjudged the effect on code they knew well, a personal sense of speed is not a stopwatch either.

One participant, writing as Ruby on LessWrong, offers a view on where the time went. In that developer's opinion, getting distracted while waiting on prompts, and fixing simple AI mistakes, ate into the gains. Waiting on a prompt feels like rest rather than lost time. That is one person's explanation, not a measured cause: the study itself does not answer where the time went.

The rerun that broke

METR tried again with newer tools, and the second attempt ran into a problem before it produced an answer. In a February 2026 update, METR said it had to change its experiment design, because developers increasingly refused to do tasks without AI. That skewed who would even sign up.

The late-2025 round had 57 developers, most of them newly recruited. The original participants came out roughly 18% faster, and the newcomers about 4% faster. Both results, METR says, are too noisy to call a reliable speedup, so the honest reading is that this round measured nothing conclusive, in either direction.

Then comes the twist. METR estimates that 30 to 50% of developers admitted avoiding putting the tasks they thought AI would help most into the no-AI condition. The experiment bent around people's belief in the tool. Developers now trust AI enough to refuse working without it, even though nobody can yet measure the gain.

If the careful science is stuck, the next place to look is the companies selling the tools.

Does AI coding save time on a blank page? The vendor's 55%

Vendors have a much sunnier number, and it is worth asking why it does not match. In 2023, GitHub-sponsored Microsoft Research testing found developers with Copilot finished a task 55% faster than a group without it, as GitHub reports it. Note who is speaking: that is a sponsored test published by the vendor, not an independent trial. And the task was a sample HTTP server: fresh, small and self-contained.

Compare that with METR's setting: big, mature codebases you have to understand before you touch them. Both numbers can be true at once. AI is quick on a blank page and slower in a codebase full of history. Which kind of work fills your week decides which number is yours.

Then there is trust. Sonar's State of Code survey of more than 1,100 professional developers (a survey run by a company in the code-quality business, and self-reported by respondents rather than measured) found AI already accounts for 42% of committed code. Yet 96% say they do not fully trust that AI code is functionally correct.

Only 48% say they always check AI-assisted code before committing it. Picture a pull request touching billing code, half of it written by a model nobody fully trusts. Somebody has to read every line, and that reading never shows up in a speed test.

Where the time goes

The saved time appears to go into review: AI makes writing code cheap, and hours pile up waiting for a human to judge it. The clearest evidence is a 2026 NBER working paper that studied over 100,000 GitHub developers, an independent academic analysis of real repository activity rather than a lab task or a survey.

With autonomous coding agents, commit activity rose by a cumulative 180%. That turned into only a 30% increase in releases shipped. The authors call it a weak-link bottleneck in human review and judgement.

They also find that AI and developers are strong complements, not substitutes. In plain words, the model does not replace your time; it moves it, from typing to reading. Amazon landed in the same place this October, telling engineers to measure success by customer and business impact rather than lines of code or token usage.

Here is the verdict. If you mostly write new, small, well-defined pieces of code, the vendor's HTTP server task looks more like your day. If you work in a large codebase you know deeply, the METR result is your warning. Use a clock on your own tasks instead of your gut feeling when the work is unfamiliar code you have to read before you change it. Do not use any of these figures as a forecast for your team: none of this material measures your codebase, and the one attempt to repeat the measurement with newer tools came back too noisy to call. And if you lead a team, stop counting commits, count what ships, and budget for review.

The line to take to your next standup: AI writes the code, but you still ship it.

Our Short on this: Macro: the PDF-editor startup that open-sourced an entire work OS.

Sources