AI Coding Tools and Developer Productivity: What the Best Evidence Actually Shows




Do AI coding tools make developers faster? Sometimes, on some kinds of work, by an amount nobody has measured reliably since mid-2025. The best-known controlled result, METR’s 19% slowdown for experienced open-source developers, describes early-2025 tools, and METR itself now labels it out of date. Its 2026 follow-up leaned toward a speedup but was too distorted by selection effects to put a number on it. What has held up is more useful than any single figure: developers misjudge their own speed, individual gains often stall in code review, and trust in AI output is conditional on being able to verify it.
Updated November 18, 2026 · Reading time: about 14 minutes
Key takeaways
- The 19% slowdown is a snapshot, not a verdict. It covered tools from February to June 2025. METR’s own 2026 update says the follow-up data gives an unreliable signal, and that developers are likely more sped up now than a year earlier.
- The perception gap is the finding that survived. Developers predicted a 24% speedup, believed afterward that they had gained about 20%, and were measured at 19% slower.
- Faster individuals do not automatically mean faster delivery. Telemetry from Faros AI and the 2025 DORA report both show gains in output colliding with review load and delivery instability.
- Trust is conditional. In Stack Overflow’s 2026 survey, 48% of developers trust AI output when they can easily verify it, and only 6.6% trust it for important work decisions.
- Learning has a cost. In a randomized trial, developers who used AI to learn a new library scored lower on a comprehension quiz than those who coded by hand.
- The right response is to measure your own results, with a design that does not let people quietly opt out of the hard tasks.
The number everyone quotes, and why it needs a date stamp
In July 2025, METR published a randomized controlled trial that became the most cited counterweight to vendor productivity claims. Sixteen experienced open-source developers worked on 246 real issues in repositories they knew well, averaging more than 22,000 GitHub stars and over a million lines of code. Each issue was randomly assigned to “AI allowed” or “AI disallowed.” With AI, tasks took 19% longer. The developers had forecast a 24% speedup before starting, and afterward still believed AI had made them about 20% faster.
Two details matter for anyone repeating this result.
First, the study measured early-2025 tools. According to METR’s paper, the main tool was Cursor Pro with Claude 3.5 and 3.7 Sonnet, which is a long way from the agentic tools many developers use today. METR has since added a banner to the original post stating that the results are out of date.
Second, the setting was deliberately demanding: experts, their own mature codebases, and tasks they already understood. That is plausibly close to the worst case for AI assistance, and the authors framed it as one snapshot rather than a general law.
What METR measured the second time
In August 2025, METR started a larger follow-up with 57 developers across 143 repositories and more than 800 tasks. Ten developers returned from the original study; 47 were newly recruited from a more varied set of projects. On February 24, 2026, METR published the results under a telling headline: it was changing its experiment design.
On METR’s scale, which expresses the change in time a task takes, positive numbers mean AI-allowed tasks took longer, and negative numbers mean they took less time.
| Study | Who | Change in task time with AI | Confidence interval |
|---|---|---|---|
| 2025 RCT (early-2025 tools) | 16 developers, 246 issues | +19% (slower) | +2% to +39% |
| 2026 follow-up, returning developers | 10 developers | −18% (faster) | −38% to +9% |
| 2026 follow-up, new developers | 47 developers | −4% | −15% to +9% |
The pattern in the intervals is the real story. The 2025 interval excludes zero, so that result was statistically distinguishable from no effect. Both 2026 intervals cross zero, so on their own they cannot separate a speedup from no effect. METR’s raw numbers point toward a speedup, but METR does not present them as a reliable measurement.
A warning about secondary coverage. Because METR reports a “speedup” of −18%, many blog summaries read the sign backwards and called it an 18% slowdown, while others rounded it up into an “18% productivity boost.” Neither is what METR wrote. Read the primary post before quoting any of these figures.
Why METR distrusted its own follow-up
METR listed the problems itself, which is what makes the update credible:
- Developers refused the no-AI condition. More developers said they would not want to do half of their work without AI, so the study was missing exactly the people who expect the largest benefit.
- Developers withheld tasks. In surveys, 30% to 50% said they skipped submitting some tasks because they did not want to do them without AI. Those are the tasks where AI likely helps most.
- Pay dropped. Participants were paid $50 an hour instead of $150, which probably worsened selection.
- Time measurement got harder. Developers running several agents at once often worked on something else while an agent ran, so “time on task” became murky.
- Some tasks were not completed. One developer finished none of the tasks assigned to the AI-disallowed condition.
METR concluded that its estimate is likely a lower bound on the true effect for these developers. It also noted that this design will keep degrading as AI gets better and developers’ expectations rise. The planned alternatives include shorter, more intensive experiments, observational data, fixed-task designs, and randomizing at the developer level.
What this means for you: if a headline tells you AI makes experienced developers slower, check the date of the tools. If it tells you AI makes everyone dramatically faster, check whether anyone measured it or only asked.
Feeling fast versus being fast
The one result that nobody has overturned is the perception gap. METR’s 2025 study found developers overestimated AI’s effect on their own task time by roughly 40 percentage points. The 2026 follow-up did not formally re-measure perception.
METR’s own May 2026 survey shows how large self-reported gains have become. Across 349 technical workers surveyed from February to April 2026, the median self-reported change in the value of their work was between 1.4x and 2x, and the median self-reported speed change was 3x. Respondents recalled about 1.3x for March 2025, rated March 2026 at about 2x, and forecast about 2.5x for March 2027.
METR was candid about the limits. The sample was a convenience sample with response rates near 2% among people it emailed. It reviewed the seven respondents who reported 10x or more on at least two measures, and in the two cases where it could inspect public output, it judged the claims overstated. METR staff themselves gave the lowest estimates, which METR suggests may reflect familiarity with its own perception-gap finding.
The survey also draws a distinction worth borrowing: speed is not value. If AI makes it cheap to build a dashboard you would never have prioritized, your task times collapse, but the value of your work barely moves. METR calls this task substitution, and it is one reason time-based measures can overstate real gains.
When individual speed stalls at the team level
Even if an individual developer is faster, the team may not ship faster. Two independent bodies of evidence point at the same bottleneck.
Faros AI telemetry
Faros AI, an engineering analytics vendor, published telemetry from more than 10,000 developers across 1,255 teams in 2025. On teams with high AI adoption, developers completed 21% more tasks and merged 98% more pull requests, but review time rose 91%, pull requests grew 154% larger, and bugs per developer rose 9%. Company-level delivery metrics did not move.
Its 2026 report, The Acceleration Whiplash, covers about 22,000 developers across more than 4,000 teams over two years. Comparing low-adoption with high-adoption settings, Faros reports median time in pull-request review up about 441%, median time to first review up about 157%, bugs per developer up 54%, and incidents per pull request more than tripled.
Two honest caveats apply. This is observational telemetry, not a randomized trial, so it cannot prove AI caused the changes. And Faros sells software that measures exactly these problems, so treat the framing with appropriate skepticism while taking the underlying direction seriously.
The DORA reports
Google’s DORA team surveyed nearly 5,000 respondents between June 13 and July 21, 2025. About 90% reported using AI at work, and more than 80% said it improved their productivity. Roughly 30% said they trust AI-generated code little or not at all.
The structural finding is the interesting one. In 2024, DORA had associated higher AI adoption with lower delivery throughput and lower stability, specifically 1.5% less throughput and 7.2% less stability for each 25% increase in adoption. In 2025, the throughput relationship turned positive, but AI adoption was still associated with higher delivery instability. DORA’s framing is that AI amplifies what an organization already is: strong foundations gain, weak ones see their problems accelerate.
DORA’s 2026 follow-up on the return on AI investment adds a timing dimension, describing a J-curve in which productivity dips during adoption, because of learning and workflow adjustment, before gains arrive. Teams that judge a tool at the bottom of that curve may draw the wrong conclusion in either direction.
Trust: adoption is high, confidence is conditional
Stack Overflow’s 2026 Developer Survey (30,903 responses from 169 countries, published October 2026) reports that 66% of respondents use coding assistants or agents, 63% use general-purpose chat tools, and 26.2% use automated agent workflows. Only 17.2% say they use no AI tools.
The trust question changed this year, so do not compare the numbers directly with 2025:
| Survey | What it asked | Result |
|---|---|---|
| 2025 | Trust in the accuracy of AI output | 46% distrust, 33% trust, 3% “highly trust”; positive sentiment fell to about 60% |
| 2026 | How much developers trust AI output in their workflow (n = 14,304) | 48.0% trust it when they can easily verify it; 6.6% trust it for many tasks including important decisions; 12.3% do not trust it for most tasks |
Developer behavior matches the stated attitude. Most developers verify AI output by running it locally (76.5%), and only about 10% use answers as they come. Just 16% say AI handles complex tasks very well, while 39% say it handles them well but not great. On governance, only 24% say their organization publishes approved tools or an AI policy, and 30% say AI use is optional and left to the individual.
The practical reading: the binding constraint on AI’s value is not generation but verifiability. Tasks where output can be checked cheaply (tests, type checks, small diffs) capture the benefit. Tasks where it cannot (architecture, concurrency, cross-file refactors) push cost into review.
Does AI-generated code lower quality?
The evidence is suggestive but not conclusive, because nearly all of it is observational.
- GitClear analyzed more than 200 million changed lines of code and reported a shift toward more copy-pasted code and less refactoring since AI assistants arrived. It is an industry report, not peer-reviewed. MIT Technology Review also noted GitClear’s finding that engineers produce roughly 10% more durable code since 2022, so the picture is mixed rather than uniformly negative.
- Faros and DORA both associate heavy AI adoption with more bugs or instability, with the causal direction unproven.
- MIT CSAIL researchers catalogued the failure modes behind this in a 2025 paper: generated code can call functions that do not exist, ignore project conventions, or pass shallow tests while hiding logic that fails in production.
The cost nobody puts on the dashboard: skill formation
In January 2026, Anthropic researchers Judy Hanwen Shen and Alex Tamkin published a randomized controlled trial of 52 developers learning an unfamiliar Python library (Trio, an asynchronous programming framework). Disclosure: Anthropic makes Claude and Claude Code, and funded this research.
Task time was similar, roughly 23 minutes with AI against 25 without, and the difference was not statistically significant. But on a follow-up quiz taken without AI, the AI group averaged about 50% against about 67% for the hand-coding group, with the largest gap on debugging questions.
How people used AI mattered more than whether they used it. Developers who delegated code generation wholesale, or who repeatedly pasted errors back into the assistant, scored below 40%. Developers who asked conceptual questions or requested explanations alongside generated code preserved much more understanding, scoring 65% or higher.
The study is small and tests learning one library, so it does not show that AI degrades expert skill over years. It does show that the “learning a new tool” moment, which is exactly when AI feels most tempting, is where a speed gain can be traded for comprehension.
What the evidence supports, task by task
Earlier versions of this article rated task categories “strong” based on practitioner consensus. That is not evidence in the same sense as a trial, so this table separates what was measured from what is widely believed.
| Type of work | What the evidence says | How much to trust it |
|---|---|---|
| Small, self-contained task | In a 2023 controlled experiment, developers using GitHub Copilot finished a JavaScript HTTP-server task 55.8% faster | One task, one language, older tools: directional |
| Experienced developer, large familiar codebase | 19% slower in METR’s 2025 trial; the 2026 follow-up leaned toward speedup but was unreliable | Best controlled evidence, now dated |
| Learning an unfamiliar library | No significant time difference; lower comprehension scores with AI | Randomized, but small (52 developers) |
| Boilerplate, test scaffolding, documentation | Widely reported as strong; I found no independent controlled trial isolating these tasks | Practitioner consensus, not measured |
| Code review and verification | Review time and queues grow sharply under heavy AI adoption in Faros telemetry | Observational, vendor data |
| Whole-team delivery | DORA 2025: throughput up, instability up; Faros 2025: delivery metrics flat | Observational |
Which tool? Honest answers about Copilot, Cursor and Claude Code
The older tiering of inline assistants, AI-native editors and autonomous agents is still a useful way to think about how tools differ, but the product lines blur quickly: editors have added agent modes and agents have added editor integrations.
Three things the evidence does and does not say:
- Cursor has been tested in a controlled trial. Earlier versions of this article claimed no randomized trial had evaluated an AI-native editor. That was wrong: METR’s 2025 study used Cursor Pro. What has not been tested is whether newer, more agentic tool generations change the result, and METR’s follow-up could not settle that.
- Agentic tools are now mainstream. METR’s February 2026 post cited an estimate that about 4% of GitHub commits are authored by Claude Code, and Stack Overflow’s 2026 survey found over a quarter of respondents using automated agent workflows. Adoption is moving faster than measurement.
- I found no independent, head-to-head controlled trial of Copilot, Cursor, Claude Code, or Codex. Vendor benchmarks and individual impressions fill the gap. Given how quickly each tool changes, any ranking has a short shelf life.
The shovelware test, revisited
In September 2025, developer Mike Judge ran a six-week self-experiment: he estimated each task’s duration, flipped a coin to decide whether to use AI, and timed the result. AI came out a median 21% slower for him, though he acknowledged a limited sample that could be due to chance. He then asked a sharper question: if AI made developers dramatically more productive, where is the flood of new software? In the public data he examined, the lines were flat.
That argument looks different now. In January 2026, Pete Goldsmith reported that “Show HN” posts, a proxy for people launching projects, had more than doubled over the prior year. One community proxy is not proof of an industry-wide boom, and it says nothing about quality. But it suggests the absence of a visible surge in 2025 may have reflected tool maturity and adoption lag, not a ceiling.
How to measure your own results
Vendor numbers, surveys, and even good research will not tell you what happens on your codebase with your team. Here is a measurement approach that incorporates what METR learned the hard way.
- Pre-register tasks before assigning conditions. METR’s biggest 2026 problem was selection: people held back tasks where they most wanted AI. List the tasks first, then randomize.
- Randomize at the task level when you can. Flip a coin per task, as Judge did, or assign alternately. Record start and end times honestly, including review and rework.
- Track completion, not just speed. If assigned-no-AI tasks go unfinished or get quietly reassigned, you are measuring selection, not productivity.
- Measure the downstream system. Track time to first review, time in review, change failure rate, reverts or follow-up fixes within two weeks, and incidents per merged change. Rising throughput with rising instability is a finding, not a success.
- Write the interpretation before the data arrives. Decide in advance what you will do if throughput rises and instability rises, and what you will do if both hold steady.
- Allow for the J-curve. DORA describes a temporary dip during adoption. Evaluate over weeks, not days, and separate tool-learning time from steady-state results.
- Protect learning. For juniors and for work in unfamiliar areas, prefer usage patterns that include explanations and conceptual questions over wholesale delegation, and test comprehension without the tool.
- Compare surveys to measurements. If your developers report large gains but cycle time and review queues say otherwise, the gap itself is the thing to investigate.
Frequently asked questions
Do AI coding tools make developers faster?
It depends on the task, the developer, and the tool generation. Controlled and observational evidence conflicts: a 2023 experiment found large gains on a self-contained task, METR’s 2025 trial found a slowdown for experts on familiar code, and METR’s 2026 follow-up leaned toward a speedup but was too affected by selection effects to quantify it.
Is METR’s 19% slowdown still valid?
It remains a valid description of early-2025 tools in that specific setting, but METR now labels it out of date. Do not cite it as the current effect of AI on developer productivity.
Did METR’s 2026 study show a speedup?
Not conclusively. METR’s raw estimates were −18% task time for returning developers and −4% for new developers, both with intervals that include zero. METR called the data an unreliable signal and said its estimate is likely a lower bound on the true effect.
Do developers trust AI-generated code?
Mostly conditionally. In Stack Overflow’s 2026 survey, 48% trust AI output when they can easily verify it, and only 6.6% trust it for many tasks, including important decisions.
Does AI hurt code quality or delivery stability?
Several observational datasets, including Faros telemetry and DORA’s 2025 report, associate heavy AI adoption with more bugs or instability, but none proves causation. Review capacity and engineering foundations appear to matter as much as the tool.
Which AI coding tool is the most productive?
No independent controlled trial compares the leading tools head to head, and the products change every few months. Evaluate candidates on your own tasks with a measurement design like the one above.
What I could not verify
Honest reporting includes the gaps:
- Faros figures come from its own reports and blog, which I read directly for some metrics and via secondary summaries for others. Faros is a commercial vendor in this space.
- I found no independent controlled study isolating AI’s effect on boilerplate, test generation, or documentation, so those remain “widely believed” rather than “measured.”
- Earlier versions of this article cited a vendor-reported “81% complete tasks faster” figure for Copilot, a specific production incident with hours-of-remediation numbers, and several quotes and token counts about autonomous agents. I could not trace those to verifiable primary sources, so they have been removed.
- Stack Overflow’s 2025 and 2026 trust questions differ, so year-over-year trust trends cannot be read from them directly.
Related: AI Coding Tools Guide: Prompting for Better Code Output
Sources
- METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (July 2025; paper: arXiv 2507.09089)
- METR, We are Changing our Developer Productivity Experiment Design (February 24, 2026)
- METR, Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity (May 11, 2026)
- METR, Task substitution and uplift (May 8, 2026)
- Stack Overflow, 2026 Developer Survey: AI and press release (October 2026)
- Stack Overflow, 2025 Developer Survey: AI and analysis of the trust gap (April 2026)
- Google Cloud DORA, State of AI-assisted Software Development 2025 and ROI of AI-assisted Software Development (2026.01)
- Faros AI, The AI Productivity Paradox Report 2025 and AI Engineering Report 2026 summary
- Shen and Tamkin, How AI Impacts Skill Formation (Anthropic, January 2026; arXiv 2601.20245)
- Peng, Kalliamvakou, Cihon and Demirer, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (2023)
- GitClear, Coding on Copilot: data suggests downward pressure on code quality
- MIT Technology Review, AI coding is now everywhere. But not everyone is convinced (December 2025)
- Gu et al., Challenges and Paths Towards AI for Software Engineering (2025)
- Mike Judge, Where’s the Shovelware? Why AI Coding Claims Don’t Add Up (September 2025)
- Pete Goldsmith, Show HN trends (January 2026), as discussed by Jerod Santo


