AI Coding Productivity — Empirical Evidence (Deep-Research Synthesis)

Compiled: 2026-07-18 by Claude Code deep-research harness (fan-out web search → source fetch → 3-vote adversarial verification → synthesis). Purpose: A deliberately ADVERSARIAL evidence dossier commissioned to counterbalance a vault of promotional sources asserting AI has "zeroed the cost of commodity software development" and made "seniors more valuable while juniors become irrelevant." Provenance note: This is a second-order synthesis, but unlike the vault's promotional videos its inputs are primary peer-reviewed papers, RCTs, and large-sample surveys. Verification stats: 6 search angles → 23 sources fetched → 104 claims extracted → 25 verified (23 confirmed, 2 refuted). Every quantitative figure below is time-scoped and caveated.

Bottom line

Rigorous evidence qualifies more than it demolishes the popular narrative. No study shows AI has zero or negative value in general. What the evidence shows: AI's real productivity effect on developers is modest (≈14–19% range), context-dependent, tilted toward NOVICES rather than experts, and systematically overstated by self-report. Two of the popular thesis's load-bearing claims are directly inverted by the best available data:

"AI zeroed the cost of skilled dev work" → the strongest developer-specific RCT found experienced devs 19% SLOWER with early-2025 AI.
"Seniors up, juniors irrelevant" → the best skill-distribution study found the least-experienced workers gain most (34% vs. ~0 for experts).

Finding 1 — METR RCT: experienced devs were 19% SLOWER with AI (confidence: HIGH, 3-0)

Claim: METR's 2025 randomized controlled trial found experienced open-source developers took 19% LONGER to complete real tasks when allowed to use early-2025 AI coding tools — AI slowed the most-skilled population down.

Evidence: Verbatim from primary source (METR blog 2025-07-10; arXiv:2507.09089): "When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown" / "allowing AI actually increases completion time by 19%—AI tooling slowed developers down."

Design: genuine RCT (gold standard). 16 experienced OSS developers (avg ~5 yrs on their OWN mature repos, ~1,500 commits; repos avg 22k+ stars, 1M+ LOC). 246 real tasks (~2h each), Feb–June 2025, randomly assigned AI-allowed vs AI-disallowed. Tools: Cursor Pro + Claude 3.5/3.7 Sonnet.
Caveats (qualify, don't refute): (1) small n=16; (2) narrow population — experts on codebases they know deeply, the cohort least likely to benefit → does NOT generalize to juniors or greenfield work; (3) non-peer-reviewed preprint; (4) authors concede experimental-artifact risk "cannot be entirely ruled out."
On the Feb 2026 follow-up: METR does NOT retract the 19% figure; it labels it "historical" (scoped to early-2025 tools) and only redesigns FUTURE experiments after finding selection bias (30–50% of devs refused no-AI conditions) in a separate Aug-2025 trial. Secondary blogs calling this "backtracking" overstate it.

Sources: metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study · arxiv.org/abs/2507.09089 · x.com/METR_Evals/status/1943360399220388093 · metr.org/blog/2026-02-24-uplift-update

Finding 2 — Perception vs. reality gap (~39 points) (confidence: HIGH, 3-0)

Claim: Developers' subjective perception of AI's benefit is badly miscalibrated. In the METR RCT they forecast a 24% speedup beforehand and still estimated a 20% speedup afterward — despite being measured 19% slower. A ~39-point perception-vs-reality gap, which is exactly the evidence base ("it feels faster") that most "AI makes devs faster" claims rest on.

Evidence: All three figures verbatim from arXiv:2507.09089 abstract (developers' own forecast 24%; post-hoc estimate 20%; measured −19%). This within-study perception gap is the study's most robust, uncontested takeaway — critiques target the external validity of the 19% magnitude, not the perception gap. Survey parallel: Stack Overflow 2025 reports 69% self-report a personal-productivity increase from AI agents even while majorities report time lost fixing flawed AI output — self-report on both sides, so a looser "parallel," not objective measurement.

Sources: arxiv.org/abs/2507.09089 · metr.org/blog/2025-07-10… · stackoverflow.blog/2025/12/29/…

Finding 3 — Brynjolfsson/Li/Raymond: AI gains tilt to the LEAST-skilled (confidence: HIGH, 3-0)

Claim: "Generative AI at Work" found gains distributed INVERSELY with skill: least-experienced workers improved most (~34%, in speed AND quality); most-experienced workers saw only small speed gains and small quality DECLINES. Overall average a modest 14–15% — not a cost-to-zero collapse. This is the direct inverse of "seniors more valuable, juniors irrelevant."

Evidence: Verbatim (NBER w31161, Apr 2023; published Quarterly Journal of Economics 140(2):889–942, 2025 — a top-5 econ journal; authors Brynjolfsson/Stanford, Li/MIT, Raymond): "increases productivity, as measured by issues resolved per hour, by 14% on average, including a 34% improvement for novice and low-skilled workers but with minimal impact on experienced and highly skilled workers" and "Less experienced and lower-skilled workers improve both the speed and quality of their output while the most experienced and highest-skilled workers see small gains in speed and small declines in quality." (QJE version: 15% / 5,172 agents; NBER WP: 14% / 5,179 — both correct to their version.)

MANDATORY SCOPE CAVEAT: population is customer-support agents, NOT software developers. "No software developers were studied." Strong analogous/directional counter-evidence to the general "seniors up, juniors out" thesis — not same-population proof about coding.
Design: quasi-experimental staggered rollout (difference-in-differences, small RCT component), single Fortune 500 firm.
Direction aligns with the developer-specific METR and Copilot findings (no directional contradiction). A Brookings follow-up ("Borrowed expertise") qualifies: AI disseminates senior expertise; gains may not persist long-run.

Sources: nber.org/papers/w31161 · arxiv.org/pdf/2304.11771 · gsb.stanford.edu/faculty-research/publications/generative-ai-work · ssrn.com/abstract=4426942 · ideas.repec.org/a/oup/qjecon/v140y2025i2p889-942

Finding 4 — GitHub Copilot RCT: 55.8% faster on a TOY task; novices benefit more (confidence: MEDIUM)

Claim: A GitHub/Microsoft-authored RCT found less-experienced developers benefited more (also heavier-coding-load and age-25–44 devs) — reinforcing "novices gain most." Its large headline (55.8% faster) came from a single narrow toy task; the gulf vs. METR's −19% is largely task realism (standardized benchmark vs. real PRs on familiar codebases).

Evidence: arXiv:2302.06590 (Peng, Kalliamvakou, Cihon, Demirer, 2023). Treated group "completed the task 55.8% faster" (95% CI 21–89%, p=0.0017; 71.17 vs 160.89 min) on a toy task (implement a JS HTTP server), n=70 analyzed (Upwork freelancers), NOT real collaborative development.

Honest caveats: the programming-EXPERIENCE coefficient — the centerpiece of "novices benefit more" — is NOT significant at 0.05 (p=0.0629); only hours/day and age-25–44 clear 0.05. Paper hedges "If this result persists…". Subgroups underpowered (~35/group); sample skews young, lower-income, India/Pakistan. Authors are Copilot's makers (COI — though this cuts toward credibility for an anti-hype "novices benefit more" result they had no incentive to manufacture).

Sources: arxiv.org/pdf/2302.06590 (critique: blueoptima.com/post/debunking-githubs-claims…)

Finding 5 — DORA 2024: AI hurt team delivery stability/throughput (confidence: MEDIUM)

Claim: DORA's 2024 research: AI adoption raised individual productivity/satisfaction but negatively impacted software delivery stability and throughput at the team level — individual speed gains did not translate to better team delivery.

Evidence: Verbatim (dora.dev/research/2024): "AI adoption significantly increases individual productivity, flow, and job satisfaction. However, it also negatively impacts software delivery stability and throughput." Magnitudes (corroborated by RedMonk/InfoQ/GetDX): a 25% increase in AI adoption ≈ 1.5% throughput decrease, 7.2% stability reduction; mechanism = AI inflating batch/changeset size. Independent primary research (Google/DORA), not vendor marketing.

Two mandatory caveats: (1) observational/self-reported cross-sectional survey → CORRELATION not causation. (2) PARTIALLY SUPERSEDED — DORA's OWN 2025 report REVERSED the throughput finding (AI now positively correlates with throughput); the negative STABILITY relationship persisted. Only the stability half remains current.

Sources: dora.dev/research/2024/dora-report

Finding 6 — Stack Overflow 2025: trust low and falling; debugging tax (confidence: HIGH, 3-0)

Claim: SO 2025 Developer Survey (~49,000 respondents): developer trust in AI output accuracy is low and eroding, with a documented "debugging tax." Only ~3% "highly trust" AI accuracy while ~46% actively distrust (up from 31% in 2024; now exceeding the ~33% who trust), even as adoption climbed to 84%; ~45% report debugging AI-generated code is time-consuming.

Evidence: Verbatim (survey.stackoverflow.co/2025/ai, n=33,244 on trust; Dec 2025). Highly trust 3.1%, somewhat 29.6%, somewhat distrust 26.1%, highly distrust 19.6% → ~46% distrust vs 32.7% trust. "Trust in the accuracy of AI has fallen from 40% in previous years to just 29% this year." Press release: "Trust in AI at an All Time Low." Adoption rose to 84% (from 76%); 51% daily among pros.

Caveats: all self-reported perception/trust, not direct output-reliability measures. SO has mild business interest in AI skepticism.
⚠ Data-hygiene note (verification caught this): two claims asserting "66% spend more time fixing almost-right AI code" were REFUTED — the 66% item measures encountering "almost right, but not quite" solutions; the cleanest "more time" figure is 45%. Cite 45%, not 66%. Several secondary summaries conflate them.

Sources: survey.stackoverflow.co/2025/ai · stackoverflow.co/company/press/archive/stack-overflow-2025-developer-survey · stackoverflow.blog/2025/12/29/…

Overall caveats (the adversarial source held to its own standard)

TIME-SENSITIVITY (biggest limit): every figure is tied to a fast-moving tool generation. METR's −19% is explicitly "historical"/early-2025 (Cursor Pro + Claude 3.5/3.7 Sonnet); the Copilot RCT uses 2022-era Copilot; DORA's 2024 throughput finding was reversed by DORA in 2025. None represents the live 2026 state of the art.
STATISTICAL POWER/SAMPLE: METR n=16 (though significant across 246 tasks); its population is precisely the cohort least likely to gain, so it does NOT show AI slows juniors/greenfield/most devs (METR is explicit about these non-conclusions). Copilot heterogeneity subgroups underpowered; key experience coefficient not significant at 0.05.
SCOPE/DOMAIN MISMATCH: the strongest skill-distribution result (Brynjolfsson, 34% novice gain) is customer-support agents, not developers — directional, not same-population proof.
DESIGN LADDER: RCTs (METR, Copilot) > quasi-experiment (Brynjolfsson) > observational surveys (DORA, SO). "Zeroed cost" is undercut by inference, not a direct cost measurement.
SOURCE BIAS: GitHub/Microsoft authored the pro-Copilot RCT (COI); SO has mild commercial interest in AI skepticism; DORA/Google is independent of the tool vendors.

Open questions

Does METR's −19% persist/reverse with mid-2026 agentic tools? Its own Feb-2026 update hints at possible speedup but calls it "very weak evidence"; a properly powered follow-up RCT on current tools is the key missing piece.
Does "novices gain most" hold for REAL software work, or invert on complex/unfamiliar codebases where juniors lack judgment to catch AI errors (the mechanism behind experts' small quality declines)?
All findings measure task SPEED/productivity — the "seniors up/juniors out" thesis is about labor-market VALUE (wages, hiring, headcount), which no source here measures. Directional counter-evidence, not same-metric refutation.
Are AI's negative team-level effects (larger batches, debugging burden, low trust) a transient learning-curve cost or a durable tax? DORA's 2025 partial reversal suggests some are transitional; persistent stability decline and falling trust suggest others may not be.

Refuted during verification (recorded for honesty)

"66% report spending MORE time fixing almost-right AI code" (0-3 and 1-2) — conflates SO's 66% "encounter almost-right solutions" item with the 45% "debugging is time-consuming" item. Use 45%.

13 KiB Raw Blame History Unescape Escape