AI Coding Productivity — Empirical Evidence (Adversarial)

#source #ai #evidence #adversarial

Source Metadata

Date: 2026-07-18 (compilation date; underlying studies span 2023–2025)
Raw path: raw/sources/AI Coding Productivity - Empirical Evidence (deep research 2026-07-18).md
Source type: Deep-research synthesis — a Claude Code deep-research run (6 search angles → 23 sources fetched → 104 claims → 25 adversarially verified, 23 confirmed / 2 refuted). Not a single external artifact.
Ingestion date: 2026-07-18
Provenance role: The vault's first deliberately adversarial source — commissioned (see 2026-07-17-best-method-first-client's lineage of open threads) to test the premise underlying both vault branches, that AI has zeroed the cost of commodity software execution. Second-order like the distillation, but its inputs are peer-reviewed papers, RCTs, and large-sample surveys rather than promotional videos — higher-provenance than any prior source, though still a synthesis.
⚠ Time-scope: every figure is tied to a specific tool generation (early-2025 and earlier). This is 2023–2025 evidence, explicitly NOT the live 2026 state of the art — METR itself labels its result "historical"; DORA reversed one 2024 finding in 2025.

Core Claims

The evidence QUALIFIES the hype, it does not demolish it. No study shows AI has zero/negative value in general. It shows the real effect is modest (≈14–19%), context-dependent, novice-tilted, and overstated by self-report.
"AI zeroed the cost of skilled dev work" is directly contradicted by the strongest developer-specific RCT: experienced devs were 19% slower with early-2025 AI ([METR]).
"Seniors up, juniors irrelevant" is directly inverted by the best skill-distribution study: the least-experienced gain most (34% vs ~0 for experts) — and a Copilot RCT independently found less-experienced devs benefit most.
Self-reported productivity gains are unreliable evidence. METR devs forecast +24%, felt +20%, measured −19% — a ~39-point perception gap. This is the exact "it feels faster" basis most promotional claims rest on.
Individual speed ≠ team delivery. DORA 2024: AI adoption correlated with reduced delivery stability (and, in 2024, throughput). Trust is low and falling (SO 2025: ~3% highly trust, ~46% distrust).

Key Evidence / Details

#	Finding	Design	Population	Headline	Confidence
1	Experienced devs 19% slower with AI	RCT	16 expert OSS maintainers, 246 real tasks, their own mature repos, early-2025 tools	−19% completion time	HIGH (3-0)
2	Perception ≫ reality	(within #1)	same	forecast +24%, post +20%, actual −19%	HIGH (3-0)
3	Gains tilt to least-skilled	quasi-exp (diff-in-diff)	5,179 customer-support agents — NOT devs	+34% novices vs ~0 experts (avg +14–15%)	HIGH (3-0)
4	Novices benefit more; toy-task speedup	RCT	70 freelancers, one JS toy task, 2022 Copilot	+55.8% (toy); exp. coeff n.s. at 0.05	MEDIUM
5	AI hurt team stability/throughput	observational survey	DORA 2024 respondents	−7.2% stability / −1.5% throughput per +25% adoption	MEDIUM (throughput reversed in 2025)
6	Trust low & falling; debugging tax	survey (~49k)	SO 2025	~3% highly trust, ~46% distrust; ~45% debugging is time-consuming	HIGH (3-0)

Key primary sources: METR (metr.org/blog/2025-07-10…, arXiv:2507.09089) · Brynjolfsson, Li & Raymond, "Generative AI at Work," QJE 140(2):889–942 2025 (NBER w31161) · GitHub Copilot RCT (arXiv:2302.06590) · DORA 2024 (dora.dev/research/2024) · Stack Overflow 2025 Developer Survey (survey.stackoverflow.co/2025/ai).

Verification honesty: 2 claims were refuted during 3-vote checking — both versions of "66% spend more time fixing almost-right AI code" (the 66% actually measures encountering almost-right output; the real debugging-time figure is 45%). Recorded so the error isn't silently repeated.

Connections

ai-productivity-evidence — the concept page this source anchors (measured vs. claimed AI productivity)
ai-market-shift — challenges the premise that a ~$200/mo AI is a genuine substitute zeroing rate competition; substantiates a doubt the vault already flagged (the "$200 substitute is asserted, not demonstrated" open question)
seniority-and-ai — inverts the productivity-gain direction of "seniors up, juniors irrelevant" (novices gain most) — while not settling the labor-market-value question, and partly supporting the "juniors can't catch AI's errors" risk mechanism
future-of-engineering-work — qualifies "coding cost → 0, teams collapse 8→2–3": effect sizes are modest and team-level delivery can worsen
Counter-pole to sebastian's standout claims and to the whole "AI killed hourly dev" school (dmitry-rodenko, ai-market-shift)
overview — flips the standing "no adversarial source" gap

Open Questions

Does METR's −19% persist/reverse with mid-2026 agentic tools? The single most important missing piece; this source is explicitly historical.
Does "novices gain most" hold for real software work, or invert on complex/unfamiliar codebases where juniors can't catch AI errors?
The findings measure speed, not labor-market value (wages/hiring/headcount) — so they are directional counter-evidence to "juniors out," not a same-metric refutation.
Transient learning-curve cost vs. durable tax for the team-level effects? DORA's 2025 partial reversal cuts both ways.
Should dmitry-rodenko-style claims ("$200/mo AI substitute") now carry a stronger Status: contested given this evidence?

Change Impact on Wiki

Thirteenth source; the vault's first adversarial / empirical source and its highest-provenance one.

Created concept ai-productivity-evidence as the durable home for the empirical counter-narrative.
Added counter-evidence sections to ai-market-shift, seniority-and-ai, future-of-engineering-work — recording contradictions explicitly (rule #5), not overwriting the sources' prior claims (Update Policy).
Flipped the long-standing gap in overview and index ("no adversarial source / 11-of-12 one school").
No entity pages created for METR / Brynjolfsson / DORA / Stack Overflow — they are cited as evidence rather than tracked as recurring subjects (consistent with the Buffett/Munger decision on mohnish-pabrai); revisit if a second study-based source arrives.
Deliberately framed as qualifying, not inverting, the thesis — the honest reading of the evidence.

6.6 KiB Raw Permalink Blame History Unescape Escape