BusinessNotes/wiki/sources/2026-07-18-ai-productivity-adversarial-evidence.md

# AI Coding Productivity — Empirical Evidence (Adversarial)

#source #ai #evidence #adversarial

## Source Metadata

- **Date:** 2026-07-18 (compilation date; underlying studies span 2023–2025)
- **Raw path:** `raw/sources/AI Coding Productivity - Empirical Evidence (deep research 2026-07-18).md`
- **Source type:** **Deep-research synthesis** — a Claude Code `deep-research` run (6 search angles → 23 sources fetched → 104 claims → 25 adversarially verified, 23 confirmed / 2 refuted). Not a single external artifact.
- **Ingestion date:** 2026-07-18
- **Provenance role:** The vault's **first deliberately adversarial source** — commissioned (see [[2026-07-17-best-method-first-client]]'s lineage of open threads) to test the premise underlying both vault branches, that *AI has zeroed the cost of commodity software execution*. Second-order like the distillation, but its inputs are peer-reviewed papers, RCTs, and large-sample surveys rather than promotional videos — **higher-provenance than any prior source**, though still a synthesis.
- **⚠ Time-scope:** every figure is tied to a specific tool generation (early-2025 and earlier). This is **2023–2025 evidence, explicitly NOT the live 2026 state of the art** — METR itself labels its result "historical"; DORA reversed one 2024 finding in 2025.

## Core Claims

1. **The evidence QUALIFIES the hype, it does not demolish it.** No study shows AI has zero/negative value in general. It shows the real effect is **modest (≈14–19%), context-dependent, novice-tilted, and overstated by self-report.**
2. **"AI zeroed the cost of skilled dev work" is directly contradicted** by the strongest developer-specific RCT: experienced devs were **19% *slower*** with early-2025 AI ([METR]).
3. **"Seniors up, juniors irrelevant" is directly inverted** by the best skill-distribution study: the **least-experienced gain most** (34% vs ~0 for experts) — and a Copilot RCT independently found less-experienced devs benefit most.
4. **Self-reported productivity gains are unreliable evidence.** METR devs forecast +24%, felt +20%, measured −19% — a ~39-point perception gap. This is the exact "it feels faster" basis most promotional claims rest on.
5. **Individual speed ≠ team delivery.** DORA 2024: AI adoption correlated with *reduced* delivery stability (and, in 2024, throughput). Trust is low and falling (SO 2025: ~3% highly trust, ~46% distrust).

## Key Evidence / Details

| # | Finding | Design | Population | Headline | Confidence |
|---|---|---|---|---|---|
| 1 | Experienced devs **19% slower** with AI | RCT | 16 expert OSS maintainers, 246 real tasks, their own mature repos, early-2025 tools | −19% completion time | HIGH (3-0) |
| 2 | Perception ≫ reality | (within #1) | same | forecast +24%, post +20%, actual −19% | HIGH (3-0) |
| 3 | Gains tilt to **least-skilled** | quasi-exp (diff-in-diff) | **5,179 customer-support agents — NOT devs** | +34% novices vs ~0 experts (avg +14–15%) | HIGH (3-0) |
| 4 | Novices benefit more; toy-task speedup | RCT | 70 freelancers, one JS toy task, 2022 Copilot | +55.8% (toy); exp. coeff n.s. at 0.05 | MEDIUM |
| 5 | AI hurt team **stability/throughput** | observational survey | DORA 2024 respondents | −7.2% stability / −1.5% throughput per +25% adoption | MEDIUM (throughput reversed in 2025) |
| 6 | Trust low & falling; debugging tax | survey (~49k) | SO 2025 | ~3% highly trust, ~46% distrust; ~45% debugging is time-consuming | HIGH (3-0) |

**Key primary sources:** METR (metr.org/blog/2025-07-10…, arXiv:2507.09089) · Brynjolfsson, Li & Raymond, "Generative AI at Work," *QJE* 140(2):889–942 2025 (NBER w31161) · GitHub Copilot RCT (arXiv:2302.06590) · DORA 2024 (dora.dev/research/2024) · Stack Overflow 2025 Developer Survey (survey.stackoverflow.co/2025/ai).

**Verification honesty:** 2 claims were **refuted** during 3-vote checking — both versions of "66% spend more time fixing almost-right AI code" (the 66% actually measures *encountering* almost-right output; the real debugging-time figure is **45%**). Recorded so the error isn't silently repeated.

## Connections

- [[ai-productivity-evidence]] — the concept page this source anchors (measured vs. claimed AI productivity)
- [[ai-market-shift]] — **challenges** the premise that a ~$200/mo AI is a genuine *substitute* zeroing rate competition; substantiates a doubt the vault already flagged (the "$200 substitute is asserted, not demonstrated" open question)
- [[seniority-and-ai]] — **inverts** the productivity-gain direction of "seniors up, juniors irrelevant" (novices gain most) — while *not* settling the labor-market-value question, and partly supporting the "juniors can't catch AI's errors" risk mechanism
- [[future-of-engineering-work]] — **qualifies** "coding cost → 0, teams collapse 8→2–3": effect sizes are modest and team-level delivery can worsen
- Counter-pole to [[sebastian]]'s standout claims and to the whole "AI killed hourly dev" school ([[dmitry-rodenko]], [[ai-market-shift]])
- [[overview]] — flips the standing "no adversarial source" gap

## Open Questions

- Does METR's −19% persist/reverse with **mid-2026 agentic tools**? The single most important missing piece; this source is explicitly historical.
- Does "novices gain most" hold for **real** software work, or invert on complex/unfamiliar codebases where juniors can't catch AI errors?
- The findings measure **speed**, not **labor-market value** (wages/hiring/headcount) — so they are directional counter-evidence to "juniors out," not a same-metric refutation.
- Transient learning-curve cost vs. durable tax for the team-level effects? DORA's 2025 partial reversal cuts both ways.
- Should [[dmitry-rodenko]]-style claims ("$200/mo AI substitute") now carry a stronger `Status: contested` given this evidence?

## Change Impact on Wiki

Thirteenth source; the vault's **first adversarial / empirical source** and its highest-provenance one.

- Created concept [[ai-productivity-evidence]] as the durable home for the empirical counter-narrative.
- Added counter-evidence sections to [[ai-market-shift]], [[seniority-and-ai]], [[future-of-engineering-work]] — recording contradictions explicitly (rule #5), **not** overwriting the sources' prior claims (Update Policy).
- Flipped the long-standing gap in [[overview]] and [[index]] ("no adversarial source / 11-of-12 one school").
- **No entity pages created** for METR / Brynjolfsson / DORA / Stack Overflow — they are cited as evidence rather than tracked as recurring subjects (consistent with the Buffett/Munger decision on [[mohnish-pabrai]]); revisit if a second study-based source arrives.
- Deliberately framed as *qualifying*, not inverting, the thesis — the honest reading of the evidence.