all

wiki/sources/2026-07-18-ai-productivity-adversarial-evidence.md

@@ -0,0 +1,62 @@
+# AI Coding Productivity — Empirical Evidence (Adversarial)
+
+#source #ai #evidence #adversarial
+
+## Source Metadata
+
+- **Date:** 2026-07-18 (compilation date; underlying studies span 2023–2025)
+- **Raw path:** `raw/sources/AI Coding Productivity - Empirical Evidence (deep research 2026-07-18).md`
+- **Source type:** **Deep-research synthesis** — a Claude Code `deep-research` run (6 search angles → 23 sources fetched → 104 claims → 25 adversarially verified, 23 confirmed / 2 refuted). Not a single external artifact.
+- **Ingestion date:** 2026-07-18
+- **Provenance role:** The vault's **first deliberately adversarial source** — commissioned (see [[2026-07-17-best-method-first-client]]'s lineage of open threads) to test the premise underlying both vault branches, that *AI has zeroed the cost of commodity software execution*. Second-order like the distillation, but its inputs are peer-reviewed papers, RCTs, and large-sample surveys rather than promotional videos — **higher-provenance than any prior source**, though still a synthesis.
+- **⚠ Time-scope:** every figure is tied to a specific tool generation (early-2025 and earlier). This is **2023–2025 evidence, explicitly NOT the live 2026 state of the art** — METR itself labels its result "historical"; DORA reversed one 2024 finding in 2025.
+
+## Core Claims
+
+1. **The evidence QUALIFIES the hype, it does not demolish it.** No study shows AI has zero/negative value in general. It shows the real effect is **modest (≈14–19%), context-dependent, novice-tilted, and overstated by self-report.**
+2. **"AI zeroed the cost of skilled dev work" is directly contradicted** by the strongest developer-specific RCT: experienced devs were **19% *slower*** with early-2025 AI ([METR]).
+3. **"Seniors up, juniors irrelevant" is directly inverted** by the best skill-distribution study: the **least-experienced gain most** (34% vs ~0 for experts) — and a Copilot RCT independently found less-experienced devs benefit most.
+4. **Self-reported productivity gains are unreliable evidence.** METR devs forecast +24%, felt +20%, measured −19% — a ~39-point perception gap. This is the exact "it feels faster" basis most promotional claims rest on.
+5. **Individual speed ≠ team delivery.** DORA 2024: AI adoption correlated with *reduced* delivery stability (and, in 2024, throughput). Trust is low and falling (SO 2025: ~3% highly trust, ~46% distrust).
+
+## Key Evidence / Details
+
+| # | Finding | Design | Population | Headline | Confidence |
+|---|---|---|---|---|---|
+| 1 | Experienced devs **19% slower** with AI | RCT | 16 expert OSS maintainers, 246 real tasks, their own mature repos, early-2025 tools | −19% completion time | HIGH (3-0) |
+| 2 | Perception ≫ reality | (within #1) | same | forecast +24%, post +20%, actual −19% | HIGH (3-0) |
+| 3 | Gains tilt to **least-skilled** | quasi-exp (diff-in-diff) | **5,179 customer-support agents — NOT devs** | +34% novices vs ~0 experts (avg +14–15%) | HIGH (3-0) |
+| 4 | Novices benefit more; toy-task speedup | RCT | 70 freelancers, one JS toy task, 2022 Copilot | +55.8% (toy); exp. coeff n.s. at 0.05 | MEDIUM |
+| 5 | AI hurt team **stability/throughput** | observational survey | DORA 2024 respondents | −7.2% stability / −1.5% throughput per +25% adoption | MEDIUM (throughput reversed in 2025) |
+| 6 | Trust low & falling; debugging tax | survey (~49k) | SO 2025 | ~3% highly trust, ~46% distrust; ~45% debugging is time-consuming | HIGH (3-0) |
+
+**Key primary sources:** METR (metr.org/blog/2025-07-10…, arXiv:2507.09089) · Brynjolfsson, Li & Raymond, "Generative AI at Work," *QJE* 140(2):889–942 2025 (NBER w31161) · GitHub Copilot RCT (arXiv:2302.06590) · DORA 2024 (dora.dev/research/2024) · Stack Overflow 2025 Developer Survey (survey.stackoverflow.co/2025/ai).
+
+**Verification honesty:** 2 claims were **refuted** during 3-vote checking — both versions of "66% spend more time fixing almost-right AI code" (the 66% actually measures *encountering* almost-right output; the real debugging-time figure is **45%**). Recorded so the error isn't silently repeated.
+
+## Connections
+
+- [[ai-productivity-evidence]] — the concept page this source anchors (measured vs. claimed AI productivity)
+- [[ai-market-shift]] — **challenges** the premise that a ~$200/mo AI is a genuine *substitute* zeroing rate competition; substantiates a doubt the vault already flagged (the "$200 substitute is asserted, not demonstrated" open question)
+- [[seniority-and-ai]] — **inverts** the productivity-gain direction of "seniors up, juniors irrelevant" (novices gain most) — while *not* settling the labor-market-value question, and partly supporting the "juniors can't catch AI's errors" risk mechanism
+- [[future-of-engineering-work]] — **qualifies** "coding cost → 0, teams collapse 8→2–3": effect sizes are modest and team-level delivery can worsen
+- Counter-pole to [[sebastian]]'s standout claims and to the whole "AI killed hourly dev" school ([[dmitry-rodenko]], [[ai-market-shift]])
+- [[overview]] — flips the standing "no adversarial source" gap
+
+## Open Questions
+
+- Does METR's −19% persist/reverse with **mid-2026 agentic tools**? The single most important missing piece; this source is explicitly historical.
+- Does "novices gain most" hold for **real** software work, or invert on complex/unfamiliar codebases where juniors can't catch AI errors?
+- The findings measure **speed**, not **labor-market value** (wages/hiring/headcount) — so they are directional counter-evidence to "juniors out," not a same-metric refutation.
+- Transient learning-curve cost vs. durable tax for the team-level effects? DORA's 2025 partial reversal cuts both ways.
+- Should [[dmitry-rodenko]]-style claims ("$200/mo AI substitute") now carry a stronger `Status: contested` given this evidence?
+
+## Change Impact on Wiki
+
+Thirteenth source; the vault's **first adversarial / empirical source** and its highest-provenance one.
+
+- Created concept [[ai-productivity-evidence]] as the durable home for the empirical counter-narrative.
+- Added counter-evidence sections to [[ai-market-shift]], [[seniority-and-ai]], [[future-of-engineering-work]] — recording contradictions explicitly (rule #5), **not** overwriting the sources' prior claims (Update Policy).
+- Flipped the long-standing gap in [[overview]] and [[index]] ("no adversarial source / 11-of-12 one school").
+- **No entity pages created** for METR / Brynjolfsson / DORA / Stack Overflow — they are cited as evidence rather than tracked as recurring subjects (consistent with the Buffett/Munger decision on [[mohnish-pabrai]]); revisit if a second study-based source arrives.
+- Deliberately framed as *qualifying*, not inverting, the thesis — the honest reading of the evidence.