111 lines
13 KiB
Markdown
111 lines
13 KiB
Markdown
# AI Coding Productivity — Empirical Evidence (Deep-Research Synthesis)
|
||
|
||
**Compiled:** 2026-07-18 by Claude Code `deep-research` harness (fan-out web search → source fetch → 3-vote adversarial verification → synthesis).
|
||
**Purpose:** A deliberately ADVERSARIAL evidence dossier commissioned to counterbalance a vault of promotional sources asserting AI has "zeroed the cost of commodity software development" and made "seniors more valuable while juniors become irrelevant."
|
||
**Provenance note:** This is a *second-order synthesis*, but unlike the vault's promotional videos its inputs are primary peer-reviewed papers, RCTs, and large-sample surveys. Verification stats: 6 search angles → 23 sources fetched → 104 claims extracted → 25 verified (23 confirmed, 2 refuted). Every quantitative figure below is time-scoped and caveated.
|
||
|
||
---
|
||
|
||
## Bottom line
|
||
|
||
Rigorous evidence **qualifies more than it demolishes** the popular narrative. No study shows AI has zero or negative value in general. What the evidence shows: AI's real productivity effect on developers is **modest (≈14–19% range), context-dependent, tilted toward NOVICES rather than experts, and systematically overstated by self-report.** Two of the popular thesis's load-bearing claims are directly inverted by the best available data:
|
||
|
||
1. **"AI zeroed the cost of skilled dev work"** → the strongest developer-specific RCT found experienced devs **19% SLOWER** with early-2025 AI.
|
||
2. **"Seniors up, juniors irrelevant"** → the best skill-distribution study found the **least-experienced workers gain most** (34% vs. ~0 for experts).
|
||
|
||
---
|
||
|
||
## Finding 1 — METR RCT: experienced devs were 19% SLOWER with AI (confidence: HIGH, 3-0)
|
||
|
||
**Claim:** METR's 2025 randomized controlled trial found experienced open-source developers took **19% LONGER** to complete real tasks when allowed to use early-2025 AI coding tools — AI slowed the most-skilled population down.
|
||
|
||
**Evidence:** Verbatim from primary source (METR blog 2025-07-10; arXiv:2507.09089): *"When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown"* / *"allowing AI actually increases completion time by 19%—AI tooling slowed developers down."*
|
||
|
||
- **Design:** genuine RCT (gold standard). 16 experienced OSS developers (avg ~5 yrs on their OWN mature repos, ~1,500 commits; repos avg 22k+ stars, 1M+ LOC). 246 real tasks (~2h each), Feb–June 2025, randomly assigned AI-allowed vs AI-disallowed. Tools: Cursor Pro + Claude 3.5/3.7 Sonnet.
|
||
- **Caveats (qualify, don't refute):** (1) small n=16; (2) narrow population — experts on codebases they know deeply, the cohort *least* likely to benefit → does NOT generalize to juniors or greenfield work; (3) non-peer-reviewed preprint; (4) authors concede experimental-artifact risk "cannot be entirely ruled out."
|
||
- **On the Feb 2026 follow-up:** METR does NOT retract the 19% figure; it labels it "historical" (scoped to early-2025 tools) and only redesigns FUTURE experiments after finding selection bias (30–50% of devs refused no-AI conditions) in a *separate* Aug-2025 trial. Secondary blogs calling this "backtracking" overstate it.
|
||
|
||
**Sources:** metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study · arxiv.org/abs/2507.09089 · x.com/METR_Evals/status/1943360399220388093 · metr.org/blog/2026-02-24-uplift-update
|
||
|
||
---
|
||
|
||
## Finding 2 — Perception vs. reality gap (~39 points) (confidence: HIGH, 3-0)
|
||
|
||
**Claim:** Developers' subjective perception of AI's benefit is badly miscalibrated. In the METR RCT they **forecast a 24% speedup** beforehand and **still estimated a 20% speedup afterward** — despite being measured **19% slower**. A ~39-point perception-vs-reality gap, which is exactly the evidence base ("it feels faster") that most "AI makes devs faster" claims rest on.
|
||
|
||
**Evidence:** All three figures verbatim from arXiv:2507.09089 abstract (developers' own forecast 24%; post-hoc estimate 20%; measured −19%). This within-study perception gap is the study's most robust, uncontested takeaway — critiques target the external validity of the 19% *magnitude*, not the perception gap. **Survey parallel:** Stack Overflow 2025 reports 69% self-report a personal-productivity increase from AI agents *even while* majorities report time lost fixing flawed AI output — self-report on both sides, so a looser "parallel," not objective measurement.
|
||
|
||
**Sources:** arxiv.org/abs/2507.09089 · metr.org/blog/2025-07-10… · stackoverflow.blog/2025/12/29/…
|
||
|
||
---
|
||
|
||
## Finding 3 — Brynjolfsson/Li/Raymond: AI gains tilt to the LEAST-skilled (confidence: HIGH, 3-0)
|
||
|
||
**Claim:** "Generative AI at Work" found gains distributed INVERSELY with skill: least-experienced workers improved most (~34%, in speed AND quality); most-experienced workers saw only small speed gains and **small quality DECLINES**. Overall average a modest 14–15% — not a cost-to-zero collapse. This is the direct inverse of "seniors more valuable, juniors irrelevant."
|
||
|
||
**Evidence:** Verbatim (NBER w31161, Apr 2023; published *Quarterly Journal of Economics* 140(2):889–942, 2025 — a top-5 econ journal; authors Brynjolfsson/Stanford, Li/MIT, Raymond): *"increases productivity, as measured by issues resolved per hour, by 14% on average, including a 34% improvement for novice and low-skilled workers but with minimal impact on experienced and highly skilled workers"* and *"Less experienced and lower-skilled workers improve both the speed and quality of their output while the most experienced and highest-skilled workers see small gains in speed and small declines in quality."* (QJE version: 15% / 5,172 agents; NBER WP: 14% / 5,179 — both correct to their version.)
|
||
|
||
- **MANDATORY SCOPE CAVEAT:** population is **customer-support agents, NOT software developers.** "No software developers were studied." Strong *analogous/directional* counter-evidence to the general "seniors up, juniors out" thesis — not same-population proof about coding.
|
||
- **Design:** quasi-experimental staggered rollout (difference-in-differences, small RCT component), single Fortune 500 firm.
|
||
- Direction aligns with the developer-specific METR and Copilot findings (no directional contradiction). A Brookings follow-up ("Borrowed expertise") qualifies: AI disseminates senior expertise; gains may not persist long-run.
|
||
|
||
**Sources:** nber.org/papers/w31161 · arxiv.org/pdf/2304.11771 · gsb.stanford.edu/faculty-research/publications/generative-ai-work · ssrn.com/abstract=4426942 · ideas.repec.org/a/oup/qjecon/v140y2025i2p889-942
|
||
|
||
---
|
||
|
||
## Finding 4 — GitHub Copilot RCT: 55.8% faster on a TOY task; novices benefit more (confidence: MEDIUM)
|
||
|
||
**Claim:** A GitHub/Microsoft-authored RCT found less-experienced developers benefited more (also heavier-coding-load and age-25–44 devs) — reinforcing "novices gain most." Its large headline (**55.8% faster**) came from a single narrow toy task; the gulf vs. METR's −19% is largely task realism (standardized benchmark vs. real PRs on familiar codebases).
|
||
|
||
**Evidence:** arXiv:2302.06590 (Peng, Kalliamvakou, Cihon, Demirer, 2023). Treated group "completed the task 55.8% faster" (95% CI 21–89%, p=0.0017; 71.17 vs 160.89 min) on a *toy* task (implement a JS HTTP server), n=70 analyzed (Upwork freelancers), NOT real collaborative development.
|
||
|
||
- **Honest caveats:** the programming-EXPERIENCE coefficient — the centerpiece of "novices benefit more" — is NOT significant at 0.05 (p=0.0629); only hours/day and age-25–44 clear 0.05. Paper hedges "If this result persists…". Subgroups underpowered (~35/group); sample skews young, lower-income, India/Pakistan. Authors are Copilot's makers (COI — though this cuts toward credibility for an anti-hype "novices benefit more" result they had no incentive to manufacture).
|
||
|
||
**Sources:** arxiv.org/pdf/2302.06590 (critique: blueoptima.com/post/debunking-githubs-claims…)
|
||
|
||
---
|
||
|
||
## Finding 5 — DORA 2024: AI hurt team delivery stability/throughput (confidence: MEDIUM)
|
||
|
||
**Claim:** DORA's 2024 research: AI adoption raised individual productivity/satisfaction but **negatively impacted software delivery stability and throughput** at the team level — individual speed gains did not translate to better team delivery.
|
||
|
||
**Evidence:** Verbatim (dora.dev/research/2024): *"AI adoption significantly increases individual productivity, flow, and job satisfaction. However, it also negatively impacts software delivery stability and throughput."* Magnitudes (corroborated by RedMonk/InfoQ/GetDX): a 25% increase in AI adoption ≈ 1.5% throughput decrease, 7.2% stability reduction; mechanism = AI inflating batch/changeset size. Independent primary research (Google/DORA), not vendor marketing.
|
||
|
||
- **Two mandatory caveats:** (1) observational/self-reported cross-sectional survey → CORRELATION not causation. (2) **PARTIALLY SUPERSEDED** — DORA's OWN 2025 report REVERSED the throughput finding (AI now positively correlates with throughput); the negative STABILITY relationship persisted. Only the stability half remains current.
|
||
|
||
**Sources:** dora.dev/research/2024/dora-report
|
||
|
||
---
|
||
|
||
## Finding 6 — Stack Overflow 2025: trust low and falling; debugging tax (confidence: HIGH, 3-0)
|
||
|
||
**Claim:** SO 2025 Developer Survey (~49,000 respondents): developer trust in AI output accuracy is low and eroding, with a documented "debugging tax." Only **~3% "highly trust"** AI accuracy while **~46% actively distrust** (up from 31% in 2024; now exceeding the ~33% who trust), even as adoption climbed to 84%; **~45% report debugging AI-generated code is time-consuming.**
|
||
|
||
**Evidence:** Verbatim (survey.stackoverflow.co/2025/ai, n=33,244 on trust; Dec 2025). Highly trust 3.1%, somewhat 29.6%, somewhat distrust 26.1%, highly distrust 19.6% → ~46% distrust vs 32.7% trust. *"Trust in the accuracy of AI has fallen from 40% in previous years to just 29% this year."* Press release: "Trust in AI at an All Time Low." Adoption rose to 84% (from 76%); 51% daily among pros.
|
||
|
||
- **Caveats:** all self-reported perception/trust, not direct output-reliability measures. SO has mild business interest in AI skepticism.
|
||
- **⚠ Data-hygiene note (verification caught this):** two claims asserting "66% spend more time fixing almost-right AI code" were **REFUTED** — the 66% item measures *encountering* "almost right, but not quite" solutions; the cleanest "more time" figure is **45%**. Cite 45%, not 66%. Several secondary summaries conflate them.
|
||
|
||
**Sources:** survey.stackoverflow.co/2025/ai · stackoverflow.co/company/press/archive/stack-overflow-2025-developer-survey · stackoverflow.blog/2025/12/29/…
|
||
|
||
---
|
||
|
||
## Overall caveats (the adversarial source held to its own standard)
|
||
|
||
- **TIME-SENSITIVITY (biggest limit):** every figure is tied to a fast-moving tool generation. METR's −19% is explicitly "historical"/early-2025 (Cursor Pro + Claude 3.5/3.7 Sonnet); the Copilot RCT uses 2022-era Copilot; DORA's 2024 throughput finding was reversed by DORA in 2025. None represents the live 2026 state of the art.
|
||
- **STATISTICAL POWER/SAMPLE:** METR n=16 (though significant across 246 tasks); its population is precisely the cohort least likely to gain, so it does NOT show AI slows juniors/greenfield/most devs (METR is explicit about these non-conclusions). Copilot heterogeneity subgroups underpowered; key experience coefficient not significant at 0.05.
|
||
- **SCOPE/DOMAIN MISMATCH:** the strongest skill-distribution result (Brynjolfsson, 34% novice gain) is customer-support agents, not developers — directional, not same-population proof.
|
||
- **DESIGN LADDER:** RCTs (METR, Copilot) > quasi-experiment (Brynjolfsson) > observational surveys (DORA, SO). "Zeroed cost" is undercut by inference, not a direct cost measurement.
|
||
- **SOURCE BIAS:** GitHub/Microsoft authored the pro-Copilot RCT (COI); SO has mild commercial interest in AI skepticism; DORA/Google is independent of the tool vendors.
|
||
|
||
## Open questions
|
||
|
||
1. Does METR's −19% persist/reverse with mid-2026 agentic tools? Its own Feb-2026 update hints at possible speedup but calls it "very weak evidence"; a properly powered follow-up RCT on current tools is the key missing piece.
|
||
2. Does "novices gain most" hold for REAL software work, or invert on complex/unfamiliar codebases where juniors lack judgment to catch AI errors (the mechanism behind experts' small quality declines)?
|
||
3. All findings measure task SPEED/productivity — the "seniors up/juniors out" thesis is about labor-market VALUE (wages, hiring, headcount), which no source here measures. Directional counter-evidence, not same-metric refutation.
|
||
4. Are AI's negative team-level effects (larger batches, debugging burden, low trust) a transient learning-curve cost or a durable tax? DORA's 2025 partial reversal suggests some are transitional; persistent stability decline and falling trust suggest others may not be.
|
||
|
||
## Refuted during verification (recorded for honesty)
|
||
|
||
- "66% report spending MORE time fixing almost-right AI code" (0-3 and 1-2) — conflates SO's 66% "encounter almost-right solutions" item with the 45% "debugging is time-consuming" item. Use 45%.
|