3314112bb9
Add two new sources with summaries, new concepts (developer-as-agent-manager, review-is-the-new-bottleneck), new entities (SWEPR, Nikolai Sheiko), and a query on the Stanford source; update related concept pages, overview, index, and log.
5.6 KiB
Stanford SWEPR — AI and the widening productivity gap
Research dossier compiled 2026-07-30. Trigger: a saved screenshot (raw/assets/G6g3O60bkAE05ZW.png, X/Twitter image filename) of a Stanford slide titled "Teams that master AI are accelerating their productivity gains, widening the gap with laggards." This document assembles what the underlying research says, from the slide itself plus public coverage. It is NOT a first-hand transcript of a talk; per-claim provenance is marked below.
Who / what
- Group: Software Engineering Productivity Research (SWEPR), Stanford University — https://softwareengineeringproductivity.stanford.edu/
- Lead researcher (public face): Yegor Denisov-Blanch — https://yegordb.com/
- Data: private Git repositories from 600+ companies, ~100,000–120,000 software engineers, tens of millions of commits, collected since 2022. (Coverage varies between "100k" and "120k+"; the group's own site says 120,000+.)
- Method: a machine-learning model trained to replicate a panel of human expert reviewers scoring every commit — measuring functionality delivered rather than commit counts or lines of code. Methodology is peer-reviewed: "Predicting Expert Evaluations in Software Code Reviews" (https://arxiv.org/pdf/2409.15152); a companion paper tests LLM determinism in code review (https://arxiv.org/pdf/2502.20747).
The slide (primary evidence — read directly from the screenshot)
Slide 3 of a deck; footer "Stanford University / SWEPR / Software Engineering Productivity Research Group." Title: "Teams that master AI are accelerating their productivity gains, widening the gap with laggards."
Chart: "Causal Impact of AI on Software Engineering Productivity: Difference-in-Differences Analysis."
- Method steps on the slide: (1) identified 46 teams that used AI; (2) matched with 46 similar non-AI teams; (3) measured net productivity gains from AI quarterly.
- Y-axis: net productivity gain (%), causal vs the matched control group; median line + 95% CI band (Q25–Q75).
- X-axis: April 2023 → July 2025, with model-release markers annotated along the top (GPT-era releases).
- April 2023: 4.8% Q1–Q3 difference. July 2025: 19% Q1–Q3 difference. Labeled "Widening Gap: 4× increase."
- Fine print: "DID Covariate Balance < 0.25".
- Early quarters (Apr 2023–Jan 2024) hover around 0% or slightly negative — the gap only opens from mid-2024 onward and then accelerates.
The slide matches Denisov-Blanch's September 2025 AI Conference talk "Will AI Replace Software Engineers?" (deck PDF: https://aiconference.com/wp-content/uploads/2025/09/Yegor-Denisov-Blanch-Will-AI-Replace-Software-Engineers_-.pptx.pdf). A video version of the material: "Can you prove AI ROI in Software Eng? (Stanford 120k Devs Study)" — https://www.youtube.com/watch?v=JvosMkuNxF8
Broader findings of the study (from public coverage of the talks)
Headline: AI coding tools deliver ~15–20% net average productivity gain — not the 10× of vendor marketing, and not zero. Gross delivered code volume rises 30–40%, but rework (fixing AI-introduced bugs) eats roughly half the gross gain.
Gains vary sharply by context (the "it depends" matrix):
| Context | Net gain |
|---|---|
| Greenfield, low complexity | 30–40% |
| Greenfield, high complexity | 10–15% |
| Brownfield (legacy), low complexity | 15–20% |
| Brownfield (legacy), high complexity | 0–10%, can be negative |
- Language popularity matters: popular languages (Python, Java, JS/TS) ~20% on simple tasks, 10–15% on complex; niche languages (COBOL, Haskell, Elixir) minimal or negative — thin AI training data.
- Codebase size matters: gains fall sharply as codebases grow from ~10k to ~10M lines — attributed to context-window limits, signal-to-noise degradation, and domain-specific logic.
- Coverage of the same research reports a 91% increase in PR review time and a ~2.6× increase in rework in AI-heavy workflows, i.e. the cost moved downstream from writing to reviewing. (Secondary coverage; not read off a primary slide.)
- Interpretation offered by the researchers in talks/coverage: quality of AI usage beats volume of AI usage; teams with clean, modular, well-tested codebases compound gains, while teams with poor code hygiene accumulate technical debt and lose trust in the tools — one proposed mechanism for the widening gap.
- Related earlier finding from the same group (separate result, widely covered ~Nov 2024): "ghost engineers" — ~9.5% of engineers in the dataset perform virtually no verifiable work. (Contextual; distinct from the AI-impact analysis.)
Status / caveats
- The 46-vs-46 difference-in-differences result is, as of this writing, talk/deck-published only — presented at conferences and webinars, not (yet) in a peer-reviewed paper. The peer-reviewed papers cover the measurement methodology.
- All numbers outside the slide itself come from secondary coverage of the talks and may compress or paraphrase.
- The study's data is proprietary (companies opt in), so independent replication is not possible from outside.
Links (all, in one place)
- Group site: https://softwareengineeringproductivity.stanford.edu/
- Sept 2025 AI Conference deck: https://aiconference.com/wp-content/uploads/2025/09/Yegor-Denisov-Blanch-Will-AI-Replace-Software-Engineers_-.pptx.pdf
- Talk video: https://www.youtube.com/watch?v=JvosMkuNxF8
- Researcher: https://yegordb.com/
- Methodology paper: https://arxiv.org/pdf/2409.15152
- Determinism paper: https://arxiv.org/pdf/2502.20747
- Screenshot that triggered this dossier:
raw/assets/G6g3O60bkAE05ZW.png