3314112bb9
Add two new sources with summaries, new concepts (developer-as-agent-manager, review-is-the-new-bottleneck), new entities (SWEPR, Nikolai Sheiko), and a query on the Stanford source; update related concept pages, overview, index, and log.
49 lines
5.1 KiB
Markdown
49 lines
5.1 KiB
Markdown
# Stanford SWEPR — AI and the widening productivity gap
|
||
|
||
#source
|
||
|
||
## Source Metadata
|
||
|
||
- **Date:** research presented through 2025 (chart data April 2023 → July 2025; deck Sept 2025); dossier compiled 2026-07-30
|
||
- **Raw path:** `raw/sources/Stanford SWEPR - AI and the widening productivity gap.md` (screenshot: `raw/assets/G6g3O60bkAE05ZW.png`)
|
||
- **Source type:** research dossier — a slide read first-hand plus public coverage of the underlying study; **not** a talk transcript. The corpus's first *quantitative outside study* (every other source is practitioner testimony or synthesis).
|
||
- **Authors:** [[swepr]] (Stanford Software Engineering Productivity Research group), public face Yegor Denisov-Blanch
|
||
- **Ingestion date:** 2026-07-30
|
||
|
||
## Core Claims
|
||
|
||
- **The gap between AI-mastering teams and laggards is widening, measurably.** Difference-in-differences analysis of 46 AI-adopting teams vs 46 matched non-AI teams: net productivity difference between top and bottom quartiles grew from **4.8% (April 2023) to 19% (July 2025) — a 4× increase** in ~2.25 years. Early quarters hover near zero or negative; the curve only takes off from mid-2024, then accelerates.
|
||
- **Average net gain from AI is ~15–20%, not 10×.** Gross delivered code volume rises 30–40%, but rework (fixing AI-introduced bugs) eats roughly half.
|
||
- **Gains are strongly context-dependent:** greenfield/low-complexity 30–40%; brownfield/high-complexity 0–10% and can be *negative*. Popular languages gain more than niche ones; gains collapse as codebases grow 10k → 10M lines (context-window limits, signal-to-noise).
|
||
- **The cost moved downstream:** coverage reports +91% PR review time and ~2.6× rework in AI-heavy workflows — writing got cheaper, reviewing got more expensive.
|
||
- **Proposed mechanism for the gap:** quality of AI usage beats volume; teams with clean, modular, well-tested code compound gains, teams with poor hygiene accumulate debt and lose trust in the tools.
|
||
|
||
## Key Evidence / Details
|
||
|
||
- Data: private Git repos, 600+ companies, ~100k–120k engineers, since 2022; ML model replicating a panel of expert reviewers, measuring *functionality delivered* (not commits/LOC). Methodology peer-reviewed (arXiv 2409.15152); the DiD result itself is talk-published only.
|
||
- The slide: "Causal Impact of AI on Software Engineering Productivity: Difference-in-Differences Analysis," DID covariate balance < 0.25, 95% CI band, model-release markers on the time axis. Deck: "Will AI Replace Software Engineers?", AI Conference, Sept 2025.
|
||
- Full numbers, links, and per-claim provenance in the raw dossier.
|
||
|
||
## Connections
|
||
|
||
- **Entity:** [[swepr]]
|
||
- **Corroborates:** [[2026-07-14-gap-between-ai-users-irreversible]] — Allie Miller's "irreversible gap" prediction is this curve, asserted 18 months earlier from advisory experience; Stanford supplies the measurement. Also the team-level twin of [[levels-of-ai-usage]] (the gap grows between *rungs of mastery*, not between license-holders and others).
|
||
- **Corroborates:** [[context-as-scarce-resource]] — the codebase-size finding (gains collapse toward 10M LOC, attributed to context-window limits and signal-to-noise) is the corpus's first outside quantitative support for context as the binding constraint.
|
||
- **Complicates:** [[make-more-cheap-code]] / [[code-as-throwaway]] — +91% PR review time and 2.6× rework externally confirm that generation moved the cost to review/verification, which is Theo's premise; but the *negative* gains in complex brownfield code sharpen the honest caveat that "code is cheap" holds least where most code lives.
|
||
- **Independently cited by a practitioner source:** [[2026-07-30-rakes-in-ai-sdlc-adoption]] builds its stakes on this study ("the Stanford study, Aug 2025: +20% speed, but the top performers pull away — the goal is to be in the top half of the chart") and anecdotally mirrors its downstream-cost finding (the +1%-despite-more-PRs rework case; review as the bottleneck — see [[review-is-the-new-bottleneck]]).
|
||
- **Queries:** [[2026-07-30-stanford-widening-gap-source]] (how this source was traced), [[2026-07-28-webinar-theses]] (thesis T7 gains its number)
|
||
|
||
## Open Questions
|
||
|
||
- Does the 46-team DiD analysis ever get a peer-reviewed publication? (Claim status upgrades when it does.)
|
||
- What exactly distinguishes the mastering teams — tooling, codebase hygiene, or skills/context practices? The proposed mechanism ("quality of usage") is asserted in talks, not decomposed. Directly relevant to whether the webinar's skills rung is *the* differentiator. Status: tentative.
|
||
- The study measures teams of engineers; how far do team-level results transfer to the webinar's non-engineer audience? Status: tentative.
|
||
|
||
## Change Impact on Wiki
|
||
|
||
- Created [[swepr]] entity.
|
||
- Added measured external evidence to [[levels-of-ai-usage]], [[context-as-scarce-resource]] and [[make-more-cheap-code]].
|
||
- Upgraded the corroboration pointer on [[2026-07-14-gap-between-ai-users-irreversible]] to cite this page.
|
||
- Added the "Stanford measured it: 4× in two years" note to thesis T7 in [[2026-07-28-webinar-theses]].
|
||
- Updated [[overview]] (11 → 12 sources; first quantitative outside study), `index.md`, `log.md`.
|