ingest: Stanford SWEPR widening-gap study and AI-in-SDLC adoption pitfalls
Add two new sources with summaries, new concepts (developer-as-agent-manager, review-is-the-new-bottleneck), new entities (SWEPR, Nikolai Sheiko), and a query on the Stanford source; update related concept pages, overview, index, and log.
This commit is contained in:
72
.obsidian/workspace.json
vendored
72
.obsidian/workspace.json
vendored
@@ -13,15 +13,28 @@
|
||||
"state": {
|
||||
"type": "markdown",
|
||||
"state": {
|
||||
"file": "index.md",
|
||||
"file": "raw/sources/Грабли во внедрении ИИ в SDLC.md",
|
||||
"mode": "source",
|
||||
"source": false
|
||||
},
|
||||
"icon": "lucide-file",
|
||||
"title": "index"
|
||||
"title": "Грабли во внедрении ИИ в SDLC"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "2d46e86fefb0f38a",
|
||||
"type": "leaf",
|
||||
"state": {
|
||||
"type": "release-notes",
|
||||
"state": {
|
||||
"currentVersion": "1.13.4"
|
||||
},
|
||||
"icon": "lucide-book-up",
|
||||
"title": "Release Notes 1.13.4"
|
||||
}
|
||||
}
|
||||
]
|
||||
],
|
||||
"currentTab": 1
|
||||
}
|
||||
],
|
||||
"direction": "vertical"
|
||||
@@ -41,7 +54,9 @@
|
||||
"type": "file-explorer",
|
||||
"state": {
|
||||
"sortOrder": "alphabetical",
|
||||
"autoReveal": false
|
||||
"autoReveal": false,
|
||||
"showSearch": false,
|
||||
"searchQuery": ""
|
||||
},
|
||||
"icon": "lucide-folder-closed",
|
||||
"title": "Files"
|
||||
@@ -183,24 +198,37 @@
|
||||
"bases:Create new base": false
|
||||
}
|
||||
},
|
||||
"active": "5463f555de76b7cd",
|
||||
"active": "2d46e86fefb0f38a",
|
||||
"lastOpenFiles": [
|
||||
"index.md.tmp.37928.2285b1ab105c",
|
||||
"index.md.tmp.37928.291fc158bc9a",
|
||||
"index.md.tmp.37928.2c928edacabb",
|
||||
"wiki/overview.md.tmp.37928.21bfbe643084",
|
||||
"wiki/overview.md.tmp.37928.58de935b7d14",
|
||||
"wiki/overview.md.tmp.37928.5dfad5760432",
|
||||
"wiki/overview.md.tmp.37928.e9644e4b8e6b",
|
||||
"wiki/overview.md.tmp.37928.fdb65c168940",
|
||||
"wiki/concepts/make-more-cheap-code.md.tmp.37928.96f5312269c7",
|
||||
"wiki/concepts/enterprise-ai-reality.md.tmp.37928.e11badf2e036",
|
||||
"wiki/concepts/enterprise-ai-reality.md.tmp.37928.a6c2340fbd91",
|
||||
"wiki/concepts/developer-as-agent-manager.md",
|
||||
"wiki/concepts/review-is-the-new-bottleneck.md",
|
||||
"wiki/entities/nikolai-sheiko.md",
|
||||
"wiki/sources/2026-07-30-rakes-in-ai-sdlc-adoption.md",
|
||||
"raw/assets/G6g3O60bkAE05ZW.png",
|
||||
"wiki/concepts/harness.md",
|
||||
"raw/sources/Грабли во внедрении ИИ в SDLC.md",
|
||||
"wiki/entities/swepr.md",
|
||||
"wiki/sources/2026-07-30-stanford-swepr-widening-gap.md",
|
||||
"raw/sources/Stanford SWEPR - AI and the widening productivity gap.md",
|
||||
"wiki/queries/2026-07-30-stanford-widening-gap-source.md",
|
||||
"index.md",
|
||||
"wiki/queries/2026-07-28-webinar-theses.md",
|
||||
"wiki/queries/2026-07-22-webinar-theses.md",
|
||||
"raw/notes/my theses.md",
|
||||
"raw/notes/Webinar Plan - From Chat Box to Your Own OS.md",
|
||||
"raw/notes/Webinar script.md",
|
||||
"raw/sources/А что если наВайб-Кодить.md",
|
||||
"index.md.tmp.28724.8b2cd0672e68",
|
||||
"index.md.tmp.28724.bf94cff6ee0a",
|
||||
"wiki/overview.md.tmp.28724.772546efd332",
|
||||
"wiki/overview.md.tmp.28724.8c8f2e4cf507",
|
||||
"wiki/overview.md.tmp.28724.0d66b3b4bb53",
|
||||
"wiki/overview.md.tmp.28724.026fed72c5e1",
|
||||
"wiki/overview.md.tmp.28724.331375f61a61",
|
||||
"wiki/concepts/code-as-throwaway.md.tmp.28724.a614242c473d",
|
||||
"wiki/concepts/code-as-throwaway.md.tmp.28724.296cfad627cf",
|
||||
"wiki/concepts/emacsification-of-software.md.tmp.28724.4ff35ff220f6",
|
||||
"wiki/concepts/maintenance-is-the-real-cost.md",
|
||||
"wiki/sources/2026-07-29-what-if-we-vibe-code-it.md",
|
||||
"raw/sources/In 1 Year, the Gap Between AI Users and Everyone Else Will Be Irreversible.md",
|
||||
@@ -208,18 +236,6 @@
|
||||
"wiki/concepts/enterprise-ai-reality.md",
|
||||
"wiki/concepts/code-as-throwaway.md",
|
||||
"wiki/lint-reports/2026-07-28-lint.md",
|
||||
"wiki/concepts/emacsification-of-software.md",
|
||||
"raw/sources/Agentic Engineering, explained by a 10x developer.md",
|
||||
"wiki/concepts/async-by-default.md",
|
||||
"wiki/concepts/explosion-of-internal-software.md",
|
||||
"wiki/concepts/build-for-the-agent-not-the-human.md",
|
||||
"wiki/concepts/shedding-weight.md",
|
||||
"wiki/entities/amp.md",
|
||||
"wiki/entities/thorsten-ball.md",
|
||||
"wiki/sources/2026-07-28-agentic-engineering-10x-developer.md",
|
||||
"dashboard.md",
|
||||
"mockups/README.md",
|
||||
"wiki/concepts/evolution-of-agent-tooling.md",
|
||||
"wiki/script-coverage.md"
|
||||
"wiki/concepts/emacsification-of-software.md"
|
||||
]
|
||||
}
|
||||
7
index.md
7
index.md
@@ -19,6 +19,8 @@ Every wiki page carries a page-type tag under its H1 (`#source`, `#entity`, `#co
|
||||
- [[2026-07-24-youre-reading-way-too-much-code]] — Theo Browne (video): make more cheap code, four tiers of code, 100:1 slop-to-ship verification _(raw: You're reading way too much code.md)_
|
||||
- [[2026-07-28-agentic-engineering-10x-developer]] — Thorsten Ball (AMP): shed weight, orbs/async, Emacsification, internal software; **rejects skills/MCP** _(raw: Agentic Engineering, explained by a 10x developer.md)_
|
||||
- [[2026-07-29-what-if-we-vibe-code-it]] — YouTube video (author unknown): maintenance is the real cost, internal service = second business, Jira→Linear pendulum, build-vs-buy checklist _(raw: А что если наВайб-Кодить.md)_
|
||||
- [[2026-07-30-stanford-swepr-widening-gap]] — Stanford SWEPR (research dossier): 46-vs-46 teams DiD, gap 4.8%→19% (4×); ~15–20% net avg gain; gains collapse in large/legacy codebases; +91% PR review time _(raw: Stanford SWEPR - AI and the widening productivity gap.md)_
|
||||
- [[2026-07-30-rakes-in-ai-sdlc-adoption]] — Nikolai Sheiko (talk): people/companies/metrics throttle real AI gains; review is the new bottleneck; measure tasks-without-rework; developer → agent manager; Agentic Evolution + context-free-subagent skill verification _(raw: Грабли во внедрении ИИ в SDLC.md)_
|
||||
|
||||
**All of `raw/sources/` is ingested.** The webinar-deliverable working notes now live in `raw/notes/` (`Ideas for webinar.md` · `Webinar Plan - From Chat Box to Your Own OS.md` · `Webinar script.md` · `my theses.md` · `introduction.md`) — treated as authored deliverables rather than sources, cited as raw where used.
|
||||
|
||||
@@ -34,6 +36,7 @@ Every wiki page carries a page-type tag under its H1 (`#source`, `#entity`, `#co
|
||||
- [[yulia]] — HR/recruiting lead; webinar organizer (name/affiliation tentative)
|
||||
- [[larysa]] — technical BA/PM, ex-mobile dev; advanced user blocked by memory + integrations
|
||||
- [[thorsten-ball]] — founding engineer at AMP; 99% AI-written code, no skills/MCP
|
||||
- [[nikolai-sheiko]] — AI-adoption practitioner/consultant; review bottleneck, agent-manager shift, Agentic Evolution
|
||||
|
||||
### Tools / Orgs
|
||||
- [[claude-code]] — reference harness (all sources)
|
||||
@@ -41,6 +44,7 @@ Every wiki page carries a page-type tag under its H1 (`#source`, `#entity`, `#co
|
||||
- [[hermes]] — skills-first, self-curating harness
|
||||
- [[virtido]] — Sebastian's outsourcing company; its HR team is the webinar audience
|
||||
- [[inspectron]] — Eugene's employer (Edge Compute / IoT)
|
||||
- [[swepr]] — Stanford Software Engineering Productivity Research group (Yegor Denisov-Blanch); the corpus's quantitative outside study
|
||||
|
||||
## Concepts
|
||||
|
||||
@@ -63,6 +67,7 @@ Every wiki page carries a page-type tag under its H1 (`#source`, `#entity`, `#co
|
||||
- [[network-from-a-standing-start]] — tentative from-zero networking protocol (v0, to be validated)
|
||||
- [[seniority-and-the-junior-squeeze]] — judgment as risk-reduction
|
||||
- [[decoupling-identity-from-profession]] — separate who you are from what you do
|
||||
- [[developer-as-agent-manager]] — CPU-bound coder → IO-bound manager of agent-employees; Judgment stays human; users vs Agentic Operations
|
||||
|
||||
**Strategy side**
|
||||
- [[think-wider-not-bigger]] — breadth over depth; match ambition to the model
|
||||
@@ -74,6 +79,7 @@ Every wiki page carries a page-type tag under its H1 (`#source`, `#entity`, `#co
|
||||
- [[emacsification-of-software]] — fork and remix rather than upstream; software becomes bespoke
|
||||
- [[explosion-of-internal-software]] — the Excel/wiki/hack layer becomes real tools; skill + token budget as the divide
|
||||
- [[maintenance-is-the-real-cost]] — writing was never the bottleneck; internal service = second business; build-vs-buy checklist
|
||||
- [[review-is-the-new-bottleneck]] — the SDLC collapses around the humans; review with the agent; measure completed tasks without rework, never LoC/PRs
|
||||
|
||||
## Timelines
|
||||
|
||||
@@ -91,6 +97,7 @@ Every wiki page carries a page-type tag under its H1 (`#source`, `#entity`, `#co
|
||||
- [[2026-07-28-webinar-theses]] — **v2, current**: 17 theses from all 10 sources; T3 reframed (*authored beats inferred*), verification / shedding-weight / ask-for-15-options added; 30-min cut + gaps in the current script
|
||||
- [[2026-07-28-verification-beat-design]] — how to add the missing verification beat: checker skill, the "right shelf" ambiguity demo, and the closing what-stays-yours half; drafted script copy
|
||||
- [[2026-07-24-non-engineer-throwaway-verification]] — non-engineer analog of throwaway verification code: generated checks not content; checker skills; drift as diagnostic
|
||||
- [[2026-07-30-stanford-widening-gap-source]] — traced the "widening gap" chart (`raw/assets/G6g3O60bkAE05ZW.png`) to Stanford SWEPR / Yegor Denisov-Blanch; first measured support for the irreversible-gap thesis; since ingested as [[2026-07-30-stanford-swepr-widening-gap]]
|
||||
|
||||
## Lint Reports
|
||||
|
||||
|
||||
38
log.md
38
log.md
@@ -230,3 +230,41 @@ Entry format:
|
||||
- Uncertainty flagged: the pendulum case is second-hand tweets with fuzzy company identification (tentative); the checklist is prescriptive, not observed; the author's own company is currently building a Datadog replacement — if it ships and survives, he becomes his own counterexample. Open question with no evidence either way in the corpus: does the maintenance objection survive *agents* doing the maintenance ([[agentic-loops]], [[async-by-default]])?
|
||||
- Effect on the last lint's watch item: [[emacsification-of-software]] and [[explosion-of-internal-software]] were merge candidates "if neither gains second-source support by the next lint" — both now have second-source engagement (as a bounding counterpoint), which argues for keeping them separate with [[maintenance-is-the-real-cost]] as the shared boundary page.
|
||||
- Next: the standing recommendations are unchanged (a `taste` concept page; extend [[theo-konstantin-allie]] to four lenses; the skills falsification test). New candidate question for the HR audience: which of their candidate tools (candidate knowledge base, transcribe→summarize) fall on the safe side of the build-vs-buy checklist — directly webinar-relevant if Q&A raises "should we build or buy?"
|
||||
|
||||
## 2026-07-30 — query (Stanford "widening gap" chart traced to source)
|
||||
- Intent: query
|
||||
- Input: user saved `raw/assets/G6g3O60bkAE05ZW.png` (X/Twitter screenshot of a Stanford slide, "Teams that master AI are accelerating their productivity gains, widening the gap with laggards") and asked to find the original research.
|
||||
- Pages created: [[2026-07-30-stanford-widening-gap-source]] (query — the slide is from Stanford's Software Engineering Productivity Research group (SWEPR), Yegor Denisov-Blanch; 600+ companies / 120k+ engineers since 2022; the chart is a 46-vs-46-team difference-in-differences analysis showing the AI-adopter productivity gap growing 4.8% → 19% (4×) from April 2023 to July 2025; slide matches his Sept 2025 AI Conference deck "Will AI Replace Software Engineers?"; primary links recorded on the page).
|
||||
- Pages updated: [[2026-07-14-gap-between-ai-users-irreversible]] (external-corroboration pointer under Connections — Allie's title claim gains its first measured, non-practitioner support), `index.md` (Queries).
|
||||
- Notes: the DiD result itself is talk/deck-published, not peer-reviewed — marked tentative; only the measurement methodology has a peer-reviewed paper (arXiv 2409.15152). Nuance recorded: same study finds ~15–20% average gains with AI *decreasing* net productivity in complex legacy codebases (rework, +91% PR review time) — honest-caveat material aligning with [[maintenance-is-the-real-cost]] and [[make-more-cheap-code]]. No concept pages changed (citation policy: the talk is not yet an ingested source).
|
||||
- Next: decide whether to ingest the talk/deck as a proper `raw/sources/` doc (would let concept pages and the webinar theses cite it as `wiki/sources/*` evidence); consider adding the "Stanford measured it: 4× in two years" line to the webinar's stakes beat; watch for a peer-reviewed version of the DiD analysis.
|
||||
|
||||
## 2026-07-30 — ingest (Stanford SWEPR — AI and the widening productivity gap)
|
||||
- Intent: ingest
|
||||
- Input: user — "Create the source. I'm giving you permission to write a new file in the sources folder." **Authorized raw write** (exception to rule 1, new file only): created `raw/sources/Stanford SWEPR - AI and the widening productivity gap.md`, a research *dossier* compiled from the saved slide screenshot (`raw/assets/G6g3O60bkAE05ZW.png`) plus public coverage — honestly marked as not-a-transcript, with per-claim provenance (slide-read vs secondary coverage). 12th source; the corpus's first quantitative outside study.
|
||||
- Pages created: [[2026-07-30-stanford-swepr-widening-gap]] (source), [[swepr]] (entity — Stanford Software Engineering Productivity Research group, Yegor Denisov-Blanch; 15th entity).
|
||||
- Pages updated: [[levels-of-ai-usage]] (evidence: measured team-level twin of the mastery gap, with the non-engineer-audience caveat), [[context-as-scarce-resource]] (evidence: gains collapse toward 10M LOC via context-window limits — first outside quantitative support for context-as-constraint), [[make-more-cheap-code]] (evidence: +91% PR review time / 2.6× rework = the cost moving downstream, measured; brownfield-negative caveat), [[2026-07-14-gap-between-ai-users-irreversible]] (corroboration pointer upgraded to cite the source page), [[2026-07-28-webinar-theses]] (T7 note: thesis upgraded from prediction to measurement, with a citable stage line), [[2026-07-30-stanford-widening-gap-source]] (ingest follow-up marked done), [[overview]] (11→12 sources; stakes-claim measurement added to the agree section; navigation), `index.md`.
|
||||
- Notes: the source *corroborates* rather than contradicts — no new contradictions logged, but three caveats recorded on the source page and carried into every citing page: the DiD analysis is talk-published, not peer-reviewed (tentative); the rework/review numbers are secondary coverage; the study measures engineering teams, not the webinar's non-engineer audience. Deliberately did **not** claim the study validates the *skills* rung specifically — Stanford asserts "quality of usage" as the differentiator without decomposing it, so it cannot arbitrate contradiction #1 (skills vs no-skills). The webinar theses' T7 gains the corpus's best number: gap 4.8% → 19% (4×), April 2023 → July 2025.
|
||||
- Next: watch for a peer-reviewed version of the DiD analysis (status upgrade); candidate webinar edit — the one-line "Stanford measured it: the gap quadrupled in two years" in the stakes beat (user's call, script is an authored deliverable); standing recommendations unchanged (a `taste` concept page; extend [[theo-konstantin-allie]] to four lenses; the skills falsification test).
|
||||
|
||||
## 2026-07-30 — maintenance (authorized raw edit: Stanford beat in the script's closing arc)
|
||||
- Intent: maintenance
|
||||
- Input: user — "good idea to add it to the webinar script. Do it."
|
||||
- **Authorized raw edit** (exception to rule 1, on direct instruction; same pattern as the 2026-07-14/07-28 script edits): `raw/notes/Webinar script.md` — inserted a Stanford beat into the closing arc, between "You don't buy it. You build it — one small tool at a time." and "We started this journey…". The beat: Stanford tracked 46 AI teams vs 46 matched non-AI teams for 2+ years → the teams that *learned* it pulled away from the ones that just *had* it → spread under 5% (spring 2023) → 19% (summer 2025) → "the gap quadrupled in two years" → callback to the script's own reveal: "everyone had the same models the whole time. The difference was never the model. It was who built something around it."
|
||||
- Placement rationale: the closing arc is where the script's "same model, different harness" argument lands, and the Stanford curve is that exact argument as data — it also answers "why start now" right before the final chat-box→OS callback. Phrasing kept factually careful: the widening spread is *among AI-using teams* (masters vs laggards), so the beat says "pulled away from the ones that just had it," not "AI users vs non-users."
|
||||
- Stage-safety/honesty note added inline (`_note:`): source pointer to [[2026-07-30-stanford-swepr-widening-gap]] plus the three Q&A caveats (talk-published not peer-reviewed; software teams not office workers; "quality of usage" asserted but not decomposed).
|
||||
- Pages changed: [[2026-07-28-webinar-theses]] (T7 note updated — dramatized in the script as of today, promoted from Q&A material to an on-stage beat). No other wiki pages changed; `index.md` unchanged (no catalog change).
|
||||
- Note on scope: the Stanford research itself was already fully ingested earlier today ([[2026-07-30-stanford-swepr-widening-gap]] + [[swepr]] — see the previous ingest entry); this operation only carries the number into the deliverable.
|
||||
- Next: unchanged from the ingest entry (peer-review watch; `taste` concept page; four-lens comparison; skills falsification test).
|
||||
|
||||
## 2026-07-30 — ingest (Грабли во внедрении ИИ в SDLC / Nikolai Sheiko)
|
||||
- Intent: ingest
|
||||
- Input: `raw/sources/Грабли во внедрении ИИ в SDLC.md` — viewer's conclusions from Nikolai Sheiko's 45:59 Russian YouTube talk ("why the AI is there but the results aren't"). 13th source; `raw/sources/` fully ingested again.
|
||||
- Pages created: [[2026-07-30-rakes-in-ai-sdlc-adoption]] (source), [[nikolai-sheiko]] (entity — 16th), [[review-is-the-new-bottleneck]] and [[developer-as-agent-manager]] (concepts — 26th and 27th).
|
||||
- Core of the source: models are already good enough — **people, companies and metrics throttle the gains by an order of magnitude**. The SDLC collapsed into days/hours but not around the two human "red squares" (reviewer, planner); the fix is review *with* the agent plus the one metric that resists gaming — **completed tasks without rework** (never LoC/commits/PRs; his outsourcer case: more PRs, +1% net). Error #0: the AI-developer is an IO-bound **manager of an agent-employee**, not a CPU-bound coder ("sit watching Claude Code work = bad employee") — and not everyone can or should switch. "Companies no longer need custom AI development — install Claude Code/Codex, configure, attach connectors, mind security." **Agentic Evolution**: walk the agent through hard tasks → "remember this and write the manual for the next one" → verify via a **context-free subagent** solving the task from the skill alone. Compaction curse on big codebases → best practices exist *for the agent* (locality, interfaces, AST search over grep); embeddings/RAG over code rejected flatly.
|
||||
- Why it matters to this vault: (1) **second practitioner vote for the skills layer**, narrowing the evidential asymmetry Thorsten's dissent enjoyed on [[skills-as-memory]] — and his verification protocol is the corpus's first described run of anything like the proposed skills falsification test (with-skill half only; no without-skill control, so the test question stands). (2) **Independent citation of the Stanford chart** ([[swepr]]) as his stakes slide, plus an anecdotal mirror of its +91%-review-time finding — recorded on [[2026-07-30-stanford-swepr-widening-gap]]. (3) Names the org-level bottleneck the corpus had only as an open question on [[async-by-default]] (parallel diffs pile up on a human) — now a page: [[review-is-the-new-bottleneck]]. (4) His anti-RAG-for-code stance supports the audience-driven reading of the skills-vs-RAG contradiction on [[context-as-scarce-resource]] (anti-RAG votes are about code/procedures; the pro-RAG vote is about business data).
|
||||
- Pages updated: [[solve-first-then-skillify]] (Agentic Evolution + 5-step verification protocol; next-question partially answered), [[skills-as-memory]] (evidence + asymmetry softened + falsification-test note), [[context-as-scarce-resource]] (compaction curse; codebase-stores-context converging with Thorsten from the opposite direction; AST search; RAG contradiction note), [[async-by-default]] (IO-bound-manager evidence; related links to both new concepts), [[enterprise-ai-reality]] (no-custom-AI-dev quote as the managed-harness market seconded; external-configurator vs teacher/curator anti-pattern; Cursor metered billing → team economizes, the metered-vs-subscription split observed organizationally; tokens-dearer-then-cheaper matching Eugene's prediction), [[make-more-cheap-code]] (related link to the review-bottleneck page), [[2026-07-30-stanford-swepr-widening-gap]] (independent-citation pointer), [[overview]] (12→13 sources; new "adoption side" bullet; agree/diverge updates; skills-dissent paragraph rebalanced), `index.md`.
|
||||
- New contradiction logged (on [[developer-as-agent-manager]]): Sheiko's "don't force everyone" vs the irreversible/compounding gap (Allie, Stanford) — opting out is legitimate *and* costly; no source reconciles the two. Also tentative: all client cases are anonymous self-reported anecdotes; "embeddings over code don't work" has no mechanism given; the users-vs-Agentic-Operations split is prediction, not observation.
|
||||
- Uncertainty about the speaker himself: affiliation unknown; "you don't need custom AI development" is also a consultant's pitch — flagged on [[nikolai-sheiko]].
|
||||
- Webinar relevance noted but not applied (script untouched): the Intelligence-vs-Judgment framing is close kin to the planned "what stays yours" closing beat, and the talk independently strengthens the case for the standing `taste` concept-page recommendation (Judgment = taste-built-over-years or domain expertise — a second source alongside Thorsten's).
|
||||
- Next: standing recommendations unchanged (a `taste` concept page — now with two sources backing it; extend [[theo-konstantin-allie]] to four lenses; the skills falsification test — half-run by this source, control still missing). New candidate question: what does review-with-the-agent look like concretely (no transcript of it done well exists in the corpus).
|
||||
|
||||
BIN
raw/assets/G6g3O60bkAE05ZW.png
Normal file
BIN
raw/assets/G6g3O60bkAE05ZW.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 343 KiB |
@@ -638,6 +638,32 @@ Your harness is unique to you.
|
||||
|
||||
You don't buy it. You build it — one small tool at a time.
|
||||
|
||||
---
|
||||
|
||||
And don't take my word for why this matters *now*.
|
||||
|
||||
Stanford measured it.
|
||||
|
||||
Their researchers tracked 46 teams working with AI — matched against 46 similar teams without it — for more than two years.
|
||||
|
||||
The teams that really learned it pulled away from the ones that just... had it.
|
||||
|
||||
In spring 2023, the spread between them was under five percent.
|
||||
|
||||
By summer 2025 — nineteen.
|
||||
|
||||
The gap quadrupled in two years. And the curve is still bending upward.
|
||||
|
||||
And remember — everyone had the same models the whole time.
|
||||
|
||||
The difference was never the model.
|
||||
|
||||
It was who built something around it.
|
||||
|
||||
_note: source — Stanford SWEPR, difference-in-differences analysis, Apr 2023 → Jul 2025 (see wiki/sources/2026-07-30-stanford-swepr-widening-gap.md). Honest caveats if asked in Q&A: talk-published, not yet peer-reviewed; measures software teams, not office workers; Stanford says "quality of usage" decides, without naming which practice._
|
||||
|
||||
---
|
||||
|
||||
We started this journey by pasting an email into a chat box.
|
||||
|
||||
We're ending it with a button that already knows what the email said.
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Stanford SWEPR — AI and the widening productivity gap
|
||||
|
||||
_Research dossier compiled 2026-07-30. Trigger: a saved screenshot (`raw/assets/G6g3O60bkAE05ZW.png`, X/Twitter image filename) of a Stanford slide titled "Teams that master AI are accelerating their productivity gains, widening the gap with laggards." This document assembles what the underlying research says, from the slide itself plus public coverage. It is NOT a first-hand transcript of a talk; per-claim provenance is marked below._
|
||||
|
||||
## Who / what
|
||||
|
||||
- **Group:** Software Engineering Productivity Research (SWEPR), Stanford University — https://softwareengineeringproductivity.stanford.edu/
|
||||
- **Lead researcher (public face):** Yegor Denisov-Blanch — https://yegordb.com/
|
||||
- **Data:** private Git repositories from **600+ companies**, **~100,000–120,000 software engineers**, tens of millions of commits, collected since 2022. (Coverage varies between "100k" and "120k+"; the group's own site says 120,000+.)
|
||||
- **Method:** a machine-learning model trained to replicate a **panel of human expert reviewers** scoring every commit — measuring *functionality delivered* rather than commit counts or lines of code. Methodology is peer-reviewed: "Predicting Expert Evaluations in Software Code Reviews" (https://arxiv.org/pdf/2409.15152); a companion paper tests LLM determinism in code review (https://arxiv.org/pdf/2502.20747).
|
||||
|
||||
## The slide (primary evidence — read directly from the screenshot)
|
||||
|
||||
Slide 3 of a deck; footer "Stanford University / SWEPR / Software Engineering Productivity Research Group." Title: **"Teams that master AI are accelerating their productivity gains, widening the gap with laggards."**
|
||||
|
||||
Chart: *"Causal Impact of AI on Software Engineering Productivity: Difference-in-Differences Analysis."*
|
||||
|
||||
- Method steps on the slide: (1) identified **46 teams that used AI**; (2) matched with **46 similar non-AI teams**; (3) measured **net productivity gains from AI quarterly**.
|
||||
- Y-axis: net productivity gain (%), causal vs the matched control group; median line + 95% CI band (Q25–Q75).
|
||||
- X-axis: April 2023 → July 2025, with model-release markers annotated along the top (GPT-era releases).
|
||||
- **April 2023: 4.8% Q1–Q3 difference. July 2025: 19% Q1–Q3 difference. Labeled "Widening Gap: 4× increase."**
|
||||
- Fine print: "DID Covariate Balance < 0.25".
|
||||
- Early quarters (Apr 2023–Jan 2024) hover around **0% or slightly negative** — the gap only opens from mid-2024 onward and then accelerates.
|
||||
|
||||
The slide matches Denisov-Blanch's September 2025 AI Conference talk **"Will AI Replace Software Engineers?"** (deck PDF: https://aiconference.com/wp-content/uploads/2025/09/Yegor-Denisov-Blanch-Will-AI-Replace-Software-Engineers_-.pptx.pdf). A video version of the material: "Can you prove AI ROI in Software Eng? (Stanford 120k Devs Study)" — https://www.youtube.com/watch?v=JvosMkuNxF8
|
||||
|
||||
## Broader findings of the study (from public coverage of the talks)
|
||||
|
||||
Headline: **AI coding tools deliver ~15–20% net average productivity gain** — not the 10× of vendor marketing, and not zero. Gross delivered code volume rises 30–40%, but **rework** (fixing AI-introduced bugs) eats roughly half the gross gain.
|
||||
|
||||
Gains vary sharply by context (the "it depends" matrix):
|
||||
|
||||
| Context | Net gain |
|
||||
| --- | --- |
|
||||
| Greenfield, low complexity | 30–40% |
|
||||
| Greenfield, high complexity | 10–15% |
|
||||
| Brownfield (legacy), low complexity | 15–20% |
|
||||
| Brownfield (legacy), high complexity | **0–10%, can be negative** |
|
||||
|
||||
- **Language popularity matters:** popular languages (Python, Java, JS/TS) ~20% on simple tasks, 10–15% on complex; niche languages (COBOL, Haskell, Elixir) minimal or negative — thin AI training data.
|
||||
- **Codebase size matters:** gains fall sharply as codebases grow from ~10k to ~10M lines — attributed to **context-window limits, signal-to-noise degradation, and domain-specific logic**.
|
||||
- Coverage of the same research reports a **91% increase in PR review time** and a **~2.6× increase in rework** in AI-heavy workflows, i.e. the cost moved downstream from writing to reviewing. _(Secondary coverage; not read off a primary slide.)_
|
||||
- Interpretation offered by the researchers in talks/coverage: quality of AI usage beats volume of AI usage; teams with clean, modular, well-tested codebases compound gains, while teams with poor code hygiene accumulate technical debt and lose trust in the tools — one proposed mechanism for the widening gap.
|
||||
- Related earlier finding from the same group (separate result, widely covered ~Nov 2024): **"ghost engineers"** — ~9.5% of engineers in the dataset perform virtually no verifiable work. _(Contextual; distinct from the AI-impact analysis.)_
|
||||
|
||||
## Status / caveats
|
||||
|
||||
- The 46-vs-46 difference-in-differences result is, as of this writing, **talk/deck-published only** — presented at conferences and webinars, not (yet) in a peer-reviewed paper. The peer-reviewed papers cover the measurement methodology.
|
||||
- All numbers outside the slide itself come from secondary coverage of the talks and may compress or paraphrase.
|
||||
- The study's data is proprietary (companies opt in), so independent replication is not possible from outside.
|
||||
|
||||
## Links (all, in one place)
|
||||
|
||||
- Group site: https://softwareengineeringproductivity.stanford.edu/
|
||||
- Sept 2025 AI Conference deck: https://aiconference.com/wp-content/uploads/2025/09/Yegor-Denisov-Blanch-Will-AI-Replace-Software-Engineers_-.pptx.pdf
|
||||
- Talk video: https://www.youtube.com/watch?v=JvosMkuNxF8
|
||||
- Researcher: https://yegordb.com/
|
||||
- Methodology paper: https://arxiv.org/pdf/2409.15152
|
||||
- Determinism paper: https://arxiv.org/pdf/2502.20747
|
||||
- Screenshot that triggered this dossier: `raw/assets/G6g3O60bkAE05ZW.png`
|
||||
198
raw/sources/Грабли во внедрении ИИ в SDLC.md
Normal file
198
raw/sources/Грабли во внедрении ИИ в SDLC.md
Normal file
@@ -0,0 +1,198 @@
|
||||
# Выводы по видео
|
||||
|
||||
**Источник:** https://www.youtube.com/watch?v=Nm3MsnngCJg
|
||||
**Название:** Грабли во внедрении ИИ в SDLC — почему ИИ есть, а результата нет и как это лечить (Николай Шейко)
|
||||
**Длительность:** 45:59
|
||||
|
||||
---
|
||||
|
||||
## Главный тезис
|
||||
|
||||
**ИИ в разработке уже даёт реальный прирост, но люди, компании и метрики тормозят его на порядок.** С декабря 2025 (Opus 4.5 / GPT-5.2 + Claude Code / Codex) начался настоящий скачок — SDLC схлопнулся в дни/часы. Но результат появляется только у тех, кто (а) перестал быть просто разработчиком и стал менеджером агента, (б) построил feedback loop, (в) меряет **выполненные задачи без rework**, а не строки кода / PR-ы, и (г) занимается **эволюцией системы** (skill-и, промпты, инструменты), а не одноразовыми настройками.
|
||||
|
||||
Ключевая цитата: *«Компаниям больше не нужна кастомная AI-разработка. Им нужно прийти, поставить Claude Code или Codex, всё настроить, прицепить коннекторы, подумать про безопасность — и это работает лучше любой кастомной».*
|
||||
|
||||
---
|
||||
|
||||
## 1. Прошлое: что уже произошло
|
||||
|
||||
| Событие | Что важно |
|
||||
|---|---|
|
||||
| Статья METR (июль 25) | Ожидали ускорение, а замер показал замедление. Но у исследования много методологических проблем |
|
||||
| Стэнфордское исследование (август 25) | +20% скорости, но **разрыв между топ-перформерами и середняками растёт** — догнать всё сложнее |
|
||||
| Твит Карпаты (ноябрь-декабрь 25) | Автор термина «vibe-coding» сам говорит: раньше было баловство, теперь 80% делает Claude Code, 20% дорабатываю руками |
|
||||
| Статья «SDLC is dead» | Стадии жизненного цикла схлопнулись из недель в дни/часы |
|
||||
|
||||
**Но SDLC схлопнулся не полностью.** Осталось два «красных квадратика» — люди:
|
||||
- **Ревьюер** — узкое место. Задачи копятся в очереди на ревью.
|
||||
- **Планировщик / продукт** — тоже человек.
|
||||
|
||||
Распределение времени разработчика сдвинулось с «кодинга посередине» к «планированию слева + проверке справа».
|
||||
|
||||
---
|
||||
|
||||
## 2. Универсальная ошибка №0: разработчик ≠ менеджер
|
||||
|
||||
- Хороший разработчик = 3-5 часов сфокусированной работы над одной фичей (CPU-bound).
|
||||
- Хороший AI-разработчик = **менеджер сотрудника-агента** (IO-bound), запускающий несколько задач параллельно.
|
||||
- Если ты запустил Claude Code и **сидишь смотришь**, как он работает — ты плохой сотрудник.
|
||||
- Не все психологически способны переключиться. **И это нормально** — не заставляйте всех.
|
||||
|
||||
---
|
||||
|
||||
## 3. Кейсы и ошибки
|
||||
|
||||
### Кейс 1: Перенос фронтенда на новый стек
|
||||
|
||||
| Ошибка | Лечение |
|
||||
|---|---|
|
||||
| Нет feedback loop у агента | Дать доступ к браузеру (Playwright / Agent Browser / Chrome DevTools) — пусть сравнивает старый vs новый фронт |
|
||||
| Кранч → баги, плохая архитектура, порочный круг фиксов | Много времени на **планирование**. 20 минут — минимум, часы — норма. Каждые 10 минут планирования экономят часы. Цель — реализация за один шаг |
|
||||
| Устаревшие инструменты (Cursor + оплата за токены → команда экономит) | Claude Code / Codex по подпискам |
|
||||
| Внешний эксперт настраивает — «магический артефакт» | Люди должны настраивать инструменты сами. Опросник, чтобы вытащить неявные знания в инструкцию агента |
|
||||
| Всех под одну гребёнку тащить в AI | Разное сопротивление — нормально. Консерваторы охраняют компанию от вайб-кодеров |
|
||||
| Взяли архитектурный проект для обучения | Архитектурные ошибки стоят дорого и живут годами. Лучше начинать не с архитектуры |
|
||||
|
||||
**Решение по обучению команды:** записать видео → скрипт для выгрузки всех сессий → эксперт отсматривает → **пишет фидбэк команде, а команда сама правит инструкции агента** → фокус на 2 топ-перформерах (час с ними даёт 10× больше пользы, чем с остальными).
|
||||
|
||||
### Кейс 2: Европейский аутсорс — PR-ов больше, а прирост +1%
|
||||
|
||||
Причина — **rework**. Быстро задеплоили, потом много доработок.
|
||||
|
||||
**Ошибки метрик** (мем всех компаний):
|
||||
|
||||
| Не мерить | Мерить |
|
||||
|---|---|
|
||||
| Строки кода | Количество **выполненных** задач в единицу времени |
|
||||
| Количество коммитов | (задача выполнена только если **не вернулась** на доработку) |
|
||||
| Число PR-ов | Время жизни задачи + время на доработки |
|
||||
|
||||
**Ревью — новое узкое место.** Ручное ревью → скапливаются задачи → выгорание → снижение качества. **Решение — ревью вместе с агентом** (не вместо, а вместе). Модель = умный студент: направляйте, задавайте гипотезы, находите проблемы.
|
||||
|
||||
### Кейс 3: Большая кодовая база + «проклятие compaction»
|
||||
|
||||
- Агент собирает контекст → окно переполняется → compaction → снова добирает → снова compaction → задача еле выполняется.
|
||||
- Компания решила: «раз агенты, best practices не нужны». **Ошибка.**
|
||||
- Best practices нужны именно для агента: локальность, изолированные модули с интерфейсами, кодовая база хранит контекст.
|
||||
- **AST Search** вместо grep на колоссальных проектах — grep выдаёт «портянку» мусора, AST даёт релевантное. Работает на разных языках.
|
||||
|
||||
### Мини-кейсы
|
||||
|
||||
- **Middle**, который внедрил AI и ускорил команду на десятки %, попросил повышение зарплаты, не получил — ушёл на сильно больше. *Если вы такой middle — задумайтесь. Если руководитель — вдвойне.*
|
||||
- **Стартап, который делает spec-driven development, не зная, что хочет.** Планирование бесполезно, если нет цели. Сначала соберите UI (даже с in-memory базой в браузере) → покликайте → поймите, что нужно → потом планируйте.
|
||||
|
||||
---
|
||||
|
||||
## 4. Что будет дальше (будущее ролей)
|
||||
|
||||
### Появление **Product-инженера**
|
||||
|
||||
Планирование съедает больше времени → нужен человек, который отвечает не «как технически сделать», а «**почему** технически делаем так, чем можем пренебречь, что упростить, на что забить».
|
||||
|
||||
### Intelligence vs Judgment
|
||||
|
||||
- **Intelligence** (последовательности действий, требующие интеллекта) — AI уже отжирает.
|
||||
- **Judgment** (вкус, различие полутонов, доменная экспертиза) — пока за людьми.
|
||||
- Мы детектим slop в интернете, потому что там нет human touch.
|
||||
- Judgment = либо **вкус, наработанный годами** (хороший код), либо **доменная экспертиза** (нефтянка, медицина).
|
||||
|
||||
### Разделение ролей: пользователь vs настройщик системы
|
||||
|
||||
Сейчас те, кто пишет код с AI, и те, кто настраивает harness — обычно одни люди. Постепенно **разделяется**:
|
||||
- **Пользователи** — планирование + верификация конкретной фичи.
|
||||
- **Agentic Operations** — настраивают SDLC, feedback loop-ы, инструменты, промпты, скиллы.
|
||||
|
||||
### Тренды на токены
|
||||
|
||||
- Пока **дорожают** — в ближайшее время. Позже, вероятно, начнут дешеветь.
|
||||
- Сейчас — **дикий запад**. Задача — оказаться в верхней половине графика Стэнфорда.
|
||||
- **Экспериментируйте на полную котлету**, пока подписки дешёвые.
|
||||
|
||||
---
|
||||
|
||||
## 5. Agentic Evolution — ключевой концепт
|
||||
|
||||
Как правильно обучать агента:
|
||||
|
||||
| Плохо | Хорошо |
|
||||
|---|---|
|
||||
| Спросить эксперта «как ты это делаешь» → он расскажет теорию | Взять нового сотрудника (агента) за ручку → провести по сложным задачам → показать грабли → потом сказать: *«Запомни всё это и напиши методичку следующему»* |
|
||||
|
||||
Без этого мы пользуемся тем, что нам дали по умолчанию. С этим — начинается вертикальный рост.
|
||||
|
||||
**Как верифицировать скилл:**
|
||||
|
||||
1. Написал скилл вместе с агентом.
|
||||
2. **Не иди обедать.**
|
||||
3. Запусти сабагента **без контекста** → пусть решит ту же задачу с нуля, используя только скилл.
|
||||
4. Основной агент смотрит, что не так → правит скилл.
|
||||
5. Так агент-ментор онбордит следующего агента.
|
||||
|
||||
---
|
||||
|
||||
## 6. Матрица «что делать / чего не делать»
|
||||
|
||||
### Ошибки разработчика
|
||||
|
||||
- Ревью полностью вручную ❌
|
||||
- Отдать ревью целиком агенту ❌
|
||||
- Слишком мало времени на планирование ❌
|
||||
- Слишком много планирования при неясной цели ❌
|
||||
- Обмазываться сабагентами / мультиагентными системами сразу ❌ (сложные системы вырастают из простых)
|
||||
- Забить на эволюцию (жить с дефолтами) ❌
|
||||
|
||||
### Ошибки компании
|
||||
|
||||
- Нанять внешнего настройщика (даст рыбу, но не удочку) → **нужен препод/куратор**
|
||||
- Устаревшие инструменты (Cursor с оплатой за токены)
|
||||
- Всех сотрудников под одну гребёнку в AI
|
||||
- Не повышать зарплату — потеряете тех, кто разобрался
|
||||
- Надеяться на быстрый результат — это долгий навык
|
||||
- Нанимать по-старому
|
||||
- Метрики, которые легко хакаются (LoC, PR count)
|
||||
|
||||
---
|
||||
|
||||
## 7. Что можно сделать **завтра**
|
||||
|
||||
1. **Замкните feedback loop**, если ещё нет.
|
||||
2. **Напишите скилл, который анализирует ваши сессии** хотя бы за день.
|
||||
3. **Автоматизируйте**: schedule в Codex или routines в Anthropic.
|
||||
4. **Голосовой ввод** (сильно больше контекста), причём **на русском** — тоже больше контекста, чем на английском.
|
||||
5. **Не гоняйтесь за каждым новым инструментом** — что не всасывается в Claude Code / Codex за пару месяцев, скорее всего бесполезно.
|
||||
|
||||
---
|
||||
|
||||
## 8. Долгосрочный план
|
||||
|
||||
- Стать «правым» на слайде — тем, кто настраивает систему, а не пользуется.
|
||||
- Вкатываться, **пока подписки дешёвые**.
|
||||
- Отлаживать систему до one-shot execution (реализация за один проход, не итерации).
|
||||
- Автоматический анализ сессий на **уровне команды** — искать общие паттерны, не только личные.
|
||||
|
||||
---
|
||||
|
||||
## 9. Из Q&A: заметки на полях
|
||||
|
||||
- **Embeddings в AI-инжиниринге / RAG поверх эмбеддингов кода** — **не работает.** Не используйте, если не понимаете *очень* хорошо, зачем.
|
||||
- **Cursor не тот инструмент**, потому что: (а) отставал от Claude Code, (б) тащит старые фичи (embedding-индекс), (в) поддержка совместимости с VS Code съедает ресурсы, (г) оплата за токены → команда экономит вместо экспериментов. ~30% дороже подписок при том же уровне.
|
||||
- **Дефицит железа**: H100/H600 не арендовать. Claude банит через раз, требует верификацию email. Прогноз докладчика: **AI назовут кибероружием**, введут лицензии как на биотех/фарму.
|
||||
- **Китайские модели — важный тренд:**
|
||||
- **GLM** (ZAI) — «красавчики», хорошая модель, есть свой GUI-клон Codex, подключается любая модель. Минус — не vision.
|
||||
- **Kimi** — рабочая, есть vision (для frontend feedback loop).
|
||||
- **Xiaomi агент** — интересная реализация памяти (стоит посмотреть). Head of AI перешёл туда из DeepSeek.
|
||||
- **Как заставить агента справиться за один шаг:**
|
||||
1. Feedback loop — must have.
|
||||
2. Новая фича **goal** — чётко формулируйте реальную цель; агент сам проверяет, достигнута ли.
|
||||
3. Скилл-верификация через субагента без контекста.
|
||||
|
||||
---
|
||||
|
||||
## 10. Кому это полезно
|
||||
|
||||
- **Тимлидам и техлидам**, которые внедряют AI и меряют строками кода (перестаньте).
|
||||
- **Разработчикам,** которые сидят и смотрят, как Claude работает — пора становиться менеджером.
|
||||
- **CTO** — не нанимайте внешнего настройщика, наймите куратора; повышайте зарплату тем, кто разобрался, иначе уйдут.
|
||||
- **Стартаперам** — не делайте spec-driven, пока не знаете, что хотите.
|
||||
- **AI-инженерам** — не суйте embeddings в код, стройте skill evolution через субагента без контекста.
|
||||
- **Всем** — замкните feedback loop и включите голосовой ввод на русском **завтра**.
|
||||
@@ -20,11 +20,12 @@ If agent work takes sixteen minutes and you are doing something else, latency st
|
||||
|
||||
- Orbs, sleep/wake, one-URL packaging, multiplayer handoff, the 16-minute live demo, the collapsed cloud-IDE objections, Quinn's proof line, paper-cut velocity, local-dev prediction, infra-margin prediction — [[2026-07-28-agentic-engineering-10x-developer]].
|
||||
- Fire-and-forget as a native harness mode; completion notifications as what makes background agents usable — [[2026-07-14-skills-based-on-git]], [[harness]].
|
||||
- The same mode as a job description: the AI-developer is IO-bound, runs tasks in parallel, and "if you launched Claude Code and sit watching it work — you're a bad employee" — [[2026-07-30-rakes-in-ai-sdlc-adoption]] (the role-shift side lives at [[developer-as-agent-manager]]).
|
||||
- Scheduled agents producing while you sleep (the non-engineer version) — [[personal-ai-operating-system]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[harness]] (async is one of its two modes), [[agentic-loops]], [[make-more-cheap-code]] (proof artifacts are throwaway code with a job), [[shedding-weight]] (async is what makes killing the backlog possible — parked agents replace queued tickets), [[personal-ai-operating-system]], [[context-as-scarce-resource]]
|
||||
- Concepts: [[harness]] (async is one of its two modes), [[agentic-loops]], [[make-more-cheap-code]] (proof artifacts are throwaway code with a job), [[shedding-weight]] (async is what makes killing the backlog possible — parked agents replace queued tickets), [[personal-ai-operating-system]], [[context-as-scarce-resource]], [[developer-as-agent-manager]] (the human role this mode implies), [[review-is-the-new-bottleneck]] (where the parallel diffs pile up)
|
||||
- Entities: [[thorsten-ball]], [[amp]], [[claude-code]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
@@ -22,6 +22,8 @@ The human role has climbed prompt-engineer → **context-engineer** → harness-
|
||||
|
||||
**Context now has a price — at fleet scale.** The same source names **token budget** as one of two variables separating winners from losers, alongside knowing how to use agents. Context has always been scarce per-request; this is the corpus's first claim that it is also scarce per-*wallet*. Scope, corrected 2026-07-28: the claim comes from **metered** usage (parallel remote sandboxes), and under a flat consumer subscription the corpus's own heavy users report no ceiling — so per-request scarcity remains the binding constraint for individuals, and per-wallet scarcity is a fleet and enterprise concern. See [[enterprise-ai-reality]] and [[explosion-of-internal-software]].
|
||||
|
||||
**The compaction curse — scarcity at codebase scale** ([[2026-07-30-rakes-in-ai-sdlc-adoption]]). On a huge codebase the agent gathers context → the window overflows → compaction → it re-gathers → compaction again, and the task barely completes. [[nikolai-sheiko]]'s cure inverts the "agents mean best practices don't matter" fallacy: **best practices exist for the agent now** — locality, isolated modules with interfaces, so that *the codebase stores the context*. This converges with [[thorsten-ball]]'s context-lives-in-the-codebase position from the opposite direction (Thorsten skips skills because his codebase carries context; Sheiko says make your codebase able to). Practical additions: **AST search instead of grep** on colossal projects (grep returns a wall of noise, AST returns the relevant slice), and a flat rejection of embeddings/RAG over code. The mechanism matches Stanford's measured gains-collapse toward 10M LOC (evidence below).
|
||||
|
||||
**The supply-side facet** ([[2026-07-22-ai-is-stupid]]): before context is *scarce* it is usually *absent*. "Intelligence without context loses to context without intelligence" — ten Nobel laureates asked about your sales month can only cite industry averages, while your rank-and-file employee answers better because they see your funnel, clients, and deals. The default "stupid AI" experience is a strong model given neither business context nor a [[harness]]; the fix is investing in context infrastructure (data, memory, integrations) before reaching for a bigger model.
|
||||
|
||||
## Evidence
|
||||
@@ -31,6 +33,8 @@ The human role has climbed prompt-engineer → **context-engineer** → harness-
|
||||
- "Intelligence without context loses"; Nobel-vs-employee analogy; invest in context before model upgrades — [[2026-07-22-ai-is-stupid]].
|
||||
- Information > model choice > effort level; the two information sources; token budget as a winner/loser variable — [[2026-07-28-agentic-engineering-10x-developer]].
|
||||
- Reading costs attention — the human-side analog of the same scarcity — [[make-more-cheap-code]], [[2026-07-24-youre-reading-way-too-much-code]].
|
||||
- **First outside quantitative support:** Stanford SWEPR finds AI productivity gains collapse as codebases grow from ~10k to ~10M lines, attributing it to context-window limits and signal-to-noise degradation — the corpus's context-is-the-constraint claim, measured at scale — [[2026-07-30-stanford-swepr-widening-gap]].
|
||||
- Compaction curse; best-practices-for-the-agent (locality, interfaces, codebase-stores-context); AST search over grep; embeddings/RAG over code rejected — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
@@ -40,7 +44,7 @@ The human role has climbed prompt-engineer → **context-engineer** → harness-
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
- "First third = smart zone" is a heuristic, not a measured boundary; likely model-dependent. Status: tentative.
|
||||
- [[2026-07-22-ai-is-stupid]] names **RAG** and long-term assistant memory as the practical context mechanisms; [[2026-07-14-skills-based-on-git]] argues [[skills-as-memory|skills]] beat RAG (load-on-activation vs pre-injection). Possibly audience-driven (business data vs procedures) rather than a real disagreement. Status: tentative.
|
||||
- [[2026-07-22-ai-is-stupid]] names **RAG** and long-term assistant memory as the practical context mechanisms; [[2026-07-14-skills-based-on-git]] argues [[skills-as-memory|skills]] beat RAG (load-on-activation vs pre-injection). Possibly audience-driven (business data vs procedures) rather than a real disagreement. Status: tentative. *(2026-07-30: [[nikolai-sheiko]] adds a hard anti-RAG data point for the code domain specifically — "embeddings over code don't work" — which supports the audience-driven reading: the anti-RAG votes are both about code/procedures, the pro-RAG vote is about business data.)*
|
||||
|
||||
## Next Questions
|
||||
|
||||
|
||||
37
wiki/concepts/developer-as-agent-manager.md
Normal file
37
wiki/concepts/developer-as-agent-manager.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# Developer as Agent Manager
|
||||
|
||||
#concept
|
||||
|
||||
## Summary
|
||||
|
||||
[[nikolai-sheiko]]'s "universal error #0": treating AI-assisted development as the same job at higher speed. A good developer is **CPU-bound** — 3–5 hours of deep focus on one feature. A good AI-developer is **IO-bound** — a *manager of an agent-employee*, running several tasks in parallel, spending their time on planning and verification instead of typing. His blunt test: "If you launched Claude Code and sit watching it work — you're a bad employee."
|
||||
|
||||
## Current Understanding
|
||||
|
||||
- **The switch is psychological, not technical** — and not everyone can make it. Sheiko is explicit that this is *fine*: don't drag everyone into AI under one brush; conservatives "guard the company from the vibe-coders." Compare [[2026-07-14-nina-interview|Nina's finding]] that adoption blocks on friction, not resistance — this page is the case where genuine resistance exists and is legitimate.
|
||||
- **Where the time goes instead:** "planning on the left, verification on the right" — the coding middle collapsed. Hence 20-minutes-minimum planning, explicit goals the agent self-checks against, and the review discipline of [[review-is-the-new-bottleneck]].
|
||||
- **The role splits further.** Today the people who *use* agents and the people who *tune the harness* are the same; Sheiko predicts a split into **users** (plan + verify a feature) and **Agentic Operations** (own the SDLC configuration, feedback loops, prompts, skills). A **Product engineer** also emerges — answers *why* we build it this way, what to simplify, what to ignore.
|
||||
- **What stays human: Judgment over Intelligence.** AI absorbs *Intelligence* (action sequences requiring intellect); *Judgment* — taste built over years, or deep domain expertise (oil & gas, medicine) — remains human for now. We detect slop precisely because it lacks human touch. This is the corpus's [[seniority-and-the-junior-squeeze|judgment-as-risk-reduction]] claim restated as a capability boundary.
|
||||
- **Corpus convergence.** The same working mode appears as [[async-by-default|orbs and parallel delegation]] ([[thorsten-ball]]), Eugene's 7 parallel project-agents, Allie's ~100 agents, and Karpathy's "80% Claude Code, 20% by hand" (cited within the talk). Sheiko's contribution is naming the *identity* shift and its HR consequences: the middle dev who mastered this, asked for a raise, was refused, and left for far more.
|
||||
|
||||
## Evidence
|
||||
|
||||
- CPU-bound vs IO-bound framing; "sit watching = bad employee"; don't-force-everyone; users vs Agentic Operations; Product engineer; Intelligence vs Judgment; the raise-refusal mini-case — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
- The same mode practiced at the frontier (parallel remote sandboxes, delegation, proof-asking) — [[2026-07-28-agentic-engineering-10x-developer]] via [[async-by-default]].
|
||||
- Non-engineer versions of parallel delegation (~100 agents, scheduled workflows) — [[2026-07-14-gap-between-ai-users-irreversible]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[async-by-default]] (the infrastructure this role runs on), [[review-is-the-new-bottleneck]] (where the manager's verification time goes), [[product-ownership]] (the webinar's human-side twin: own outcomes, not tickets), [[seniority-and-the-junior-squeeze]] (Judgment as the durable half), [[levels-of-ai-usage]] (the non-engineer's version of the same climb), [[harness]]
|
||||
- Entities: [[nikolai-sheiko]], [[thorsten-ball]], [[eugene]], [[allie-miller]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
- **"Don't force everyone" vs the widening gap.** Sheiko permits opting out; [[2026-07-14-gap-between-ai-users-irreversible|Allie]] calls the gap irreversible and [[2026-07-30-stanford-swepr-widening-gap|Stanford measured it quadrupling]]. If both are right, opting out is legitimate *and* costly, and the source doesn't reconcile the two. Status: tentative.
|
||||
- The users / Agentic-Operations split is a prediction, not an observation — today's evidence (Eugene, Thorsten) is of people doing both. Status: tentative.
|
||||
- Intelligence-vs-Judgment is a moving boundary asserted "for now"; the corpus has no criterion for where it stops moving.
|
||||
|
||||
## Next Questions
|
||||
|
||||
- What does the manager's day actually look like — is there a source with a concrete parallel-task routine (queue depth, check-in cadence) rather than the mode's name?
|
||||
- Does the Agentic-Operations role match the webinar's promise that non-engineers can self-serve ([[levels-of-ai-usage]]), or does it re-centralize harness-tuning in specialists?
|
||||
@@ -14,17 +14,19 @@ The indie/practitioner world and the regulated-enterprise world diverge sharply.
|
||||
- **Adjacent constraints:** the [[seniority-and-the-junior-squeeze|"read what you approve"]] security concern is amplified at scale; safety-critical/regulated code is the clear exception to [[code-as-throwaway|"most code isn't high-value"]].
|
||||
- **A second divide: the token budget** (added 2026-07-28; **scope corrected 2026-07-28** — see below). [[thorsten-ball]] names two variables separating winners from losers — knowing how to use agents, and **having the token budget to do it**. It cuts both ways for this page: an enterprise can buy budget an individual cannot, while a locked-down enterprise may withhold it from the people who would use it best. Whoever controls the budget controls how far [[explosion-of-internal-software|internal software]] spreads. Thorsten names the variable and says nothing about who pays.
|
||||
- **Scoping correction.** This was first written here as "the divide is also a *spending* gap," which overstates it. Thorsten's pricing regime is **metered**: [[amp]] sells usage, and his working pattern is parallel remote sandboxes and parked orbs ([[async-by-default]]) — a fleet cost, not a seat cost. Under a **flat consumer subscription** the corpus's own evidence points the other way: [[eugene]] runs 7 project-agents in parallel on a $200 plan, [[allie-miller]] runs ~100 agents and 36 workflows, and neither reports hitting a cost ceiling — while [[2026-07-14-sebastian-eugene-interview]] frames levelling as "a 20-year veteran and a fresh grad **on the same subscription**." For individual and small-team use the budget is one subscription; the token-budget variable bites at fleet scale and under metered pricing, which is where Thorsten sits and where enterprises will land.
|
||||
- **The market claim, seconded — and sharpened into a quote** (added 2026-07-30). [[nikolai-sheiko]], from multi-company adoption work: *"Companies no longer need custom AI development. Come in, install Claude Code or Codex, configure everything, attach connectors, think about security — and it works better than any custom build."* This is Sebastian's company-managed-harness market stated as a service playbook. Two adoption anti-patterns attached: **the external configurator** who leaves a "magic artifact" nobody on the team owns (what a company should buy is a *teacher/curator*; the team must configure its own tools — the [[solve-first-then-skillify|evolution]] has to happen in their hands), and **metered pricing shaping behaviour** — a team on Cursor's per-token billing economizes instead of experimenting (~30% dearer than subscriptions at the same level), which is the metered-vs-subscription split from the scoping correction above observed as an organizational failure mode. His pricing prediction — tokens get dearer near-term, cheaper later; "experiment at full throttle while subscriptions are cheap" — matches [[eugene]]'s price-rise prediction already flagged in the token-budget question.
|
||||
- **The frontier's advice does not transfer.** [[shedding-weight]] — kill the backlog, kill CI that repeats the agent's tests, kill local dev in favour of remote sandboxes ([[async-by-default]]) — describes a startup that owns its own process. In a regulated shop the pipeline, the audit trail and the ticket history frequently *are* the deliverable to a regulator, and code sitting in a vendor's remote sandbox is precisely what Sebastian's clients forbid. The gap between what the frontier recommends and what compliance permits is the same gap this page calls the market.
|
||||
|
||||
## Evidence
|
||||
|
||||
- Managed VMs / zero self-install, Roche ~1,200 engineers, banks banned→adopting, "company-managed resource," "the interesting market" — [[2026-07-14-sebastian-eugene-interview]].
|
||||
- Token budget as a winner/loser variable; the frontier playbook (kill backlog/CI/local dev, remote sandboxes) that compliance cannot follow — [[2026-07-28-agentic-engineering-10x-developer]].
|
||||
- "No custom AI development needed" quote; external-configurator anti-pattern vs teacher/curator; Cursor per-token billing → team economizes; tokens-dearer-then-cheaper prediction — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[harness]], [[seniority-and-the-junior-squeeze]], [[code-as-throwaway]], [[shedding-weight]], [[async-by-default]], [[explosion-of-internal-software]], [[context-as-scarce-resource]]
|
||||
- Entities: [[sebastian]], [[virtido]], [[eugene]], [[thorsten-ball]]
|
||||
- Entities: [[sebastian]], [[virtido]], [[eugene]], [[thorsten-ball]], [[nikolai-sheiko]]
|
||||
- Tools: [[claude-code]], [[amp]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
@@ -20,6 +20,7 @@ Supporting practices at the plateau: keep CLAUDE.md self-maintaining ("always ke
|
||||
- Webinar title and non-programmer audience confirm the ladder as the webinar's spine — [[2026-07-14-nina-interview]].
|
||||
- Convergent structure (foundation docs + skills as the non-engineer's OS) — [[2026-07-14-gap-between-ai-users-irreversible]].
|
||||
- A high-rung user missing the skills rung, and the memory pain that results — [[2026-07-21-larysa-interview]].
|
||||
- **Measured, team-level corroboration that the gap grows between rungs of mastery**, not between license-holders and others: Stanford's 46-vs-46-team analysis shows the productivity gap between AI-mastering and lagging teams growing 4.8% → 19% (4×) over ~2.25 years — [[2026-07-30-stanford-swepr-widening-gap]]. *(Caveat: measures engineering teams, not this ladder's non-programmer audience; and the study asserts "quality of usage" as the differentiator without decomposing which rung supplies it.)*
|
||||
|
||||
## Related Pages
|
||||
|
||||
|
||||
@@ -23,10 +23,11 @@
|
||||
- All claims, ratios, tier table, slop patterns, Dax/Shao citations — [[2026-07-24-youre-reading-way-too-much-code]].
|
||||
- Groundwork (code disposable, kill without guilt, G-brain markdown tier) — [[2026-07-14-everything-we-knew-about-software-has-changed]].
|
||||
- 15 icon variants, Midjourney rounds, "ask the agent for proof" — [[2026-07-28-agentic-engineering-10x-developer]].
|
||||
- **External measurement of the cost moving downstream:** Stanford SWEPR coverage reports +91% PR review time and ~2.6× rework in AI-heavy workflows — writing got cheaper, reviewing got dearer, which is this page's premise measured rather than asserted — [[2026-07-30-stanford-swepr-widening-gap]]. *(Secondary-coverage numbers; and note the same study's negative gains in complex brownfield code — "code is cheap" holds least where most code lives.)*
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[code-as-throwaway]] (parent claim: cost → zero; this page is its *discipline* — what cheap code is actually for), [[think-wider-not-bigger]] (same breadth logic applied to generation volume rather than ambition), [[product-ownership]] (verifying as the human's remaining job), [[solve-first-then-skillify]] (contrast: slop is frozen into nothing; skills freeze the procedure), [[leave-less-room-for-imagination]] (tension — see below), [[context-as-scarce-resource]], [[async-by-default]] (proof artifacts as the delegated form of the same move)
|
||||
- Concepts: [[code-as-throwaway]] (parent claim: cost → zero; this page is its *discipline* — what cheap code is actually for), [[think-wider-not-bigger]] (same breadth logic applied to generation volume rather than ambition), [[product-ownership]] (verifying as the human's remaining job), [[solve-first-then-skillify]] (contrast: slop is frozen into nothing; skills freeze the procedure), [[leave-less-room-for-imagination]] (tension — see below), [[context-as-scarce-resource]], [[async-by-default]] (proof artifacts as the delegated form of the same move), [[review-is-the-new-bottleneck]] (the org-level form of the cost shift this page manages individually — with the completed-without-rework metric as its answer)
|
||||
- Entities: [[theo-browne]], [[eugene]], [[thorsten-ball]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
38
wiki/concepts/review-is-the-new-bottleneck.md
Normal file
38
wiki/concepts/review-is-the-new-bottleneck.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Review Is the New Bottleneck
|
||||
|
||||
#concept
|
||||
|
||||
## Summary
|
||||
|
||||
When agents write the code, the SDLC doesn't collapse completely — it collapses *around the humans*. [[nikolai-sheiko]]'s two remaining "red squares" are the **reviewer** and the **planner**: tasks pile up in the review queue, reviewers burn out, quality drops, and headline output metrics (PRs, LoC) rise while real throughput barely moves. The organizational fix is twofold: **review with the agent** (not fully manual, not fully delegated) and **measure completed tasks without rework** rather than anything volume-based.
|
||||
|
||||
## Current Understanding
|
||||
|
||||
- **The mechanism.** Generation got ~free, so the cost moved downstream to verification — and at team scale, downstream is a *person* with a queue. Sheiko's European-outsourcer case: more PRs than ever, net gain +1%, because rework consumed the difference. Manual-only review starts a spiral (queue → burnout → rubber-stamping → more rework); fully delegated review is the opposite error ([[async-by-default]]'s "proof produced by the thing being checked").
|
||||
- **The middle path: review together with the agent.** Treat the model as a smart student — direct it, pose hypotheses, locate problems jointly. This is the org-level sibling of Theo's reading economics ([[make-more-cheap-code]]: AI reviews before humans, per-file summaries, read only what's worth reading).
|
||||
- **The metric that resists gaming:** a task counts as done only if it **doesn't come back for rework**; track task lifetime and rework time. LoC, commit count and PR count are all trivially hacked and all rise *because* of the bottleneck, not despite it.
|
||||
- **Externally measured:** Stanford SWEPR coverage reports **+91% PR review time** and ~2.6× rework in AI-heavy workflows, and finds gross code volume up 30–40% while net gains are ~15–20% — the same rework-eats-half story Sheiko tells anecdotally ([[2026-07-30-stanford-swepr-widening-gap]]).
|
||||
- **Planning is the other red square.** Time redistributes from coding to "planning on the left + verification on the right" — which is why Sheiko prescribes 20-minutes-minimum planning and why a **Product engineer** role emerges ([[developer-as-agent-manager]]).
|
||||
|
||||
## Evidence
|
||||
|
||||
- Reviewer/planner as the remaining red squares; review-with-the-agent; the metrics table; the +1% outsourcer case — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
- +91% PR review time, 2.6× rework, gross-vs-net gap — [[2026-07-30-stanford-swepr-widening-gap]] *(secondary-coverage numbers)*.
|
||||
- Reading as the scarce human resource; AI-review-before-human-review; tiered reading discipline — [[2026-07-24-youre-reading-way-too-much-code]] via [[make-more-cheap-code]].
|
||||
- Parallel agents multiplying diffs without multiplying review capacity — the attention pile-up already logged on [[async-by-default]]; this page names that open question as the bottleneck it becomes at team scale.
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[make-more-cheap-code]] (individual-level discipline for the same cost shift), [[async-by-default]] (the generation side that feeds the queue), [[developer-as-agent-manager]] (the role shift on the human side of the queue), [[seniority-and-the-junior-squeeze]] ("read what you approve" — why review can't just be dropped), [[maintenance-is-the-real-cost]] (kindred move: the visible activity was never the expensive part)
|
||||
- Entities: [[nikolai-sheiko]], [[swepr]], [[theo-browne]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
- **How much review survives?** Theo/Dax hold that agent diff-summaries surface anomalies; [[eugene]] holds that drift is precisely what summaries miss ([[leave-less-room-for-imagination]]); Sheiko's review-with-the-agent is a third position between them — asserted, not tested. Status: tentative.
|
||||
- The completed-without-rework metric is better than LoC/PRs but still gameable (e.g. by inflating task granularity or quietly reclassifying rework as new tasks); the source doesn't address it. Status: tentative.
|
||||
- Whether review-as-bottleneck is transitional (until verification is agentized) or structural (a human must always sign off — the [[seniority-and-the-junior-squeeze|accountability]] view) is open across the corpus.
|
||||
|
||||
## Next Questions
|
||||
|
||||
- What does review-with-the-agent look like concretely — a checklist, a dialogue pattern, a skill? The corpus has the prescription but no transcript of it done well.
|
||||
- Is there a non-engineer analog (the webinar audience reviews documents, not PRs)? The checker-skill design in [[2026-07-24-non-engineer-throwaway-verification]] may be it.
|
||||
@@ -27,6 +27,7 @@ The **method** for populating skills is [[solve-first-then-skillify]]: reach the
|
||||
- ~3-message correction-loop heuristic; skills as the non-programmer ceiling (with CLAUDE.md) — [[2026-07-14-yulia-interview]].
|
||||
- Cross-session memory loss as the #1 practitioner pain; "memory is the worst thing agents have"; skills committed as the webinar remedy — [[2026-07-21-larysa-interview]].
|
||||
- Counter-evidence: no skills, no MCP, no slash commands at a 99%-AI-written company; `AGENTS.md` + codebase + rich prompt as the substitute — [[2026-07-28-agentic-engineering-10x-developer]].
|
||||
- **A second practitioner vote *for* the skills layer** (2026-07-30): [[nikolai-sheiko]]'s "Agentic Evolution" makes skill-building-plus-verification the difference between living on defaults and "vertical growth," and he flatly rejects embeddings/RAG over code ("don't use them unless you understand *very* well why") — siding with the load-on-activation camp in the skills-vs-RAG contradiction below. His verification protocol (context-free subagent re-solves the task from the skill alone) is the closest thing yet to the falsification test this page asks for — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
@@ -43,10 +44,10 @@ The **method** for populating skills is [[solve-first-then-skillify]]: reach the
|
||||
1. **Situational.** He works daily in *one codebase he controls*, where context can live in the code and `AGENTS.md`. Skills earn their keep when work is spread across many ad-hoc tasks with no codebase to encode into — which is exactly the corpus's HR/BA audience ([[nina]], [[yulia]], [[larysa]]). Under this reading both are right and the disagreement is about who is speaking.
|
||||
2. **The abstraction is premature.** Skills are scaffolding for models that needed it; a strong model plus a rich prompt plus a good repo may simply beat a skills library, making the whole layer a 2025 artifact. This is the uncomfortable reading for the webinar's central promise.
|
||||
3. **He has skills under another name.** AMP's Oracle/Painter/Puck sub-agents and a maintained `AGENTS.md` *are* curated, reusable, two-stage context — just authored by the vendor and the team rather than the user. Under this reading the dispute is about who curates, not whether curation is needed.
|
||||
Status: tentative. Note the evidential asymmetry — his is a first-hand report of daily practice at scale, where the pro-skills case rests on Konstantin's architecture argument plus self-reported individual workflows. **Presentation-safe restatement:** [[2026-07-28-webinar-theses]] reframes the claim as *context you author beats context that's inferred*, which holds under all three readings — Konstantin's skills, Allie's foundation docs, Eugene's anti-memory position and Thorsten's `AGENTS.md` are all authored context.
|
||||
Status: tentative. Note the evidential asymmetry — his is a first-hand report of daily practice at scale, where the pro-skills case rests on Konstantin's architecture argument plus self-reported individual workflows. *(Asymmetry softened 2026-07-30: [[nikolai-sheiko]] adds a second practitioner voice on the pro-skills side, from multi-company adoption work rather than one codebase — though his cases are anonymous anecdotes, so the readings above remain unsettled.)* **Presentation-safe restatement:** [[2026-07-28-webinar-theses]] reframes the claim as *context you author beats context that's inferred*, which holds under all three readings — Konstantin's skills, Allie's foundation docs, Eugene's anti-memory position and Thorsten's `AGENTS.md` are all authored context.
|
||||
|
||||
## Next Questions
|
||||
|
||||
- ~~What's a starter skill set for a non-engineer?~~ Answered in [[2026-07-14-best-first-skill-for-beginner]] (skill-creator as meta-skill; tone-of-voice + anti-AI-language as first content skill).
|
||||
- Do skills actually solve *cross-project* context, or only per-procedure recall? Larysa's complaint may be the former, which skills don't obviously address.
|
||||
- Is there a test that would separate reading 1 from reading 2 above? The cheapest one available: give a non-engineer the same task with and without a skill and compare drift — the corpus has never run it, and the webinar's promise rests on the answer.
|
||||
- Is there a test that would separate reading 1 from reading 2 above? The cheapest one available: give a non-engineer the same task with and without a skill and compare drift — the corpus has never run it, and the webinar's promise rests on the answer. *(Adjacent evidence 2026-07-30: Sheiko's context-free-subagent protocol — [[solve-first-then-skillify]] — runs the with-skill half in practice, but never the without-skill control, so the question stands.)*
|
||||
|
||||
@@ -15,17 +15,28 @@ The recurring beginner mistake is writing the skill first and then trying to "sh
|
||||
|
||||
The payoff goes beyond reuse: a packaged skill is a **handoff and de-risking asset** — "a person with not even a third of your HR experience can deliver a decent result," which cuts onboarding and lets the expert take a vacation. This is how [[skills-as-memory]] gets *populated* in practice — the method side of that architecture, and the fix for "don't teach the AI abstractly."
|
||||
|
||||
**Agentic Evolution — the strongest formulation, plus the missing verification step** (added 2026-07-30). [[nikolai-sheiko]] frames the same method as onboarding an employee: asking the expert "how do you do this?" yields theory; instead **take the new employee (the agent) by the hand through hard real tasks, show it the rakes, then say: "remember all of this and write the manual for the next one."** Without this you live on defaults; with it "vertical growth begins." He then adds what the corpus's earlier heuristics lacked — a **verification protocol** for the frozen skill:
|
||||
|
||||
1. Write the skill together with the agent.
|
||||
2. Don't go to lunch.
|
||||
3. Launch a **subagent with no context** — it must solve the same task from scratch using only the skill.
|
||||
4. The main agent watches what fails and fixes the skill.
|
||||
5. The mentor agent thus onboards the next agent.
|
||||
|
||||
This is the first source to describe actually *running* something close to the skills falsification test proposed on [[skills-as-memory]] (same task, with-skill vs from-scratch) — though it tests the skill's completeness for one task, not whether the skill beats no-skill. His do-tomorrow extension: a skill that analyses your own sessions daily, automated via schedules/routines — evolution as a standing loop rather than a one-time freeze.
|
||||
|
||||
## Evidence
|
||||
|
||||
- "You first solve a task with Claude; the moment you reach the final solution, you say — now create a skill from this"; ~3-message heuristic — [[2026-07-14-yulia-interview]].
|
||||
- Do-the-task-then-freeze framing; skill-as-handoff to a junior hire; vacation/de-risking angle — [[2026-07-14-nina-interview]].
|
||||
- >5-tool-calls auto-creation heuristic and curator pruning — [[2026-07-14-skills-based-on-git]].
|
||||
- Skills prescribed specifically as the workaround for cross-session memory loss, and as the constraint on drift — [[2026-07-21-larysa-interview]].
|
||||
- Agentic Evolution (walk the agent through tasks → have it write the manual); the context-free-subagent verification protocol; session-analysis skill as a daily loop — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[skills-as-memory]] (the architecture this method feeds), [[levels-of-ai-usage]] (skills are the top practical rung), [[personal-ai-operating-system]], [[leave-less-room-for-imagination]] (why a *proven* spec beats a written-ahead one)
|
||||
- Entities: [[eugene]], [[nina]], [[konstantin]], [[larysa]]
|
||||
- Entities: [[eugene]], [[nina]], [[konstantin]], [[larysa]], [[nikolai-sheiko]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
@@ -33,5 +44,6 @@ The payoff goes beyond reuse: a packaged skill is a **handoff and de-risking ass
|
||||
|
||||
## Next Questions
|
||||
|
||||
- What does a good "create a skill from this" prompt look like — does the corpus contain a concrete example transcript?
|
||||
- What does a good "create a skill from this" prompt look like — does the corpus contain a concrete example transcript? *(Partially answered 2026-07-30: Sheiko's "remember all of this and write the manual for the next one" after a guided run is the best prompt-shape the corpus has.)*
|
||||
- How do the ~3-message and >5-tool-call heuristics compare in practice; is one strictly better for non-programmers?
|
||||
- Does the context-free-subagent check catch skill *quality*, or only completeness for the one task it was frozen from?
|
||||
|
||||
36
wiki/entities/nikolai-sheiko.md
Normal file
36
wiki/entities/nikolai-sheiko.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# Nikolai Sheiko
|
||||
|
||||
#entity
|
||||
|
||||
## Summary
|
||||
|
||||
Speaker of the talk "Грабли во внедрении ИИ в SDLC" ([[2026-07-30-rakes-in-ai-sdlc-adoption]]). Russian-speaking AI-adoption practitioner/consultant who works with client companies (a frontend-migration team, a European outsourcer, a large-codebase company) on getting real results from AI in the software lifecycle. Background beyond the talk unknown.
|
||||
|
||||
## Current Understanding
|
||||
|
||||
His through-line: the models are already good enough — **people, companies and metrics are what throttle the gains**. Signature positions:
|
||||
|
||||
- The AI-developer is a **manager of an agent-employee**, IO-bound and parallel ([[developer-as-agent-manager]]).
|
||||
- **Review is the new bottleneck**; review *with* the agent; measure completed-tasks-without-rework, never LoC/PRs ([[review-is-the-new-bottleneck]]).
|
||||
- **Agentic Evolution**: walk the agent through hard tasks, then have it write the manual; verify skills with a context-free subagent ([[solve-first-then-skillify]]).
|
||||
- Companies should install and configure Claude Code / Codex rather than build custom AI tooling; buy a **teacher/curator**, not an external configurator ([[enterprise-ai-reality]]).
|
||||
- Best practices matter *more* with agents (compaction curse, AST search over grep); embeddings/RAG over code don't work ([[context-as-scarce-resource]]).
|
||||
- AI eats **Intelligence**; **Judgment** (taste or domain expertise) stays human for now.
|
||||
|
||||
## Evidence
|
||||
|
||||
- All positions — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[developer-as-agent-manager]], [[review-is-the-new-bottleneck]], [[solve-first-then-skillify]], [[enterprise-ai-reality]], [[context-as-scarce-resource]]
|
||||
- Entities: [[swepr]] (he cites their Stanford chart as the stakes — "be in the top half"), [[thorsten-ball]] (fellow frontier practitioner; they disagree on skills — Sheiko builds skill evolution, Thorsten uses none)
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
- His client cases are anonymous and self-reported; no numbers are verifiable. Status: tentative.
|
||||
- Identity/affiliation beyond the talk unknown (the raw doc names only the talk itself). Status: tentative.
|
||||
|
||||
## Next Questions
|
||||
|
||||
- Who is he professionally — independent consultant, agency, vendor? Affects how to weigh the "you don't need custom AI development" claim (it is also a consultant's pitch).
|
||||
35
wiki/entities/swepr.md
Normal file
35
wiki/entities/swepr.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# SWEPR (Stanford Software Engineering Productivity Research)
|
||||
|
||||
#entity
|
||||
|
||||
## Summary
|
||||
|
||||
Research group at Stanford University measuring software-engineering productivity from private Git data — 600+ companies, ~100k–120k engineers since 2022 — using an ML model that replicates a panel of expert code reviewers. Public face: researcher **Yegor Denisov-Blanch**. Source of the corpus's only quantitative outside study of AI's productivity impact.
|
||||
|
||||
## Current Understanding
|
||||
|
||||
- Site: https://softwareengineeringproductivity.stanford.edu/ — offers an "AI Practices Benchmark" and "AI Impact" research to participating companies.
|
||||
- Known for three results: the **widening-gap DiD analysis** (46 vs 46 teams, gap 4.8% → 19%, 4×, Apr 2023–Jul 2025), the **~15–20% net average gain** figure (after rework; 30–40% gross), and the earlier **"ghost engineers"** finding (~9.5% of engineers show virtually no verifiable output).
|
||||
- Methodology is peer-reviewed (arXiv 2409.15152, 2502.20747); the headline AI-impact analyses are talk/deck-published (Sept 2025 AI Conference deck "Will AI Replace Software Engineers?").
|
||||
- Their measurement philosophy — functionality delivered, not commits or LOC — aligns with the corpus's own suspicion of volume metrics ([[make-more-cheap-code]]: generation volume is the *cheap* part).
|
||||
|
||||
## Evidence
|
||||
|
||||
- All claims and links — [[2026-07-30-stanford-swepr-widening-gap]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Sources: [[2026-07-30-stanford-swepr-widening-gap]]
|
||||
- Concepts: [[levels-of-ai-usage]] (team-level twin of the mastery gap), [[context-as-scarce-resource]] (codebase-size finding), [[make-more-cheap-code]] (review-time shift)
|
||||
- Queries: [[2026-07-30-stanford-widening-gap-source]]
|
||||
- Entities: [[allie-miller]] (her prediction, their measurement)
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
- Data is proprietary and opt-in — no outside replication possible; participating companies may skew toward the measurement-friendly. Status: tentative.
|
||||
- The widening-gap analysis itself is not peer-reviewed as of 2026-07-30. Status: tentative.
|
||||
|
||||
## Next Questions
|
||||
|
||||
- Track whether the DiD analysis lands in a peer-reviewed venue.
|
||||
- What do they say *causes* the gap (tooling vs hygiene vs practices)? The talks assert "quality of usage" without decomposing it.
|
||||
@@ -10,7 +10,7 @@ A high-signal personal knowledge base. `raw/` holds immutable source materials;
|
||||
|
||||
## The through-line
|
||||
|
||||
Across eleven sources — five talks/videos/interviews from practitioners (two of them Theo's), five interviews conducted for this project, and a business-facing short — one spine recurs:
|
||||
Across thirteen sources — six talks/videos/interviews from practitioners (two of them Theo's), five interviews conducted for this project, a business-facing short, and one quantitative outside study ([[swepr|Stanford SWEPR]]) — one spine recurs:
|
||||
|
||||
> **As the cost of writing code goes to zero, value migrates from *producing* software to *directing and verifying* it — and the durable human assets become judgment, ownership, taste, and in-person relationships.**
|
||||
|
||||
@@ -20,23 +20,24 @@ Everything else hangs off that:
|
||||
- **The human side** — what stays yours: [[product-ownership]] over outcomes, [[connections-as-moat|in-person connections]] as the last non-commoditized asset, [[seniority-and-the-junior-squeeze|judgment as risk-reduction]], and the need to [[decoupling-identity-from-profession|decouple identity from profession]].
|
||||
- **The strategy side** — where to point it: [[think-wider-not-bigger|think wider not bigger]], treat [[code-as-throwaway|code as throwaway]], and mind [[enterprise-ai-reality|enterprise compliance reality]] (the company-managed-harness market). Theo's second video supplies the *verifying* half of the spine its method: [[make-more-cheap-code]] — keep hand-verification of what ships, and generate orders of magnitude more never-shipped code to verify and explore.
|
||||
- **The frontier side** — what it looks like at the far end, from [[thorsten-ball]] at [[amp]] (99% of their code AI-written): [[shedding-weight|shed weight]] by deleting every process that only existed because humans were the bottleneck; [[build-for-the-agent-not-the-human|build for the agent, not the human]]; work [[async-by-default|async by default]] in remote sandboxes and ask for proof rather than claims. His two mechanisms for software becoming *personal* — [[emacsification-of-software|remixing what exists]] and [[explosion-of-internal-software|building what never did]] — are the corpus's strongest outside validation of the webinar's own thesis, "little tools you make for yourself." He is also its sharpest dissenter: he uses **no skills, no MCP, no slash commands**. Both mechanisms now carry a sourced counterweight — [[maintenance-is-the-real-cost]]: writing code was never the bottleneck, maintenance is, and an internal service is a second business. The reconciliation is a threshold, not a winner: tiny personal tools pass, replacing your Jira does not.
|
||||
- **The adoption side** — what goes wrong when organizations try this, from [[nikolai-sheiko]]'s multi-company casework ([[2026-07-30-rakes-in-ai-sdlc-adoption]]): the SDLC collapses *around the humans* — [[review-is-the-new-bottleneck|review becomes the bottleneck]] and volume metrics (LoC, PRs) go anti-informative, so measure **completed tasks without rework**; the developer's job flips from CPU-bound coding to [[developer-as-agent-manager|IO-bound agent management]]; and the winning company move is not custom AI development but installing and *evolving* a standard harness ([[enterprise-ai-reality]]) — with skills grown by walking the agent through real tasks and verified by a context-free subagent ([[solve-first-then-skillify]]).
|
||||
- **The demand side** — three interviews ground it all in a real audience. The two HR ones ([[2026-07-14-nina-interview|Nina]], [[2026-07-14-yulia-interview|Yulia]]) supply pain points (interview write-ups, job descriptions, sourcing) that collapse into "a candidate knowledge base plus search," teachable via [[levels-of-ai-usage]] and [[solve-first-then-skillify]]. Their key finding: **adoption is blocked by friction, not resistance.** The [[2026-07-21-larysa-interview|Larysa interview]] adds the *advanced* user's version of the same story: past the friction, the remaining walls are structural — no durable memory, [[integration-dead-ends|integrations that dead-end]], and drift on loose specs ([[leave-less-room-for-imagination]]). Her diagnosis matters because she is technically deep yet skipped the skills rung, which is exactly what her "the agent forgot" complaint reduces to.
|
||||
|
||||
See [[ai-agent-evolution]] for how the capability curve got here.
|
||||
|
||||
## Where sources agree vs diverge
|
||||
|
||||
- **Agree:** code is cheap/disposable; harnesses are the unit of work; skills-as-memory (Konstantin ↔ Allie ↔ Eugene); human relationships rise in value (Sebastian ↔ Allie ↔ Eugene, who lands there independently in the Yulia interview); solve-first-then-skillify (Eugene ↔ Konstantin's heuristics); context is the constraint — Thorsten's version is the bluntest: **the dominant variable in output quality is the information you put in**, not the model or the effort level. The [[2026-07-22-ai-is-stupid|"AI is stupid!" short]] independently compresses the machine-side spine into a business one-liner: **model + context + harness = employee-level answer**. Slop is a human problem, not an AI defect (Theo ↔ Thorsten, from verification discipline and from taste respectively). Software becomes personal — "little tools you make for yourself" (Eugene's webinar arc ↔ Thorsten's club app and bespoke forks ↔ Allie's personal OS).
|
||||
- **Diverge:** personal vs company-managed vs vendor-managed harness ([[eugene]] vs [[sebastian]] vs [[amp]]); online vs in-person networking (Eugene/Sebastian); OSS as marketing vs OSS growth; built-in agent memory as anti-feature (Eugene) vs persistent context docs used without complaint (Allie); tight specs ([[leave-less-room-for-imagination]]) vs wide latitude ([[think-wider-not-bigger]]); agent diff-summaries as sufficient review (Theo/Dax) vs invisible drift as the core danger (Eugene); model choice as a real lever (Eugene runs 4.7 over 4.8) vs a distraction past the frontier (Thorsten); local consolidated workspace (Eugene) vs local dev disappearing into remote sandboxes (Thorsten); build-your-own-tools ([[thorsten-ball]], the webinar arc) vs [[maintenance-is-the-real-cost|buy anything that needs ongoing support]] (the vibe-coding video, with the corpus's only observed reversal: an in-house Jira clone abandoned for Linear in four months). These live under "Contradictions" on the relevant pages.
|
||||
- **The one that matters most for the webinar:** [[thorsten-ball]] runs a 99%-AI-written codebase with **no skills, no MCP servers and no slash commands** — his context lives in the codebase and `AGENTS.md`. That is the corpus's first credible rejection of the mechanism the webinar's central promise rests on. Three readings (situational / premature abstraction / same thing under another name) are logged on [[skills-as-memory]]; none is settled, and the evidential asymmetry favours him — his is first-hand daily practice at scale.
|
||||
- **Agree:** code is cheap/disposable; harnesses are the unit of work; skills-as-memory (Konstantin ↔ Allie ↔ Eugene); human relationships rise in value (Sebastian ↔ Allie ↔ Eugene, who lands there independently in the Yulia interview); solve-first-then-skillify (Eugene ↔ Konstantin's heuristics); context is the constraint — Thorsten's version is the bluntest: **the dominant variable in output quality is the information you put in**, not the model or the effort level. The [[2026-07-22-ai-is-stupid|"AI is stupid!" short]] independently compresses the machine-side spine into a business one-liner: **model + context + harness = employee-level answer**. Slop is a human problem, not an AI defect (Theo ↔ Thorsten, from verification discipline and from taste respectively). Software becomes personal — "little tools you make for yourself" (Eugene's webinar arc ↔ Thorsten's club app and bespoke forks ↔ Allie's personal OS). And the corpus's central *stakes* claim now has outside measurement: [[2026-07-30-stanford-swepr-widening-gap|Stanford SWEPR]] finds the productivity gap between AI-mastering and lagging teams grew 4.8% → 19% (4×) from April 2023 to July 2025 — Allie's prediction, measured; the same study's codebase-size finding independently supports [[context-as-scarce-resource|context as the binding constraint]]. [[2026-07-30-rakes-in-ai-sdlc-adoption|Sheiko]] cites that same Stanford chart as his stakes slide and lands on the spine independently — "companies no longer need custom AI development, install Claude Code or Codex and configure it" is harness-over-model as a service playbook, and his codebase-stores-context prescription converges with Thorsten's from the opposite direction. His review-bottleneck casework (+1% net despite more PRs) is SWEPR's +91%-review-time finding told anecdotally.
|
||||
- **Diverge:** personal vs company-managed vs vendor-managed harness ([[eugene]] vs [[sebastian]] vs [[amp]]); online vs in-person networking (Eugene/Sebastian); OSS as marketing vs OSS growth; built-in agent memory as anti-feature (Eugene) vs persistent context docs used without complaint (Allie); tight specs ([[leave-less-room-for-imagination]]) vs wide latitude ([[think-wider-not-bigger]]); agent diff-summaries as sufficient review (Theo/Dax) vs invisible drift as the core danger (Eugene); model choice as a real lever (Eugene runs 4.7 over 4.8) vs a distraction past the frontier (Thorsten); local consolidated workspace (Eugene) vs local dev disappearing into remote sandboxes (Thorsten); build-your-own-tools ([[thorsten-ball]], the webinar arc) vs [[maintenance-is-the-real-cost|buy anything that needs ongoing support]] (the vibe-coding video, with the corpus's only observed reversal: an in-house Jira clone abandoned for Linear in four months); permit opting out of the agent-manager switch ([[nikolai-sheiko]] — "don't force everyone") vs the gap is irreversible and compounding ([[allie-miller]], [[swepr|Stanford]]) — see [[developer-as-agent-manager]]. These live under "Contradictions" on the relevant pages.
|
||||
- **The one that matters most for the webinar:** [[thorsten-ball]] runs a 99%-AI-written codebase with **no skills, no MCP servers and no slash commands** — his context lives in the codebase and `AGENTS.md`. That is the corpus's first credible rejection of the mechanism the webinar's central promise rests on. Three readings (situational / premature abstraction / same thing under another name) are logged on [[skills-as-memory]]; none is settled. The evidential asymmetry that favoured him narrowed on 2026-07-30: [[nikolai-sheiko]] is a second practitioner voice on the pro-skills side — his "Agentic Evolution" (guided tasks → agent writes the manual → context-free-subagent verification) is the corpus's first described *test* of a skill, though his cases are anonymous anecdotes where Thorsten's is first-hand daily practice at scale.
|
||||
|
||||
## Navigation
|
||||
|
||||
- **[[index]]** — content catalog
|
||||
- **Sources (11):** [[2026-07-14-everything-we-knew-about-software-has-changed|Theo Browne]] · [[2026-07-14-gap-between-ai-users-irreversible|Allie Miller]] · [[2026-07-14-sebastian-eugene-interview|Sebastian interview]] · [[2026-07-14-skills-based-on-git|Konstantin (git skills)]] · [[2026-07-14-nina-interview|Nina interview]] · [[2026-07-14-yulia-interview|Yulia interview]] · [[2026-07-21-larysa-interview|Larysa interview]] · [[2026-07-22-ai-is-stupid|"AI is stupid!" short]] · [[2026-07-24-youre-reading-way-too-much-code|Theo Browne (reading code)]] · [[2026-07-28-agentic-engineering-10x-developer|Thorsten Ball (agentic engineering)]] · [[2026-07-29-what-if-we-vibe-code-it|"What if we vibe-code it?" (maintenance trap)]]
|
||||
- **People:** [[theo-browne]] · [[allie-miller]] · [[sebastian]] · [[eugene]] · [[konstantin]] · [[nina]] · [[yulia]] · [[larysa]] · [[thorsten-ball]]
|
||||
- **Tools/orgs:** [[claude-code]] · [[amp]] · [[hermes]] · [[virtido]] · [[inspectron]]
|
||||
- **Concepts:** see the through-line above (25 pages) · **Timeline:** [[ai-agent-evolution]] · **Comparison:** [[theo-konstantin-allie]]
|
||||
- **Sources (13):** [[2026-07-14-everything-we-knew-about-software-has-changed|Theo Browne]] · [[2026-07-14-gap-between-ai-users-irreversible|Allie Miller]] · [[2026-07-14-sebastian-eugene-interview|Sebastian interview]] · [[2026-07-14-skills-based-on-git|Konstantin (git skills)]] · [[2026-07-14-nina-interview|Nina interview]] · [[2026-07-14-yulia-interview|Yulia interview]] · [[2026-07-21-larysa-interview|Larysa interview]] · [[2026-07-22-ai-is-stupid|"AI is stupid!" short]] · [[2026-07-24-youre-reading-way-too-much-code|Theo Browne (reading code)]] · [[2026-07-28-agentic-engineering-10x-developer|Thorsten Ball (agentic engineering)]] · [[2026-07-29-what-if-we-vibe-code-it|"What if we vibe-code it?" (maintenance trap)]] · [[2026-07-30-stanford-swepr-widening-gap|Stanford SWEPR (widening gap)]] · [[2026-07-30-rakes-in-ai-sdlc-adoption|Nikolai Sheiko (rakes in SDLC adoption)]]
|
||||
- **People:** [[theo-browne]] · [[allie-miller]] · [[sebastian]] · [[eugene]] · [[konstantin]] · [[nina]] · [[yulia]] · [[larysa]] · [[thorsten-ball]] · [[nikolai-sheiko]]
|
||||
- **Tools/orgs:** [[claude-code]] · [[amp]] · [[hermes]] · [[virtido]] · [[inspectron]] · [[swepr]]
|
||||
- **Concepts:** see the through-line above (27 pages) · **Timeline:** [[ai-agent-evolution]] · **Comparison:** [[theo-konstantin-allie]]
|
||||
|
||||
## Open Questions (vault-level)
|
||||
|
||||
|
||||
@@ -61,7 +61,8 @@ The audience's real objection is "can I trust it?", and the current script has n
|
||||
|
||||
**T7. The gap between AI users and everyone else compounds — and is becoming irreversible.**
|
||||
The person who builds their OS this week fears no release, because each capability slots into a system that already knows them. *(Thorsten names **token budget** as a second winner/loser variable, but that is a claim about metered agent-fleet work; for this audience the budget is one consumer subscription — keep the thesis on the skill gap. See [[enterprise-ai-reality]].)*
|
||||
— [[2026-07-14-gap-between-ai-users-irreversible]]
|
||||
*(Measured 2026-07-30: this thesis is no longer prediction-only — Stanford SWEPR's 46-vs-46-team analysis shows the gap growing 4.8% → 19% (4×) from April 2023 to July 2025. One citable stage line: "Stanford measured it: the gap quadrupled in two years." See [[2026-07-30-stanford-swepr-widening-gap]]. **Added to the script's closing arc 2026-07-30** — T7 is now dramatized, upgrading it from Q&A material to an on-stage beat.)*
|
||||
— [[2026-07-14-gap-between-ai-users-irreversible]] · [[2026-07-30-stanford-swepr-widening-gap]]
|
||||
|
||||
**T8. The more the world is mediated by AI proxies, the more valuable real human connection becomes.**
|
||||
The "market of one" raises, not lowers, the price of being human.
|
||||
|
||||
55
wiki/queries/2026-07-30-stanford-widening-gap-source.md
Normal file
55
wiki/queries/2026-07-30-stanford-widening-gap-source.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# Stanford "widening gap" chart — original research located
|
||||
|
||||
#query
|
||||
|
||||
## Question asked
|
||||
|
||||
The user saved a chart screenshot (`raw/assets/G6g3O60bkAE05ZW.png`, filename pattern = X/Twitter image) of a Stanford slide titled *"Teams that master AI are accelerating their productivity gains, widening the gap with laggards"* and asked: **find the original Stanford research behind it.**
|
||||
|
||||
## Answer
|
||||
|
||||
The slide is from **Stanford's Software Engineering Productivity Research group (SWEPR)**, led by researcher **Yegor Denisov-Blanch**. Since 2022 the group has analyzed private Git repositories from 600+ companies and 120,000+ engineers, scoring every commit with an ML model trained to replicate a panel of human expert reviewers.
|
||||
|
||||
The specific chart is their **causal difference-in-differences analysis**: 46 teams that adopted AI, matched against 46 similar non-AI teams, with net productivity gains measured quarterly.
|
||||
|
||||
- **April 2023:** 4.8% gap between top-quartile and bottom-quartile AI adopters.
|
||||
- **July 2025:** 19% gap — a **4× increase** in ~2.25 years.
|
||||
- Slide footer: Stanford University / SWEPR; the specific slide (page 3) matches Denisov-Blanch's September 2025 AI Conference deck.
|
||||
|
||||
**Primary sources:**
|
||||
|
||||
- Research group home: <https://softwareengineeringproductivity.stanford.edu/> (AI Impact research, AI Practices Benchmark)
|
||||
- Talk the slide comes from: *"Will AI Replace Software Engineers?"*, AI Conference, Sept 2025 — [slide deck PDF](https://aiconference.com/wp-content/uploads/2025/09/Yegor-Denisov-Blanch-Will-AI-Replace-Software-Engineers_-.pptx.pdf)
|
||||
- Video walkthrough: [Can you prove AI ROI in Software Eng? (Stanford 120k Devs Study)](https://www.youtube.com/watch?v=JvosMkuNxF8)
|
||||
- Researcher site: <https://yegordb.com/>
|
||||
- Peer-reviewed methodology paper: [Predicting Expert Evaluations in Software Code Reviews](https://arxiv.org/pdf/2409.15152)
|
||||
|
||||
**Caveat (Status: tentative):** the 46-vs-46 difference-in-differences result itself has been presented via talks, webinars and decks — not (yet) a peer-reviewed paper. The peer-reviewed publications cover the *measurement methodology*, not this specific analysis.
|
||||
|
||||
**Surrounding findings from the same study** (useful nuance): AI raises developer productivity ~15–20% on average, with high variance — largest gains on greenfield/simple tasks in popular languages; AI can *decrease* net productivity in complex legacy codebases (rework eats the gains, ~2.6× increase in rework reported).
|
||||
|
||||
## Why this matters to the vault
|
||||
|
||||
This is the **first quantitative, external, longitudinal measurement** of a claim the corpus so far held only as practitioner assertion:
|
||||
|
||||
- [[2026-07-14-gap-between-ai-users-irreversible]] — [[allie-miller]]'s central prediction ("in 12 months the gap will be irreversible") is the same shape as this curve, asserted from advisory experience. Stanford now supplies measured team-level data pointing the same direction.
|
||||
- [[2026-07-28-webinar-theses]] — the "stakes" thesis group (irreversible gap) gains a citable number: *4.8% → 19%, 4× in about two years*. A Stanford chart is far stronger webinar ammunition than "an ex-Amazon AI leader predicts…".
|
||||
- The "AI can decrease productivity in complex legacy codebases" finding is honest-caveat material aligning with the vault's recorded tensions ([[maintenance-is-the-real-cost]], rework costs; [[make-more-cheap-code]]'s verification burden — cf. the study's 91% increase in PR review time).
|
||||
- The mechanism Stanford implies (teams that *master* AI compound, laggards stall) is the team-level twin of [[levels-of-ai-usage]] — the gap grows between rungs, not between haves and have-nots of licenses.
|
||||
|
||||
## Evidence trail
|
||||
|
||||
- Screenshot: `raw/assets/G6g3O60bkAE05ZW.png` (raw asset; likely captured from an X/Twitter post sharing the talk)
|
||||
- Web search + fetch of the SWEPR site and the AI Conference deck (2026-07-30); slide title, footer, chart annotations and page number all match the deck's era (data ends July 2025)
|
||||
|
||||
## Follow-up questions
|
||||
|
||||
- ~~Ingest-worthy?~~ **Done, same day:** the user authorized a new raw source file; the dossier lives at `raw/sources/Stanford SWEPR - AI and the widening productivity gap.md` and is ingested as [[2026-07-30-stanford-swepr-widening-gap]] — concept pages now cite it directly.
|
||||
- Does the webinar want the number? One line — "Stanford measured it: the gap 4×'d in two years" — would upgrade the stakes beat from prediction to measurement.
|
||||
- Watch for a peer-reviewed version of the difference-in-differences analysis; the claim's status upgrades from tentative when it lands.
|
||||
|
||||
## Whether this output changed existing pages
|
||||
|
||||
- [[2026-07-14-gap-between-ai-users-irreversible]] — added an external-corroboration pointer to this page under Connections.
|
||||
- `index.md` (Queries section) and `log.md` updated.
|
||||
- No concept pages changed — deliberately, since the underlying talk is not yet ingested as a source (citation policy: concept evidence should point at `wiki/sources/*`).
|
||||
@@ -37,6 +37,7 @@
|
||||
- **Concepts:** [[personal-ai-operating-system]] · [[skills-as-memory]] · [[context-as-scarce-resource]] · [[connections-as-moat]] (human-relationship side effect)
|
||||
- **Related sources:** [[2026-07-14-skills-based-on-git]] (skills-as-memory from the engineering side — strong overlap) · [[2026-07-14-everything-we-knew-about-software-has-changed]] (markdown-as-skill ≈ G-brain markdown tier)
|
||||
- **Tools mentioned:** [[claude-code]], Claude Cowork, Codex, Flint
|
||||
- **External corroboration:** [[2026-07-30-stanford-swepr-widening-gap]] — Stanford SWEPR's difference-in-differences study measured the gap between AI-mastering and lagging teams growing 4.8% → 19% (4×) from April 2023 to July 2025; the title claim's first measured, non-practitioner support. (Traced via [[2026-07-30-stanford-widening-gap-source]].)
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
55
wiki/sources/2026-07-30-rakes-in-ai-sdlc-adoption.md
Normal file
55
wiki/sources/2026-07-30-rakes-in-ai-sdlc-adoption.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# Грабли во внедрении ИИ в SDLC — Rakes in AI Adoption in the SDLC (Nikolai Sheiko)
|
||||
|
||||
#source
|
||||
|
||||
## Source Metadata
|
||||
|
||||
- **Date:** talk published 2026 (references events through Dec 2025); conclusions doc saved 2026-07-30
|
||||
- **Raw path:** `raw/sources/Грабли во внедрении ИИ в SDLC.md`
|
||||
- **Source type:** viewer's conclusions from a Russian-language YouTube talk (45:59) — https://www.youtube.com/watch?v=Nm3MsnngCJg — "Грабли во внедрении ИИ в SDLC — почему ИИ есть, а результата нет и как это лечить" ("why the AI is there but the results aren't, and how to treat it"). Not a transcript.
|
||||
- **Speaker:** [[nikolai-sheiko]] — AI-adoption consultant/practitioner (works with client companies on SDLC adoption; background otherwise unknown)
|
||||
- **Ingestion date:** 2026-07-30
|
||||
|
||||
## Core Claims
|
||||
|
||||
- **AI in development already delivers real gains, but people, companies and metrics throttle it by an order of magnitude.** The real jump started Dec 2025 (Opus 4.5 / GPT-5.2 + Claude Code / Codex); the SDLC collapsed into days/hours — but *not fully*: two human "red squares" remain — the **reviewer** (tasks queue at review) and the **planner/product person**. Developer time redistributed from "coding in the middle" to "planning on the left + verification on the right." See [[review-is-the-new-bottleneck]].
|
||||
- **Universal error #0: a developer is not a manager.** A good developer is 3–5 hours of CPU-bound focus on one feature; a good AI-developer is an IO-bound **manager of an agent-employee**, running several tasks in parallel. "If you launched Claude Code and sit watching it work — you're a bad employee." Not everyone can make the psychological switch, *and that's fine* — don't force everyone. See [[developer-as-agent-manager]].
|
||||
- **Measure completed tasks without rework** — never LoC, commit count or PR count (all trivially hacked; the European-outsourcing case shipped more PRs for a +1% gain because rework ate everything). A task counts only if it doesn't come back for fixes; also track task lifetime + rework time.
|
||||
- **Review with the agent, not instead of it and not fully by hand.** Manual-only review → queue → burnout → quality collapse; fully delegated review is the opposite error. Treat the model as a smart student: direct it, pose hypotheses, find problems together.
|
||||
- **Companies no longer need custom AI development.** Key quote: *"Come in, install Claude Code or Codex, configure everything, attach connectors, think about security — and it works better than any custom build."* Corollary anti-pattern: hiring an external configurator who leaves behind a "magic artifact" nobody owns — teams must configure their own tools; what a company should buy is a **teacher/curator**, not a setup.
|
||||
- **Agentic Evolution** — the key concept. Don't ask the expert to explain how they work (you get theory); instead take the new employee (the agent) by the hand through hard tasks, show it the rakes, then say: *"remember all of this and write the manual for the next one."* Verify a skill by launching a **context-free subagent** that must solve the same task from scratch using only the skill; the mentor agent watches what fails and fixes the skill. Without evolution you live on defaults; with it, vertical growth begins.
|
||||
- **Best practices matter more with agents, not less.** The "compaction curse": on a huge codebase the agent gathers context → window overflows → compaction → re-gathers → compaction again, and the task barely completes. The cure is locality, isolated modules with interfaces — **the codebase stores the context**. Use AST search instead of grep on colossal projects. And **embeddings/RAG over code do not work** — don't use them unless you understand *very* well why.
|
||||
- **Role futures:** a **Product engineer** emerges (answers *why* we build it this way, what to cut, what to ignore); users vs **Agentic Operations** (who tune the SDLC, feedback loops, prompts, skills) gradually split; AI eats **Intelligence** (action sequences requiring intellect) while **Judgment** (taste built over years, or domain expertise — oil & gas, medicine) stays human for now.
|
||||
- **Tokens get more expensive near-term, cheaper later. It's the wild west — experiment at full throttle while subscriptions are cheap**; the goal is to land in the top half of the Stanford chart. Don't chase every new tool: what Claude Code / Codex doesn't absorb within a couple of months is probably useless.
|
||||
|
||||
## Key Evidence / Details
|
||||
|
||||
- **Timeline the talk builds on:** METR study (Jul 2025, measured a *slowdown*, but methodologically contested) → Stanford study (Aug 2025, +20% speed but top-performers pull away — this is [[swepr]]'s research, cited independently) → Karpathy's tweet (Nov–Dec 2025: "80% Claude Code, 20% by hand") → "SDLC is dead" article (stages collapse into days/hours).
|
||||
- **Case 1 (frontend migration):** no feedback loop → give the agent a browser (Playwright / Chrome DevTools) to compare old vs new front; crunch → invest in planning (20 min minimum, hours are normal; every 10 min of planning saves hours; target one-shot implementation); Cursor with per-token billing → team economizes instead of experimenting (~30% dearer than subscriptions at the same level); training rollout: record sessions → expert reviews → *writes feedback, team fixes their own agent instructions* → focus on the top 2 performers (an hour with them is worth 10× more).
|
||||
- **Case 2 (European outsourcer):** more PRs, +1% — rework was the cause; the metrics table (don't: LoC/commits/PRs; do: completed-tasks-without-rework).
|
||||
- **Case 3 (large codebase):** the compaction curse; "agents mean we can drop best practices" is exactly wrong.
|
||||
- **Mini-cases:** the middle dev who sped the team up by tens of %, was refused a raise, and left for much more ("if you're that middle — think; if you're the manager — think twice"); the startup doing spec-driven development without knowing what it wants — build the UI first (even with an in-memory browser DB), click around, *then* plan.
|
||||
- **Do-tomorrow list:** close the feedback loop; write a skill that analyses your own sessions daily; automate it (Codex schedules / Anthropic routines); voice input (more context, and Russian gives more context than English); ignore tool churn.
|
||||
- **Q&A notes:** GLM (good, no vision), Kimi (works, has vision), Xiaomi's agent (interesting memory implementation; ex-DeepSeek Head of AI) as the Chinese-model trend; hardware deficit (H100s unrentable); speaker's prediction that AI will be classed as a cyber-weapon with biotech-style licensing.
|
||||
- **One-shot recipe:** feedback loop (must-have) + explicit real *goal* the agent self-checks against + skill verification via the context-free subagent.
|
||||
|
||||
## Connections
|
||||
|
||||
- **Entities:** [[nikolai-sheiko]], [[swepr]] (its Stanford study is the talk's central chart — "be in the top half"), [[claude-code]]
|
||||
- **New concepts:** [[review-is-the-new-bottleneck]], [[developer-as-agent-manager]]
|
||||
- **Corroborates:** [[2026-07-30-stanford-swepr-widening-gap]] (independent practitioner citation of the widening-gap result *and* of review-as-downstream-cost); [[solve-first-then-skillify]] (Agentic Evolution is its strongest formulation, plus the missing verification step); [[skills-as-memory]] (a second frontier-practitioner vote *for* the skills/evolution layer, against [[thorsten-ball]]'s dissent — and a vote *against* embeddings/RAG over code, siding with Konstantin in the skills-vs-RAG contradiction); [[context-as-scarce-resource]] (compaction curse; codebase-stores-context converges with Thorsten from the opposite direction); [[async-by-default]] (IO-bound parallel management as the working mode); [[enterprise-ai-reality]] ("no custom AI development needed" names the same managed-harness market; per-token billing shapes behaviour — the metered-vs-subscription split already logged there); [[harness]] (install-and-configure beats custom builds); [[leave-less-room-for-imagination]] (explicit goal + planning discipline).
|
||||
- **Complicates:** [[make-more-cheap-code]] (rework-free-completion as the metric is the org-level answer to "generation moved the cost downstream").
|
||||
|
||||
## Open Questions
|
||||
|
||||
- The cases are anonymous client anecdotes with self-reported numbers (+1%, "tens of %"); none are verifiable. Status: tentative.
|
||||
- "Embeddings/RAG over code don't work" is stated flatly with no mechanism given — strong claim, no evidence in the doc. Status: tentative.
|
||||
- Does the context-free-subagent verification protocol actually measure skill quality, or only skill *completeness* for one task? Closely related to the corpus's proposed skills falsification test ([[skills-as-memory]]) — this is the first source to describe running one.
|
||||
- The Dec-2025 "real jump" periodization is the speaker's own; the corpus's other timeline ([[ai-agent-evolution]]) slices eras differently.
|
||||
- Predictions (AI as licensed cyber-weapon; token prices up then down) are speculation. Status: tentative.
|
||||
|
||||
## Change Impact on Wiki
|
||||
|
||||
- Created [[nikolai-sheiko]] (entity), [[review-is-the-new-bottleneck]] and [[developer-as-agent-manager]] (concepts).
|
||||
- Updated [[solve-first-then-skillify]] (Agentic Evolution + skill-verification protocol), [[skills-as-memory]] (second practitioner vote for skills; anti-RAG-for-code), [[context-as-scarce-resource]] (compaction curse, AST search, codebase-stores-context), [[async-by-default]] (IO-bound manager evidence), [[enterprise-ai-reality]] (no-custom-AI-dev quote; external-configurator anti-pattern; token-price prediction), [[make-more-cheap-code]] (related link), [[2026-07-30-stanford-swepr-widening-gap]] (independent citation).
|
||||
- Updated [[overview]] (12 → 13 sources), `index.md`, `log.md`.
|
||||
48
wiki/sources/2026-07-30-stanford-swepr-widening-gap.md
Normal file
48
wiki/sources/2026-07-30-stanford-swepr-widening-gap.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# Stanford SWEPR — AI and the widening productivity gap
|
||||
|
||||
#source
|
||||
|
||||
## Source Metadata
|
||||
|
||||
- **Date:** research presented through 2025 (chart data April 2023 → July 2025; deck Sept 2025); dossier compiled 2026-07-30
|
||||
- **Raw path:** `raw/sources/Stanford SWEPR - AI and the widening productivity gap.md` (screenshot: `raw/assets/G6g3O60bkAE05ZW.png`)
|
||||
- **Source type:** research dossier — a slide read first-hand plus public coverage of the underlying study; **not** a talk transcript. The corpus's first *quantitative outside study* (every other source is practitioner testimony or synthesis).
|
||||
- **Authors:** [[swepr]] (Stanford Software Engineering Productivity Research group), public face Yegor Denisov-Blanch
|
||||
- **Ingestion date:** 2026-07-30
|
||||
|
||||
## Core Claims
|
||||
|
||||
- **The gap between AI-mastering teams and laggards is widening, measurably.** Difference-in-differences analysis of 46 AI-adopting teams vs 46 matched non-AI teams: net productivity difference between top and bottom quartiles grew from **4.8% (April 2023) to 19% (July 2025) — a 4× increase** in ~2.25 years. Early quarters hover near zero or negative; the curve only takes off from mid-2024, then accelerates.
|
||||
- **Average net gain from AI is ~15–20%, not 10×.** Gross delivered code volume rises 30–40%, but rework (fixing AI-introduced bugs) eats roughly half.
|
||||
- **Gains are strongly context-dependent:** greenfield/low-complexity 30–40%; brownfield/high-complexity 0–10% and can be *negative*. Popular languages gain more than niche ones; gains collapse as codebases grow 10k → 10M lines (context-window limits, signal-to-noise).
|
||||
- **The cost moved downstream:** coverage reports +91% PR review time and ~2.6× rework in AI-heavy workflows — writing got cheaper, reviewing got more expensive.
|
||||
- **Proposed mechanism for the gap:** quality of AI usage beats volume; teams with clean, modular, well-tested code compound gains, teams with poor hygiene accumulate debt and lose trust in the tools.
|
||||
|
||||
## Key Evidence / Details
|
||||
|
||||
- Data: private Git repos, 600+ companies, ~100k–120k engineers, since 2022; ML model replicating a panel of expert reviewers, measuring *functionality delivered* (not commits/LOC). Methodology peer-reviewed (arXiv 2409.15152); the DiD result itself is talk-published only.
|
||||
- The slide: "Causal Impact of AI on Software Engineering Productivity: Difference-in-Differences Analysis," DID covariate balance < 0.25, 95% CI band, model-release markers on the time axis. Deck: "Will AI Replace Software Engineers?", AI Conference, Sept 2025.
|
||||
- Full numbers, links, and per-claim provenance in the raw dossier.
|
||||
|
||||
## Connections
|
||||
|
||||
- **Entity:** [[swepr]]
|
||||
- **Corroborates:** [[2026-07-14-gap-between-ai-users-irreversible]] — Allie Miller's "irreversible gap" prediction is this curve, asserted 18 months earlier from advisory experience; Stanford supplies the measurement. Also the team-level twin of [[levels-of-ai-usage]] (the gap grows between *rungs of mastery*, not between license-holders and others).
|
||||
- **Corroborates:** [[context-as-scarce-resource]] — the codebase-size finding (gains collapse toward 10M LOC, attributed to context-window limits and signal-to-noise) is the corpus's first outside quantitative support for context as the binding constraint.
|
||||
- **Complicates:** [[make-more-cheap-code]] / [[code-as-throwaway]] — +91% PR review time and 2.6× rework externally confirm that generation moved the cost to review/verification, which is Theo's premise; but the *negative* gains in complex brownfield code sharpen the honest caveat that "code is cheap" holds least where most code lives.
|
||||
- **Independently cited by a practitioner source:** [[2026-07-30-rakes-in-ai-sdlc-adoption]] builds its stakes on this study ("the Stanford study, Aug 2025: +20% speed, but the top performers pull away — the goal is to be in the top half of the chart") and anecdotally mirrors its downstream-cost finding (the +1%-despite-more-PRs rework case; review as the bottleneck — see [[review-is-the-new-bottleneck]]).
|
||||
- **Queries:** [[2026-07-30-stanford-widening-gap-source]] (how this source was traced), [[2026-07-28-webinar-theses]] (thesis T7 gains its number)
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Does the 46-team DiD analysis ever get a peer-reviewed publication? (Claim status upgrades when it does.)
|
||||
- What exactly distinguishes the mastering teams — tooling, codebase hygiene, or skills/context practices? The proposed mechanism ("quality of usage") is asserted in talks, not decomposed. Directly relevant to whether the webinar's skills rung is *the* differentiator. Status: tentative.
|
||||
- The study measures teams of engineers; how far do team-level results transfer to the webinar's non-engineer audience? Status: tentative.
|
||||
|
||||
## Change Impact on Wiki
|
||||
|
||||
- Created [[swepr]] entity.
|
||||
- Added measured external evidence to [[levels-of-ai-usage]], [[context-as-scarce-resource]] and [[make-more-cheap-code]].
|
||||
- Upgraded the corroboration pointer on [[2026-07-14-gap-between-ai-users-irreversible]] to cite this page.
|
||||
- Added the "Stanford measured it: 4× in two years" note to thesis T7 in [[2026-07-28-webinar-theses]].
|
||||
- Updated [[overview]] (11 → 12 sources; first quantitative outside study), `index.md`, `log.md`.
|
||||
Reference in New Issue
Block a user