ingest: Stanford SWEPR widening-gap study and AI-in-SDLC adoption pitfalls

Add two new sources with summaries, new concepts (developer-as-agent-manager,
review-is-the-new-bottleneck), new entities (SWEPR, Nikolai Sheiko), and a
query on the Stanford source; update related concept pages, overview, index,
and log.
This commit is contained in:
EugeneTes
2026-07-31 08:33:56 +02:00
parent 62d0f06a2d
commit 3314112bb9
24 changed files with 719 additions and 45 deletions

View File

@@ -20,11 +20,12 @@ If agent work takes sixteen minutes and you are doing something else, latency st
- Orbs, sleep/wake, one-URL packaging, multiplayer handoff, the 16-minute live demo, the collapsed cloud-IDE objections, Quinn's proof line, paper-cut velocity, local-dev prediction, infra-margin prediction — [[2026-07-28-agentic-engineering-10x-developer]].
- Fire-and-forget as a native harness mode; completion notifications as what makes background agents usable — [[2026-07-14-skills-based-on-git]], [[harness]].
- The same mode as a job description: the AI-developer is IO-bound, runs tasks in parallel, and "if you launched Claude Code and sit watching it work — you're a bad employee" — [[2026-07-30-rakes-in-ai-sdlc-adoption]] (the role-shift side lives at [[developer-as-agent-manager]]).
- Scheduled agents producing while you sleep (the non-engineer version) — [[personal-ai-operating-system]].
## Related Pages
- Concepts: [[harness]] (async is one of its two modes), [[agentic-loops]], [[make-more-cheap-code]] (proof artifacts are throwaway code with a job), [[shedding-weight]] (async is what makes killing the backlog possible — parked agents replace queued tickets), [[personal-ai-operating-system]], [[context-as-scarce-resource]]
- Concepts: [[harness]] (async is one of its two modes), [[agentic-loops]], [[make-more-cheap-code]] (proof artifacts are throwaway code with a job), [[shedding-weight]] (async is what makes killing the backlog possible — parked agents replace queued tickets), [[personal-ai-operating-system]], [[context-as-scarce-resource]], [[developer-as-agent-manager]] (the human role this mode implies), [[review-is-the-new-bottleneck]] (where the parallel diffs pile up)
- Entities: [[thorsten-ball]], [[amp]], [[claude-code]]
## Contradictions / Uncertainty

View File

@@ -22,6 +22,8 @@ The human role has climbed prompt-engineer → **context-engineer** → harness-
**Context now has a price — at fleet scale.** The same source names **token budget** as one of two variables separating winners from losers, alongside knowing how to use agents. Context has always been scarce per-request; this is the corpus's first claim that it is also scarce per-*wallet*. Scope, corrected 2026-07-28: the claim comes from **metered** usage (parallel remote sandboxes), and under a flat consumer subscription the corpus's own heavy users report no ceiling — so per-request scarcity remains the binding constraint for individuals, and per-wallet scarcity is a fleet and enterprise concern. See [[enterprise-ai-reality]] and [[explosion-of-internal-software]].
**The compaction curse — scarcity at codebase scale** ([[2026-07-30-rakes-in-ai-sdlc-adoption]]). On a huge codebase the agent gathers context → the window overflows → compaction → it re-gathers → compaction again, and the task barely completes. [[nikolai-sheiko]]'s cure inverts the "agents mean best practices don't matter" fallacy: **best practices exist for the agent now** — locality, isolated modules with interfaces, so that *the codebase stores the context*. This converges with [[thorsten-ball]]'s context-lives-in-the-codebase position from the opposite direction (Thorsten skips skills because his codebase carries context; Sheiko says make your codebase able to). Practical additions: **AST search instead of grep** on colossal projects (grep returns a wall of noise, AST returns the relevant slice), and a flat rejection of embeddings/RAG over code. The mechanism matches Stanford's measured gains-collapse toward 10M LOC (evidence below).
**The supply-side facet** ([[2026-07-22-ai-is-stupid]]): before context is *scarce* it is usually *absent*. "Intelligence without context loses to context without intelligence" — ten Nobel laureates asked about your sales month can only cite industry averages, while your rank-and-file employee answers better because they see your funnel, clients, and deals. The default "stupid AI" experience is a strong model given neither business context nor a [[harness]]; the fix is investing in context infrastructure (data, memory, integrations) before reaching for a bigger model.
## Evidence
@@ -31,6 +33,8 @@ The human role has climbed prompt-engineer → **context-engineer** → harness-
- "Intelligence without context loses"; Nobel-vs-employee analogy; invest in context before model upgrades — [[2026-07-22-ai-is-stupid]].
- Information > model choice > effort level; the two information sources; token budget as a winner/loser variable — [[2026-07-28-agentic-engineering-10x-developer]].
- Reading costs attention — the human-side analog of the same scarcity — [[make-more-cheap-code]], [[2026-07-24-youre-reading-way-too-much-code]].
- **First outside quantitative support:** Stanford SWEPR finds AI productivity gains collapse as codebases grow from ~10k to ~10M lines, attributing it to context-window limits and signal-to-noise degradation — the corpus's context-is-the-constraint claim, measured at scale — [[2026-07-30-stanford-swepr-widening-gap]].
- Compaction curse; best-practices-for-the-agent (locality, interfaces, codebase-stores-context); AST search over grep; embeddings/RAG over code rejected — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
## Related Pages
@@ -40,7 +44,7 @@ The human role has climbed prompt-engineer → **context-engineer** → harness-
## Contradictions / Uncertainty
- "First third = smart zone" is a heuristic, not a measured boundary; likely model-dependent. Status: tentative.
- [[2026-07-22-ai-is-stupid]] names **RAG** and long-term assistant memory as the practical context mechanisms; [[2026-07-14-skills-based-on-git]] argues [[skills-as-memory|skills]] beat RAG (load-on-activation vs pre-injection). Possibly audience-driven (business data vs procedures) rather than a real disagreement. Status: tentative.
- [[2026-07-22-ai-is-stupid]] names **RAG** and long-term assistant memory as the practical context mechanisms; [[2026-07-14-skills-based-on-git]] argues [[skills-as-memory|skills]] beat RAG (load-on-activation vs pre-injection). Possibly audience-driven (business data vs procedures) rather than a real disagreement. Status: tentative. *(2026-07-30: [[nikolai-sheiko]] adds a hard anti-RAG data point for the code domain specifically — "embeddings over code don't work" — which supports the audience-driven reading: the anti-RAG votes are both about code/procedures, the pro-RAG vote is about business data.)*
## Next Questions

View File

@@ -0,0 +1,37 @@
# Developer as Agent Manager
#concept
## Summary
[[nikolai-sheiko]]'s "universal error #0": treating AI-assisted development as the same job at higher speed. A good developer is **CPU-bound** — 35 hours of deep focus on one feature. A good AI-developer is **IO-bound** — a *manager of an agent-employee*, running several tasks in parallel, spending their time on planning and verification instead of typing. His blunt test: "If you launched Claude Code and sit watching it work — you're a bad employee."
## Current Understanding
- **The switch is psychological, not technical** — and not everyone can make it. Sheiko is explicit that this is *fine*: don't drag everyone into AI under one brush; conservatives "guard the company from the vibe-coders." Compare [[2026-07-14-nina-interview|Nina's finding]] that adoption blocks on friction, not resistance — this page is the case where genuine resistance exists and is legitimate.
- **Where the time goes instead:** "planning on the left, verification on the right" — the coding middle collapsed. Hence 20-minutes-minimum planning, explicit goals the agent self-checks against, and the review discipline of [[review-is-the-new-bottleneck]].
- **The role splits further.** Today the people who *use* agents and the people who *tune the harness* are the same; Sheiko predicts a split into **users** (plan + verify a feature) and **Agentic Operations** (own the SDLC configuration, feedback loops, prompts, skills). A **Product engineer** also emerges — answers *why* we build it this way, what to simplify, what to ignore.
- **What stays human: Judgment over Intelligence.** AI absorbs *Intelligence* (action sequences requiring intellect); *Judgment* — taste built over years, or deep domain expertise (oil & gas, medicine) — remains human for now. We detect slop precisely because it lacks human touch. This is the corpus's [[seniority-and-the-junior-squeeze|judgment-as-risk-reduction]] claim restated as a capability boundary.
- **Corpus convergence.** The same working mode appears as [[async-by-default|orbs and parallel delegation]] ([[thorsten-ball]]), Eugene's 7 parallel project-agents, Allie's ~100 agents, and Karpathy's "80% Claude Code, 20% by hand" (cited within the talk). Sheiko's contribution is naming the *identity* shift and its HR consequences: the middle dev who mastered this, asked for a raise, was refused, and left for far more.
## Evidence
- CPU-bound vs IO-bound framing; "sit watching = bad employee"; don't-force-everyone; users vs Agentic Operations; Product engineer; Intelligence vs Judgment; the raise-refusal mini-case — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
- The same mode practiced at the frontier (parallel remote sandboxes, delegation, proof-asking) — [[2026-07-28-agentic-engineering-10x-developer]] via [[async-by-default]].
- Non-engineer versions of parallel delegation (~100 agents, scheduled workflows) — [[2026-07-14-gap-between-ai-users-irreversible]].
## Related Pages
- Concepts: [[async-by-default]] (the infrastructure this role runs on), [[review-is-the-new-bottleneck]] (where the manager's verification time goes), [[product-ownership]] (the webinar's human-side twin: own outcomes, not tickets), [[seniority-and-the-junior-squeeze]] (Judgment as the durable half), [[levels-of-ai-usage]] (the non-engineer's version of the same climb), [[harness]]
- Entities: [[nikolai-sheiko]], [[thorsten-ball]], [[eugene]], [[allie-miller]]
## Contradictions / Uncertainty
- **"Don't force everyone" vs the widening gap.** Sheiko permits opting out; [[2026-07-14-gap-between-ai-users-irreversible|Allie]] calls the gap irreversible and [[2026-07-30-stanford-swepr-widening-gap|Stanford measured it quadrupling]]. If both are right, opting out is legitimate *and* costly, and the source doesn't reconcile the two. Status: tentative.
- The users / Agentic-Operations split is a prediction, not an observation — today's evidence (Eugene, Thorsten) is of people doing both. Status: tentative.
- Intelligence-vs-Judgment is a moving boundary asserted "for now"; the corpus has no criterion for where it stops moving.
## Next Questions
- What does the manager's day actually look like — is there a source with a concrete parallel-task routine (queue depth, check-in cadence) rather than the mode's name?
- Does the Agentic-Operations role match the webinar's promise that non-engineers can self-serve ([[levels-of-ai-usage]]), or does it re-centralize harness-tuning in specialists?

View File

@@ -14,17 +14,19 @@ The indie/practitioner world and the regulated-enterprise world diverge sharply.
- **Adjacent constraints:** the [[seniority-and-the-junior-squeeze|"read what you approve"]] security concern is amplified at scale; safety-critical/regulated code is the clear exception to [[code-as-throwaway|"most code isn't high-value"]].
- **A second divide: the token budget** (added 2026-07-28; **scope corrected 2026-07-28** — see below). [[thorsten-ball]] names two variables separating winners from losers — knowing how to use agents, and **having the token budget to do it**. It cuts both ways for this page: an enterprise can buy budget an individual cannot, while a locked-down enterprise may withhold it from the people who would use it best. Whoever controls the budget controls how far [[explosion-of-internal-software|internal software]] spreads. Thorsten names the variable and says nothing about who pays.
- **Scoping correction.** This was first written here as "the divide is also a *spending* gap," which overstates it. Thorsten's pricing regime is **metered**: [[amp]] sells usage, and his working pattern is parallel remote sandboxes and parked orbs ([[async-by-default]]) — a fleet cost, not a seat cost. Under a **flat consumer subscription** the corpus's own evidence points the other way: [[eugene]] runs 7 project-agents in parallel on a $200 plan, [[allie-miller]] runs ~100 agents and 36 workflows, and neither reports hitting a cost ceiling — while [[2026-07-14-sebastian-eugene-interview]] frames levelling as "a 20-year veteran and a fresh grad **on the same subscription**." For individual and small-team use the budget is one subscription; the token-budget variable bites at fleet scale and under metered pricing, which is where Thorsten sits and where enterprises will land.
- **The market claim, seconded — and sharpened into a quote** (added 2026-07-30). [[nikolai-sheiko]], from multi-company adoption work: *"Companies no longer need custom AI development. Come in, install Claude Code or Codex, configure everything, attach connectors, think about security — and it works better than any custom build."* This is Sebastian's company-managed-harness market stated as a service playbook. Two adoption anti-patterns attached: **the external configurator** who leaves a "magic artifact" nobody on the team owns (what a company should buy is a *teacher/curator*; the team must configure its own tools — the [[solve-first-then-skillify|evolution]] has to happen in their hands), and **metered pricing shaping behaviour** — a team on Cursor's per-token billing economizes instead of experimenting (~30% dearer than subscriptions at the same level), which is the metered-vs-subscription split from the scoping correction above observed as an organizational failure mode. His pricing prediction — tokens get dearer near-term, cheaper later; "experiment at full throttle while subscriptions are cheap" — matches [[eugene]]'s price-rise prediction already flagged in the token-budget question.
- **The frontier's advice does not transfer.** [[shedding-weight]] — kill the backlog, kill CI that repeats the agent's tests, kill local dev in favour of remote sandboxes ([[async-by-default]]) — describes a startup that owns its own process. In a regulated shop the pipeline, the audit trail and the ticket history frequently *are* the deliverable to a regulator, and code sitting in a vendor's remote sandbox is precisely what Sebastian's clients forbid. The gap between what the frontier recommends and what compliance permits is the same gap this page calls the market.
## Evidence
- Managed VMs / zero self-install, Roche ~1,200 engineers, banks banned→adopting, "company-managed resource," "the interesting market" — [[2026-07-14-sebastian-eugene-interview]].
- Token budget as a winner/loser variable; the frontier playbook (kill backlog/CI/local dev, remote sandboxes) that compliance cannot follow — [[2026-07-28-agentic-engineering-10x-developer]].
- "No custom AI development needed" quote; external-configurator anti-pattern vs teacher/curator; Cursor per-token billing → team economizes; tokens-dearer-then-cheaper prediction — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
## Related Pages
- Concepts: [[harness]], [[seniority-and-the-junior-squeeze]], [[code-as-throwaway]], [[shedding-weight]], [[async-by-default]], [[explosion-of-internal-software]], [[context-as-scarce-resource]]
- Entities: [[sebastian]], [[virtido]], [[eugene]], [[thorsten-ball]]
- Entities: [[sebastian]], [[virtido]], [[eugene]], [[thorsten-ball]], [[nikolai-sheiko]]
- Tools: [[claude-code]], [[amp]]
## Contradictions / Uncertainty

View File

@@ -20,6 +20,7 @@ Supporting practices at the plateau: keep CLAUDE.md self-maintaining ("always ke
- Webinar title and non-programmer audience confirm the ladder as the webinar's spine — [[2026-07-14-nina-interview]].
- Convergent structure (foundation docs + skills as the non-engineer's OS) — [[2026-07-14-gap-between-ai-users-irreversible]].
- A high-rung user missing the skills rung, and the memory pain that results — [[2026-07-21-larysa-interview]].
- **Measured, team-level corroboration that the gap grows between rungs of mastery**, not between license-holders and others: Stanford's 46-vs-46-team analysis shows the productivity gap between AI-mastering and lagging teams growing 4.8% → 19% (4×) over ~2.25 years — [[2026-07-30-stanford-swepr-widening-gap]]. *(Caveat: measures engineering teams, not this ladder's non-programmer audience; and the study asserts "quality of usage" as the differentiator without decomposing which rung supplies it.)*
## Related Pages

View File

@@ -23,10 +23,11 @@
- All claims, ratios, tier table, slop patterns, Dax/Shao citations — [[2026-07-24-youre-reading-way-too-much-code]].
- Groundwork (code disposable, kill without guilt, G-brain markdown tier) — [[2026-07-14-everything-we-knew-about-software-has-changed]].
- 15 icon variants, Midjourney rounds, "ask the agent for proof" — [[2026-07-28-agentic-engineering-10x-developer]].
- **External measurement of the cost moving downstream:** Stanford SWEPR coverage reports +91% PR review time and ~2.6× rework in AI-heavy workflows — writing got cheaper, reviewing got dearer, which is this page's premise measured rather than asserted — [[2026-07-30-stanford-swepr-widening-gap]]. *(Secondary-coverage numbers; and note the same study's negative gains in complex brownfield code — "code is cheap" holds least where most code lives.)*
## Related Pages
- Concepts: [[code-as-throwaway]] (parent claim: cost → zero; this page is its *discipline* — what cheap code is actually for), [[think-wider-not-bigger]] (same breadth logic applied to generation volume rather than ambition), [[product-ownership]] (verifying as the human's remaining job), [[solve-first-then-skillify]] (contrast: slop is frozen into nothing; skills freeze the procedure), [[leave-less-room-for-imagination]] (tension — see below), [[context-as-scarce-resource]], [[async-by-default]] (proof artifacts as the delegated form of the same move)
- Concepts: [[code-as-throwaway]] (parent claim: cost → zero; this page is its *discipline* — what cheap code is actually for), [[think-wider-not-bigger]] (same breadth logic applied to generation volume rather than ambition), [[product-ownership]] (verifying as the human's remaining job), [[solve-first-then-skillify]] (contrast: slop is frozen into nothing; skills freeze the procedure), [[leave-less-room-for-imagination]] (tension — see below), [[context-as-scarce-resource]], [[async-by-default]] (proof artifacts as the delegated form of the same move), [[review-is-the-new-bottleneck]] (the org-level form of the cost shift this page manages individually — with the completed-without-rework metric as its answer)
- Entities: [[theo-browne]], [[eugene]], [[thorsten-ball]]
## Contradictions / Uncertainty

View File

@@ -0,0 +1,38 @@
# Review Is the New Bottleneck
#concept
## Summary
When agents write the code, the SDLC doesn't collapse completely — it collapses *around the humans*. [[nikolai-sheiko]]'s two remaining "red squares" are the **reviewer** and the **planner**: tasks pile up in the review queue, reviewers burn out, quality drops, and headline output metrics (PRs, LoC) rise while real throughput barely moves. The organizational fix is twofold: **review with the agent** (not fully manual, not fully delegated) and **measure completed tasks without rework** rather than anything volume-based.
## Current Understanding
- **The mechanism.** Generation got ~free, so the cost moved downstream to verification — and at team scale, downstream is a *person* with a queue. Sheiko's European-outsourcer case: more PRs than ever, net gain +1%, because rework consumed the difference. Manual-only review starts a spiral (queue → burnout → rubber-stamping → more rework); fully delegated review is the opposite error ([[async-by-default]]'s "proof produced by the thing being checked").
- **The middle path: review together with the agent.** Treat the model as a smart student — direct it, pose hypotheses, locate problems jointly. This is the org-level sibling of Theo's reading economics ([[make-more-cheap-code]]: AI reviews before humans, per-file summaries, read only what's worth reading).
- **The metric that resists gaming:** a task counts as done only if it **doesn't come back for rework**; track task lifetime and rework time. LoC, commit count and PR count are all trivially hacked and all rise *because* of the bottleneck, not despite it.
- **Externally measured:** Stanford SWEPR coverage reports **+91% PR review time** and ~2.6× rework in AI-heavy workflows, and finds gross code volume up 3040% while net gains are ~1520% — the same rework-eats-half story Sheiko tells anecdotally ([[2026-07-30-stanford-swepr-widening-gap]]).
- **Planning is the other red square.** Time redistributes from coding to "planning on the left + verification on the right" — which is why Sheiko prescribes 20-minutes-minimum planning and why a **Product engineer** role emerges ([[developer-as-agent-manager]]).
## Evidence
- Reviewer/planner as the remaining red squares; review-with-the-agent; the metrics table; the +1% outsourcer case — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
- +91% PR review time, 2.6× rework, gross-vs-net gap — [[2026-07-30-stanford-swepr-widening-gap]] *(secondary-coverage numbers)*.
- Reading as the scarce human resource; AI-review-before-human-review; tiered reading discipline — [[2026-07-24-youre-reading-way-too-much-code]] via [[make-more-cheap-code]].
- Parallel agents multiplying diffs without multiplying review capacity — the attention pile-up already logged on [[async-by-default]]; this page names that open question as the bottleneck it becomes at team scale.
## Related Pages
- Concepts: [[make-more-cheap-code]] (individual-level discipline for the same cost shift), [[async-by-default]] (the generation side that feeds the queue), [[developer-as-agent-manager]] (the role shift on the human side of the queue), [[seniority-and-the-junior-squeeze]] ("read what you approve" — why review can't just be dropped), [[maintenance-is-the-real-cost]] (kindred move: the visible activity was never the expensive part)
- Entities: [[nikolai-sheiko]], [[swepr]], [[theo-browne]]
## Contradictions / Uncertainty
- **How much review survives?** Theo/Dax hold that agent diff-summaries surface anomalies; [[eugene]] holds that drift is precisely what summaries miss ([[leave-less-room-for-imagination]]); Sheiko's review-with-the-agent is a third position between them — asserted, not tested. Status: tentative.
- The completed-without-rework metric is better than LoC/PRs but still gameable (e.g. by inflating task granularity or quietly reclassifying rework as new tasks); the source doesn't address it. Status: tentative.
- Whether review-as-bottleneck is transitional (until verification is agentized) or structural (a human must always sign off — the [[seniority-and-the-junior-squeeze|accountability]] view) is open across the corpus.
## Next Questions
- What does review-with-the-agent look like concretely — a checklist, a dialogue pattern, a skill? The corpus has the prescription but no transcript of it done well.
- Is there a non-engineer analog (the webinar audience reviews documents, not PRs)? The checker-skill design in [[2026-07-24-non-engineer-throwaway-verification]] may be it.

View File

@@ -27,6 +27,7 @@ The **method** for populating skills is [[solve-first-then-skillify]]: reach the
- ~3-message correction-loop heuristic; skills as the non-programmer ceiling (with CLAUDE.md) — [[2026-07-14-yulia-interview]].
- Cross-session memory loss as the #1 practitioner pain; "memory is the worst thing agents have"; skills committed as the webinar remedy — [[2026-07-21-larysa-interview]].
- Counter-evidence: no skills, no MCP, no slash commands at a 99%-AI-written company; `AGENTS.md` + codebase + rich prompt as the substitute — [[2026-07-28-agentic-engineering-10x-developer]].
- **A second practitioner vote *for* the skills layer** (2026-07-30): [[nikolai-sheiko]]'s "Agentic Evolution" makes skill-building-plus-verification the difference between living on defaults and "vertical growth," and he flatly rejects embeddings/RAG over code ("don't use them unless you understand *very* well why") — siding with the load-on-activation camp in the skills-vs-RAG contradiction below. His verification protocol (context-free subagent re-solves the task from the skill alone) is the closest thing yet to the falsification test this page asks for — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
## Related Pages
@@ -43,10 +44,10 @@ The **method** for populating skills is [[solve-first-then-skillify]]: reach the
1. **Situational.** He works daily in *one codebase he controls*, where context can live in the code and `AGENTS.md`. Skills earn their keep when work is spread across many ad-hoc tasks with no codebase to encode into — which is exactly the corpus's HR/BA audience ([[nina]], [[yulia]], [[larysa]]). Under this reading both are right and the disagreement is about who is speaking.
2. **The abstraction is premature.** Skills are scaffolding for models that needed it; a strong model plus a rich prompt plus a good repo may simply beat a skills library, making the whole layer a 2025 artifact. This is the uncomfortable reading for the webinar's central promise.
3. **He has skills under another name.** AMP's Oracle/Painter/Puck sub-agents and a maintained `AGENTS.md` *are* curated, reusable, two-stage context — just authored by the vendor and the team rather than the user. Under this reading the dispute is about who curates, not whether curation is needed.
Status: tentative. Note the evidential asymmetry — his is a first-hand report of daily practice at scale, where the pro-skills case rests on Konstantin's architecture argument plus self-reported individual workflows. **Presentation-safe restatement:** [[2026-07-28-webinar-theses]] reframes the claim as *context you author beats context that's inferred*, which holds under all three readings — Konstantin's skills, Allie's foundation docs, Eugene's anti-memory position and Thorsten's `AGENTS.md` are all authored context.
Status: tentative. Note the evidential asymmetry — his is a first-hand report of daily practice at scale, where the pro-skills case rests on Konstantin's architecture argument plus self-reported individual workflows. *(Asymmetry softened 2026-07-30: [[nikolai-sheiko]] adds a second practitioner voice on the pro-skills side, from multi-company adoption work rather than one codebase — though his cases are anonymous anecdotes, so the readings above remain unsettled.)* **Presentation-safe restatement:** [[2026-07-28-webinar-theses]] reframes the claim as *context you author beats context that's inferred*, which holds under all three readings — Konstantin's skills, Allie's foundation docs, Eugene's anti-memory position and Thorsten's `AGENTS.md` are all authored context.
## Next Questions
- ~~What's a starter skill set for a non-engineer?~~ Answered in [[2026-07-14-best-first-skill-for-beginner]] (skill-creator as meta-skill; tone-of-voice + anti-AI-language as first content skill).
- Do skills actually solve *cross-project* context, or only per-procedure recall? Larysa's complaint may be the former, which skills don't obviously address.
- Is there a test that would separate reading 1 from reading 2 above? The cheapest one available: give a non-engineer the same task with and without a skill and compare drift — the corpus has never run it, and the webinar's promise rests on the answer.
- Is there a test that would separate reading 1 from reading 2 above? The cheapest one available: give a non-engineer the same task with and without a skill and compare drift — the corpus has never run it, and the webinar's promise rests on the answer. *(Adjacent evidence 2026-07-30: Sheiko's context-free-subagent protocol — [[solve-first-then-skillify]] — runs the with-skill half in practice, but never the without-skill control, so the question stands.)*

View File

@@ -15,17 +15,28 @@ The recurring beginner mistake is writing the skill first and then trying to "sh
The payoff goes beyond reuse: a packaged skill is a **handoff and de-risking asset** — "a person with not even a third of your HR experience can deliver a decent result," which cuts onboarding and lets the expert take a vacation. This is how [[skills-as-memory]] gets *populated* in practice — the method side of that architecture, and the fix for "don't teach the AI abstractly."
**Agentic Evolution — the strongest formulation, plus the missing verification step** (added 2026-07-30). [[nikolai-sheiko]] frames the same method as onboarding an employee: asking the expert "how do you do this?" yields theory; instead **take the new employee (the agent) by the hand through hard real tasks, show it the rakes, then say: "remember all of this and write the manual for the next one."** Without this you live on defaults; with it "vertical growth begins." He then adds what the corpus's earlier heuristics lacked — a **verification protocol** for the frozen skill:
1. Write the skill together with the agent.
2. Don't go to lunch.
3. Launch a **subagent with no context** — it must solve the same task from scratch using only the skill.
4. The main agent watches what fails and fixes the skill.
5. The mentor agent thus onboards the next agent.
This is the first source to describe actually *running* something close to the skills falsification test proposed on [[skills-as-memory]] (same task, with-skill vs from-scratch) — though it tests the skill's completeness for one task, not whether the skill beats no-skill. His do-tomorrow extension: a skill that analyses your own sessions daily, automated via schedules/routines — evolution as a standing loop rather than a one-time freeze.
## Evidence
- "You first solve a task with Claude; the moment you reach the final solution, you say — now create a skill from this"; ~3-message heuristic — [[2026-07-14-yulia-interview]].
- Do-the-task-then-freeze framing; skill-as-handoff to a junior hire; vacation/de-risking angle — [[2026-07-14-nina-interview]].
- >5-tool-calls auto-creation heuristic and curator pruning — [[2026-07-14-skills-based-on-git]].
- Skills prescribed specifically as the workaround for cross-session memory loss, and as the constraint on drift — [[2026-07-21-larysa-interview]].
- Agentic Evolution (walk the agent through tasks → have it write the manual); the context-free-subagent verification protocol; session-analysis skill as a daily loop — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
## Related Pages
- Concepts: [[skills-as-memory]] (the architecture this method feeds), [[levels-of-ai-usage]] (skills are the top practical rung), [[personal-ai-operating-system]], [[leave-less-room-for-imagination]] (why a *proven* spec beats a written-ahead one)
- Entities: [[eugene]], [[nina]], [[konstantin]], [[larysa]]
- Entities: [[eugene]], [[nina]], [[konstantin]], [[larysa]], [[nikolai-sheiko]]
## Contradictions / Uncertainty
@@ -33,5 +44,6 @@ The payoff goes beyond reuse: a packaged skill is a **handoff and de-risking ass
## Next Questions
- What does a good "create a skill from this" prompt look like — does the corpus contain a concrete example transcript?
- What does a good "create a skill from this" prompt look like — does the corpus contain a concrete example transcript? *(Partially answered 2026-07-30: Sheiko's "remember all of this and write the manual for the next one" after a guided run is the best prompt-shape the corpus has.)*
- How do the ~3-message and >5-tool-call heuristics compare in practice; is one strictly better for non-programmers?
- Does the context-free-subagent check catch skill *quality*, or only completeness for the one task it was frozen from?

View File

@@ -0,0 +1,36 @@
# Nikolai Sheiko
#entity
## Summary
Speaker of the talk "Грабли во внедрении ИИ в SDLC" ([[2026-07-30-rakes-in-ai-sdlc-adoption]]). Russian-speaking AI-adoption practitioner/consultant who works with client companies (a frontend-migration team, a European outsourcer, a large-codebase company) on getting real results from AI in the software lifecycle. Background beyond the talk unknown.
## Current Understanding
His through-line: the models are already good enough — **people, companies and metrics are what throttle the gains**. Signature positions:
- The AI-developer is a **manager of an agent-employee**, IO-bound and parallel ([[developer-as-agent-manager]]).
- **Review is the new bottleneck**; review *with* the agent; measure completed-tasks-without-rework, never LoC/PRs ([[review-is-the-new-bottleneck]]).
- **Agentic Evolution**: walk the agent through hard tasks, then have it write the manual; verify skills with a context-free subagent ([[solve-first-then-skillify]]).
- Companies should install and configure Claude Code / Codex rather than build custom AI tooling; buy a **teacher/curator**, not an external configurator ([[enterprise-ai-reality]]).
- Best practices matter *more* with agents (compaction curse, AST search over grep); embeddings/RAG over code don't work ([[context-as-scarce-resource]]).
- AI eats **Intelligence**; **Judgment** (taste or domain expertise) stays human for now.
## Evidence
- All positions — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
## Related Pages
- Concepts: [[developer-as-agent-manager]], [[review-is-the-new-bottleneck]], [[solve-first-then-skillify]], [[enterprise-ai-reality]], [[context-as-scarce-resource]]
- Entities: [[swepr]] (he cites their Stanford chart as the stakes — "be in the top half"), [[thorsten-ball]] (fellow frontier practitioner; they disagree on skills — Sheiko builds skill evolution, Thorsten uses none)
## Contradictions / Uncertainty
- His client cases are anonymous and self-reported; no numbers are verifiable. Status: tentative.
- Identity/affiliation beyond the talk unknown (the raw doc names only the talk itself). Status: tentative.
## Next Questions
- Who is he professionally — independent consultant, agency, vendor? Affects how to weigh the "you don't need custom AI development" claim (it is also a consultant's pitch).

35
wiki/entities/swepr.md Normal file
View File

@@ -0,0 +1,35 @@
# SWEPR (Stanford Software Engineering Productivity Research)
#entity
## Summary
Research group at Stanford University measuring software-engineering productivity from private Git data — 600+ companies, ~100k120k engineers since 2022 — using an ML model that replicates a panel of expert code reviewers. Public face: researcher **Yegor Denisov-Blanch**. Source of the corpus's only quantitative outside study of AI's productivity impact.
## Current Understanding
- Site: https://softwareengineeringproductivity.stanford.edu/ — offers an "AI Practices Benchmark" and "AI Impact" research to participating companies.
- Known for three results: the **widening-gap DiD analysis** (46 vs 46 teams, gap 4.8% → 19%, 4×, Apr 2023Jul 2025), the **~1520% net average gain** figure (after rework; 3040% gross), and the earlier **"ghost engineers"** finding (~9.5% of engineers show virtually no verifiable output).
- Methodology is peer-reviewed (arXiv 2409.15152, 2502.20747); the headline AI-impact analyses are talk/deck-published (Sept 2025 AI Conference deck "Will AI Replace Software Engineers?").
- Their measurement philosophy — functionality delivered, not commits or LOC — aligns with the corpus's own suspicion of volume metrics ([[make-more-cheap-code]]: generation volume is the *cheap* part).
## Evidence
- All claims and links — [[2026-07-30-stanford-swepr-widening-gap]].
## Related Pages
- Sources: [[2026-07-30-stanford-swepr-widening-gap]]
- Concepts: [[levels-of-ai-usage]] (team-level twin of the mastery gap), [[context-as-scarce-resource]] (codebase-size finding), [[make-more-cheap-code]] (review-time shift)
- Queries: [[2026-07-30-stanford-widening-gap-source]]
- Entities: [[allie-miller]] (her prediction, their measurement)
## Contradictions / Uncertainty
- Data is proprietary and opt-in — no outside replication possible; participating companies may skew toward the measurement-friendly. Status: tentative.
- The widening-gap analysis itself is not peer-reviewed as of 2026-07-30. Status: tentative.
## Next Questions
- Track whether the DiD analysis lands in a peer-reviewed venue.
- What do they say *causes* the gap (tooling vs hygiene vs practices)? The talks assert "quality of usage" without decomposing it.

View File

@@ -10,7 +10,7 @@ A high-signal personal knowledge base. `raw/` holds immutable source materials;
## The through-line
Across eleven sources — five talks/videos/interviews from practitioners (two of them Theo's), five interviews conducted for this project, and a business-facing short — one spine recurs:
Across thirteen sources — six talks/videos/interviews from practitioners (two of them Theo's), five interviews conducted for this project, a business-facing short, and one quantitative outside study ([[swepr|Stanford SWEPR]]) — one spine recurs:
> **As the cost of writing code goes to zero, value migrates from *producing* software to *directing and verifying* it — and the durable human assets become judgment, ownership, taste, and in-person relationships.**
@@ -20,23 +20,24 @@ Everything else hangs off that:
- **The human side** — what stays yours: [[product-ownership]] over outcomes, [[connections-as-moat|in-person connections]] as the last non-commoditized asset, [[seniority-and-the-junior-squeeze|judgment as risk-reduction]], and the need to [[decoupling-identity-from-profession|decouple identity from profession]].
- **The strategy side** — where to point it: [[think-wider-not-bigger|think wider not bigger]], treat [[code-as-throwaway|code as throwaway]], and mind [[enterprise-ai-reality|enterprise compliance reality]] (the company-managed-harness market). Theo's second video supplies the *verifying* half of the spine its method: [[make-more-cheap-code]] — keep hand-verification of what ships, and generate orders of magnitude more never-shipped code to verify and explore.
- **The frontier side** — what it looks like at the far end, from [[thorsten-ball]] at [[amp]] (99% of their code AI-written): [[shedding-weight|shed weight]] by deleting every process that only existed because humans were the bottleneck; [[build-for-the-agent-not-the-human|build for the agent, not the human]]; work [[async-by-default|async by default]] in remote sandboxes and ask for proof rather than claims. His two mechanisms for software becoming *personal* — [[emacsification-of-software|remixing what exists]] and [[explosion-of-internal-software|building what never did]] — are the corpus's strongest outside validation of the webinar's own thesis, "little tools you make for yourself." He is also its sharpest dissenter: he uses **no skills, no MCP, no slash commands**. Both mechanisms now carry a sourced counterweight — [[maintenance-is-the-real-cost]]: writing code was never the bottleneck, maintenance is, and an internal service is a second business. The reconciliation is a threshold, not a winner: tiny personal tools pass, replacing your Jira does not.
- **The adoption side** — what goes wrong when organizations try this, from [[nikolai-sheiko]]'s multi-company casework ([[2026-07-30-rakes-in-ai-sdlc-adoption]]): the SDLC collapses *around the humans* — [[review-is-the-new-bottleneck|review becomes the bottleneck]] and volume metrics (LoC, PRs) go anti-informative, so measure **completed tasks without rework**; the developer's job flips from CPU-bound coding to [[developer-as-agent-manager|IO-bound agent management]]; and the winning company move is not custom AI development but installing and *evolving* a standard harness ([[enterprise-ai-reality]]) — with skills grown by walking the agent through real tasks and verified by a context-free subagent ([[solve-first-then-skillify]]).
- **The demand side** — three interviews ground it all in a real audience. The two HR ones ([[2026-07-14-nina-interview|Nina]], [[2026-07-14-yulia-interview|Yulia]]) supply pain points (interview write-ups, job descriptions, sourcing) that collapse into "a candidate knowledge base plus search," teachable via [[levels-of-ai-usage]] and [[solve-first-then-skillify]]. Their key finding: **adoption is blocked by friction, not resistance.** The [[2026-07-21-larysa-interview|Larysa interview]] adds the *advanced* user's version of the same story: past the friction, the remaining walls are structural — no durable memory, [[integration-dead-ends|integrations that dead-end]], and drift on loose specs ([[leave-less-room-for-imagination]]). Her diagnosis matters because she is technically deep yet skipped the skills rung, which is exactly what her "the agent forgot" complaint reduces to.
See [[ai-agent-evolution]] for how the capability curve got here.
## Where sources agree vs diverge
- **Agree:** code is cheap/disposable; harnesses are the unit of work; skills-as-memory (Konstantin ↔ Allie ↔ Eugene); human relationships rise in value (Sebastian ↔ Allie ↔ Eugene, who lands there independently in the Yulia interview); solve-first-then-skillify (Eugene ↔ Konstantin's heuristics); context is the constraint — Thorsten's version is the bluntest: **the dominant variable in output quality is the information you put in**, not the model or the effort level. The [[2026-07-22-ai-is-stupid|"AI is stupid!" short]] independently compresses the machine-side spine into a business one-liner: **model + context + harness = employee-level answer**. Slop is a human problem, not an AI defect (Theo ↔ Thorsten, from verification discipline and from taste respectively). Software becomes personal — "little tools you make for yourself" (Eugene's webinar arc ↔ Thorsten's club app and bespoke forks ↔ Allie's personal OS).
- **Diverge:** personal vs company-managed vs vendor-managed harness ([[eugene]] vs [[sebastian]] vs [[amp]]); online vs in-person networking (Eugene/Sebastian); OSS as marketing vs OSS growth; built-in agent memory as anti-feature (Eugene) vs persistent context docs used without complaint (Allie); tight specs ([[leave-less-room-for-imagination]]) vs wide latitude ([[think-wider-not-bigger]]); agent diff-summaries as sufficient review (Theo/Dax) vs invisible drift as the core danger (Eugene); model choice as a real lever (Eugene runs 4.7 over 4.8) vs a distraction past the frontier (Thorsten); local consolidated workspace (Eugene) vs local dev disappearing into remote sandboxes (Thorsten); build-your-own-tools ([[thorsten-ball]], the webinar arc) vs [[maintenance-is-the-real-cost|buy anything that needs ongoing support]] (the vibe-coding video, with the corpus's only observed reversal: an in-house Jira clone abandoned for Linear in four months). These live under "Contradictions" on the relevant pages.
- **The one that matters most for the webinar:** [[thorsten-ball]] runs a 99%-AI-written codebase with **no skills, no MCP servers and no slash commands** — his context lives in the codebase and `AGENTS.md`. That is the corpus's first credible rejection of the mechanism the webinar's central promise rests on. Three readings (situational / premature abstraction / same thing under another name) are logged on [[skills-as-memory]]; none is settled, and the evidential asymmetry favours him — his is first-hand daily practice at scale.
- **Agree:** code is cheap/disposable; harnesses are the unit of work; skills-as-memory (Konstantin ↔ Allie ↔ Eugene); human relationships rise in value (Sebastian ↔ Allie ↔ Eugene, who lands there independently in the Yulia interview); solve-first-then-skillify (Eugene ↔ Konstantin's heuristics); context is the constraint — Thorsten's version is the bluntest: **the dominant variable in output quality is the information you put in**, not the model or the effort level. The [[2026-07-22-ai-is-stupid|"AI is stupid!" short]] independently compresses the machine-side spine into a business one-liner: **model + context + harness = employee-level answer**. Slop is a human problem, not an AI defect (Theo ↔ Thorsten, from verification discipline and from taste respectively). Software becomes personal — "little tools you make for yourself" (Eugene's webinar arc ↔ Thorsten's club app and bespoke forks ↔ Allie's personal OS). And the corpus's central *stakes* claim now has outside measurement: [[2026-07-30-stanford-swepr-widening-gap|Stanford SWEPR]] finds the productivity gap between AI-mastering and lagging teams grew 4.8% → 19% (4×) from April 2023 to July 2025 — Allie's prediction, measured; the same study's codebase-size finding independently supports [[context-as-scarce-resource|context as the binding constraint]]. [[2026-07-30-rakes-in-ai-sdlc-adoption|Sheiko]] cites that same Stanford chart as his stakes slide and lands on the spine independently — "companies no longer need custom AI development, install Claude Code or Codex and configure it" is harness-over-model as a service playbook, and his codebase-stores-context prescription converges with Thorsten's from the opposite direction. His review-bottleneck casework (+1% net despite more PRs) is SWEPR's +91%-review-time finding told anecdotally.
- **Diverge:** personal vs company-managed vs vendor-managed harness ([[eugene]] vs [[sebastian]] vs [[amp]]); online vs in-person networking (Eugene/Sebastian); OSS as marketing vs OSS growth; built-in agent memory as anti-feature (Eugene) vs persistent context docs used without complaint (Allie); tight specs ([[leave-less-room-for-imagination]]) vs wide latitude ([[think-wider-not-bigger]]); agent diff-summaries as sufficient review (Theo/Dax) vs invisible drift as the core danger (Eugene); model choice as a real lever (Eugene runs 4.7 over 4.8) vs a distraction past the frontier (Thorsten); local consolidated workspace (Eugene) vs local dev disappearing into remote sandboxes (Thorsten); build-your-own-tools ([[thorsten-ball]], the webinar arc) vs [[maintenance-is-the-real-cost|buy anything that needs ongoing support]] (the vibe-coding video, with the corpus's only observed reversal: an in-house Jira clone abandoned for Linear in four months); permit opting out of the agent-manager switch ([[nikolai-sheiko]] — "don't force everyone") vs the gap is irreversible and compounding ([[allie-miller]], [[swepr|Stanford]]) — see [[developer-as-agent-manager]]. These live under "Contradictions" on the relevant pages.
- **The one that matters most for the webinar:** [[thorsten-ball]] runs a 99%-AI-written codebase with **no skills, no MCP servers and no slash commands** — his context lives in the codebase and `AGENTS.md`. That is the corpus's first credible rejection of the mechanism the webinar's central promise rests on. Three readings (situational / premature abstraction / same thing under another name) are logged on [[skills-as-memory]]; none is settled. The evidential asymmetry that favoured him narrowed on 2026-07-30: [[nikolai-sheiko]] is a second practitioner voice on the pro-skills side — his "Agentic Evolution" (guided tasks → agent writes the manual → context-free-subagent verification) is the corpus's first described *test* of a skill, though his cases are anonymous anecdotes where Thorsten's is first-hand daily practice at scale.
## Navigation
- **[[index]]** — content catalog
- **Sources (11):** [[2026-07-14-everything-we-knew-about-software-has-changed|Theo Browne]] · [[2026-07-14-gap-between-ai-users-irreversible|Allie Miller]] · [[2026-07-14-sebastian-eugene-interview|Sebastian interview]] · [[2026-07-14-skills-based-on-git|Konstantin (git skills)]] · [[2026-07-14-nina-interview|Nina interview]] · [[2026-07-14-yulia-interview|Yulia interview]] · [[2026-07-21-larysa-interview|Larysa interview]] · [[2026-07-22-ai-is-stupid|"AI is stupid!" short]] · [[2026-07-24-youre-reading-way-too-much-code|Theo Browne (reading code)]] · [[2026-07-28-agentic-engineering-10x-developer|Thorsten Ball (agentic engineering)]] · [[2026-07-29-what-if-we-vibe-code-it|"What if we vibe-code it?" (maintenance trap)]]
- **People:** [[theo-browne]] · [[allie-miller]] · [[sebastian]] · [[eugene]] · [[konstantin]] · [[nina]] · [[yulia]] · [[larysa]] · [[thorsten-ball]]
- **Tools/orgs:** [[claude-code]] · [[amp]] · [[hermes]] · [[virtido]] · [[inspectron]]
- **Concepts:** see the through-line above (25 pages) · **Timeline:** [[ai-agent-evolution]] · **Comparison:** [[theo-konstantin-allie]]
- **Sources (13):** [[2026-07-14-everything-we-knew-about-software-has-changed|Theo Browne]] · [[2026-07-14-gap-between-ai-users-irreversible|Allie Miller]] · [[2026-07-14-sebastian-eugene-interview|Sebastian interview]] · [[2026-07-14-skills-based-on-git|Konstantin (git skills)]] · [[2026-07-14-nina-interview|Nina interview]] · [[2026-07-14-yulia-interview|Yulia interview]] · [[2026-07-21-larysa-interview|Larysa interview]] · [[2026-07-22-ai-is-stupid|"AI is stupid!" short]] · [[2026-07-24-youre-reading-way-too-much-code|Theo Browne (reading code)]] · [[2026-07-28-agentic-engineering-10x-developer|Thorsten Ball (agentic engineering)]] · [[2026-07-29-what-if-we-vibe-code-it|"What if we vibe-code it?" (maintenance trap)]] · [[2026-07-30-stanford-swepr-widening-gap|Stanford SWEPR (widening gap)]] · [[2026-07-30-rakes-in-ai-sdlc-adoption|Nikolai Sheiko (rakes in SDLC adoption)]]
- **People:** [[theo-browne]] · [[allie-miller]] · [[sebastian]] · [[eugene]] · [[konstantin]] · [[nina]] · [[yulia]] · [[larysa]] · [[thorsten-ball]] · [[nikolai-sheiko]]
- **Tools/orgs:** [[claude-code]] · [[amp]] · [[hermes]] · [[virtido]] · [[inspectron]] · [[swepr]]
- **Concepts:** see the through-line above (27 pages) · **Timeline:** [[ai-agent-evolution]] · **Comparison:** [[theo-konstantin-allie]]
## Open Questions (vault-level)

View File

@@ -61,7 +61,8 @@ The audience's real objection is "can I trust it?", and the current script has n
**T7. The gap between AI users and everyone else compounds — and is becoming irreversible.**
The person who builds their OS this week fears no release, because each capability slots into a system that already knows them. *(Thorsten names **token budget** as a second winner/loser variable, but that is a claim about metered agent-fleet work; for this audience the budget is one consumer subscription — keep the thesis on the skill gap. See [[enterprise-ai-reality]].)*
— [[2026-07-14-gap-between-ai-users-irreversible]]
*(Measured 2026-07-30: this thesis is no longer prediction-only — Stanford SWEPR's 46-vs-46-team analysis shows the gap growing 4.8% → 19% (4×) from April 2023 to July 2025. One citable stage line: "Stanford measured it: the gap quadrupled in two years." See [[2026-07-30-stanford-swepr-widening-gap]]. **Added to the script's closing arc 2026-07-30** — T7 is now dramatized, upgrading it from Q&A material to an on-stage beat.)*
— [[2026-07-14-gap-between-ai-users-irreversible]] · [[2026-07-30-stanford-swepr-widening-gap]]
**T8. The more the world is mediated by AI proxies, the more valuable real human connection becomes.**
The "market of one" raises, not lowers, the price of being human.

View File

@@ -0,0 +1,55 @@
# Stanford "widening gap" chart — original research located
#query
## Question asked
The user saved a chart screenshot (`raw/assets/G6g3O60bkAE05ZW.png`, filename pattern = X/Twitter image) of a Stanford slide titled *"Teams that master AI are accelerating their productivity gains, widening the gap with laggards"* and asked: **find the original Stanford research behind it.**
## Answer
The slide is from **Stanford's Software Engineering Productivity Research group (SWEPR)**, led by researcher **Yegor Denisov-Blanch**. Since 2022 the group has analyzed private Git repositories from 600+ companies and 120,000+ engineers, scoring every commit with an ML model trained to replicate a panel of human expert reviewers.
The specific chart is their **causal difference-in-differences analysis**: 46 teams that adopted AI, matched against 46 similar non-AI teams, with net productivity gains measured quarterly.
- **April 2023:** 4.8% gap between top-quartile and bottom-quartile AI adopters.
- **July 2025:** 19% gap — a **4× increase** in ~2.25 years.
- Slide footer: Stanford University / SWEPR; the specific slide (page 3) matches Denisov-Blanch's September 2025 AI Conference deck.
**Primary sources:**
- Research group home: <https://softwareengineeringproductivity.stanford.edu/> (AI Impact research, AI Practices Benchmark)
- Talk the slide comes from: *"Will AI Replace Software Engineers?"*, AI Conference, Sept 2025 — [slide deck PDF](https://aiconference.com/wp-content/uploads/2025/09/Yegor-Denisov-Blanch-Will-AI-Replace-Software-Engineers_-.pptx.pdf)
- Video walkthrough: [Can you prove AI ROI in Software Eng? (Stanford 120k Devs Study)](https://www.youtube.com/watch?v=JvosMkuNxF8)
- Researcher site: <https://yegordb.com/>
- Peer-reviewed methodology paper: [Predicting Expert Evaluations in Software Code Reviews](https://arxiv.org/pdf/2409.15152)
**Caveat (Status: tentative):** the 46-vs-46 difference-in-differences result itself has been presented via talks, webinars and decks — not (yet) a peer-reviewed paper. The peer-reviewed publications cover the *measurement methodology*, not this specific analysis.
**Surrounding findings from the same study** (useful nuance): AI raises developer productivity ~1520% on average, with high variance — largest gains on greenfield/simple tasks in popular languages; AI can *decrease* net productivity in complex legacy codebases (rework eats the gains, ~2.6× increase in rework reported).
## Why this matters to the vault
This is the **first quantitative, external, longitudinal measurement** of a claim the corpus so far held only as practitioner assertion:
- [[2026-07-14-gap-between-ai-users-irreversible]] — [[allie-miller]]'s central prediction ("in 12 months the gap will be irreversible") is the same shape as this curve, asserted from advisory experience. Stanford now supplies measured team-level data pointing the same direction.
- [[2026-07-28-webinar-theses]] — the "stakes" thesis group (irreversible gap) gains a citable number: *4.8% → 19%, 4× in about two years*. A Stanford chart is far stronger webinar ammunition than "an ex-Amazon AI leader predicts…".
- The "AI can decrease productivity in complex legacy codebases" finding is honest-caveat material aligning with the vault's recorded tensions ([[maintenance-is-the-real-cost]], rework costs; [[make-more-cheap-code]]'s verification burden — cf. the study's 91% increase in PR review time).
- The mechanism Stanford implies (teams that *master* AI compound, laggards stall) is the team-level twin of [[levels-of-ai-usage]] — the gap grows between rungs, not between haves and have-nots of licenses.
## Evidence trail
- Screenshot: `raw/assets/G6g3O60bkAE05ZW.png` (raw asset; likely captured from an X/Twitter post sharing the talk)
- Web search + fetch of the SWEPR site and the AI Conference deck (2026-07-30); slide title, footer, chart annotations and page number all match the deck's era (data ends July 2025)
## Follow-up questions
- ~~Ingest-worthy?~~ **Done, same day:** the user authorized a new raw source file; the dossier lives at `raw/sources/Stanford SWEPR - AI and the widening productivity gap.md` and is ingested as [[2026-07-30-stanford-swepr-widening-gap]] — concept pages now cite it directly.
- Does the webinar want the number? One line — "Stanford measured it: the gap 4×'d in two years" — would upgrade the stakes beat from prediction to measurement.
- Watch for a peer-reviewed version of the difference-in-differences analysis; the claim's status upgrades from tentative when it lands.
## Whether this output changed existing pages
- [[2026-07-14-gap-between-ai-users-irreversible]] — added an external-corroboration pointer to this page under Connections.
- `index.md` (Queries section) and `log.md` updated.
- No concept pages changed — deliberately, since the underlying talk is not yet ingested as a source (citation policy: concept evidence should point at `wiki/sources/*`).

View File

@@ -37,6 +37,7 @@
- **Concepts:** [[personal-ai-operating-system]] · [[skills-as-memory]] · [[context-as-scarce-resource]] · [[connections-as-moat]] (human-relationship side effect)
- **Related sources:** [[2026-07-14-skills-based-on-git]] (skills-as-memory from the engineering side — strong overlap) · [[2026-07-14-everything-we-knew-about-software-has-changed]] (markdown-as-skill ≈ G-brain markdown tier)
- **Tools mentioned:** [[claude-code]], Claude Cowork, Codex, Flint
- **External corroboration:** [[2026-07-30-stanford-swepr-widening-gap]] — Stanford SWEPR's difference-in-differences study measured the gap between AI-mastering and lagging teams growing 4.8% → 19% (4×) from April 2023 to July 2025; the title claim's first measured, non-practitioner support. (Traced via [[2026-07-30-stanford-widening-gap-source]].)
## Open Questions

View File

@@ -0,0 +1,55 @@
# Грабли во внедрении ИИ в SDLC — Rakes in AI Adoption in the SDLC (Nikolai Sheiko)
#source
## Source Metadata
- **Date:** talk published 2026 (references events through Dec 2025); conclusions doc saved 2026-07-30
- **Raw path:** `raw/sources/Грабли во внедрении ИИ в SDLC.md`
- **Source type:** viewer's conclusions from a Russian-language YouTube talk (45:59) — https://www.youtube.com/watch?v=Nm3MsnngCJg — "Грабли во внедрении ИИ в SDLC — почему ИИ есть, а результата нет и как это лечить" ("why the AI is there but the results aren't, and how to treat it"). Not a transcript.
- **Speaker:** [[nikolai-sheiko]] — AI-adoption consultant/practitioner (works with client companies on SDLC adoption; background otherwise unknown)
- **Ingestion date:** 2026-07-30
## Core Claims
- **AI in development already delivers real gains, but people, companies and metrics throttle it by an order of magnitude.** The real jump started Dec 2025 (Opus 4.5 / GPT-5.2 + Claude Code / Codex); the SDLC collapsed into days/hours — but *not fully*: two human "red squares" remain — the **reviewer** (tasks queue at review) and the **planner/product person**. Developer time redistributed from "coding in the middle" to "planning on the left + verification on the right." See [[review-is-the-new-bottleneck]].
- **Universal error #0: a developer is not a manager.** A good developer is 35 hours of CPU-bound focus on one feature; a good AI-developer is an IO-bound **manager of an agent-employee**, running several tasks in parallel. "If you launched Claude Code and sit watching it work — you're a bad employee." Not everyone can make the psychological switch, *and that's fine* — don't force everyone. See [[developer-as-agent-manager]].
- **Measure completed tasks without rework** — never LoC, commit count or PR count (all trivially hacked; the European-outsourcing case shipped more PRs for a +1% gain because rework ate everything). A task counts only if it doesn't come back for fixes; also track task lifetime + rework time.
- **Review with the agent, not instead of it and not fully by hand.** Manual-only review → queue → burnout → quality collapse; fully delegated review is the opposite error. Treat the model as a smart student: direct it, pose hypotheses, find problems together.
- **Companies no longer need custom AI development.** Key quote: *"Come in, install Claude Code or Codex, configure everything, attach connectors, think about security — and it works better than any custom build."* Corollary anti-pattern: hiring an external configurator who leaves behind a "magic artifact" nobody owns — teams must configure their own tools; what a company should buy is a **teacher/curator**, not a setup.
- **Agentic Evolution** — the key concept. Don't ask the expert to explain how they work (you get theory); instead take the new employee (the agent) by the hand through hard tasks, show it the rakes, then say: *"remember all of this and write the manual for the next one."* Verify a skill by launching a **context-free subagent** that must solve the same task from scratch using only the skill; the mentor agent watches what fails and fixes the skill. Without evolution you live on defaults; with it, vertical growth begins.
- **Best practices matter more with agents, not less.** The "compaction curse": on a huge codebase the agent gathers context → window overflows → compaction → re-gathers → compaction again, and the task barely completes. The cure is locality, isolated modules with interfaces — **the codebase stores the context**. Use AST search instead of grep on colossal projects. And **embeddings/RAG over code do not work** — don't use them unless you understand *very* well why.
- **Role futures:** a **Product engineer** emerges (answers *why* we build it this way, what to cut, what to ignore); users vs **Agentic Operations** (who tune the SDLC, feedback loops, prompts, skills) gradually split; AI eats **Intelligence** (action sequences requiring intellect) while **Judgment** (taste built over years, or domain expertise — oil & gas, medicine) stays human for now.
- **Tokens get more expensive near-term, cheaper later. It's the wild west — experiment at full throttle while subscriptions are cheap**; the goal is to land in the top half of the Stanford chart. Don't chase every new tool: what Claude Code / Codex doesn't absorb within a couple of months is probably useless.
## Key Evidence / Details
- **Timeline the talk builds on:** METR study (Jul 2025, measured a *slowdown*, but methodologically contested) → Stanford study (Aug 2025, +20% speed but top-performers pull away — this is [[swepr]]'s research, cited independently) → Karpathy's tweet (NovDec 2025: "80% Claude Code, 20% by hand") → "SDLC is dead" article (stages collapse into days/hours).
- **Case 1 (frontend migration):** no feedback loop → give the agent a browser (Playwright / Chrome DevTools) to compare old vs new front; crunch → invest in planning (20 min minimum, hours are normal; every 10 min of planning saves hours; target one-shot implementation); Cursor with per-token billing → team economizes instead of experimenting (~30% dearer than subscriptions at the same level); training rollout: record sessions → expert reviews → *writes feedback, team fixes their own agent instructions* → focus on the top 2 performers (an hour with them is worth 10× more).
- **Case 2 (European outsourcer):** more PRs, +1% — rework was the cause; the metrics table (don't: LoC/commits/PRs; do: completed-tasks-without-rework).
- **Case 3 (large codebase):** the compaction curse; "agents mean we can drop best practices" is exactly wrong.
- **Mini-cases:** the middle dev who sped the team up by tens of %, was refused a raise, and left for much more ("if you're that middle — think; if you're the manager — think twice"); the startup doing spec-driven development without knowing what it wants — build the UI first (even with an in-memory browser DB), click around, *then* plan.
- **Do-tomorrow list:** close the feedback loop; write a skill that analyses your own sessions daily; automate it (Codex schedules / Anthropic routines); voice input (more context, and Russian gives more context than English); ignore tool churn.
- **Q&A notes:** GLM (good, no vision), Kimi (works, has vision), Xiaomi's agent (interesting memory implementation; ex-DeepSeek Head of AI) as the Chinese-model trend; hardware deficit (H100s unrentable); speaker's prediction that AI will be classed as a cyber-weapon with biotech-style licensing.
- **One-shot recipe:** feedback loop (must-have) + explicit real *goal* the agent self-checks against + skill verification via the context-free subagent.
## Connections
- **Entities:** [[nikolai-sheiko]], [[swepr]] (its Stanford study is the talk's central chart — "be in the top half"), [[claude-code]]
- **New concepts:** [[review-is-the-new-bottleneck]], [[developer-as-agent-manager]]
- **Corroborates:** [[2026-07-30-stanford-swepr-widening-gap]] (independent practitioner citation of the widening-gap result *and* of review-as-downstream-cost); [[solve-first-then-skillify]] (Agentic Evolution is its strongest formulation, plus the missing verification step); [[skills-as-memory]] (a second frontier-practitioner vote *for* the skills/evolution layer, against [[thorsten-ball]]'s dissent — and a vote *against* embeddings/RAG over code, siding with Konstantin in the skills-vs-RAG contradiction); [[context-as-scarce-resource]] (compaction curse; codebase-stores-context converges with Thorsten from the opposite direction); [[async-by-default]] (IO-bound parallel management as the working mode); [[enterprise-ai-reality]] ("no custom AI development needed" names the same managed-harness market; per-token billing shapes behaviour — the metered-vs-subscription split already logged there); [[harness]] (install-and-configure beats custom builds); [[leave-less-room-for-imagination]] (explicit goal + planning discipline).
- **Complicates:** [[make-more-cheap-code]] (rework-free-completion as the metric is the org-level answer to "generation moved the cost downstream").
## Open Questions
- The cases are anonymous client anecdotes with self-reported numbers (+1%, "tens of %"); none are verifiable. Status: tentative.
- "Embeddings/RAG over code don't work" is stated flatly with no mechanism given — strong claim, no evidence in the doc. Status: tentative.
- Does the context-free-subagent verification protocol actually measure skill quality, or only skill *completeness* for one task? Closely related to the corpus's proposed skills falsification test ([[skills-as-memory]]) — this is the first source to describe running one.
- The Dec-2025 "real jump" periodization is the speaker's own; the corpus's other timeline ([[ai-agent-evolution]]) slices eras differently.
- Predictions (AI as licensed cyber-weapon; token prices up then down) are speculation. Status: tentative.
## Change Impact on Wiki
- Created [[nikolai-sheiko]] (entity), [[review-is-the-new-bottleneck]] and [[developer-as-agent-manager]] (concepts).
- Updated [[solve-first-then-skillify]] (Agentic Evolution + skill-verification protocol), [[skills-as-memory]] (second practitioner vote for skills; anti-RAG-for-code), [[context-as-scarce-resource]] (compaction curse, AST search, codebase-stores-context), [[async-by-default]] (IO-bound manager evidence), [[enterprise-ai-reality]] (no-custom-AI-dev quote; external-configurator anti-pattern; token-price prediction), [[make-more-cheap-code]] (related link), [[2026-07-30-stanford-swepr-widening-gap]] (independent citation).
- Updated [[overview]] (12 → 13 sources), `index.md`, `log.md`.

View File

@@ -0,0 +1,48 @@
# Stanford SWEPR — AI and the widening productivity gap
#source
## Source Metadata
- **Date:** research presented through 2025 (chart data April 2023 → July 2025; deck Sept 2025); dossier compiled 2026-07-30
- **Raw path:** `raw/sources/Stanford SWEPR - AI and the widening productivity gap.md` (screenshot: `raw/assets/G6g3O60bkAE05ZW.png`)
- **Source type:** research dossier — a slide read first-hand plus public coverage of the underlying study; **not** a talk transcript. The corpus's first *quantitative outside study* (every other source is practitioner testimony or synthesis).
- **Authors:** [[swepr]] (Stanford Software Engineering Productivity Research group), public face Yegor Denisov-Blanch
- **Ingestion date:** 2026-07-30
## Core Claims
- **The gap between AI-mastering teams and laggards is widening, measurably.** Difference-in-differences analysis of 46 AI-adopting teams vs 46 matched non-AI teams: net productivity difference between top and bottom quartiles grew from **4.8% (April 2023) to 19% (July 2025) — a 4× increase** in ~2.25 years. Early quarters hover near zero or negative; the curve only takes off from mid-2024, then accelerates.
- **Average net gain from AI is ~1520%, not 10×.** Gross delivered code volume rises 3040%, but rework (fixing AI-introduced bugs) eats roughly half.
- **Gains are strongly context-dependent:** greenfield/low-complexity 3040%; brownfield/high-complexity 010% and can be *negative*. Popular languages gain more than niche ones; gains collapse as codebases grow 10k → 10M lines (context-window limits, signal-to-noise).
- **The cost moved downstream:** coverage reports +91% PR review time and ~2.6× rework in AI-heavy workflows — writing got cheaper, reviewing got more expensive.
- **Proposed mechanism for the gap:** quality of AI usage beats volume; teams with clean, modular, well-tested code compound gains, teams with poor hygiene accumulate debt and lose trust in the tools.
## Key Evidence / Details
- Data: private Git repos, 600+ companies, ~100k120k engineers, since 2022; ML model replicating a panel of expert reviewers, measuring *functionality delivered* (not commits/LOC). Methodology peer-reviewed (arXiv 2409.15152); the DiD result itself is talk-published only.
- The slide: "Causal Impact of AI on Software Engineering Productivity: Difference-in-Differences Analysis," DID covariate balance < 0.25, 95% CI band, model-release markers on the time axis. Deck: "Will AI Replace Software Engineers?", AI Conference, Sept 2025.
- Full numbers, links, and per-claim provenance in the raw dossier.
## Connections
- **Entity:** [[swepr]]
- **Corroborates:** [[2026-07-14-gap-between-ai-users-irreversible]] — Allie Miller's "irreversible gap" prediction is this curve, asserted 18 months earlier from advisory experience; Stanford supplies the measurement. Also the team-level twin of [[levels-of-ai-usage]] (the gap grows between *rungs of mastery*, not between license-holders and others).
- **Corroborates:** [[context-as-scarce-resource]] — the codebase-size finding (gains collapse toward 10M LOC, attributed to context-window limits and signal-to-noise) is the corpus's first outside quantitative support for context as the binding constraint.
- **Complicates:** [[make-more-cheap-code]] / [[code-as-throwaway]] — +91% PR review time and 2.6× rework externally confirm that generation moved the cost to review/verification, which is Theo's premise; but the *negative* gains in complex brownfield code sharpen the honest caveat that "code is cheap" holds least where most code lives.
- **Independently cited by a practitioner source:** [[2026-07-30-rakes-in-ai-sdlc-adoption]] builds its stakes on this study ("the Stanford study, Aug 2025: +20% speed, but the top performers pull away — the goal is to be in the top half of the chart") and anecdotally mirrors its downstream-cost finding (the +1%-despite-more-PRs rework case; review as the bottleneck — see [[review-is-the-new-bottleneck]]).
- **Queries:** [[2026-07-30-stanford-widening-gap-source]] (how this source was traced), [[2026-07-28-webinar-theses]] (thesis T7 gains its number)
## Open Questions
- Does the 46-team DiD analysis ever get a peer-reviewed publication? (Claim status upgrades when it does.)
- What exactly distinguishes the mastering teams — tooling, codebase hygiene, or skills/context practices? The proposed mechanism ("quality of usage") is asserted in talks, not decomposed. Directly relevant to whether the webinar's skills rung is *the* differentiator. Status: tentative.
- The study measures teams of engineers; how far do team-level results transfer to the webinar's non-engineer audience? Status: tentative.
## Change Impact on Wiki
- Created [[swepr]] entity.
- Added measured external evidence to [[levels-of-ai-usage]], [[context-as-scarce-resource]] and [[make-more-cheap-code]].
- Upgraded the corroboration pointer on [[2026-07-14-gap-between-ai-users-irreversible]] to cite this page.
- Added the "Stanford measured it: 4× in two years" note to thesis T7 in [[2026-07-28-webinar-theses]].
- Updated [[overview]] (11 → 12 sources; first quantitative outside study), `index.md`, `log.md`.