ingest: Stanford SWEPR widening-gap study and AI-in-SDLC adoption pitfalls
Add two new sources with summaries, new concepts (developer-as-agent-manager, review-is-the-new-bottleneck), new entities (SWEPR, Nikolai Sheiko), and a query on the Stanford source; update related concept pages, overview, index, and log.
This commit is contained in:
@@ -20,11 +20,12 @@ If agent work takes sixteen minutes and you are doing something else, latency st
|
||||
|
||||
- Orbs, sleep/wake, one-URL packaging, multiplayer handoff, the 16-minute live demo, the collapsed cloud-IDE objections, Quinn's proof line, paper-cut velocity, local-dev prediction, infra-margin prediction — [[2026-07-28-agentic-engineering-10x-developer]].
|
||||
- Fire-and-forget as a native harness mode; completion notifications as what makes background agents usable — [[2026-07-14-skills-based-on-git]], [[harness]].
|
||||
- The same mode as a job description: the AI-developer is IO-bound, runs tasks in parallel, and "if you launched Claude Code and sit watching it work — you're a bad employee" — [[2026-07-30-rakes-in-ai-sdlc-adoption]] (the role-shift side lives at [[developer-as-agent-manager]]).
|
||||
- Scheduled agents producing while you sleep (the non-engineer version) — [[personal-ai-operating-system]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[harness]] (async is one of its two modes), [[agentic-loops]], [[make-more-cheap-code]] (proof artifacts are throwaway code with a job), [[shedding-weight]] (async is what makes killing the backlog possible — parked agents replace queued tickets), [[personal-ai-operating-system]], [[context-as-scarce-resource]]
|
||||
- Concepts: [[harness]] (async is one of its two modes), [[agentic-loops]], [[make-more-cheap-code]] (proof artifacts are throwaway code with a job), [[shedding-weight]] (async is what makes killing the backlog possible — parked agents replace queued tickets), [[personal-ai-operating-system]], [[context-as-scarce-resource]], [[developer-as-agent-manager]] (the human role this mode implies), [[review-is-the-new-bottleneck]] (where the parallel diffs pile up)
|
||||
- Entities: [[thorsten-ball]], [[amp]], [[claude-code]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
@@ -22,6 +22,8 @@ The human role has climbed prompt-engineer → **context-engineer** → harness-
|
||||
|
||||
**Context now has a price — at fleet scale.** The same source names **token budget** as one of two variables separating winners from losers, alongside knowing how to use agents. Context has always been scarce per-request; this is the corpus's first claim that it is also scarce per-*wallet*. Scope, corrected 2026-07-28: the claim comes from **metered** usage (parallel remote sandboxes), and under a flat consumer subscription the corpus's own heavy users report no ceiling — so per-request scarcity remains the binding constraint for individuals, and per-wallet scarcity is a fleet and enterprise concern. See [[enterprise-ai-reality]] and [[explosion-of-internal-software]].
|
||||
|
||||
**The compaction curse — scarcity at codebase scale** ([[2026-07-30-rakes-in-ai-sdlc-adoption]]). On a huge codebase the agent gathers context → the window overflows → compaction → it re-gathers → compaction again, and the task barely completes. [[nikolai-sheiko]]'s cure inverts the "agents mean best practices don't matter" fallacy: **best practices exist for the agent now** — locality, isolated modules with interfaces, so that *the codebase stores the context*. This converges with [[thorsten-ball]]'s context-lives-in-the-codebase position from the opposite direction (Thorsten skips skills because his codebase carries context; Sheiko says make your codebase able to). Practical additions: **AST search instead of grep** on colossal projects (grep returns a wall of noise, AST returns the relevant slice), and a flat rejection of embeddings/RAG over code. The mechanism matches Stanford's measured gains-collapse toward 10M LOC (evidence below).
|
||||
|
||||
**The supply-side facet** ([[2026-07-22-ai-is-stupid]]): before context is *scarce* it is usually *absent*. "Intelligence without context loses to context without intelligence" — ten Nobel laureates asked about your sales month can only cite industry averages, while your rank-and-file employee answers better because they see your funnel, clients, and deals. The default "stupid AI" experience is a strong model given neither business context nor a [[harness]]; the fix is investing in context infrastructure (data, memory, integrations) before reaching for a bigger model.
|
||||
|
||||
## Evidence
|
||||
@@ -31,6 +33,8 @@ The human role has climbed prompt-engineer → **context-engineer** → harness-
|
||||
- "Intelligence without context loses"; Nobel-vs-employee analogy; invest in context before model upgrades — [[2026-07-22-ai-is-stupid]].
|
||||
- Information > model choice > effort level; the two information sources; token budget as a winner/loser variable — [[2026-07-28-agentic-engineering-10x-developer]].
|
||||
- Reading costs attention — the human-side analog of the same scarcity — [[make-more-cheap-code]], [[2026-07-24-youre-reading-way-too-much-code]].
|
||||
- **First outside quantitative support:** Stanford SWEPR finds AI productivity gains collapse as codebases grow from ~10k to ~10M lines, attributing it to context-window limits and signal-to-noise degradation — the corpus's context-is-the-constraint claim, measured at scale — [[2026-07-30-stanford-swepr-widening-gap]].
|
||||
- Compaction curse; best-practices-for-the-agent (locality, interfaces, codebase-stores-context); AST search over grep; embeddings/RAG over code rejected — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
@@ -40,7 +44,7 @@ The human role has climbed prompt-engineer → **context-engineer** → harness-
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
- "First third = smart zone" is a heuristic, not a measured boundary; likely model-dependent. Status: tentative.
|
||||
- [[2026-07-22-ai-is-stupid]] names **RAG** and long-term assistant memory as the practical context mechanisms; [[2026-07-14-skills-based-on-git]] argues [[skills-as-memory|skills]] beat RAG (load-on-activation vs pre-injection). Possibly audience-driven (business data vs procedures) rather than a real disagreement. Status: tentative.
|
||||
- [[2026-07-22-ai-is-stupid]] names **RAG** and long-term assistant memory as the practical context mechanisms; [[2026-07-14-skills-based-on-git]] argues [[skills-as-memory|skills]] beat RAG (load-on-activation vs pre-injection). Possibly audience-driven (business data vs procedures) rather than a real disagreement. Status: tentative. *(2026-07-30: [[nikolai-sheiko]] adds a hard anti-RAG data point for the code domain specifically — "embeddings over code don't work" — which supports the audience-driven reading: the anti-RAG votes are both about code/procedures, the pro-RAG vote is about business data.)*
|
||||
|
||||
## Next Questions
|
||||
|
||||
|
||||
37
wiki/concepts/developer-as-agent-manager.md
Normal file
37
wiki/concepts/developer-as-agent-manager.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# Developer as Agent Manager
|
||||
|
||||
#concept
|
||||
|
||||
## Summary
|
||||
|
||||
[[nikolai-sheiko]]'s "universal error #0": treating AI-assisted development as the same job at higher speed. A good developer is **CPU-bound** — 3–5 hours of deep focus on one feature. A good AI-developer is **IO-bound** — a *manager of an agent-employee*, running several tasks in parallel, spending their time on planning and verification instead of typing. His blunt test: "If you launched Claude Code and sit watching it work — you're a bad employee."
|
||||
|
||||
## Current Understanding
|
||||
|
||||
- **The switch is psychological, not technical** — and not everyone can make it. Sheiko is explicit that this is *fine*: don't drag everyone into AI under one brush; conservatives "guard the company from the vibe-coders." Compare [[2026-07-14-nina-interview|Nina's finding]] that adoption blocks on friction, not resistance — this page is the case where genuine resistance exists and is legitimate.
|
||||
- **Where the time goes instead:** "planning on the left, verification on the right" — the coding middle collapsed. Hence 20-minutes-minimum planning, explicit goals the agent self-checks against, and the review discipline of [[review-is-the-new-bottleneck]].
|
||||
- **The role splits further.** Today the people who *use* agents and the people who *tune the harness* are the same; Sheiko predicts a split into **users** (plan + verify a feature) and **Agentic Operations** (own the SDLC configuration, feedback loops, prompts, skills). A **Product engineer** also emerges — answers *why* we build it this way, what to simplify, what to ignore.
|
||||
- **What stays human: Judgment over Intelligence.** AI absorbs *Intelligence* (action sequences requiring intellect); *Judgment* — taste built over years, or deep domain expertise (oil & gas, medicine) — remains human for now. We detect slop precisely because it lacks human touch. This is the corpus's [[seniority-and-the-junior-squeeze|judgment-as-risk-reduction]] claim restated as a capability boundary.
|
||||
- **Corpus convergence.** The same working mode appears as [[async-by-default|orbs and parallel delegation]] ([[thorsten-ball]]), Eugene's 7 parallel project-agents, Allie's ~100 agents, and Karpathy's "80% Claude Code, 20% by hand" (cited within the talk). Sheiko's contribution is naming the *identity* shift and its HR consequences: the middle dev who mastered this, asked for a raise, was refused, and left for far more.
|
||||
|
||||
## Evidence
|
||||
|
||||
- CPU-bound vs IO-bound framing; "sit watching = bad employee"; don't-force-everyone; users vs Agentic Operations; Product engineer; Intelligence vs Judgment; the raise-refusal mini-case — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
- The same mode practiced at the frontier (parallel remote sandboxes, delegation, proof-asking) — [[2026-07-28-agentic-engineering-10x-developer]] via [[async-by-default]].
|
||||
- Non-engineer versions of parallel delegation (~100 agents, scheduled workflows) — [[2026-07-14-gap-between-ai-users-irreversible]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[async-by-default]] (the infrastructure this role runs on), [[review-is-the-new-bottleneck]] (where the manager's verification time goes), [[product-ownership]] (the webinar's human-side twin: own outcomes, not tickets), [[seniority-and-the-junior-squeeze]] (Judgment as the durable half), [[levels-of-ai-usage]] (the non-engineer's version of the same climb), [[harness]]
|
||||
- Entities: [[nikolai-sheiko]], [[thorsten-ball]], [[eugene]], [[allie-miller]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
- **"Don't force everyone" vs the widening gap.** Sheiko permits opting out; [[2026-07-14-gap-between-ai-users-irreversible|Allie]] calls the gap irreversible and [[2026-07-30-stanford-swepr-widening-gap|Stanford measured it quadrupling]]. If both are right, opting out is legitimate *and* costly, and the source doesn't reconcile the two. Status: tentative.
|
||||
- The users / Agentic-Operations split is a prediction, not an observation — today's evidence (Eugene, Thorsten) is of people doing both. Status: tentative.
|
||||
- Intelligence-vs-Judgment is a moving boundary asserted "for now"; the corpus has no criterion for where it stops moving.
|
||||
|
||||
## Next Questions
|
||||
|
||||
- What does the manager's day actually look like — is there a source with a concrete parallel-task routine (queue depth, check-in cadence) rather than the mode's name?
|
||||
- Does the Agentic-Operations role match the webinar's promise that non-engineers can self-serve ([[levels-of-ai-usage]]), or does it re-centralize harness-tuning in specialists?
|
||||
@@ -14,17 +14,19 @@ The indie/practitioner world and the regulated-enterprise world diverge sharply.
|
||||
- **Adjacent constraints:** the [[seniority-and-the-junior-squeeze|"read what you approve"]] security concern is amplified at scale; safety-critical/regulated code is the clear exception to [[code-as-throwaway|"most code isn't high-value"]].
|
||||
- **A second divide: the token budget** (added 2026-07-28; **scope corrected 2026-07-28** — see below). [[thorsten-ball]] names two variables separating winners from losers — knowing how to use agents, and **having the token budget to do it**. It cuts both ways for this page: an enterprise can buy budget an individual cannot, while a locked-down enterprise may withhold it from the people who would use it best. Whoever controls the budget controls how far [[explosion-of-internal-software|internal software]] spreads. Thorsten names the variable and says nothing about who pays.
|
||||
- **Scoping correction.** This was first written here as "the divide is also a *spending* gap," which overstates it. Thorsten's pricing regime is **metered**: [[amp]] sells usage, and his working pattern is parallel remote sandboxes and parked orbs ([[async-by-default]]) — a fleet cost, not a seat cost. Under a **flat consumer subscription** the corpus's own evidence points the other way: [[eugene]] runs 7 project-agents in parallel on a $200 plan, [[allie-miller]] runs ~100 agents and 36 workflows, and neither reports hitting a cost ceiling — while [[2026-07-14-sebastian-eugene-interview]] frames levelling as "a 20-year veteran and a fresh grad **on the same subscription**." For individual and small-team use the budget is one subscription; the token-budget variable bites at fleet scale and under metered pricing, which is where Thorsten sits and where enterprises will land.
|
||||
- **The market claim, seconded — and sharpened into a quote** (added 2026-07-30). [[nikolai-sheiko]], from multi-company adoption work: *"Companies no longer need custom AI development. Come in, install Claude Code or Codex, configure everything, attach connectors, think about security — and it works better than any custom build."* This is Sebastian's company-managed-harness market stated as a service playbook. Two adoption anti-patterns attached: **the external configurator** who leaves a "magic artifact" nobody on the team owns (what a company should buy is a *teacher/curator*; the team must configure its own tools — the [[solve-first-then-skillify|evolution]] has to happen in their hands), and **metered pricing shaping behaviour** — a team on Cursor's per-token billing economizes instead of experimenting (~30% dearer than subscriptions at the same level), which is the metered-vs-subscription split from the scoping correction above observed as an organizational failure mode. His pricing prediction — tokens get dearer near-term, cheaper later; "experiment at full throttle while subscriptions are cheap" — matches [[eugene]]'s price-rise prediction already flagged in the token-budget question.
|
||||
- **The frontier's advice does not transfer.** [[shedding-weight]] — kill the backlog, kill CI that repeats the agent's tests, kill local dev in favour of remote sandboxes ([[async-by-default]]) — describes a startup that owns its own process. In a regulated shop the pipeline, the audit trail and the ticket history frequently *are* the deliverable to a regulator, and code sitting in a vendor's remote sandbox is precisely what Sebastian's clients forbid. The gap between what the frontier recommends and what compliance permits is the same gap this page calls the market.
|
||||
|
||||
## Evidence
|
||||
|
||||
- Managed VMs / zero self-install, Roche ~1,200 engineers, banks banned→adopting, "company-managed resource," "the interesting market" — [[2026-07-14-sebastian-eugene-interview]].
|
||||
- Token budget as a winner/loser variable; the frontier playbook (kill backlog/CI/local dev, remote sandboxes) that compliance cannot follow — [[2026-07-28-agentic-engineering-10x-developer]].
|
||||
- "No custom AI development needed" quote; external-configurator anti-pattern vs teacher/curator; Cursor per-token billing → team economizes; tokens-dearer-then-cheaper prediction — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[harness]], [[seniority-and-the-junior-squeeze]], [[code-as-throwaway]], [[shedding-weight]], [[async-by-default]], [[explosion-of-internal-software]], [[context-as-scarce-resource]]
|
||||
- Entities: [[sebastian]], [[virtido]], [[eugene]], [[thorsten-ball]]
|
||||
- Entities: [[sebastian]], [[virtido]], [[eugene]], [[thorsten-ball]], [[nikolai-sheiko]]
|
||||
- Tools: [[claude-code]], [[amp]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
@@ -20,6 +20,7 @@ Supporting practices at the plateau: keep CLAUDE.md self-maintaining ("always ke
|
||||
- Webinar title and non-programmer audience confirm the ladder as the webinar's spine — [[2026-07-14-nina-interview]].
|
||||
- Convergent structure (foundation docs + skills as the non-engineer's OS) — [[2026-07-14-gap-between-ai-users-irreversible]].
|
||||
- A high-rung user missing the skills rung, and the memory pain that results — [[2026-07-21-larysa-interview]].
|
||||
- **Measured, team-level corroboration that the gap grows between rungs of mastery**, not between license-holders and others: Stanford's 46-vs-46-team analysis shows the productivity gap between AI-mastering and lagging teams growing 4.8% → 19% (4×) over ~2.25 years — [[2026-07-30-stanford-swepr-widening-gap]]. *(Caveat: measures engineering teams, not this ladder's non-programmer audience; and the study asserts "quality of usage" as the differentiator without decomposing which rung supplies it.)*
|
||||
|
||||
## Related Pages
|
||||
|
||||
|
||||
@@ -23,10 +23,11 @@
|
||||
- All claims, ratios, tier table, slop patterns, Dax/Shao citations — [[2026-07-24-youre-reading-way-too-much-code]].
|
||||
- Groundwork (code disposable, kill without guilt, G-brain markdown tier) — [[2026-07-14-everything-we-knew-about-software-has-changed]].
|
||||
- 15 icon variants, Midjourney rounds, "ask the agent for proof" — [[2026-07-28-agentic-engineering-10x-developer]].
|
||||
- **External measurement of the cost moving downstream:** Stanford SWEPR coverage reports +91% PR review time and ~2.6× rework in AI-heavy workflows — writing got cheaper, reviewing got dearer, which is this page's premise measured rather than asserted — [[2026-07-30-stanford-swepr-widening-gap]]. *(Secondary-coverage numbers; and note the same study's negative gains in complex brownfield code — "code is cheap" holds least where most code lives.)*
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[code-as-throwaway]] (parent claim: cost → zero; this page is its *discipline* — what cheap code is actually for), [[think-wider-not-bigger]] (same breadth logic applied to generation volume rather than ambition), [[product-ownership]] (verifying as the human's remaining job), [[solve-first-then-skillify]] (contrast: slop is frozen into nothing; skills freeze the procedure), [[leave-less-room-for-imagination]] (tension — see below), [[context-as-scarce-resource]], [[async-by-default]] (proof artifacts as the delegated form of the same move)
|
||||
- Concepts: [[code-as-throwaway]] (parent claim: cost → zero; this page is its *discipline* — what cheap code is actually for), [[think-wider-not-bigger]] (same breadth logic applied to generation volume rather than ambition), [[product-ownership]] (verifying as the human's remaining job), [[solve-first-then-skillify]] (contrast: slop is frozen into nothing; skills freeze the procedure), [[leave-less-room-for-imagination]] (tension — see below), [[context-as-scarce-resource]], [[async-by-default]] (proof artifacts as the delegated form of the same move), [[review-is-the-new-bottleneck]] (the org-level form of the cost shift this page manages individually — with the completed-without-rework metric as its answer)
|
||||
- Entities: [[theo-browne]], [[eugene]], [[thorsten-ball]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
38
wiki/concepts/review-is-the-new-bottleneck.md
Normal file
38
wiki/concepts/review-is-the-new-bottleneck.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# Review Is the New Bottleneck
|
||||
|
||||
#concept
|
||||
|
||||
## Summary
|
||||
|
||||
When agents write the code, the SDLC doesn't collapse completely — it collapses *around the humans*. [[nikolai-sheiko]]'s two remaining "red squares" are the **reviewer** and the **planner**: tasks pile up in the review queue, reviewers burn out, quality drops, and headline output metrics (PRs, LoC) rise while real throughput barely moves. The organizational fix is twofold: **review with the agent** (not fully manual, not fully delegated) and **measure completed tasks without rework** rather than anything volume-based.
|
||||
|
||||
## Current Understanding
|
||||
|
||||
- **The mechanism.** Generation got ~free, so the cost moved downstream to verification — and at team scale, downstream is a *person* with a queue. Sheiko's European-outsourcer case: more PRs than ever, net gain +1%, because rework consumed the difference. Manual-only review starts a spiral (queue → burnout → rubber-stamping → more rework); fully delegated review is the opposite error ([[async-by-default]]'s "proof produced by the thing being checked").
|
||||
- **The middle path: review together with the agent.** Treat the model as a smart student — direct it, pose hypotheses, locate problems jointly. This is the org-level sibling of Theo's reading economics ([[make-more-cheap-code]]: AI reviews before humans, per-file summaries, read only what's worth reading).
|
||||
- **The metric that resists gaming:** a task counts as done only if it **doesn't come back for rework**; track task lifetime and rework time. LoC, commit count and PR count are all trivially hacked and all rise *because* of the bottleneck, not despite it.
|
||||
- **Externally measured:** Stanford SWEPR coverage reports **+91% PR review time** and ~2.6× rework in AI-heavy workflows, and finds gross code volume up 30–40% while net gains are ~15–20% — the same rework-eats-half story Sheiko tells anecdotally ([[2026-07-30-stanford-swepr-widening-gap]]).
|
||||
- **Planning is the other red square.** Time redistributes from coding to "planning on the left + verification on the right" — which is why Sheiko prescribes 20-minutes-minimum planning and why a **Product engineer** role emerges ([[developer-as-agent-manager]]).
|
||||
|
||||
## Evidence
|
||||
|
||||
- Reviewer/planner as the remaining red squares; review-with-the-agent; the metrics table; the +1% outsourcer case — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
- +91% PR review time, 2.6× rework, gross-vs-net gap — [[2026-07-30-stanford-swepr-widening-gap]] *(secondary-coverage numbers)*.
|
||||
- Reading as the scarce human resource; AI-review-before-human-review; tiered reading discipline — [[2026-07-24-youre-reading-way-too-much-code]] via [[make-more-cheap-code]].
|
||||
- Parallel agents multiplying diffs without multiplying review capacity — the attention pile-up already logged on [[async-by-default]]; this page names that open question as the bottleneck it becomes at team scale.
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[make-more-cheap-code]] (individual-level discipline for the same cost shift), [[async-by-default]] (the generation side that feeds the queue), [[developer-as-agent-manager]] (the role shift on the human side of the queue), [[seniority-and-the-junior-squeeze]] ("read what you approve" — why review can't just be dropped), [[maintenance-is-the-real-cost]] (kindred move: the visible activity was never the expensive part)
|
||||
- Entities: [[nikolai-sheiko]], [[swepr]], [[theo-browne]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
- **How much review survives?** Theo/Dax hold that agent diff-summaries surface anomalies; [[eugene]] holds that drift is precisely what summaries miss ([[leave-less-room-for-imagination]]); Sheiko's review-with-the-agent is a third position between them — asserted, not tested. Status: tentative.
|
||||
- The completed-without-rework metric is better than LoC/PRs but still gameable (e.g. by inflating task granularity or quietly reclassifying rework as new tasks); the source doesn't address it. Status: tentative.
|
||||
- Whether review-as-bottleneck is transitional (until verification is agentized) or structural (a human must always sign off — the [[seniority-and-the-junior-squeeze|accountability]] view) is open across the corpus.
|
||||
|
||||
## Next Questions
|
||||
|
||||
- What does review-with-the-agent look like concretely — a checklist, a dialogue pattern, a skill? The corpus has the prescription but no transcript of it done well.
|
||||
- Is there a non-engineer analog (the webinar audience reviews documents, not PRs)? The checker-skill design in [[2026-07-24-non-engineer-throwaway-verification]] may be it.
|
||||
@@ -27,6 +27,7 @@ The **method** for populating skills is [[solve-first-then-skillify]]: reach the
|
||||
- ~3-message correction-loop heuristic; skills as the non-programmer ceiling (with CLAUDE.md) — [[2026-07-14-yulia-interview]].
|
||||
- Cross-session memory loss as the #1 practitioner pain; "memory is the worst thing agents have"; skills committed as the webinar remedy — [[2026-07-21-larysa-interview]].
|
||||
- Counter-evidence: no skills, no MCP, no slash commands at a 99%-AI-written company; `AGENTS.md` + codebase + rich prompt as the substitute — [[2026-07-28-agentic-engineering-10x-developer]].
|
||||
- **A second practitioner vote *for* the skills layer** (2026-07-30): [[nikolai-sheiko]]'s "Agentic Evolution" makes skill-building-plus-verification the difference between living on defaults and "vertical growth," and he flatly rejects embeddings/RAG over code ("don't use them unless you understand *very* well why") — siding with the load-on-activation camp in the skills-vs-RAG contradiction below. His verification protocol (context-free subagent re-solves the task from the skill alone) is the closest thing yet to the falsification test this page asks for — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
@@ -43,10 +44,10 @@ The **method** for populating skills is [[solve-first-then-skillify]]: reach the
|
||||
1. **Situational.** He works daily in *one codebase he controls*, where context can live in the code and `AGENTS.md`. Skills earn their keep when work is spread across many ad-hoc tasks with no codebase to encode into — which is exactly the corpus's HR/BA audience ([[nina]], [[yulia]], [[larysa]]). Under this reading both are right and the disagreement is about who is speaking.
|
||||
2. **The abstraction is premature.** Skills are scaffolding for models that needed it; a strong model plus a rich prompt plus a good repo may simply beat a skills library, making the whole layer a 2025 artifact. This is the uncomfortable reading for the webinar's central promise.
|
||||
3. **He has skills under another name.** AMP's Oracle/Painter/Puck sub-agents and a maintained `AGENTS.md` *are* curated, reusable, two-stage context — just authored by the vendor and the team rather than the user. Under this reading the dispute is about who curates, not whether curation is needed.
|
||||
Status: tentative. Note the evidential asymmetry — his is a first-hand report of daily practice at scale, where the pro-skills case rests on Konstantin's architecture argument plus self-reported individual workflows. **Presentation-safe restatement:** [[2026-07-28-webinar-theses]] reframes the claim as *context you author beats context that's inferred*, which holds under all three readings — Konstantin's skills, Allie's foundation docs, Eugene's anti-memory position and Thorsten's `AGENTS.md` are all authored context.
|
||||
Status: tentative. Note the evidential asymmetry — his is a first-hand report of daily practice at scale, where the pro-skills case rests on Konstantin's architecture argument plus self-reported individual workflows. *(Asymmetry softened 2026-07-30: [[nikolai-sheiko]] adds a second practitioner voice on the pro-skills side, from multi-company adoption work rather than one codebase — though his cases are anonymous anecdotes, so the readings above remain unsettled.)* **Presentation-safe restatement:** [[2026-07-28-webinar-theses]] reframes the claim as *context you author beats context that's inferred*, which holds under all three readings — Konstantin's skills, Allie's foundation docs, Eugene's anti-memory position and Thorsten's `AGENTS.md` are all authored context.
|
||||
|
||||
## Next Questions
|
||||
|
||||
- ~~What's a starter skill set for a non-engineer?~~ Answered in [[2026-07-14-best-first-skill-for-beginner]] (skill-creator as meta-skill; tone-of-voice + anti-AI-language as first content skill).
|
||||
- Do skills actually solve *cross-project* context, or only per-procedure recall? Larysa's complaint may be the former, which skills don't obviously address.
|
||||
- Is there a test that would separate reading 1 from reading 2 above? The cheapest one available: give a non-engineer the same task with and without a skill and compare drift — the corpus has never run it, and the webinar's promise rests on the answer.
|
||||
- Is there a test that would separate reading 1 from reading 2 above? The cheapest one available: give a non-engineer the same task with and without a skill and compare drift — the corpus has never run it, and the webinar's promise rests on the answer. *(Adjacent evidence 2026-07-30: Sheiko's context-free-subagent protocol — [[solve-first-then-skillify]] — runs the with-skill half in practice, but never the without-skill control, so the question stands.)*
|
||||
|
||||
@@ -15,17 +15,28 @@ The recurring beginner mistake is writing the skill first and then trying to "sh
|
||||
|
||||
The payoff goes beyond reuse: a packaged skill is a **handoff and de-risking asset** — "a person with not even a third of your HR experience can deliver a decent result," which cuts onboarding and lets the expert take a vacation. This is how [[skills-as-memory]] gets *populated* in practice — the method side of that architecture, and the fix for "don't teach the AI abstractly."
|
||||
|
||||
**Agentic Evolution — the strongest formulation, plus the missing verification step** (added 2026-07-30). [[nikolai-sheiko]] frames the same method as onboarding an employee: asking the expert "how do you do this?" yields theory; instead **take the new employee (the agent) by the hand through hard real tasks, show it the rakes, then say: "remember all of this and write the manual for the next one."** Without this you live on defaults; with it "vertical growth begins." He then adds what the corpus's earlier heuristics lacked — a **verification protocol** for the frozen skill:
|
||||
|
||||
1. Write the skill together with the agent.
|
||||
2. Don't go to lunch.
|
||||
3. Launch a **subagent with no context** — it must solve the same task from scratch using only the skill.
|
||||
4. The main agent watches what fails and fixes the skill.
|
||||
5. The mentor agent thus onboards the next agent.
|
||||
|
||||
This is the first source to describe actually *running* something close to the skills falsification test proposed on [[skills-as-memory]] (same task, with-skill vs from-scratch) — though it tests the skill's completeness for one task, not whether the skill beats no-skill. His do-tomorrow extension: a skill that analyses your own sessions daily, automated via schedules/routines — evolution as a standing loop rather than a one-time freeze.
|
||||
|
||||
## Evidence
|
||||
|
||||
- "You first solve a task with Claude; the moment you reach the final solution, you say — now create a skill from this"; ~3-message heuristic — [[2026-07-14-yulia-interview]].
|
||||
- Do-the-task-then-freeze framing; skill-as-handoff to a junior hire; vacation/de-risking angle — [[2026-07-14-nina-interview]].
|
||||
- >5-tool-calls auto-creation heuristic and curator pruning — [[2026-07-14-skills-based-on-git]].
|
||||
- Skills prescribed specifically as the workaround for cross-session memory loss, and as the constraint on drift — [[2026-07-21-larysa-interview]].
|
||||
- Agentic Evolution (walk the agent through tasks → have it write the manual); the context-free-subagent verification protocol; session-analysis skill as a daily loop — [[2026-07-30-rakes-in-ai-sdlc-adoption]].
|
||||
|
||||
## Related Pages
|
||||
|
||||
- Concepts: [[skills-as-memory]] (the architecture this method feeds), [[levels-of-ai-usage]] (skills are the top practical rung), [[personal-ai-operating-system]], [[leave-less-room-for-imagination]] (why a *proven* spec beats a written-ahead one)
|
||||
- Entities: [[eugene]], [[nina]], [[konstantin]], [[larysa]]
|
||||
- Entities: [[eugene]], [[nina]], [[konstantin]], [[larysa]], [[nikolai-sheiko]]
|
||||
|
||||
## Contradictions / Uncertainty
|
||||
|
||||
@@ -33,5 +44,6 @@ The payoff goes beyond reuse: a packaged skill is a **handoff and de-risking ass
|
||||
|
||||
## Next Questions
|
||||
|
||||
- What does a good "create a skill from this" prompt look like — does the corpus contain a concrete example transcript?
|
||||
- What does a good "create a skill from this" prompt look like — does the corpus contain a concrete example transcript? *(Partially answered 2026-07-30: Sheiko's "remember all of this and write the manual for the next one" after a guided run is the best prompt-shape the corpus has.)*
|
||||
- How do the ~3-message and >5-tool-call heuristics compare in practice; is one strictly better for non-programmers?
|
||||
- Does the context-free-subagent check catch skill *quality*, or only completeness for the one task it was frozen from?
|
||||
|
||||
Reference in New Issue
Block a user