B.6 — the dogfood loop: fail, then grow

Mon, Jun 15 2026

Not a pass-rate bench — a loop bench. model + forest + graders + retry → result → train → system. Every graded attempt becomes an experience that drives two feedback channels: certified training pairs (model) and gated grow/prune proposals (forest). The headline is a deterministic demo: the forest mis-retrieves the wrong API, the loop teaches it, and the gap closes on screen. Plus the receipts: B.4 6/6, B.5 6/6 held green throughout.

T.M.B. #6 — the loop is the experiment. B.1–B.5 measured does the library beat the model. B.6 measures does the system improve itself — and shows it, with a before/after you can re-run. Retrieval is deterministic, so the demo is exact.

What’s being measured

The thesis all along: model + forest + graders + retry loop → result → train → system. B.6 is the first bench of the last two arrows — the part where the result feeds back and the system gets better. Two feedback channels:

MODEL channel (the angler). Every passing run is distilled into a certified (prompt, code) pair — verified-correct by an un-bribable grader, not a vibe. That’s the training set for a model of our own. (The fine-tune is the next brick; the data flows now.)
FOREST channel. Closes end-to-end with no GPU. The library grows (crystallize) and prunes (garbage-collect), and — heeding the rule that self-improvement without a gate tears down — it only ever proposes.

The headline: fail → grow, on screen

A real gap in the forest as it shipped: ask for “list all files recursively including every subdirectory” and the word “recursively” lexically pulls fs.cp (copy) to the top, burying the right answer fs/promises.readdir at rank #3. The coder weights the top hits — so this mis-ranking is a real failure.

The loop catches it, crystallizes the fix (an alias teaching the forest the task-language its docstring lacks), re-embeds, and re-measures:

ROUND 1  (forest as-is)
  retrieves:  fs.cp, fs/promises.cp, fs/promises.readdir, fs.mkdir
  fs/promises.readdir ranks #3   ❌ buried below the wrong API

CRYSTALLIZE → teach the forest the right word, re-embed the corpus

ROUND 2  (forest grew)
  retrieves:  fs/promises.readdir, fs.readdir, fs.cp, fs/promises.cp
  fs/promises.readdir ranks #1   ✅ displaced fs.cp — gap closed

That’s the whole thesis in eight lines: the system got something wrong, learned the right word from its own experience, and is permanently better — every future request for recursive listing now retrieves the correct API. And the learning is a reviewable one-line diff, not an inscrutable weight update. That legibility is the point — you can see exactly what the library learned, approve it, and revert it.

The safety that makes it not tear down

Self-modification is the dangerous part — so the forest channel is conservative by construction:

guard	what it does	proven
propose-only	writes a review queue, never edits silently	crystallize + GC both
archive, don’t delete	GC moves to `archive/`, append-only revert	a throwaway verb archived + logged
cold-start guard	GC refuses to act below N observations	at 4 obs it flagged 48 good aliases as “unused” — and held
grace period	freshly-authored capsules are exempt	reported 0 unused verbs while flagging stale aliases

The cold-start number is the honest one to dwell on: with sparse usage data, everything looks unused. A naive GC would have gutted the library. The guard is the difference between a system that builds up and one that eats itself — which is the whole “does a self-modifying system tear down?” question, answered in code.

The receipts still hold

The pass-rate benches ran green through all the plumbing changes — the gate redesign and the shipped retrieval didn’t regress anything:

bench	blind	forest
B.4 — JS stdlib	5 / 6	6 / 6
B.5 — Node core	5 / 6	6 / 6

Honest scope: B.6 measures the loop’s machinery, not a new pass-rate. The “fail” is a retrieval mis-rank (deterministic, exact), not a code failure — a strong 7B rarely fails outright on common tasks, which is itself the consistent finding across every bench: the forest’s edge is narrow-but-real on a competent model, and the loop’s job is to keep widening the library where the model is weak. The model channel emits certified pairs but nothing trains on them yet (next brick). Code in the mmm-poc repo: forest.mjs, distill.mjs, gc.mjs, grow_demo.mjs — run the demo yourself.