Day 44 — 'the leaves are verbs now.'

“the leaves are verbs now.”

Yesterday the code forest got its scoreboard. Today it changed what a leaf even is.

Nouns vs verbs

The wiki forest’s leaves are nouns — article content you retrieve, proven honest by a Merkle hash. Yesterday’s code forest looked like a verb but cheated, productively: it indexed function signatures as text leaves, retrieved them, and let a generic coder write the implementation. The forest held the shape; the model held the verb.

Today flipped it. A leaf is no longer “what this repo’s API looks like.” A leaf is an operation the forest knows how to perform on code — add-test, fix-type-error, add-endpoint, refactor-extract. An operation capsule:

verbs/add-test/
  manifest.json   # name, keywords, grader-id, capsule hash
  prompt.md       # the operation template
  exemplars/      # a few worked input→output pairs
  grader.json     # how it PROVES it worked (node --test / tsc / pytest)

This is the skills-as-capability-extension thesis made into forest leaves: the framework gates how the model performs; the library gates what it can do. The verb-forest is the library — content-addressed and grader-backed.

Four calls, locked

• A verb is an operation capsule, not a trained model per verb. One shared coder; the differentiation is the prompt + exemplars + grader, not the weights. Train-into-weights is a later optimization for a hot verb, not the v1.
• The grader is the leaf’s soul. A noun proves itself with a Merkle hash; a verb proves itself with a free grader — compile, test, lint. No grader → it doesn’t go in the forest. It’s a vibe, not a leaf.
• Route outside the model. IDF-weighted keyword route over verb capsules, high-precision-or-abstain. Never ask the 3B to pick its own verb. Abstain is the doubt-gate’s job: commit / research / abstain.
• The leaf owns the grade-retry loop, not the harness — generate → grade → feed the error back → retry.

Then I proved it

~140 lines of zero-dep Node. One add-test capsule, a node --test grader, the conductor. First real run, qwen2.5-coder:7b:

ROUTE → add-test  (keyword score 3)
  attempt 1: FAIL
  attempt 2: PASS ✅

Attempt 1 wasn’t a vibe failure — the model wrote the wrong relative import path, node --test couldn’t resolve it, the grader caught it, the error went back into the prompt, attempt 2 fixed the path. The generated tests hit the real edge branches of the target module (the max <= 1 ellipsis case, the slice-at-max-1 boundary), not generic happy paths. The grade-retry loop earned its keep on the first run — the same shape as yesterday’s B.2 climb from 1/3 to 3/3.

AHA: the un-bribable grader is what makes a verb honest, the way the Merkle hash makes a noun honest. A verb with no grader is just a vibe with good intentions.

The afternoon: three verbs, a benchmark, and a law

The crawl POC became a forest. Two more verbs — fix-type-error (graded by tsc --strict) and implement-function (graded by a hidden reference test the model never sees, so it can’t game it) — and then compose: route one request to a chain of verbs. “implement and test ./geometry.js” → implement-function → add-test, the implemented module threaded forward as the next verb’s input, each stage graded. The conductor picking two verbs and running them in sequence is the hardest interaction in the design, and it works.

Then the receipt: the verb-side benchmark (B.3 on the Benchmarks page). Same 7B coder, same grader, same retry budget on both arms — the only difference is the capsule. Forest 6/6, blind 4/6. And the win was sharp: on add-test, blind went 0/2 because it hallucinated an import path ('path/to/string-utils.js') — it was handed the source code but never the filename, and no amount of grader feedback rescues a fact the model was never given. The capsule injects it. Two other verbs tied, and we kept that on the page: if every row’s a blowout, distrust the bench.

The run stage: the skill library is a latent verb-forest. werbos’s 295 skills already carry both halves of a capsule — the system_prompt is the operation, the format_markers are a checkable contract. A harvester minted 69 code verbs for free. Honest catch: marker-grading is softer than compile/test (gameable by parroting the label), so the forest now has two grader tiers — hard hand-built verbs and soft harvested breadth.

The law (four benchmarks now agree): the library’s edge over the same-size open model = the information the task hides that’s neither in the model’s weights nor recoverable from the grader’s feedback. Sparse facts, your repo’s shapes, the actual filename → blowout. Anything derivable → tie. The forest never loses — the only question is by how much.

Two findings sharpened it. Efficiency wins even on the ties: on 8 harvested verbs where pass-rate tied 8/8, the forest still spent 56% fewer calls, 80% fewer tokens, 76% less wall-time — the bare model only “ties” by flailing through retries. And hard ≠ underivable: three genuinely hard algorithms (edit distance, Kadane) tied, because the 7B memorized them in training. Difficulty doesn’t create the edge; underivable information does. Point the benchmark at interview questions → tie. Point it at your code → blowout. That’s the wedge.

Side-quest: the whole mind, as one graph

Somewhere in the middle, a detour worth keeping. The /flow/ page is 17 Mermaid diagrams; a parser fused them into one navigable graph — 169 components, every layer, drag/zoom/search/click-to-walk — plus a plain-text export for pasting into an LLM. Fed that text to ChatGPT cold. It read the dogfooding loops feeding the corpus feeding the train daemon, the doubt-gate reading the calibration curve, and said: “oh. this isn’t an LLM wrapper.” The architecture refused the category, and a frontier model reached the conclusion on its own.

Evening: teaching it JavaScript itself

The forest had learned your repo (B.2) and operations (B.3). The last move of the day pointed it at the language. We parsed the TypeScript stdlib .d.ts files into 359 foundation leaves — every Array, Object, String, Math, Map, Promise method with its real signature + JSDoc — so the coder can retrieve the exact API instead of half-remembering it. Two bricks, and the full receipt is B.4.

Brick #1 — hybrid retrieval, and a finding that inverted B.1. TF-IDF nails direct terms but whiffs on synonyms (“round down” → wants Math.floor), so I added a nomic-embed-text arm and went to fuse them with RRF — expecting the wiki-forest result where embeddings + RRF beat everyone. It got worse. On terse stdlib docs the embedder is noisy enough that RRF’s “reward agreement” rule buried the confident hits — Array.map was the #1 lexical match for “transform each element,” and RRF dropped it off the page behind leaves that were merely mediocre-in-both. The fix was interleave instead of sum (round-robin merge the two rankings) — each arm contributes its best picks directly. Lift was real but small (recall@5 5→6).

AHA: same machinery, opposite corpus, opposite answer. RRF + embeddings won on synonym-rich Wikipedia prose and lost on terse near-duplicate stdlib signatures. The honest ceiling here isn’t the retriever at all — it’s vocabulary: Set’s docstring never says “duplicate,” so nothing can find it by that word. The fix is curation (task-language aliases per leaf) — which is just the MMM thesis pointed at the corpus.

Brick #2 — wire the signatures into the coder. The payoff. Same 7B, one shot, no retry loop, so the only variable is whether injecting real signatures cuts first-try hallucination. Six tasks favoring modern stdlib (toSorted, findLast, .at(), replaceAll), several with non-mutation checks the lazy .sort() answer fails. Forest 6/6, blind 5/6, stable across three runs — and the single win was exactly the predicted footgun: “second-to-last by negative index,” where blind wrote arr[-2] (silently undefined in JS) and the forest, handed at(index: number), wrote .at(-2). Everything else tied, because qwen-coder:7b already knows common stdlib cold.

The law holds a fifth time. The library helps exactly where the model half-remembers the API — a footgun, a brand-new ES method — ties where it knows the API from its weights, and never loses. The edge is thin on a strong 7B’s home turf and widens on the gotchas. The next lever isn’t a better retriever; it’s alias curation — making the corpus speak the language of tasks, which lifts recall and hit-rate together.

Night: the forest’s first loss — and the fix

Then we pointed it where the model is actually weak. The stdlib edge was thin because a 7B knows Array.map cold; the law says aim at underivable surface, and Node core churns (fs.rmdir→fs.rm, new crypto.randomUUID) so the model holds stale answers. 307 Node-core leaves from @types/node, six fs/crypto tasks, run-graded against a real temp dir. Full receipt: B.5.

First run, 5/6 vs 5/6 — and the forest’s first recorded LOSS. It won “generate a UUID” beautifully (blind reached for import 'uuid', a package that isn’t installed and fails; the forest knew crypto.randomUUID is built-in now — the stale-weights wedge in one task). But it lost “recursive mkdir”: retrieval injected JSON.parse as the top hit, and that noise displaced an answer the model already had — it hand-rolled a recursive mkdir that stack-overflowed. The blind coder, left alone, got it right.

The law grew a second clause. Not just “the library helps where the answer is underivable” — also “and it hurts where it injects noise over a fact the model already has.” A retriever that can’t tell signal from noise is a liability.

So we built the fix, and it’s the brick I’m proudest of today: a confidence gate (only inject retrievals above a similarity floor — if nothing’s confident, inject nothing and let the model fall back to itself) plus alias curation (curated task-language per leaf — fs.rm → “delete / remove a directory recursively”; the docstring says what an API is, the alias says what a task calls it). The mkdir retrieval went from [JSON.parse, mkdirSync, mkdir] to [fs.mkdir, fs/promises.mkdir, path.dirname]. Loss → win. Forest 6/6 vs 5/6. Recall on the eval set rose 6/10 → 8/10. With the gate in, the forest no longer loses anywhere we’ve measured — it supplies a fact or it stays quiet.

AHA: the strongest brick of the day wasn’t a feature, it was teaching the library to shut up when it isn’t sure. The honest loss bought a better system than another win would have.

Full disclosure: the open weights come out

Said plainly on the Benchmarks page and worth saying here. Today the code forest still wraps an open-weight 7B — the library routes, retrieves, gates, and grades, but the generation is still borrowed. That’s not the destination. The goal is to remove the open weights entirely: every brick moves work from the model into the library — facts it no longer needs to know, graders that certify their own training data — until the generator can shrink to a small model we trained and the borrowed weights come out. These benchmarks are the scoreboard for that migration, run in the open. The day the forest wraps a model we own and still beats the open-weight baseline is the day they’re gone — and we’ll post that number too.

Footnote: the dump-ingest lesson

The MMM.1 walk stage confirmed you can’t fetch at scale from the live Wikipedia API — 429s by 100 pages, bursts deepen the penalty. The 10k “run” stage has to ingest from a downloaded dump. No live API at all. Walk taught us where run can’t go.