B.5 — Node core: the forest's first loss, and the fix

T.M.B. #5 — the judge runs the function against a real disk. Every task is graded by importing the generated async function and running it against a fresh temp directory. The wrong idiom doesn’t look wrong — it throws, or returns undefined, or stack-overflows. No vibes.

Aim where the model is weak

B.4 found the JS-stdlib edge thin: a 7B knows Array.map cold, so retrieving its signature ties. The law says the win lives in underivable surface — and Node core is exactly that. It churns (fs.rmdir → fs.rm, new fs.cp / crypto.randomUUID), so a model trained on older code holds the wrong answer in its weights. We parsed @types/node into 307 Node-core leaves (fs, fs/promises, crypto, path, os, util, stream, …) and pointed the same coder at six file-system / crypto tasks.

The first run: a tie with a trade — and our first loss

Same qwen2.5-coder:7b, one shot, no retry loop. The result was 5/6 vs 5/6, and we’re putting both halves on the record:

The win is the thesis in one task. For “generate a UUID,” the blind coder wrote import { v4 } from 'uuid' — the model’s training says npm-install a package — which isn’t installed and fails at import. The forest knew crypto.randomUUID() is built-in now. That’s the stale-weights wedge: the local library holds the current answer the model’s weights got wrong.

The loss is the more valuable half. For “recursive mkdir,” retrieval injected noise — JSON.parse ranked #1 for the phrase, plus the callback-form fs.mkdir — and that noise displaced an answer the model already had. It hand-rolled a recursive mkdir that stack-overflowed. The blind coder, left alone, got it right from memory. Retrieval made the model worse.

The law needed a second clause. It’s not only “the library helps where the answer is underivable.” It’s also: "…and it hurts where it injects noise over a fact the model already has." A retriever that can’t tell signal from noise is a liability, not an asset.

The fix: a confidence gate + alias curation

Two changes, both aimed at injection quality:

• A confidence gate. Only inject retrievals above an absolute similarity floor (kills JSON.parse showing up for a mkdir task) and within a relative cliff of the top hit. If nothing clears the floor, inject nothing — the coder falls back to its own knowledge, which the loss proved is often right. The library is no longer allowed to talk when it isn’t sure.
• Alias curation. Curated task-language per leaf, folded into both retrieval arms — the docstring says what an API is, the alias says what a task calls it (fs.rm → “delete / remove a directory recursively”; crypto.randomUUID → “uuid / guid”). This is the MMM thesis applied to vocabulary: hand-curation doing the work the corpus’s own terse words can’t.

The mechanism is visible in the retrieval itself — the mkdir task went from [JSON.parse, mkdirSync, mkdir] to [fs.mkdir, fs/promises.mkdir, path.dirname].

The result

Stable across three runs each. Retrieval recall on the eval set also rose 6/10 → 8/10 from the aliases. And B.4 held at 6/6 with cleaner retrieval — no regression. With the gate in place, the forest no longer loses anywhere we’ve measured: it either supplies a fact the model lacks, or it stays quiet.

Honest leftovers: “deduplicate an array” exposed that a method-leaf corpus can’t represent multi-call idioms (Array.from(new Set(arr))) — folded onto Array.from as a stopgap, “idiom leaf” noted as a future type. “Get unique values” still misses (common words drown the rare one). The gate’s floor is one tuned constant, not a learned threshold. N = 6 per bench, 1-shot. Both arms: qwen2.5-coder:7b, temp 0.1. Code in the mmm-poc repo — run it yourself.

Full disclosure — where this is going

Be clear about the contestant: today the forest still wraps an open-weight 7B (qwen2.5-coder:7b). The library routes, retrieves, gates, and grades — but the generation is still an open model’s. So what are we actually measuring?

Progress toward removing the open weights entirely. The end state is a forest whose library carries enough — capsules, real signatures, graders, curated vocabulary — that the generator can shrink from a 7B open model to our own small model, and eventually be dropped to nothing borrowed at all. Every brick here is a step: each fact the library supplies is one the generator no longer has to know; each grader is verified training data we can distill into a model of our own (the “angler” move — the grader manufactures its own ground truth).

These benchmarks are the scoreboard for that migration. “Forest beats blind” isn’t the finish line — it’s the evidence that the library is taking over the work the model used to do. The day the forest wraps a model we trained, on data the graders certified, and still beats the open-weight baseline — that’s when the open weights come out. We’ll post that number too.