B.3 — verb forest: graded operations, not retrieved signatures

T.M.B. #3 — the judge is the test runner and the compiler. A verb either makes the code compile / the tests pass, or it doesn’t. node --test, tsc --strict, and a hidden reference test do the grading. No vibes.

What’s different from B.2

B.2 retrieved function signatures (nouns) and let a generic coder do the work. B.3 flips it: the leaf IS the operation (a verb). add-test, fix-type-error, implement-function — each an operation capsule: a curated prompt + worked exemplars + a free grader, run in a grade-retry loop the leaf owns. The conductor routes a request to a verb outside the model, then runs it.

The setup

Six tasks, two per verb, on fresh code fixtures. Two contestants — the same qwen2.5-coder:7b on both arms:

• forest — routed to the verb capsule (curated prompt + exemplars + a deterministic fact like the exact import path), then the grade-retry loop.
• blind — a bare, one-line instruction (“write tests for this module”), the same model, the same grader, the same retry budget (×3).

That design is deliberate: both arms get the free grader and the retry loop, so a forest win can’t be credited to the loop — it isolates the value of the library itself (the capsule’s prompt, exemplars, and the facts it injects).

The results

What it says

The win concentrates where the model needs a fact it can’t derive. On add-test, forest went 2/2 and blind went 0/2 — and the reason is sharp: the blind coder was handed the source code but never the filename, so it hallucinated an import ('path/to/string-utils.js') and the grader could not rescue it. No amount of “module not found” feedback recovers a filename the model was never given. The capsule injects that fact ({import_path}) deterministically. The retry loop fixes what the model can fix; the library supplies what the model can’t.

Two verbs tied — and we’re keeping that on the page. On fix-type-error and implement-function, the bare prompt + grader-retry loop was enough: blind went 2/2 there too. The capsule didn’t help, because the model could derive everything it needed from the code plus the compiler/test feedback. That’s an honest wash, and it’s the credible part — if every row were a blowout, you should distrust the bench. The library earns its keep on the tasks with a hidden gotcha, not uniformly.

So the honest claim is narrow and true: a routed operation capsule beats a bare prompt exactly where the operation hides a fact the model can’t reverse-engineer — graded by tools that can’t be sweet-talked. As the verb library grows, the number of those facts it encodes grows with it.

Efficiency — the win that survives the ties

Pass/fail is one axis. It’s also the one where an easy, derivable task will eventually tie — the bare model flails through retries until the grader’s feedback drags it to the answer. So we tracked the axes it can’t fake: tries, tokens, wall-time. Here’s a run of 8 verbs harvested straight from werbos’s own skill catalog (graded by each skill’s format contract), where pass-rate tied 8/8 vs 8/8:

Same outcome, radically different cost. The bare model gets there — by burning 5× the tokens and 4× the wall-clock retrying until feedback corrects it. The capsule nails the contract on the first try. So even on a dead-even pass-rate, the forest wins every cost metric. Local inference is $0 in API spend; the savings that matter are tries, tokens, and seconds — and those favor the library on every task, including the ties.

What difficulty does NOT buy

A tempting next move: “throw harder questions at it, the gap will widen.” It doesn’t. We gave implement-function three genuinely hard algorithms — edit distance, longest-unique-substring, Kadane’s max-subarray with the all-negative trap — graded by a hidden reference test. Result: forest and blind both passed, first try, identical tokens. A dead tie.

Why: those algorithms are canonical. The 7B coder has seen them a thousand times in training — the answer is in its weights, derivable from memory. Hard is not the same as underivable. A textbook-hard question the model memorized ties; a trivial question about your repo’s filename is a blowout (see add-test). The forest’s edge tracks underivable information — your code, your conventions, your domain — not textbook difficulty. Point a benchmark at interview questions and you’ll measure a tie; point it at your codebase and the library is load-bearing. That’s the whole wedge.

Honest caveats: N = 6 (2 per verb — small; more tasks per verb would harden it). No claude arm this round — B.3 measures the question that matters most here, “does the library beat the same model’s raw self?”, not the frontier ceiling. Both arms: qwen2.5-coder:7b, temp 0.1, ≤3 retries. Graders: node --test, tsc --noEmit --strict, and a hidden reference test. Code + fixtures in the mmm-poc repo — run it yourself.