Day 43 — 'trust me bro — except here's the hash.'

“trust me bro — except here’s the hash.”

The day werbos got a scoreboard — and the scoreboard told the truth, including the parts I got wrong.

The forests, reachable

Deployed the two harmless forests as sibling apps on the tailnet — the MMM glass-box (the trained angler + trekker leaves, route/compose/abstain) and the ~1000-page wiki forest — and added 🌲 Forest links to the dashboard sidebar. One discipline held all night: the toy forests get networked; Mythos — the local cited hacking brain — stays offline. The cage is the feature.

A scoreboard, frozen and hashed

Then the real work: a benchmark you can’t fudge. werbos(mmm) vs raw Ollama vs Claude, 11 questions, split into corpus (obscure facts that are in the forest) and general (common knowledge that isn’t). The question set hashed, the corpus pinned, the metric — did the forest retrieve the answer — deterministic.

First clean run (local Ollama, so the open model actually swung instead of timing out): on its own domain, the forest’s retrieval beat the raw 3B 6/8 to 2/8 — 3× — and tied Claude at 6/8. Off-domain it scored 0–1/3 and abstained — “no leaf covers that” — instead of bluffing. The honest specialist: crushes its turf, shuts up off it.

Breaking the ceiling — with a wash in the middle

The upgrade arc, every step on the same hashed eval, one variable changed:

The middle row is the whole point. I bet embeddings would jump it to 8; they came back 6 — flat. They caught two “meaning” queries (scientific name of…) and lost two “exact number” queries, because semantic matching blurs specifics. A hype deck buries that and says “embeddings made it better.” The hashed eval wouldn’t let me — v2 said 6, the questions were identical, I had to post the flat line. And that’s what made v3 real: hybrid retrieval (lexical + semantic, fused by reciprocal rank fusion) kept both — and crossed over Claude’s 6 to 7. The in-domain factual-recall ceiling, broken, with the wash sitting right there as proof the number wasn’t massaged.

Then it learned to code

The best part. Pointed the same hybrid forest at the dashboard’s TypeScript — 446 declaration leaves, one per function/type/interface. Ask it “how do I fetch the dream journal” and it returns interface Dream, getDreams, the <Dreams/> component — the real signatures, cited. Not a hallucinated API. Your repo, which no frontier model has.

Then the verb-domain benchmark, and the grader is the thing that makes code special: tsc. No fragment-match, no vibes — it typechecks or it doesn’t. Three tasks that require the repo’s real shapes. Three contestants. Same coder model where it counts:

Same model — with your repo 3/3, blind 0/3. And the blind 7B couldn’t recover even with compiler-error retries — it got errors that spell out the real type and still failed all three. A small model can’t reverse-engineer your codebase from tsc messages; it needs the retrieval. A compile-feedback loop (generate → tsc → feed the error back → retry) took the forest from 1/3 to 3/3, tying frontier. The honest claim isn’t “beats Claude” — it’s “a small local coder + your repo matches frontier on your codebase, and is infinitely better than the same model without the library,” graded by a compiler that can’t be sweet-talked.

The AHA

The whole industry sells restraint it can’t prove — “too dangerous to release,” stock goes up, nobody sees the product. werbos posted the numbers: frozen, hashed, re-runnable. And they said something real — a 3B + a library ties frontier on factual recall in its domain, a small coder + your repo matches frontier on your code — measured, not asserted. The benchmark even caught my wrong bet and made me show it. That’s the opposite of trust me bro. That’s here’s the hash.