B.1 — werbos(mmm) vs ollama vs claude · v1 → v3

T.M.B. #1. The first frozen baseline — and the one every future run gets diffed against. Question set hashed (f5a1d970), forest pinned (36f2dc71…, 8,903 leaves), dated. Re-runnable. No vibes.

The setup

11 questions, three contestants, graded on whether the answer contains the known fragment. The question set is split on purpose:

• 8 corpus questions — obscure facts that are in the forest’s ~1000 Wikipedia pages (a sturgeon’s discovery year, an alligator gar’s binomial, a state park’s length). The kind of thing a 3B can’t recall from weights.
• 3 general questions — common knowledge not in the corpus (capital of France, who wrote Hamlet, gold’s symbol). The forest should abstain.

Two forest rows: w-find = did it retrieve the answer (deterministic, no model — the baseline line); w-say = did the reader synthesize it.

The results

The green w-find line is the story: 6 → 6 → 7. Tied Claude on plain TF-IDF. Stayed tied when we swapped in embeddings (a wash — gained two “meaning” queries, lost two “exact number” ones). Then hybrid retrieval (lexical + semantic, fused by Reciprocal Rank Fusion) kept both — and crossed over Claude’s flat 6.

The upgrade log (why each version moved or didn’t)

• v1 · TF-IDF — lexical word overlap. w-find 6/8, tied Claude. Missed the two “concept” queries (a fish’s scientific name) where the question and the answer share no literal words.
• v2 · embeddings — semantic routing (nomic). Caught both concept queries… but lost two exact-number queries (a founding year, a park’s length), because semantic match blurs specifics. Net 6/8 — a wash. Posted honestly, because the frozen eval wouldn’t let it be hidden.
• v3 · hybrid (RRF) — rank by lexical and semantic, fuse the ranks. Keeps the concept wins and the exact-fact wins. w-find 7/8 — past Claude’s 6/8. The in-domain factual-recall ceiling, broken on the hashed eval, one variable changed.

Read the honest caveats: the corpus column (7/8) is the clean signal — the overall number is inflated by a 2-char fragment (gold’s Au) that substring-matches half the corpus, so we don’t quote it. And w-say is stuck at 5/8: the forest now finds 7 but the 3B reader only says 5. The next lever isn’t routing — it’s the reader.

What it says

The library works — 3×. On corpus facts, werbos’s retrieval went 6/8; the same 3B without the library went 2/8. Identical model, identical weight class. The only difference is the forest. That’s the entire thesis, measured against the opponent that actually matters.

It tied frontier on its own turf. werbos w-find 6/8 = Claude 6/8 on the corpus questions. A 3B + a 6.8 MB file matched a frontier model on factual recall in its domain. Not 7–10× behind. Even.

Off-domain, it shut up. On general knowledge the forest scored 0–1/3 — it abstained (“no leaf covers that”) instead of bluffing, while Ollama and Claude (3/3 each) just knew it. That’s not a loss; it’s the specialist working as designed. A tool that crushes its domain and stays quiet off it.

Claude wins overall (9/11) — as it should. It’s the ceiling, not the opponent. The fight that matters is the other two rows, and werbos took it: 7/11 vs the raw model’s 5/11.

How to beat this baseline

The single number to move is w-find corpus = 6/8. It only changes when the corpus or the routing changes — not when the homelab is busy. Add embeddings, add pages, tighten the gate, then:

python battle.py --save --version v2

…and the line above grows a second point. That’s the chart that goes up — and unlike the IPO deck, every point is hashed.

Conditions: reader on local llama3.2:3b, all 11 reader calls fired. Saved to baselines.jsonl.