Day 62 — 'the difference between a wash and a win is the reader.'
We picked up the Month 2–3 plan — fix the input side of the brain, then close the local-reader gap — and the day became a clinic on why you measure instead of guess. A query-expander that looked like a clean +33pp win got rejected by the live gate because, fed to the small local reader, it lied exactly as often as it fixed. Then the frontier reader read the same evidence with zero lies. Same library, same candidates, opposite outcome: the gap was never the retrieval, it's the hand that reads it — and that hand is distillable. Along the way a 3-row smoke test lied by being small, the drift-proof fix turned out to be union not threshold, and the harness that measures the reader gap turned out to also harvest the training set to close it.
“the difference between a wash and a win is the reader.”
Yesterday we took a frontier model apart and found werbos owns every part except the reasoning weld. Today we picked up the plan that follows from that — Month 2 (fix the brain’s input side) and Month 3 (close the gap between the local reader and the frontier) — and it turned into a running lesson on measuring instead of guessing.
DIAGNOSE — the loop learns where to fix
The Honest Loop already caught a lie, turned it into a living-gold row, and gated a fix. But
a captured miss said nothing about which layer to fix — enriching a leaf for a fact that
isn’t in the corpus, or adding corpus facts for something that was fetched but misread, is
motion without progress. diagnose.py classifies each miss deterministically — no model, so
it can’t bribe its own verdict — with two coverage probes: is the answer in the corpus at
all? then is it in the top-k? That splits every miss into library_gap, retrieval,
reading, or abstain_miscal, weakest link first. The per-class tally over the living gold
is the Month-2 scoreboard.
The input compiler — enrich the question from the leaves
The biggest lie source isn’t a weak reader — it’s asking the library the wrong question. So we enrich the question from the library itself: rank the raw query, read the top hits, harvest their highest-signal terms, append them, re-rank. Classic pseudo-relevance feedback — the MMM.3 lesson (leaf enrichment works, routing doesn’t) turned on the input side.
It didn’t work the way we expected, and that was the interesting part. Blanket expansion drifted: it lifted some queries and dragged an already-perfect one from 100% to 50%. The obvious fix — only expand when retrieval looks weak — died on contact with the data: top-1 cosine score is inverted here, a confidently-wrong query outscoring a quietly-right one, so no threshold can tell the two apart. With no answer key at inference time, that’s just the retrieval problem restated. The honest fix isn’t to choose between raw and expanded — it’s to union them: keep the raw top-k verbatim, then append only the expanded query’s novel hits. Because the raw set is always retained, recall can only rise. Regression-proof by construction. (+33pp on the real corpus, no drift.)
The gate did its job — twice
Then we asked the real question: does a wider candidate set make the reader lie? We threaded a retrieval seam through the reasoner (default unchanged, so 145 tests stayed green) and ran the live 35-row bench on the local 7B, raw retrieval versus fused. Verdict: lies 6 = 6 — a dead-even wash. Fusion induced two lies and fixed two others; accuracy dipped two points; the gate rejected it. And a 3-row smoke we’d run first had screamed “fused causes lies” — it had only seen the induced side. The n=3 smoke lied by being small. Never conclude a live-reader A/B from three rows.
The reader-gap — measure the moat, harvest the training set in one pass
The wash is the whole story: the same wider evidence made the reader fix some rows and break
others. The only variable is whether the reader can take the answer and reject the distractor.
So we built reader_gap.py — read the same gold over the same retrieval with two readers, and
flag every row the teacher gets and the student misses. On the exact rows the local reader lied,
a frontier reader read the identical fused candidates with zero lies (75% vs 25%). That
rules out the alternatives cleanly: not retrieval (the answer was in the set), not the grader
fence (same fence — a loose fence would’ve let the frontier lie too). It’s reading
comprehension, and reading comprehension is distillable. The payoff: the rows where teacher
beats student are the distillation set — the teacher’s grounded, cited answer is the label.
The pass that measures the gap builds the set that closes it.
Where it leaves us
Month 2 shipped (diagnose, input compiler, drift-proof fusion). Month 3 has its thesis proven and its first instrument built: fused retrieval is a wash on the local reader and a win on the frontier, and the difference is distillable. Nothing is wired to the hot path — every seam defaults to the safe behavior — and the gate refused everything that didn’t earn its place. The loop worked exactly as designed: it caught a plausible win that wasn’t, and a small sample that lied. The reader is the moat now, and we can measure it.