B.11 — three rulers: what was broken the whole time?

T.M.B. #11 — the rulers turned on ourselves. B.1 through B.10 asked “is the model good?” This one asks “is anything we believe about the system actually true?” We built three deterministic benchmarks in a day and pointed them at code that had been shipping for weeks. Both production bugs they found were one-line fixes. Neither was findable by reading. Frozen probe sets, deterministic metrics, no model in the verdict — re-run them yourself.

The claim

The system has an un-bribable ruler for the reader (the Gauntlet) and held-out loss for the talk model. Everything else — which card retrieval picks, what the owned model actually says, which tier of the cascade answers a turn — was judged by reading replies and forming an opinion.

The claim under test: taste is not a ruler. If you cannot put a number on the surface that ships, you are not measuring it, and things break in ways nobody notices.

The setup

Three benches, all deterministic, all with no model in the verdict — the discipline from VERDICTS.md: a ruler you can talk your way past is not a ruler.

• chat_bench — 68-row frozen gold set (chat_gold.jsonl), labelled by behaviour not answer text: must-abstain (small talk, word problems, nonsense, uncovered facts, collision traps) vs must-route (the winning card must come from a plausible library). Scores ABSTAIN accuracy (a bluff = a confident card where silence was right) and ROUTING accuracy.
• talk_bench — 34 frozen probes across 7 owned checkpoints, five axes: CLEAN, RESPONSIVE, ADJACENCY, STABLE, OOD. Babble is judged against the 236,456-word system dictionary, which predates this repo and nothing here can influence. Floors, not a composite — fluency and responsiveness trade off, so an average lets a model buy a score by being pleasant filler.
• cascade_bench — 28 turns through the real brain_reply, asserting only which tier answered. Deterministic by sealing: real library lookup, but seek returns nothing, compute abstains, verify refuses, and cache raises — so resolve()’s G2/G3 honesty guarantees are asserted rather than assumed. No web, no Claude, no ollama.

Results

Retrieval — the gates were the problem, not the libraries.

Decomposition of the 16 routing failures: threshold 8, min_gap 5, genuine coverage gaps 2, library sprawl 1. The 55,457-card, 127-library surface everyone assumed was the problem cost one point out of twenty-one. Two tuning constants cost thirteen.

The owned talk model — one constant, 0/7 → 7/7.

Cascade tier routing — 26/28 (92.9%). abstain 7/7 · reasoning 4/4 · social 3/3 · intelligence 2/2 · code 2/2 · library 2/3 · owned-talk 6/7.

What actually moved the number

A relative test making an absolute decision. min_gap abstained whenever the top two cards scored close, reading a tie as ambiguity. In a well-covered library a tie usually means redundant correct coverage — two cards that both answer the question. “how do i treat a minor burn?”: the right card at .488, another right card at .454, gap .034 → None. It punished the library for being thorough and got worse every time a library was deepened. Ties are an artifact of cosine geometry on short queries, not a correctness signal: the same scorer returns a .744 hit with a .395 gap on “hey there”.

A character budget wearing a token budget’s clothes. reply() capped generation at max_new=64 tokens, and every shipped talk checkpoint is byte-level (vocab=256) — so 64 tokens is 64 characters, roughly ten words:

max_new=64   'I know right? I hope it feels like the planning of the day at a'
max_new=400  'I know right? I hope it feels like the planning of the day at a time.'

The models were not failing to finish. They emit their stop token happily a few characters past the cap; we stopped listening. This was live in the chat’s owned-talk tier.

And it overturned B.10’s own read. Comparing truncated output made the 5M look more fluent than the 30M. Once both can finish a sentence, the 30M is clearly the better model — talk-base-clean/talk-5m answers both “tell me about the ocean” and “tell me about mountains” with the identical “Sounds like a plan! What’s your favorite part of the day?”. Textbook filler collapse: low loss, zero conversational value. B.10’s verdict — ship the 30M — stands, for a reason B.10 could not see.

One ruler caught another lying. chat_bench filtered out rows the cascade answers upstream, to avoid measuring the library against inputs it never receives. It asked whether speech_act classified the turn. “you’re really good at this” classifies as act='other' — but respond() returns None for it, so brain_reply falls straight through to retrieval. The row was being excluded as handled while reaching the library in production. Classified is not handled. Fixed, the abstain set grew 24 → 31 rows and still scores 31/31.

The honest caveats

• All three measure floors, never ceilings. Every axis is a necessary condition. None is sufficient. Answer quality is unmeasured by every bench in this repo.
• RESPONSIVE measures variation, not relevance. Two different generic fillers both score as responsive. Judging whether a reply is actually about the ocean needs a judge model, and a model in the verdict is the one thing these must not have.
• The talk floors are made up and currently lenient. With the truncation bug fixed, 7/7 pass. A gate everything passes is decoration; they need raising against this baseline.
• The labels are one operator’s judgment. One was already wrong — I assumed sewing was uncovered; everyday-life holds a button-sewing sequence. A disagreement with a gold set is a claim to check, not a verdict.
• Thin. 21 routing rows, 28 cascade turns, 34 talk probes. A single label flip moves a headline several points.
• The web and Claude paths are sealed, therefore untested. Sealing is what makes cascade_bench deterministic, and it means the live seek → read → verify → cache loop — the part that actually learns — never runs. An abstain here may be a grounded answer in production.
• OOD is 0/7 and gated at zero. No checkpoint signals confusion at “glunglu florp?”. Tracked as debt, not solved.

The verdict

Both production bugs were one-line fixes, and neither was findable by reading. A tie-breaker that looked like prudence was costing 8 of 21 correct answers. A generation cap that looked like a sane default was truncating every reply over ten words. A test asserting “never bluff on nonsense” had been red on main and unnoticed, because library growth quietly invalidated its assumption.

The pattern is the uncomfortable part: none of this was hard. The engineering was fine. The feedback loops were the weak link — and the fix for a missing feedback loop is never clever, it is just doing it.

What these rulers still cannot settle: factual routing is 5/14, and the right card frequently exists and loses — geology holds “What is an earthquake?” and still misses “what causes earthquakes?”. Nothing shows the 55,457 cards earn their keep. And the biggest assumption in the project — that honest-abstain is the moat — remains untested by construction, because no bench can tell you whether abstaining reads as trustworthy or as broken. That one needs a person who isn’t us.