B.22 — the abstain autopsy: honest gap, or a card we could not reach?

Tue, Aug 25 2026

Honest-abstain is the moat, and we had no way to tell a principled silence from a retrieval failure — library_lookup returns the identical None for both. A second, independent reader over the same 167,493 cards splits them apart. First run: on the adversarial set, ZERO of thirteen silences were honest gaps. Every typo, homophone and idiom miss is a card we already own and cannot reach, which disproves the layman-vocabulary minting plan queued the day before.

T.M.B. #22 — measuring the thing we are proudest of. Every prior bench asked whether werbos answers correctly. This one asks whether its silences are honest, and finds that most of them are not — they are reachability failures wearing an abstain’s clothes. Frozen gold sets, model-free scoring, and a verdict that can only fire against a library the fixture named in advance. Re-run it: python scripts/reachability.py --verbose.

The claim under test

Honest-abstain is the moat. It is in every README, it is the reason a 336M model is not handed a microphone, and it is the property the whole cascade exists to protect.

It also has a hole in it that nothing had ever measured. An abstain has two causes that want opposite fixes:

HONEST GAP     nothing in 167k cards covers this      -> MINT CARDS
UNREACHABLE    the card is there and the gate blocked  -> FIX RETRIEVAL

library_lookup returns the identical None for both, because it hands back the one best card and discards the ranking. So a near-miss and a genuine absence are indistinguishable, and “we abstained honestly” has been covering for both.

The claim under test: when werbos stays silent, the library genuinely does not know.

The setup


Corpus	375 libraries, 167,493 cards (`werbos-libraries`)
Gold	`chat_gold.jsonl` 52 rows · `drunk_gold.jsonl` 25 rows — the same frozen sets B.21 used
Instrument A	`library_lookup` — TF-IDF cosine, `threshold=0.50`, returns one card or `None`
Instrument B	`knowledge_hub.search` — BM25-ish over a term index, no cosine gate, reports per-card `coverage`
Verdict rule	UNREACHABLE only when B returns a coverage-1.0 card in the library the gold row already named
Scoring	model-free and deterministic — no reader booted, no model in the verdict

Two readers is the whole method. Neither instrument can diagnose itself; the gap between them is the diagnosis. B exists because the knowledge hub shipped the same day — 375 libraries had been an answer substrate since day one and nothing could read them.

What makes a verdict falsifiable. “The hub found something” proves nothing: a term index over 167k cards finds something for lol. A row only counts when the hit lands in the library the gold fixture named before this bench existed, so there is no moving the target after seeing the result.

Results

                    silent   unreachable   partial   honest-gap
  chat_gold           11          4           4          3
  drunk_gold          13          3          10          0

silent = a routing row that came back empty. Zero honest gaps on the entire Drunk Jason set.

Every typo, homophone and idiom miss is a card we already own and cannot reach. That directly disproves the plan queued at the end of B.21 — more layman-vocabulary minting would have fixed none of them.

The sharpest single row:

'what does calling abort() on an AbortController do?'          -> ABSTAIN
hub [web-api] 'What does calling the abort() method on an AbortController do?'
              coverage 1.00

Near-verbatim, right library, and silent. Not an honest abstain by any reading.

What actually moved the number

Nothing in this bench changes behaviour — it is an instrument, not a fix. But it pointed at one, and the fix is measurable both ways.

The AbortController row abstains at threshold=0.30, not just 0.50, which rules out the gate as the cause: the card never enters the pool. B.21’s charness scope killed the big-library bias and installed a small-library one — my-recipes (131 cards) outranking web-api (22,458) on the words calling and do. Damping the denominator (size ** 0.90, swept in B.23) moves this bench without being tuned against it:

  chat_gold      before      after
  silent           11          8
  unreachable       4          2
  answered         10         13

Two independent instruments, agreeing.

The honest caveats

coverage is measured over indexed terms, not all query terms. knowledge_hub.search drops a term no card holds from the denominator (deliberately — otherwise every result caps below 1.0 and the ranking flattens). So coverage 1.0 means “every term the corpus can match”, not “every word you typed”, and UNREACHABLE is therefore an over-count in principle. The AbortController row survives inspection; the other five were not individually audited.
partial is not a verdict. 10 of 13 drunk rows land there, and the entry does not claim to know which way they go. Reading “0 honest gaps” as “the corpus is complete” would be wrong — it says only that no silence was provably a gap.
The hub would bluff. On rows werbos correctly stayed silent for, the hub returns a coverage-1.0 hit 6 times on chat_gold and 4 on drunk_gold — lol → an HTML <abbr>LOL</abbr> card, what is a flurbnax made of? → an accounting card about debits. This number is reported rather than hidden because it is the evidence that the hub is a diagnostic and must never be wired to resolve().
77 gold rows is thin. chat_bench’s own docstring says a single label flip moves the headline ~5 points. Treat a 1–2 row movement here as noise.
drunk_gold grew 20 → 25 rows since B.21, so its silent counts are not directly comparable across the two entries.

The verdict

The moat is narrower than advertised. werbos’s silences are honest in intent — nothing bluffs, no card gets invented — but on the adversarial set not one of them was a genuine gap in the library. The library knew. The cursor could not find it.

That reframes the roadmap: the next unit of retrieval work is worth more than the next unit of minting, and until today there was no way to know which.

Standing rule this establishes: an abstain is what a healthy honest system looks like, which is exactly why it must never be able to mean something upstream broke. Same day, a missing library on the deploy box produced a day of principled-sounding silence with a green dashboard. Two faces of one failure, and this bench is the only thing that can tell them apart.