B.21 — the router had no libraries: charness on the card path

T.M.B. #21 — the moat had no shelves. Every prior bench measured whether the library holds the right card. This one measures whether retrieval can find it among 374 libraries, and the answer was that nothing in the answer path knew libraries existed. Frozen gold sets, model-free scoring, both directions reported. Re-run it yourself: python scripts/chat_bench.py.

The claim under test

werbos’s thesis is that the library is the product and the model is a cursor. That claim has a load-bearing assumption nobody had measured: that the cursor can find the right shelf. B.19 scored the composer, B.20 scored the weights. Neither asked whether 374 libraries could be routed among.

The claim under test: retrieval quality scales with the library, not against it.

It does not. It degrades — and the degradation is structural, not a tuning problem.

The setup

drunk_gold exists because the operator’s actual input is not the clean-prompt distribution: typos, homophones, idioms, bare tokens. Every trap row is paired with its literal twin — fishing for grouper next to this fishing email looks sus — so a fix that trades one misroute for another cannot hide in the headline.

Results

                     chat_gold                          drunk_gold
              ABSTAIN   ROUTING   OVERALL          ABSTAIN  ROUTING  OVERALL
  before       21/31      8/21     29/52            2/8      2/12     4/20
  after        28/31      7/21     35/52            6/8      1/12     7/20

Ten bluffs became three. Overall +6 on chat_gold, +3 on drunk_gold. By kind, the movement is not uniform:

What actually moved the number

The pool was flat. _load_cards concatenated all 374 libraries into one list and make_library_lookup took a global cosine argmax over it. There was no per-library step anywhere in the answer path. web-api (22,458 cards) therefore held ~478× more argmax tickets than angular-basics (47), and the pooled IDF deflated exactly the terms that make a small library distinctive.

Observable, not theoretical — eight consecutive misses landed on spanish, a vocabulary library whose 13,242 cards gloss most common English words:

"which bucket should i use to mop the kitchen floor?"  ->  el trapeador
"what is my wife's birthday?"                          ->  El cumpleaños
"that pitch was fire"                                  ->  presentación de negocio
"touch grass"                                          ->  el tacto

Maximum lexical surface area, near-zero topical relevance. Nothing was wrong with those cards.

The fix already existed and could not be reached. intent.select_library has scored cross_idf(t) × df(t,L)/size(L) since Day 12, with tests. But it consumes CorpusIndex chunks, not cards, and resolve() has no path to it. Dividing by library size drops spanish 45× below first-aid on the mop query and 178× below singing on “that pitch was fire”.

Two corrections the measurement forced, neither of which was obvious beforehand:

An ambiguous top must narrow to a shortlist, not surrender. Falling back to the full pool on a tie readmitted all 374 libraries and spanish won again — first-aid and everyday-life tied at 1.004×, which is not evidence for the 372 libraries that scored nothing. Swept: shortlist 3, 5 and 10 are identical; 1 loses routing; unbounded gives back the abstains.

A library rarely names itself. javascript’s 10,454 cards discuss closures and promises and seldom repeat the word “javascript”, because it is the ambient context. So the single token that most clearly identifies the shelf had near-zero charness on that shelf, and “what is a closure in javascript?” scoped to python — smaller, so its handful of closure cards divided to a bigger number. Adding the card’s own lang to the profile recovered code 2/6 → 4/6. Adding topic as well cost factual 4/14 → 2/14: it is per-card, so it pours hundreds of terms into every profile and stops discriminating.

The honest caveats

The bench was broken while it was measuring this. chat_bench.py hardcoded ROOT/"libraries", a path that stopped existing when the cards moved to a sibling repo on Day 82. lang_to_dir() returned an empty map, every routing row failed a comparison it could never pass, and ROUTING reported 0/21. The real number was 8/21. A clean zero on one axis with normal numbers on the other is the signature of a dead lookup, not a collapse — and it would have made this entry’s headline a fabrication.

ROUTING went down. 8/21 → 7/21. The system now abstains where it used to answer from spanish, which is the trade this project wants — but it is a real row and it is reported as a loss, not buried.

code went 5/6 → 4/6 even after the lang fix. One coding answer was traded for the abstain gains. That is the cost of scoping.

72 gold rows is thin. chat_bench’s own docstring says a single label flip moves the headline ~5 points. Treat any 1–2 row difference here as noise, including the ROUTING loss.

The labels are not the operator’s. Every expect value in both gold sets is a judgement call made by the author of the bench. One in chat_gold was already found wrong. Disagreements with the gold set are a claim to check, not a verdict.

This did not fix the thing that prompted it. layman-vocab stayed 0/2 and homophone 1/4, because hack appears 4 times in 167,389 cards and in no security library. No routing change reaches an answer the corpus cannot phrase. That is a minting problem and it is not solved here.

The verdict

The claim fails, and the failure mode is the interesting part. Retrieval quality does not scale with the library — it degrades, because a flat pool turns library size into retrieval authority. The moat gets deeper and the cursor gets worse at reading it.

Charness fixes the arithmetic. It does not fix the deeper thing this bench exposed: a lexical match is not evidence of topical relevance, and the retriever has no other kind of evidence. It now abstains far more honestly (ten bluffs to three) and still cannot tell “let me cook” from a recipe question without help from outside the query.

What would change the verdict: wiring the compiled term→library tables (termindex.py, built alongside this bench — 61% of the corpus vocabulary lives in exactly one library and nothing had compiled it) into resolve(), plus a conversation-branch signal so a three-token query has context to disambiguate against. If ROUTING still sits near 8/21 with both of those live, the ceiling is not retrieval and this bench found the wrong floor.