B.23 — the size exponent: charness traded one bias for the other

T.M.B. #23 — the fix for the fix. B.21’s charness scope was a real win that quietly created its own failure mode, and eyeballing an exponent is how you pick the number that flatters the query you happened to look at. Seven points swept over two frozen gold sets, three axes reported because they trade against each other, and the widest plateau chosen over the best single point. Re-run it: python scripts/sweep_libsize.py.

The claim under test

B.21 found that the retriever flattened 374 libraries into one pool, so the winner was whoever held the most tickets — spanish, 13,242 cards of English-word glosses, won eight benchmark queries outright. The fix scored charness: cross_idf(t) × df(t,L) / size(L). Dividing by library size drops spanish 45× below first-aid on “which bucket should i use to mop the kitchen floor?”.

Dividing by size also does something nobody measured: it makes small libraries cheap to win with.

The claim under test: the charness denominator is a fix, not a trade.

It is a trade. The exponent is where you choose which way to be wrong.

The setup

silent is new here and it is the point. B.21 counted a wrong library and an empty hand as the same miss. They are different defects — one is a bluff, the other is a reachability failure — and charness moves them independently. B.22 is what made the distinction visible.

Results

   exp  | chat ABSTAIN  chat ROUTE  silent | drunk ABSTAIN  drunk ROUTE  silent
  ------+-----------------------------------+---------------------------------
   1.00 |    28/31         7/21       11    |     8/10          2/15      13     <- B.21
   0.95 |    28/31         8/21       10    |     8/10          2/15      13
   0.90 |    28/31         9/21        8    |     8/10          2/15      13     <- chosen
   0.85 |    28/31         8/21        9    |     8/10          2/15      12
   0.80 |    26/31         7/21        9    |     8/10          2/15      11
   0.75 |    26/31         7/21        9    |     7/10          2/15      11
   0.70 |    27/31         7/21       10    |     7/10          2/15      11

0.90 buys three unreachable rows and two routing rows for zero abstain cost. Below 0.85 the ABSTAIN column collapses 28 → 26: that is the big-library bias returning and bluffing again, which is the exact defect B.21 existed to remove.

Two rows that were silent at 1.00 and answer at 0.90:

'what does calling abort() on an AbortController do?'  -> Web API — AbortController/abort
'how do i center a div with flexbox?'                  -> CSS — Use_cases

And the B.21 wins that must survive, and do:

'which bucket should i use to mop the kitchen floor?'  -> ABSTAIN   (not el trapeador)
'touch grass'                                          -> ABSTAIN

What actually moved the number

A 131-card library was beating a 22,458-card one on filler words. my-recipes won “how do i center a div” and “what does calling abort() on an AbortController do?” — a handful of hits on calling / do / center divided by 131 beats real coverage divided by 22k.

The consequence is worse than a misroute. The scope excludes the right library from the pool entirely, so the query abstains at threshold=0.30, not just 0.50 — which is why this looked like a gate problem for most of a day and was not one.

Two things the sweep corrected about the obvious implementation:

• The scope ranking is not the contract. At 0.90, my-recipes still edges css on the flexbox row (0.111 vs 0.089) — and it does not matter, because lib_shortlist=5 keeps css in the pool and the card-level cosine picks the right card out of it. A test asserting the scope winner fails on a query the system answers correctly.
• drunk_gold does not move at 0.90. Its silent count only drops at exp ≤ 0.85, where ABSTAIN starts to cost. That is consistent with B.22: the drunk set’s misses are phrasing, not scoping, and no denominator will fix them.

The honest caveats

• Fitting a knob against the same 77 rows that grade it is overfitting. Mitigated by taking the widest plateau rather than the best point — 0.95, 0.90 and 0.85 all hold ABSTAIN at 28/31, so 0.90 is mid-plateau and not a spike. It is still 77 rows.
• This is an interpolation, not a solution. exp slides between two failure modes; it does not remove either. A corpus that grows lopsidedly will need it re-swept, which is why lib_size_exp stays a keyword argument and 1.0 remains reachable as the pre-fix behaviour.
• ROUTING 7 → 9 is inside the noise band chat_bench documents (~5 points per label flip). The number worth trusting here is silent 11 → 8, because B.22 independently confirms it (unreachable 4 → 2, answered 10 → 13) using a different instrument.
• drunk_gold grew 20 → 25 rows since B.21. Its columns are not comparable across entries.
• ABSTAIN 28/31 is the number to protect, never trade. Any future sweep that raises ROUTING while dropping ABSTAIN has re-broken B.21 and should be rejected on sight.

The verdict

B.21 was right and incomplete. Normalising by library size is correct; normalising by it linearly just relocates the bias to the other tail, and the version that shipped for a day made a 131-card personal recipe file the most authoritative shelf in the corpus for anything phrased with common words.

The broader lesson is about how the two benches relate. B.21’s headline (29/52 → 35/52) was a real win that concealed a new defect, and no amount of staring at that headline would have surfaced it — it took a second instrument asking a different question (B.22) to make the defect countable, and then a sweep to price it.

One ruler tells you whether you improved. It takes two to tell you what you broke.