B.25 — a 1.35M router that knows when it doesn't know

T.M.B. #25 — a gate is judged on its refusals, not its accuracy. 56% on a 375-way choice reads like a weak classifier. It is not being used as one. The question a router answers is act or refuse, and the measure is whether the confidence separates the two — high on what it knows, low on what it doesn’t. It does. Re-run it: python scripts/train_router.py --steps 8000.

The claim under test

A model far too small to hold 167k facts (the reader wall settled that: 57M scored 6/19 where 1M numpy scored 12/19) can still learn to route — given a query, name the domain it belongs to — and, more importantly, know when it can’t. The falsifiable half: if confidence on clear in-domain queries is no higher than on gibberish, there is no gate, just a coin that shows its edge.

The setup

scripts/train_router.py — a 1.35M transformer trained from scratch on the 167k-card library as domain-tagged prose, one line per card, the library name a tag the model learns to emit. Scored two ways, both on held-out cards the model never trained on (a 10% split), gold = the card’s real library, so no model is in the verdict:

• full 375-way: name the correct library out of all 375. The hardest possible framing.
• distinct-domain gate: the few-way sets it would actually ship as — adult-vs-not, a handful of unrelated verticals — where “route or refuse” is the real job.

The label is read from a single-token label head: one reserved token per library above the BPE range, one forward pass, argmax over the 375 label tokens. This replaced a text-label likelihood readout that collapsed on a BPE [ boundary and a length bias — the fix that took the same model from a reported 7% to its real 56%, the classifier hiding behind a broken ruler.

Results

full 375-way (one forward pass per query)
  train      58%
  held-out   56%     gap 2 points  (learning, not memorising)   chance 0.27%

distinct-domain gate (held-out)
  2-way   websecurity vs cooking            100%   (chance 50%)
  5-way   security/cooking/spanish/yoga/mc   96–100%  (chance 20%)
  4-way   javascript/css/python/web-api      24%   (chance 25%)  ← FAILS

confidence, by query kind
  "how do you say hello in spanish"   → spanish   ~100%   commit
  "what is a closure in javascript"   → concepts   61%    commit
  "how do I sear a steak"             →            33%    FALL CLOSED
  "asdf gibberish"                    →            13%    FALL CLOSED

56% held-out on 375 fine-grained libraries is 207× chance, with a 2-point gap between train and held-out — the signature of a model that learned the domains rather than memorising the rows.

What actually moved the number

The trainer init, first. The first cut opened every small model at 8–14× chance — a tied output head paired with an embedding initialised at std 1.0, so the model started far from the ln(vocab) loss that is chance and spent training climbing back to a fair starting line. Fixing the init (std 0.02, tied head) opened it at exactly 1.00× chance. That bug had silently hobbled every small model trained here; the router is just the one that made it visible.

Then the readout. The 375-way score was reported at 7% while the model had actually learned — the text-label likelihood scorer preferred short labels and broke on the [ token boundary. The single-token label head reads the distribution directly: 3ms per query across all 375 candidates, and the 7% became 56%. The classifier hadn’t improved; the ruler had stopped lying.

The honest caveats

• Adjacent domains are the wall, and it is at chance. The four coding libraries share so much vocabulary (function, return, array) that the 4-way score is 24% against a 25% chance — no signal. This is the exact adjacency failure B.23 found in select_library’s tf-idf, now reproduced in weights. The recipe is a distinct-domain gate; adjacent routing needs a bigger model or a lexical tie-breaker, and this benchmark does not pretend otherwise.
• The distinct sets are chosen to be distinct. The 100% 2-way and 5-way numbers are on hand-picked unrelated verticals. That is the honest shape of the use case (adult-vs-not, a B2B vertical stamp) — but it is not a claim that any few libraries separate cleanly; adjacent ones demonstrably don’t.
• 56% is not a usable 375-way classifier. Nearly half of fine-grained routes are wrong. The result is only meaningful because the confidence separates commit from refuse; used as a hard 375-way pick with no floor, it would be wrong 44% of the time and say so with a straight face.
• The holdout is a single 10% split. Tighter would be k-fold; this is one draw.

The verdict

As a 375-way classifier: a curiosity at 207× chance. As the thing it ships as — a distinct-domain gate that acts when confident and falls closed when not — it is the recipe, proven on held-out cards. It is the KNOWING half of the own-voice·cards lane: the router decides which library (~3ms, with a confidence), retrieval scopes to that one, and below a 0.5 floor the whole lane refuses rather than guesses. A small model cannot be an oracle; it can be an honest doorman.