B.9 — the cross-domain reader: 11 domains, still the wall
B.8 trained a model that's ours for small talk. B.9 asks whether a from-scratch READER — the model that picks which passage answers a question — can be trained off chance by BREADTH: 2,242 pick-a-passage fences across 11 domains (JS, TS, HTML, CSS, SCSS, React, DOM, Python, Node, Git, regex, SQL), vs the single-domain runs that hit chance before. The honest number is a held-out, deterministic pick-accuracy gate. Headline: 5M = 38%, 15M = 31% — chance (≈33% for 3 candidates), and bigger is WORSE. The wall holds at cross-domain scale. It's the task, not the data — so the knowledge stays in the library (retrieval answers in ~20ms, cited), and the reader stays a small cursor.
T.M.B. #9 — a negative result, logged on purpose. The reader is the one piece we keep renting Claude for. This bench tests the honest hope that more breadth (11 domains instead of 1) would teach a tiny from-scratch reader to actually read. It didn’t. We publish the wall the same way we publish the wins — with a date on it — because a benchmark you can’t fail is a benchmark that means nothing. Frozen; re-run it yourself.
The claim under test
Earlier single-domain runs (B.8-era, JS/TS) trained a pick-a-passage reader — given a question and 3–4 candidate answers, emit the index of the correct one — and it sat at chance (30–34%), with a TF-IDF baseline also at chance (no lexical shortcut: the distractors are plausible, same style as the correct answer). The open question was whether that was a data limit. So we tripled-and-broadened the corpus: 2,242 fences across 11 domains, minted by the domain-spec factory, and re-ran the size sweep with a deterministic gate (the emitted index either equals the gold or it doesn’t — no model in the verdict).
The number
| model | params | held-out pick-accuracy | vs chance (~33%) |
|---|---|---|---|
| 5M | 4.9M | 38% | chance |
| 15M | 14.4M | 31% | chance — worse |
Training loss fell (the model memorized the training fences), while held-out stayed at chance and the bigger model did worse — the textbook over-memorization signature. Sample held-out picks from the 5M: “what does the CSS box model consist of?” gold 0 → got 1; “specificity of an inline style?” gold 2 → got 1. It learned to guess a varied index, not to read.
What it means
Breadth is not the missing ingredient. Volume wasn’t either (an earlier 272→650 fence jump moved nothing). And method isn’t lexical (TF-IDF is also at chance). The honest conclusion, now confirmed across 11 domains and 2,242 examples:
A tiny from-scratch model cannot do abstractive-comprehension discrimination — pick the correct explanation among plausible-wrong ones. It’s the task, not the data.
That’s not a failure of the project — it’s the thesis, measured on the widest test yet. The knowledge belongs in the library, where retrieval answers a covered question in ~20ms, cited (and these same 11 domains do exactly that, live). The reader stays a small cursor.
The honest next step
The wrong task was baked into the fences: abstractive comprehension (all candidates are fluent explanations). The reader’s real job is extractive — pick the retrieved passage that literally contains the answer, where a lexical/answer-type signal exists (the task the 1M numpy champion won at 12/19). B.10, when we build it, mints extractive fences and re-runs this gate. Until then, the reader is rented by choice, and the library carries the load.
Footnote: the infra was honest too
Getting this number took fighting the GPU box — sustained runs at batch 64 pinned the 16 GB card
and hung (a 5M model needs megabytes, so that was a memory-ceiling stall, not our result). A
memory-safe config (batch 16, nctx 384 — the fences are ≤455 bytes, so the old nctx 1024 was
pure waste) ran clean. The dead-GPU hours are in the ledger too.