B.13 — loss ranked four models exactly backwards

Tue, Jul 28 2026

Four talk models trained on one corpus with identical budgets, varying only the parameter count. Held-out loss fell monotonically with size — 1.157, 1.109, 1.020, 0.963 — and ranked the models exactly backwards. The 30M has the best loss and is the only model that fails its floors, on babble x10. The 1M has the worst loss and the cleanest output, and now ships at one twenty-ninth the size and fourteen times the speed. Selecting on loss, which is what B.10 did, ships the worst model in the set. Includes the invalid run from the night before and why it inverted.

T.M.B. #13 — the metric was working perfectly and measuring the wrong thing. Earlier benches asked whether a model was good enough. This one asks whether the number used to CHOOSE models can be trusted, and the answer is no in the most direct way available: a controlled sweep where loss and quality are anti-correlated across a 30× size range. Frozen; same probes, same floors.

The setup

One corpus: 216,068 pairs / 22.9M tokens (byte-level, so bytes are tokens). Identical 16,880 steps — five real epochs — at batch 64. The ONLY variable is parameter count.

size	params	tok/param	% of Chinchilla	held-out loss	clean	input-sens	adjacency	stable	latency	verdict
1M	1.09M	21.0	105%	1.157	0.96	0.90	1.00	1.00	35ms	SHIPPABLE
2M	1.93M	11.8	59%	1.109	0.88	0.92	1.00	1.00	30ms	SHIPPABLE
5M	4.94M	4.6	23%	1.020	0.92	0.90	0.92	1.00	79ms	SHIPPABLE
30M	31.92M	0.7	4%	0.963	0.77	0.94	1.00	1.00	503ms	FAIL: clean

Loss is perfectly monotonic in size. 1.157 → 1.109 → 1.020 → 0.963. Textbook.

Clean inverts it. The best-loss model is the only failure — babble ×10, ten of forty-eight replies containing words that are not words. The worst-loss model produces the cleanest output ever measured here.

Why loss cannot see this

Loss measures how surprised a model is by held-out text. A model that answers everything with “Sounds like a plan!” is barely surprised by anything, because high-frequency filler is the safest continuation available — so it scores well and says nothing.

That is not a noisy estimate or a calibration issue. The failure mode is invisible to the metric by construction. No amount of more careful loss measurement recovers it.

This bench exists because B.10 chose the 30M on held-out loss and shipped it. Its title was “does bigger help on rich data?” and it answered its own question wrong, using the metric it picked to answer with. B.13 is that comparison re-run with floors instead.

Why smaller wins

At 22.9M tokens, Chinchilla-optimal is roughly 1.1M parameters.

The 30M is not a large model trained well. It is a model trained on 4% of the data it needs, and the surplus capacity has to go somewhere — into memorising fragments it cannot stitch back together. That is precisely what babble ×10 versus babble ×2 is, measured.

Every scaling result this project has produced was taken on starved models: the reader wall, the 85M dead weight, B.10 itself. They compared degrees of starvation and read it as a size preference.

The invalid run, and what it nearly cost

The first attempt ran 3,000 steps at batch 8: 24,000 sequences against 177,780 pairs = 0.13 epochs. Results were non-monotonic noise (5M 0.81, 10M 0.35), and the honest reading was that it measured training budget, not size.

The near-miss is worth recording. Every small model in that run scored input-sensitive 1.00 — perfect, and it looked like confirmation. It was babble: noise is maximally varied, so stem-diversity peaks when output is garbage. It was only caught because clean failed alongside it.

One composite score would have hidden this. A weighted average of five axes would have let a babbling model buy a passing grade with a perfect diversity score. Floors on every axis is what made the contradiction visible.

Against the incumbent

Same 48 probes, same floors, no allowances:

llama3.2:3b     clean 0.21   input-sens 0.73   20,618ms
werbos 1M       clean 0.96   input-sens 0.90       35ms

480× faster at 4.5× the clean score, from a model roughly three thousand times smaller. The 3B fails the same floor the 30M does, for a different reason (run-on ×33 — it writes paragraphs where a conversational turn is wanted; see B.12’s correction on that point).

What this does NOT show

clean 0.96 is a floor, not a ceiling. Forty-six of forty-eight replies contain only real words and finish their sentences. They are not therefore good. Verbatim from the shipped model: “It’s a secret of changer and cheese too.” That clears every floor.
ood is 0.00 for every model here, as it is for the 3B. Nothing signals confusion at nonsense. Two attempts to train it in failed — the first learned that short means nonsense, the second still refused “hey there”. Reported as debt.
One corpus, one domain. Conversational turns. Nothing here says a 1M model is right for code, retrieval, or reasoning — and the coding tier measured separately shows the opposite shape.
48 probes, one operator’s phrasing, and a clean axis whose dictionary check can pass a sentence that is grammatical nonsense.

The lesson

A metric that cannot see your failure mode will rank your models backwards.

Not “will be noisy”. Not “will need a tiebreaker”. Backwards — a perfect inversion across four models and a 30× size range. The fix is not a better loss function; it is floors on axes that name the failures out loud, and the willingness to ship the model with the worst number on the metric everyone else uses.