Day 78 — 'loss ranked them exactly backwards'

The model with the best loss is the only one that fails. The model with the worst loss now ships.

Four sizes. One corpus — 216,068 pairs, 22.9M tokens. Identical 16,880 steps, identical batch 64. The only thing that varied was the parameter count.

Read the loss column top to bottom: 1.157, 1.109, 1.020, 0.963. Perfectly monotonic. Bigger is better, every time, exactly as the textbook says.

Now read the clean column. The ordering inverts.

The 30M wins the metric and loses the job. It fails on babble ×10 — ten of forty-eight replies containing words that are not words. The 1M, dead last on loss, produces the cleanest output this project has ever measured.

Why loss lied

Loss measures one thing honestly: how surprised the model is by held-out text. A model that answers everything with “Sounds like a plan!” is barely surprised by anything — high-frequency filler is the safest possible continuation — so it scores beautifully and says nothing.

That failure mode is invisible to loss by construction. It isn’t a calibration problem or a noisy estimate. The metric is working perfectly and measuring something adjacent to what we care about.

Which is uncomfortable, because this repo’s earlier size sweep picked the 30M on held-out loss and shipped it. The sweep’s own title was “does bigger help on rich data?” — and it got the answer wrong using the metric it chose to answer with.

Why smaller wins

At 22.9M tokens, Chinchilla-optimal is roughly 1.1M parameters.

 1M    21.0 tokens/param   105% of optimal
 2M    11.8                 59%
 5M     4.6                 23%
30M     0.7                  4%

The 30M isn’t a big model trained well. It’s a model trained on 4% of the data it needs, and the extra capacity goes somewhere — into memorising high-frequency fragments it can’t stitch together. That is what babble ×10 versus ×2 actually is.

Every scaling result this project has ever produced was measured on starved models. The reader wall, the 85M dead weight, the sweep that chose the 30M — all of them compared degrees of starvation and called it a size preference.

The night before, we got this exactly wrong

The first attempt at this sweep ran 3,000 steps at batch 8. Against 177,780 pairs that is 24,000 sequences — 0.13 epochs. Every model saw thirteen percent of one pass through the data.

The numbers came back non-monotonic garbage (5M at 0.81, 10M at 0.35) and the honest read was that the experiment measured training budget, not size. Fixing the budget flipped the answer.

Worth naming the near-miss: every small model in that invalid run scored input-sensitive 1.00. Perfect. It looked like the thesis confirmed. It was babble — noise is maximally varied, so stem-diversity is perfect when the output is garbage. The only reason it wasn’t read as a win is that clean failed alongside it, which is the entire argument for floors on every axis instead of one composite score.

What shipped

default_model_dir() now returns a 1.09M-parameter model. It replaces a 31.9M one: 1/29th the size, 14× the speed, and better on the axis that decides whether a reply is usable.

It also fixes something latent — the old default pointed at talk-large-30m, a directory that does not exist on this machine. Everything worked only because every caller passed an explicit path.

Against the incumbent open-weight model on the same probes under the same floors: llama3.2:3b takes 20,618ms and scores 0.21 on clean. The 1M takes 35ms and scores 0.96. That is 480× faster at four and a half times the quality, from a model three thousand times smaller.

The honest part

clean 0.96 means forty-six of forty-eight replies contain only real words and finish their sentences. It does not mean they are good. Here is the shipped model, verbatim:

"hey there"                    -> "Soot too! What about you?"
"what did you do this weekend?" -> "I love it! It's a secret of changer and cheese too."

Those clear every floor. talk_bench measures a floor, not a ceiling — its own docstring says so — and the claim being made today is “best of four by a wide margin, at 1/29th the size”, not “good”.

A metric that cannot see your failure mode will rank your models backwards.

The fix isn’t a cleverer loss. It’s floors on axes that name the failures out loud, and the willingness to ship the model with the worst number on the metric everyone else uses.