B.15 — more data moved the optimum, and the token math called it in advance

T.M.B. #15 — the same sweep on twice the data, and the winner moved. B.13 found that loss ranks talk models backwards. This repeats that on a 67%-larger corpus and asks the next question: does the best SIZE move when the data grows? It does, and it moved to where tokens-per-parameter said it would. Frozen corpus, identical budgets, same floors.

The setup

One corpus: 361,619 pairs / ~45M byte-tokens (byte-level, so bytes are tokens; the count is estimated from the 56.2 MB snapshot less JSON overhead). Identical 16,880 steps, batch 64, nctx 512 for every tier — only n_embd/n_head/n_layer change. Trained on one RTX 5070 Ti in a single unattended run.

python scripts/train_talk_standalone.py --corpus talk_all_snapshot.jsonl \
       --sizes 1,2,5 --steps 16880 --batch 64 --out brains/talk-sweep-v4
python scripts/talk_bench.py brains/talk-sweep-v4/talk-{1m,2m,5m}

The prediction, written down first

Before the run, from the token math alone: at ~20 tokens per parameter the compute-optimal model for this corpus is ~2.2M params, so 2M should beat 5M on the floors — and if 5M won instead, B.13 needed revisiting. Recording it in advance is the only thing that makes it a prediction.

Results

        params   held-out loss   clean   input-sens   adjacency   stable   ood
  1M     1.09M       1.170        0.88      0.90        1.00       1.00    0.00
  2M     1.93M       1.128        0.92      0.98        1.00       1.00    0.00
  5M     4.94M       1.032        0.92      0.94        1.00       1.00    0.00

  3/3 SHIPPABLE — every checkpoint clears every floor

Loss ranked them backwards, again

The 5M has the best held-out loss by a clear margin and the worse input-sensitivity. Selecting on loss — the obvious thing to do, and what B.10 did — ships the wrong model for the second sweep running.

It is visible without any metric at all. The 5M, on held-out openers it never trained on:

"trying to decide what to cook tonight"     -> "What kind of plant are you in the mood for?"
"barely left the couch and still feel dra"  -> "That's awesome!"

Topic drift, and tone inverted on a plainly negative statement — from the model that “learned more”.

What is new: the optimum moved

  B.13    22.9M tokens   optimal ~1.15M   ->  1M won
  B.15    ~45M tokens    optimal ~2.2M    ->  2M won

In B.13 the 1M was the champion. Here it is the weakest model on both quality floors — clean 0.88 against 0.92, input-sensitivity 0.90 against 0.98. Nothing about the 1M changed; the corpus doubled, and a model that was correctly sized became under-sized.

So the lesson was never “smaller is better”. That shorthand has been carried since B.13 and it is wrong. The lesson is match the size to the data, and the direction of the correction depends entirely on which side of optimal you are standing on. Every tier in B.13 was above optimal, so every result there pointed the same way and looked like a law.

What the ledger caught that the sweep could not

Recording every model ever trained (the champion curve) surfaced something no single sweep can show, because the thing that got worse is not in the sweep:

                champion   clean   input-sens
  B.13  07-28      1M       0.96      0.90
  B.15  07-31      2M       0.92      0.98      <- clean FELL

We promoted a model with a worse clean score than the previous champion. And the same 1M, at the same size and the same 16,880 steps, fell from 0.96 to 0.88 on the larger corpus.

That is not what “more data” is supposed to do, and it sharpens a caveat this entry made too gently. The corpus did not only grow — its mix changed. B.15’s snapshot took proportionally more from conversation.jsonl than B.13’s did, so “45M tokens instead of 23M” is also “a different blend”, and clean is evidently sensitive to the blend in a way input-sensitive is not.

So the size conclusion above stands on its own comparison (three sizes, one corpus, one day), but the cross-sweep claim that the corpus is simply better does not. On the cleanliness floor it is measurably worse, and B.10 has no points on this chart at all because the floors did not exist when it was chosen.

What did NOT move

  ood   0.00   0/6 signalled confusion   — all three models

Not one model says “I don’t know” on out-of-distribution input, and doubling the corpus did not touch it. Every sweep on this project has produced the same zero. The retrieval side solved its version of this on Day 80 with a floor-and-agreement rule; the weights still have no equivalent, and more data is now measured — not assumed — to be the wrong lever for it.

The honest caveats

The token count is an estimate. 45M is derived from the snapshot file size less JSON overhead, not counted directly. The optimal-size arithmetic inherits that error, so “~2.2M optimal, 2M won” is not precise enough to claim the prediction landed on the nose — only that it named the right tier out of three.

Two sweeps is not a curve. Two points agreeing with a formula is weak evidence for the formula. The claim here is bounded: the winner moved up when the data grew, in the direction predicted. A 90M corpus should then favour ~4.5M, and if it does not, this entry is the one that was wrong.

The floors are known-lenient. All three pass everything except ood, so the gate is not currently separating these models — the ranking rests on two fractional scores, not on pass/fail.

Same corpus, same generator. The pairs come from the same minting pipeline as B.13, so “more data” here means more of the same KIND of data. It does not test whether more varied data behaves the same way.

The verdict

2M promoted. It wins input-sensitivity outright, ties clean, and is 2.5× smaller than the model with the better loss. And the live narrator was found still pointing at talk-large-30m — the tier B.13 named worst — so the previous sweep’s conclusion had never been deployed at all. A benchmark nobody acts on is a blog post.