B.10 — the talk-model size sweep: does bigger help on rich data?
B.8 trained a 5M talk model on 530 small-talk pairs. B.10 asks the scaling question with real data: mint a 77,000-pair conversation corpus, then train the SAME architecture at four sizes — 5M → 30M → 85M → 125M — same data, same 6,000 steps, held-out loss the only judge. The naive expectation ('bigger is closer to GPT-2, so 125M wins') was wrong. The 30M won (held-out 1.183); 85M and 125M got WORSE. And the train-loss column says why: the big models' TRAINING loss was higher too, so they're not overfitting — they're undertrained, starved of steps by a CPU-bound data loader that made the 125M take 2.36 hours for 6k steps. The honest verdict: ship the 30M, fix the loader before scaling, and don't confuse 'undertrained' with 'too small.'
T.M.B. #10 — the honest ruler judged against the plan. We’d spent the week minting a big conversation corpus to grow toward GPT-2 size. So we ran the whole ladder at once — same data, same steps, held-out loss the judge — and it told us the model we were building toward is not the one to ship. That’s what an un-bribable ruler is for. Frozen corpus, deterministic metric, re-run it yourself.
The claim
More capacity should help — a 125M model is ~4× a 30M. The naive read of “we’re building toward GPT-2 (124M)” is that the biggest model wins. The claim under test: on 77k pairs at a fixed training budget, does bigger actually help — or is it dead weight?
The setup
- Model: one architecture (byte-level causal decoder,
train_talk_standalone.py, torch, from scratch — no open weights), scaled by width/depth to four sizes. - Data: 77,384 minted conversation pairs (the two-lane mint corpus, gathered + graded by
werbos convo pull). 66,329 train / 11,055 held-out, split deterministically. - Budget: identical for every size — 6,000 steps, batch 32. This is the key control: it’s a fixed-step sweep, not a compute-matched one (that distinction is the whole finding).
- Metric: held-out cross-entropy (lower = predicts unseen conversations better). There is no deterministic correctness grader for open-ended chat — and refusing to fake one is the honest position. The samples below are the qualitative read.
Results
| size | params | held-out loss | train loss | wall-clock |
|---|---|---|---|---|
| 5M | 4.9M | 1.307 | 1.159 | 98s |
| 30M | 31.9M | 1.183 ← best | 1.026 | 328s |
| 85M | 85.7M | 1.225 | 1.038 | 1,256s |
| 125M | 135.7M | 1.231 | 1.079 | 8,490s |
The 30M won. Bigger got worse. And the samples make it visible — the 30M was coherent, the 125M looped:
30M "no plans this weekend honestly" → "That's awesome! What kind of artist are you in the mood for?"
125M "no plans this weekend honestly" → "That's awesome! Any favorites you're excited to see?"
125M "ugh weekend is already over" → "that's a lot of the workout of the world of the day…" (looping)
What actually moved the number
Read the train-loss column — that’s the whole finding. If the big models were overfitting, they’d nail the training data and miss held-out (low train loss, big gap). Instead their training loss is higher too. They’re not overfit — they’re undertrained: at a fixed 6,000 steps, a 135M model can’t even fit the data as well as a 30M does.
Why they starved: the 125M took 8,490 seconds — 2.36 hours for its 6k steps, because the trainer is CPU-data-loading bound (100% GPU utilization at a telltale ~77W, a fraction of the card’s draw — the GPU was waiting on a Python batch loop, not computing). Bigger model, same step budget, far less convergence per step of wall-clock. The steps ran out before the big models learned the data.
So the fix is not “4× the data” first. It’s fix the loader → add steps → then add data, in that order — because more data can’t help a model that hasn’t converged on the data it already has.
The honest caveats
- This is a fixed-step sweep, not compute-matched. It does not prove a 125M can’t beat a 30M — only that at equal steps, on this loader, the 30M does. A step-matched or compute-matched rerun (after the loader fix) is the fair test of scale, and it’s the next job.
- Held-out loss, not a task score. Lower loss = better next-token prediction on unseen conversations; it’s a fluency proxy, not a correctness grade (there isn’t an honest one for open chat). The samples corroborate the ranking.
- The corpus is still small for a 125M. Rough compute-optimal (~20 tokens/param) says a 135M model wants billions of tokens; we’re feeding it ~11M. Even the 30M is data-light — it just sits in a happy enough regime. Growing the corpus is real work the mint lanes are already doing.
The verdict
Ship the 30M. It’s smaller, faster, and the best model on this data — the werbos thesis one more time: a small model on good data, judged by an honest ruler, beats the bigger thing you assumed you wanted. The 125M isn’t dead; it’s blocked on a data-loading bottleneck, and naming that honestly is worth more than a bigger number would have been.