B.16 — undertraining is invisible, and the collapse that lowered the loss

T.M.B. #16 — the same model twice, and the loss lied twice. This started as a plain retrain: take the shipped 2M, add the 58k conversation pairs minted overnight, ship the better one. It became two lessons about what held-out loss does and does not see. Frozen openers, identical architecture, same 5070 Ti.

The setup

The shipped talk model is a 1.93M-param byte-level GPT (n_embd 192, n_head 6, n_layer 4), trained on talk_all_snapshot.jsonl for 16,880 steps, held-out loss 1.128. Overnight minting (a qwen2.5:14b teacher, distilled, never shipped) added 57,614 pairs the snapshot did not contain. The obvious move: merge and retrain the same 2M.

python scripts/train_talk_standalone.py --corpus talk_snapshot_plus_night.jsonl \
       --sizes 2 --steps 16880 --out talk_v3
python scripts/eval_talk.py talk_live/talk-2m talk_v3/talk-2m   # same novel openers

Every reply below is to an opener neither model trained on.

Finding 1 — undertraining is invisible in the config

The first retrain used the trainer’s default 4,000 steps. The result read as a broken model:

"my flight got delayed three hours ugh"  ->  "I know that feeling the feel the feel the way the that."
"just adopted a kitten this weekend"     ->  "It's a great that feel the way the the the start a comforthing"

Byte-salad. Held-out loss 1.594. The same architecture at 16,880 steps on the same data:

"just adopted a kitten this weekend"     ->  "Wow, that's awesome! What did you think of the day?"
"made way too much pasta for dinner"     ->  "I hear ya! What's the first thing you like to do the day?"

Coherent, held-out 1.185. Nothing in the checkpoint’s config distinguishes the two — same params, same shape, same corpus. The only difference was training budget, and it is the difference between a shippable model and garbage. A size sweep that varies architecture at a fixed low step count would have measured starvation and blamed the sizes. Steps are a hidden axis, and this one was 4x short.

Finding 2 — the fresh data raised the loss and improved the model

With both trained to the full 16,880 steps, the head-to-head:

The live model has the lower loss. Here is what it bought:

live:  "my flight got delayed three hours ugh"   ->  "That's awesome! What did you think of the way?"
       "cant stop thinking about that meeting"    ->  "That's awesome! What do you think of the way?"
       "the wifi has been down all morning"       ->  "That's awesome! What did you think of the wait?"
       -> "That's awesome!" to SEVEN of eight openers, including three complaints.

The live model collapsed to a safe high-frequency phrase, and that collapse is why its loss is lower — a bland reply that fits everywhere hedges cross-entropy. Calling a delayed flight awesome is coherent grammar and tone-deaf conversation. The new data broke the collapse:

new:   "made way too much pasta for dinner"       ->  "I hear ya!"                    (matches the tone)
       "the wifi has been down all morning"       ->  "I know that feeling..."        (reads the frustration)
       "just adopted a kitten this weekend"        ->  "Wow, that's awesome!..."       (correctly enthusiastic)

Six distinct openers, tone-appropriate, does not call a complaint awesome. It costs 0.057 nats of held-out loss and one repetition slip (a “coffee or some cooking for a coffee” loop). This is B.13 — loss ranks talk models backwards — reproduced a third time, now with a causal knob: adding variety to the corpus raised the loss and improved the conversation, because the metric rewards the collapse the variety undoes.

What shipped, and why

The new 2M is now live. We shipped the model with the higher loss on the argument B.13 and B.15 already established: for open conversation, held-out cross-entropy under-rewards the one axis that matters, input-sensitivity, and over-rewards safe templating. The live model’s lower perplexity was it being blander. The new model reads the input, was trained on strictly more and fresher data, and holds coherence at the same step budget.

The honest caveats

• The held-out splits differ. Each model’s loss is on its own corpus’s split, and the new split contains the harder, more varied night data — so some of the 0.057-nat gap is a tougher test, not a worse model. The splits are not directly comparable; the direction (more variety, higher loss, better replies) is the finding, not the exact delta.
• Eight openers is a read, not a metric. A proper input-sensitivity score (distinct-reply rate over hundreds of openers) would quantify what these eight illustrate. The qualitative gap is large enough to act on; the number would make it defensible to a skeptic.
• Both models still degrade. Neither sustains a coherent paragraph — this is a 1.93M byte-level model, and the ceiling from B.9 (a tiny reader cannot do abstractive comprehension) has not moved. What improved is conversational reflex, not depth.
• The previous model is backed up, not deleted (talk-2m-live.bak-20260801), so the ship is reversible if the input-sensitivity win does not hold up in use.

The verdict

Two things a config file cannot show you. Undertraining leaves the architecture pristine and the output broken — steps are a silent axis, and this run was 4x short before it was right. And the metric that ranks these models rewards the failure: the blander model scored better, so the fresh data made the loss worse on its way to making the model better. The one we shipped is the one that does not call a delayed flight awesome.