B.20 — filling a starved model: book pretraining, measured

T.M.B. #20 — the first werbos model that was actually fed. Every prior bench measured a cursor reading from a library. This one measures the cursor itself, and the finding is unflattering in a useful way: the babble that killed the 85M and capped the 30M was never a size problem. It was starvation, plus four pieces of bookkeeping. Two of the four are ablated below against the real pre-fix code, not a reconstruction. Frozen; re-run it yourself.

The claim under test

werbos has said for months that knowledge belongs in the library and the model stays a cursor — and every attempt to make the model itself more capable hit a wall that looked like a capacity wall. B.8 found bigger got worse on thin data. The 85M was abandoned. The shipped 30M sat at 0.64 tokens per parameter.

That last number is the claim. Chinchilla-optimal is ~20 tok/param. 0.64 is 3% of it. Every size sweep werbos has ever run was measuring degrees of starvation, so “does bigger help?” was never actually asked. The claim under test: the babble is data starvation and bookkeeping, not size.

The setup

No model is in the grader. Cross-entropy on held-out tokens is arithmetic.

python fetch_gutenberg.py --dest <books>          # or pairs_to_text.py from an existing pair corpus
python build_book_corpus.py --src <books> --out corpus.txt
python build_bpe.py --text corpus.txt --chat conversation.jsonl --vocab 16384 --out bpe.json
python tokenize_corpus.py --text corpus.txt --tokenizer bpe.json --out tokens
python train_talk_standalone.py --lm tokens/train.bin --val-bin tokens/val.bin \
       --tokenizer bpe.json --sizes 35 --steps 19393 --batch 24 --grad-accum 2 \
       --nctx 512 --amp bf16 --ckpt-every 1000

Results

step     0    10.0124      <- chance is ln(16384) = 9.7041
step 19392     2.1183
train-loss 2.118 | HELD-OUT 2.263 (ppl 9.6) | 131k tok/s | 1.0h

Held-out 2.263. Train↔held-out gap 0.145 nats. A small gap is the whole point — it means the model generalized to text it never saw rather than memorizing the corpus, and it held that even at 3.3 epochs.

The qualitative gate, unedited, on prompts it never trained on:

“She had never seen anything like” → “it before. She was now very proud of it, and she had been afraid that she should not forget it. I have no doubt that it was very hard for me ever to speak to her again.”

“The old house at the end of” → “the room was a great room with a large black table around it… where Mr. Pickwick sat at work with Mr. Weller’s friend, the old gentleman who had been left to perform some part of the journey”

Grammar, punctuation, dialogue with quotation marks, clauses that resolve. The same model family the same morning produced @@@@@@@@ and a.a.a.a.a.

What actually moved the number

Two of the four fixes are ablated directly against the real pre-fix module (git show 9a284b0:scripts/train_talk_standalone.py), same seed, same batch, same corpus — scripts/ablate_init_amp.py:

The pre-fix trainer opened 32× worse than random guessing. The mechanism is exact: torch’s stock nn.Embedding init is N(0,1), and head.weight = wte.weight ties the output head to it — the tie discards nn.Linear’s small init, so logits are a dot product over 512 dims of two unit-normal vectors, std ≈ √512 ≈ 22.6 (measured: 22.74). That is the entire 307 that killed the first run.

bf16 is worth 2.36× — in a fixed three-day window, the difference between ~0.9B and ~2.2B tokens.

The other two fixes have no scalar to report and mattered more than either:

• --lm mode. The pair trainer is completion-masked — loss only on the response half. Books shaped into {input, response} throw away half the gradient and cap effective context near 40 tokens regardless of --nctx, because every row is an independent sentence pair. There is no version of that run that learns paragraph structure.
• Atomic checkpoints. kill -9 during torch.save left a truncated zip; the resume died on failed finding central directory. The checkpoint feature costing the whole run is the exact failure it existed to prevent. Verified fixed across three consecutive kill cycles.

The honest caveats

Read these before quoting the 2.263.

• This is a 46M model, not the 350M target. It is a faithful 10×-down (same aspect ratio, same dims/head), which makes it a rehearsal, not the result.
• The corpus cannot fill the target. 144M tokens is 10.3 tok/param here but 0.4 tok/param for a 350M — worse than the 30M that babbled. This bench validates the pipeline; the 350M is blocked on books, not on compute. A 61,498-book pull (~9–10B tokens) was in flight when this was written.
• Held-out is the tail 1% of the same 973 books. Same distribution, same authors, same era. It proves the model didn’t memorize; it does not prove cross-domain generalization. B.9’s wall is untouched by this.
• Perplexity 9.6 is not comparable to anyone else’s. It is measured on a 16k BPE at 2.49 bytes/token. Cross-tokenizer perplexity comparisons are meaningless, including against werbos’s own earlier byte-level numbers.
• There is no completed control run. The pre-fix ablation is an opening loss, not a finished model — the old configuration crashed six minutes in and never produced weights to compare. What is measured is why it crashed, not what it would have converged to.
• The register is 19th-century literature. Gutenberg is overwhelmingly pre-1929. The model writes like Dickens because it read Dickens.
• Above the sentence, it drifts. “She had lived too long with her child to be a man and not to be a man” is grammatical and means nothing. Tables change colour mid-paragraph. Pronouns slip.
• 3.3 epochs. Repetition is doing some of the work in that final loss.

The verdict

The claim holds, and it is smaller than it sounds. The babble was never a size problem. Every failure in this arc said size and every one turned out to be bookkeeping — an init, a dtype, an fsync, a byte offset. The 307 was N(0,1) where N(0,0.02) belonged.

But what a fed model buys is phrasing, not knowing. This base holds a sentence and loses the thread across a paragraph; it can render a fact fluently and has no way to be right about one. That is not a disappointment — it is the thesis, now measured from the other direction. B.9 showed a tiny reader can’t be taught to know by scaling data breadth. B.10 shows a tiny model can be taught to speak by feeding it properly.

Knowledge stays in the library. The model stays a cursor. It is now an articulate one.

What would change the verdict: a 350M trained on the full ~9B-token pull, gated on the same held-out cross-entropy, plus a cross-domain held-out set drawn from text that isn’t Gutenberg. If above-sentence coherence still fails at 20 tok/param on 350M, the ceiling is architectural and this bench will have found the wrong floor.