Day 91 — the model finally got fed

“i think we are final approaching werbos2.0 phase”

Six days in one entry, and they tell one story: the composer got measured, then the model got fed. Every training run before this was starving, and nobody had the number that said so.

The suite that became a spoke

capability_suite.py began as a behavioral contract — assert each capability fires end-to-end against the real libraries and routing, which the layer unit tests miss. Thirteen pass, plus four xfail guards tracking bugs found while building it, so fixing one flips it to XPASS rather than letting it stay quietly broken.

Then it stopped being a test. Expanded to 35 probes across the whole surface and wired in as the skill_routing source for werbos mind-bench — replacing a vague reflect() self-observation with a measured 0.66 appended to the time series. The Mind Portrait’s phantom limb became a real spoke that grows as guards flip green. The suite didn’t just find bugs; it became werbos’s self-measurement.

Three lanes, one disease

Casual chat kept falling through into lexical retrieval. “what are you doing later?” matched a social favorites pair. “im going to watch tv” came back with a coding card, because “watch” drowns in the 22k web-api library — big-library bias, again.

The fix was a composed medium-talk continuer that runs before resolve(), so a chat line can never reach the library at all, plus a new plans speech act. Suite 23→27, Routing fit 60% → 69%, zero regressions.

Then the math lane closed a confident-wrong. “solve 5x + 7 = 32 for x” had no exact card, so retrieval bluffed the nearest inequality. Now linear equations are solved with no model in the path: recover each side’s slope and intercept by exact Fraction arithmetic, prove linearity by equal first differences, solve — then verify by substitution. Anything it can’t prove (x^2 = 9, multi-variable) abstains instead of falling through to bluff.

A wrong math answer stated confidently is the precise opposite of the moat.

Then the run failed

The 35M book-fill died six minutes in, at step 0, having printed:

step 0  loss 307.1179

Chance for a 16k vocab is 9.70. It opened thirty times worse than random, and the whole chain collapsed behind it.

Four bugs, and every one of them would have eaten the 350M

The trainer could not pretrain at all. pack() is completion-masked — loss only on the response half. Books shaped into sentence pairs threw away half the gradient and capped effective context near 40 tokens no matter what --nctx claimed. That trains a very fluent sentence-completer that can never hold a paragraph, and long-range coherence is the whole point.

There was no init. Torch’s default left the tied embedding at N(0,1), so the initial logits were a 512-dim dot product of two unit-normal vectors. That is where 307 came from.

The stack was pure fp32. No autocast anywhere — roughly a 2–3× throughput tax on a fixed window.

The checkpoints were not atomic. A kill -9 during torch.save left a truncated zip, and the resume died on failed finding central directory. The checkpoint feature costing the entire run — which is exactly the failure it existed to prevent. Found by actually killing it, not by reading it.

Fixed: --lm mode (one flat uint16 stream, random windows, loss on every token), bf16 autocast, GPT-2 init with 1/sqrt(2·n_layer) damping on the residual writers, and .tmp + os.replace — verified across three consecutive kill cycles, resuming 120 → 225 → 330.

A corpus factory, and one bug that would have made the scoreboard lie

Four stages, none of which existed. pairs_to_text.py recovers flowing prose from the sentence-paired book file — the pairing is a sliding window, so response[N] == input[N+1] and the books come back whole. build_book_corpus.py strips Gutenberg licence boilerplate (leave it in and it’s the most frequent document in your corpus — the model spends real capacity learning to recite a licence) and rejoins hard-wrapped lines, because leaving them teaches the model to emit a newline every 70 characters, which looks exactly like babble. tokenize_corpus.py memoises and multiprocesses the BPE encode — 144M tokens in 12 seconds. And fetch_gutenberg.py pulls the catalog-filtered English set from a mirror: never crawled, atomic, resumable.

The bug worth naming: the train/val split routed whole chunks by byte offset. At 4MB chunks and a 0.5% tail, the val slice was smaller than a single chunk — so nothing ever crossed the boundary and val.bin came out empty. Held-out loss would have silently echoed train loss for the entire run. The scoreboard reading healthy at precisely the moment it was measuring nothing.

Then it ran

One hour. 46.48M params, 19,393 steps, a steady 131k tok/s.

step     0    10.0124     <- ln(16384) = 9.70, exactly chance
step 19392     2.1183
train-loss 2.118 | HELD-OUT 2.263 (ppl 9.6)

The train↔held-out gap is 0.145 nats — small gap means it generalized rather than memorized, even at 3.3 epochs. And it writes English:

“She had never seen anything like” → “it before. She was now very proud of it, and she had been afraid that she should not forget it. I have no doubt that it was very hard for me ever to speak to her again.”

Grammar, punctuation, dialogue with quotation marks, clauses that resolve. Against @@@@@@@ and a.a.a.a.a. from the same morning.

Frozen as B.20 — filling a starved model, with both mechanistic claims ablated against the real pre-fix module rather than a reconstruction: the old code opens at 311.05 (logit std 22.74, embedding std 1.000) where the new one opens at 9.7201 against a chance of 9.7041 — 32× worse than guessing — and bf16 measures 2.36× on throughput. The first attempt at that ablation rebuilt the old init by walking model.modules() and calling reset_parameters(), and reported 9.81 — chance — because head.weight is wte.weight and the head’s small init landed on the shared tensor last, erasing the thing under test. It would have disproved a real bug. Import the old module; don’t reconstruct it.

The honest ceiling: sentence-level coherence is solved, and above the sentence it drifts. “She had lived too long with her child to be a man and not to be a man” is grammatical and means nothing. Tables change from black to red mid-paragraph. The pop-in is fixed; the pre-loader is not. This model phrases. It still doesn’t know — which is the argument for the library, not for more parameters.

A security summit while the books downloaded

The overdue list came back clean. scp injection already guarded. zip-slip structurally inapplicable — the archive parser never extracts to disk — on top of traversal rejection, tarfile.data_filter, symlink and device rejection, and a streaming budget charged against actual decompressed bytes rather than the claimed size. skill-packs-execute certified: custom skills have no exec field, body is a system prompt, they are data and not programs. The old node:net / .env red flags are gone.

Two argument-injection holes closed, both the same class as the scp one — an argument reaching a parser rather than a shell, where shell=False buys you nothing. The skill-run URL path param was interpolated straight into a dispatched command, and WERBOS_NPM_PIN went verbatim into uv tool run <spec>, where a leading - is parsed as an option.

And the finding that outranks all of it: server/gatekeep_auth.py exists and is imported by nothing. What actually holds the line is one control — serve.py refusing any non-loopback bind without --allow-remote-bind. It’s well-built and fail-closed, and it means the entire security posture is “nobody else can reach it.” Correct for local-first single-user. Not a posture you can put a client on.

The 350M isn’t what stands between here and client-ready. That is.

Numbers worth keeping

The AHA

Every failure this week said size. Every one of them was actually bookkeeping — an init, a dtype, an fsync, a byte offset. The 307 wasn’t a hard problem. It was N(0,1) where N(0,0.02) belonged.

Werbos 2.0 doesn’t start when the model gets bigger. It starts now, because the model finally has something to eat and a scoreboard that isn’t lying about it.