Day 71 — 'the 30M won, and the train loss said why.'
The day the owned talk model got trained for real — and the honest ruler delivered a verdict against the obvious plan. We'd spent the week minting a 77k-pair conversation corpus to grow toward GPT-2 size, so we ran the size sweep: 5M → 30M → 85M → 125M, same data, same 6,000 steps, held-out loss the judge. The naive expectation — 'bigger is closer to GPT-2, so 125M wins' — was wrong. The 30M won (held-out 1.183), and 85M and 125M got WORSE. The train-loss column told us why: the big models' training loss was HIGHER too, so they weren't overfitting — they were undertrained, starved of steps by a CPU-bound data loader that made the 125M take 2.36 hours for 6k steps. So the answer isn't '4× the data' first; it's 'fix the loader, add steps, THEN add data' — and meanwhile, ship the 30M, because it's smaller, faster, and the best model we have. Also today: `werbos convo pull` (a git-pull for training data that gathers both mint lanes and grades them), the whole compose→mint→talk experience went live on the site at /demo (with a bring-your-own-document mode that runs a honest-abstain reader over your own text, no server, no claude), and the marketplace's npm-arc got blueprinted.
“the point of an un-bribable ruler is that it tells you the truth even when the truth is ’not the thing you were building toward.’”
We spent the week feeding the machine so it could grow a bigger talk model. Today we trained the whole ladder at once and let the honest ruler judge — and it judged against the plan, which is exactly what it’s for.
The size sweep — does bigger help on rich data?
Four models, same 77k-pair corpus, same 6,000 steps, held-out loss the only judge:
| size | params | held-out loss | train loss |
|---|---|---|---|
| 5M | 4.9M | 1.307 | 1.159 |
| 30M | 31.9M | 1.183 ← best | 1.026 |
| 85M | 85.7M | 1.225 | 1.038 |
| 125M | 135.7M | 1.231 | 1.079 |
The 30M won. Bigger got worse. And the honest, important part is why — read the train-loss column. If the big models were overfitting, they’d nail the training data and miss held-out. Instead their training loss is higher too. They’re not overfit — they’re undertrained: at a fixed 6,000 steps, a 135M model can’t even fit the data as well as a 30M. The samples show it — the 30M was coherent, the 125M looped (“what do you think of the most of the most?”).
The kicker: the 125M took 2.36 hours for its 6k steps, because the trainer is CPU-data-loading bound (100% GPU util at a telltale 77W). That’s what starved it. So the fix isn’t “4× the data” first — it’s fix the loader → add steps → then add data. And the product call is immediate: ship the 30M. Smaller, faster, better. The thesis holds again — small model, good data, honest measurement.
werbos convo pull — a git-pull for training data
The two mint lanes generate on two machines; this gathers them into one. Pull each SSH source, merge (dedup by input), run a deterministic quality gate that drops teacher junk with a reason (malformed / echo / refusal / artifact), and write a clean corpus plus a quarantine file. Proven live: 5,822 (PC-11 14B) + 180 (SR8 3B) → merged and graded, and over the following days the corpus climbed past 77,000 clean pairs — the data was the long pole, and it stopped being one.
The demo went live: compose → mint → talk — and bring-your-own-doc
The miner + runner shipped as a real page: /demo. Pick from the real 47-topic library,
watch it mint, and talk to the composed model right on the page — a client-side honest-abstain
reader that answers from your packs, cites the pack, and refuses everything else (topics it has
cards for but you didn’t compose → “not composed”; true unknowns → “not in library”). Then the mode
that matters: Import a document. Drop or paste a ~5-page .txt / .md / .docx (in-browser
.docx unzip, no library), it chunks the text into passages, and the same reader answers from your
document and abstains on everything else. No server, no LLM, no claude, nothing uploaded — the
honest-abstain moat, proven on a real Mandated Reporter Training doc (it quotes the material or says
“not in the material,” which is the only acceptable behavior for a compliance bot).
Blueprint: the npm arc
And a piece of the marketplace future got pinned to the wall: the npm arc — versions, updates, dependencies, forks, ratings, but for a registry that can grade its own packages. The differentiator no package manager has: grader-gated updates (an update can’t land unless the sealed eval says it didn’t regress). npm ships opaque code it can’t vouch for; werbos ships graded, signed knowledge.
Shipped:
werbos convo pull— gather + grade the conversation corpus across machines (d410e01)- Self-contained GPU talk trainer + the size sweep — the honest scaling verdict (a00e8ec)
/demolive: compose → mint → talk, honest-abstain, on the site (67c4073)/demotwo modes: pick topics or import a document (client-side.docx, no claude) (88d6dae)/demofixes: local accent tokens (the site never exposed them), stricter doc-mode abstain (3969ed0, 9b01c05)- Blueprint: the npm arc — a package registry that grades its own packages (8de316b)
- Finding: on 77k pairs at 6k steps, the 30M is the sweet spot; bigger is undertrained, not overfit — fix the loader before scaling.