Day 92 — the ruler was wrong twice

“a corpus 200k vs 1.1M ..isnt a fair fight if youre only comparing via punctuated sentances”

Two of today’s biggest findings were bugs in the measuring device, not the thing measured.

The EOT fix worked, and was only 15% of the problem

The 46M stopped mid-sentence. Yesterday’s diagnosis: \n does double duty — paragraph break in books, turn terminator in chat — so a base pretrained on prose learns that \n is always plausible and the fine-tune can’t reclaim it.

A reserved terminator (byte 0x01, which cannot occur in the corpus) fixed the mechanism, and because tokenize_corpus emits it at document seams, pretraining teaches it — the base inherits a prior that means “a complete unit ended”, which is exactly what a turn end is.

Result: clean 0.46 → 0.54, input-sensitivity 0.85 → 0.94. Better. Still failing.

The other 85% was the corpus, and the operator spotted it

talk_bench’s clean floor requires terminal punctuation. It therefore structurally cannot pass a model trained on the casual register, however good that model is.

Same base weights, only the fine-tune corpus changed: clean 0.54 → 0.96.

I had made this worse, not better — my earlier “the corpus is 92.4% punctuated” came from reading the first 60,000 rows of a file that isn’t uniform.

A new champion, and an axis that had never moved

talk-2m-live          clean 0.96  input-sens 0.90  ood 0.00   (previous)
talk-46m-eot-merged   clean 0.96  input-sens 0.92  ood 0.17   SHIPPABLE

ood 0.17 is the first non-zero score any checkpoint has ever produced. talk_bench’s own docstring lists it as standing debt: “No checkpoint signals confusion at glunglu florp?. That is tracked as debt, not solved.” It moved on the model that read 973 books. One probe of six — don’t over-read it — but zero models had ever done it.

Then the ruler was wrong a second time

score_clean inferred truncation from the text: no terminal punctuation and more than five words meant “truncated”. Measured properly — 4,000 real corpus responses against the same responses cut at a random interior word — every word-list variant is bad. The length clause is wrong on three quarters of a casual corpus; dangling-word-only catches 26% of real cut-offs.

The signal was never in the text. The bench generates the replies, so it already knows: ended on the stop token = complete, hit the length cap = cut off. Ground truth, free.

                    clean (before -> after)
talk-2m-live         0.96 -> 0.96    unchanged — its 2 misses are real mojibake
talk-46m-eot         0.54 -> 1.00    all 22 "truncations" were false positives
talk-46m-eot-merged  0.96 -> 0.98

The champion being unchanged is the control that matters: the fix moved verdicts only where the heuristic was lying.

Turn two of every conversation returned Core Web Vitals

“good, what about you?” → "[library] LCP measures loading performance; a good LCP score is 2.5 seconds or less."

what about you classified as None, fell through to a 22k-card library, and "good" matched “a good LCP score”. The reciprocals list had how about you but not what about you, and you, hbu, wbu.

The tempting fix — gate the library behind “can you help me find…” — is the same substring disease that caused it. The real defect is that retrieval is the fallback: anything no lane claims lands in the library, and the library always answers. There’s no confidence floor on that path.

The farm, sized for the second node

mint_group_talk asked the teacher for a 12-turn group conversation and then flattened it into isolated consecutive pairs — generating exactly the multi-turn data the model needed and discarding all of it. It now carries the prior turns as context, which the trainer’s existing --context flag already encodes.

Three more, because 1→2 nodes is where the coordination bugs live and finding them on hardware you own is free:

• A dyad is not a small group. The prompt said “GROUP” at every size, so even 2-person runs produced “you guys” — and that register leaked into 1:1 chat the moment context was present.
• --shard i/N partitions the 900 (activity, tone) combos by stable hash — verified disjoint and complete at N=2/4/8, no coordination, a node can die without stranding work.
• --workers N fans generation over threads: 3.87x on orchestration (I/O-bound scaling; on one card expect 1.5–2.5x, and measure it).

Three surfaces lying about state outside the container

/mind still named a reflect() source replaced days earlier. The metric store had split in two — host mind-bench writes ~/werbos/experiences, the dashboard reads /data/experiences, so a snapshot taken the obvious way is invisible forever. And /models listed Ollama, MLX and cloud API keys: every model except the ones we trained.

Also: Path("F:\\...").name doesn’t split backslashes on POSIX, so no BPE model trained on the GPU box could ever be served from SR8 or the Mac. That one was a hard blocker for the entire book-pretrain line.

The AHA

A ruler that penalises a register, and a truncation test that reads the text instead of asking the generator, both produce numbers that look exactly like results. The corpus comparison and the truncation rule were each wrong in a way that made a good model look broken — and neither would have surfaced from reading the code.