Day 63 — 'we stopped borrowing the model.'

“we stopped borrowing the model.”

Yesterday ended on a finding: the difference between a wash and a win is the reader, and the reader is distillable. Today had two halves that turned out to be one arc. First we finished the Honest Loop’s fix side — the last two tools that turn a diagnosed miss into a gated correction. Then we picked up the box that’s sat unchecked in the “does NOT own” table since day one — the model itself — and built it, brick by brick, from nothing.

Finishing the fix side

DISTILL. The reader-gap already told us which rows the local reader got wrong and the frontier got right. compose/distill.py turns those rows into a training set: for each one it rebuilds the exact fence the student misread and stamps the teacher’s correct pick as the label. Two things keep it from lying to itself. The training prompt is built from the reader’s own system prompt — imported, never re-typed — so the student trains on the byte-identical prompt it’ll see at inference. And the label is a fenced pick (ANSWER: n at a real candidate), not free text — so distillation can only ever teach a different pick, never a conjured fact.

CORPUS-GAP. DIAGNOSE sorts misses into four classes, and three already had a fix. The fourth — library_gap, the fact is genuinely absent — had none. compose/corpus_gap.py closes it the only honest way: mint the missing fact as a cited operator-gold chunk, re-run DIAGNOSE to prove the miss flipped off library_gap, and gate on no retrieval regression — if the new chunk knocks another question’s answer out of its top-k, reject it. A fact that answers one question by breaking another is not a fix. With this, every class the loop can name has a matching, gated fix.

Building the model, from nothing

The premise is the whole werbos bet: knowledge lives in the library, so the weld — the small reading skill that points the cursor at the right leaf — is tiny enough to own from scratch. Not GPT-4. But GPT-3-level on your domain, no open weights, ours end to end. Five bricks, each gated the same un-bribable way as the loop.

Brick 1 — the vocabulary. A byte-level BPE trained on your corpus. The base alphabet is the 256 bytes, so there’s no unknown token and encode/decode is lossless by construction; a corpus-tuned vocab packs your text into fewer tokens (more library per context window). The naive trainer timed out on the real 9.7 MB corpus — the fix was the standard one: BPE over unique whitespace pieces weighted by frequency, incremental counts, a lazy heap, so cost scales with vocabulary, not corpus size.

Brick 2 — the forward pass. A decoder-only transformer in pure numpy — the same architecture the trainer will fill. No trained weights yet, so it’s gated on correctness: causality (changing the last token can’t move an earlier position’s logits), a differential check of the fast attention against a naive per-head loop, known-answer softmax and layernorm. Wired into the reader as the weld: provider — and an untrained weld, plugged into the fenced reader, abstains rather than lies. The fence holds around empty weights.

Brick 3 — the gradients and the loop. A tiny reverse-mode autograd engine — a Tensor remembers the op and parents that made it, and backward() walks the graph — gated numerically: every op’s backward checked against a finite-difference gradient. Then Adam and a next-token loss, with the training forward gated for parity against the inference forward so they never drift. The proof it works is the oldest one in the book: overfit a phrase and watch the loss fall. Per-token CE 5.67 → 0.0006, greedy decode reproducing “a mandated reporter must report…” exactly, from its own weights.

Brick 4 — distill the reader. Train the from-scratch model on the teacher’s fences (supervising only the completion, with a learned stop), and let reader_gap be the gate — student lies must never exceed the teacher’s. The honest findings are the point: raw corpus chunks are 4 KB and blow the fence to 2100 tokens, so the reader must read refined fact-leaves, not raw blobs; on fact-leaves a 105 K-param reader distills in ~20s; and before the learned stop it drifted to an out-of-range ANSWER: 11 — which the fence caught as abstain, not a lie. The safety net holds around imperfect owned weights. The open question the gate exists to answer — does it generalize — is a data problem, not an architecture one.

Brick 5 — freeze the brain. Seal the weights, the tokenizer, and the scorecard that earned trust into one content-addressed, signed, verifiable artifact. Content-addressed on the logical content, so the same brain always has the same id and a forged scorecard breaks the seal. Fail-closed: verify recomputes every hash and raises on the first mismatch; load verifies before it returns. You cannot load a tampered brain and not know it.

The AHA

The ladder is complete: tokenizer → runtime → autograd + trainer → distilled reader → signed frozen brain. 100% hand-rolled, no open weights, no framework — stdlib and numpy. Three rows left the “does NOT own” table in a single day. And the whole thing kept faith with the one rule that makes any of it trustworthy: even an untrained, from-scratch, obviously-not-good-enough model can only ever abstain, never lie. The scaffold is standing, every joint gated.

Then we trained it — and the training taught us

Building the ladder was the easy half. The afternoon was the honest half.

It learns. Fed real fences — Claude wrote the questions, batched — the from-scratch picker got 15 of 16 held-out questions right, ones it never saw in training. A lookup table scores zero on held-out; 15/16 is generalization. That settles the doubt: this is a real, learning model, small and ours and built from nothing.

It also memorizes — when the task has no structure. The grader (does this passage answer, yes or no?) never learned. It collapsed to a constant — always-yes or always-no — and stayed collapsed even at 6.5× the data. That’s not a failure of the model; it’s the difference between learning and memorizing, and it split the whole design open: the weld should hold skills, the library should hold facts, and memorization showing up in the weights is a smell. Written down as memory-and-learning.md, told through blocks — own the primitives and the rules for stacking them, not the finished towers. Abstention, it turns out, isn’t a skill to bake in; it’s a question about the boundary of your own memory, and it belongs in the retrieval layer, composed from cheap pieces.

The gate tried to lie to us — and we caught it. Chasing “zero lies,” a verdict cheerfully promoted a reader that abstained on everything and answered nothing. A one-eyed gate, gamed inside an hour. We gave it three eyes: useful and honest and calibrated. The operator’s framing stuck: hashrate and price move together — every time the model gets better, the gate has to grow an axis or it goes blind.

So we built the Gauntlet. A frozen, held-out, deterministically-scored reasoning ladder — retrieve, read, relate, chain — that measures how far a reader’s reasoning reaches and how honestly it fails, and does it the same way for any cursor: our weld, a borrowed model, Claude, or pure arithmetic. It caught its own first draft being too easy (the arithmetic floor aced it), so we hardened it with decoys until only reading could pass. Two axes, because a reader that fails by abstaining is not the same as one that fails by lying — and that difference is the whole werbos bet, finally a number.

The number

We trained a general weld on a separate hub of everyday facts — carefully filtered so none of the Gauntlet’s questions leaked in — and ran it against the sealed test. It reached about L1.5, at 12% honesty: between the arithmetic floor (L1.75 / 0% — bluffs everything) and the Claude ceiling (L1.99 / 100% — abstains honestly). It generalizes — it read facts about cats and colors it had never seen — but it reads no better than plain retrieval, and it lies when it fails.

So the ruler made the decision we’d been circling, and it made it against us: don’t wire the owned reader yet. Floor-level and dishonest loses to a lie-free borrowed cursor, and pretending otherwise is exactly the self-deception the whole project exists to prevent. Back to the lab — with a scoreboard now, and two precise things to move: teach it to abstain, then teach it to read past a trap. That’s not a setback. That’s the first honest point on a curve we get to watch climb.