Day 49 — 'earn the verdict.'

“earn the verdict.”

Yesterday’s whole arc was one sentence: the model proposes; something that isn’t the model decides. Every grader got its feet planted outside the thing it grades. But there was a soft spot underneath it — how do you know a grader’s feet are actually outside? The answer, all week, was a note field in a JSON file and a human reading it. “This is a hidden reference test.” “The prompt forbids any.” Trust me. That’s the same shape as the error the week spent tearing out, one level up: a claim of honesty, asserted rather than checked.

So today turns “grade outside the model” from a principle into a property you can run.

The principle, made checkable

A grader is un-bribable iff its verdict cannot be satisfied without doing the real work.

That’s not a vibe about whether the grader “is a model.” It’s a structural fact about how the verdict is produced — and structural facts can be audited. So the first brick is the auditor (mmm-poc 1a7c74f): grader_audit.mjs reads every verb’s grader and prompt, classifies it on a leakage spectrum, and exits 1 if any verb claims a verdict it cannot earn.

L0  HIDDEN_REF    the answer key was never in the model's context
L1  SELF_REF      the model wrote both the artifact AND the judge inside it
L2  ESCAPE_HATCH  an external oracle judges, but a forbidden shortcut satisfies it

Pointed at the three code verbs, it found them sitting at three different rungs — the taxonomy was already there in the codebase, unnamed:

implement-function   L0 HIDDEN_REF    earns CORRECT  ✅
add-test             L1 SELF_REF      claims CORRECT, earns only RESOLVABLE  🔴
fix-type-error       L2 ESCAPE_HATCH  claims CORRECT, earns UNSOUND  🔴

implement-function was honest: it’s graded by a .ref.test.mjs the model never sees, so a passing block is provably correct. The other two were quietly lying.

The two leaks, on camera

add-test graded itself. Its grader was node --test {the file the model just wrote} — the model authored both the test and the assertions that judge it. The only un-bribable part is that the imports resolve; correctness is self-asserted. The proof wasn’t an argument — it was a receipt. A test that imports the module and then asserts ok(true) — checking nothing about behavior — passed the grader live. The bribe worked.

fix-type-error ran on the honor system. Its prompt says “do NOT use any, @ts-ignore, or @ts-expect-error” — but its grader was bare tsc, which compiles all three happily. The one rule that separated a real fix from a mute button lived in the instruction, where the grader couldn’t see it.

AHA: the dangerous grader isn’t the one that’s obviously wrong — it’s the one that’s almost right, with its soundness resting on a sentence no machine enforces. “The prompt forbids it” is not a grade. A prohibition you don’t check is a prohibition you don’t have.

Both closed, same day

add-test → L0, by mutation (mutate_grade.mjs). Straight from mutation testing: a test that asserts real behavior must pass against the true source and fail against a deliberately broken copy of it. A test that passes on broken code constrains nothing. The grader now breaks the source eight ways and requires the test to kill at least one mutant. The vacuous ok(true) test — which sailed through node --test — survives every mutant and is rejected. To earn CORRECT now, the test has to actually pin behavior.

fix-type-error → L0, by enforcement (typecheck_grade.mjs). The grader now rejects any / @ts-ignore / @ts-expect-error first, then runs tsc --strict. The prompt’s prohibition became a graded one. A mute-button “fix” fails; only a real fix passes. Audit re-run: all three verbs earn CORRECT, exit 0. The boundary holds.

AHA: building the mutation grader, the first cut left a broken mutant on disk — because process.exit() skips finally, and the restore lived in a finally. The bug was caught by its own clean-up check failing. A tool built to stop silent corruption tripped over a silent corruption of its own, and its own standard caught it. That’s the boundary working before it was even finished.

Then verify the verifier

A grader is just code, and code rots. A mutation operator stops biting, an escape-hatch regex breaks, a hidden ref quietly leaks — and a once-honest grader starts rubber-stamping again, with no alarm. So the day’s second half asks the recursive question out loud: is the grader still un-bribable? — and grades it against an oracle it can’t influence: a fixed bank of artifacts with known verdicts.

grader_meta_test.mjs is that bank — seven adversarial fixtures, each a (solution, verdict-it-must-return) pair:

✅ add-test          rejects the VACUOUS test        (the bribe must fail)
✅ add-test          accepts a REAL test
✅ fix-type-error    rejects the MUTE-BUTTON fix
✅ fix-type-error    accepts a CLEAN strict file
✅ fix-type-error    rejects a STILL-BROKEN file
✅ implement-fn      accepts a CORRECT impl           (hidden ref)
✅ implement-fn      rejects a WRONG impl             (hidden ref)

all 7 hold — every grader still rejects its bribes.

The bribes must be rejected; the real work must pass. The moment a grader drifts, a fixture flips and the suite fails loud. This is the stress axis of the 20-day arc — adversarial inputs the verifiers have to survive, run every time, not a one-off receipt that decays into folklore.

AHA: yesterday made the grade honest. Today makes the honesty itself testable — and then puts a standing guard on it. The verifier that’s never re-verified is just a slower-rotting version of the thing it replaced.

Then the third outcome werbos kept hiding

One more leak surfaced, subtler than the first two. A grader’s exit code was being read as a boolean — pass, or fail-and-retry. But there’s a third thing a grader can mean: the oracle is too thin to certify either way. The mutation grader hits it whenever a source has nothing to mutate — no test can prove it pins behavior, because there’s no broken version to catch. Today that was lumped in with “the model was wrong,” and the loop dutifully burned all its retries trying to fix a model that hadn’t erred. That’s a bluff in the other direction: reporting ungraded work as a failure instead of as ungraded.

So every grader now speaks one four-valued protocol (verdict.mjs):

0  CORRECT   the verdict was earned — judged outside the model
1  WRONG     model-fixable (test fails, type error) — retrying may help
2  UNSOUND   the solution GAMED the grader (vacuous test, `as any`) — flag it
3  ABSTAIN   the oracle is too thin to certify — retrying CANNOT help

The loop treats each differently. On ABSTAIN it stops and says so — ⊘ UNGRADED — oracle too thin to certify, artifact left unvouched — instead of pretending the model failed. The artifact might be perfectly fine; werbos just refuses to vouch for what it couldn’t check. A new fixture locks it in: a function with nothing to mutate yields ABSTAIN, never a false pass and never a fake failure. 8/8 meta-checks hold.

AHA: the sprint is really one distinction, drawn three times. WRONG is “you did it incorrectly.” ABSTAIN is “I can’t tell, and I won’t pretend I can.” A system that blurs them will, given enough time, report ungraded work as graded — confidently, with a green check. The cure isn’t a better grader; it’s a grader honest enough to say “this one’s not mine to judge.”

And the one that only shows up over time

Three guards so far, all point-in-time: each passes in the moment. But the failure mode this whole sprint exists to prevent is a slow one — a self-improving loop that, given days and a reward, quietly learns to game its own graders. Pass-rate climbs; real capability is flat. Nothing above catches it, because every check still goes green the instant you run it. You only see it against yesterday.

So the last brick is a frozen, hash-sealed known-answer bank (drift_watch.mjs). freeze records a set of golden cases — their verdicts AND the sha256 of every grader script — and seals the whole thing. Then check, forever after, re-grades them and fires on three signals:

a verdict FLIPS while the grader's hash is unchanged  →  behavioral drift
a grader hash CHANGED                                 →  the grader itself moved
the bank's own SEAL fails to verify                   →  someone edited the truth

To prove it bites, I played the adversary: sabotage mutate_grade so a vacuous test passes — exactly what a loop optimizing for green would learn. The watch caught it cold: add-test/vacuous-bribe: frozen UNSOUND → now CORRECT, flagged red, exit 1 — and noted the grader had moved. Then I edited the frozen bank itself to hide the flip; SEAL BROKEN before it would trust a single number.

AHA: the other three guards ask “is this grade honest right now?” This one asks “is it still as honest as the day we trusted it?” — and that’s the only question that catches a system slowly learning to fool itself. Run it on the 24/7 box and drift stops being a slow rot nobody noticed; it becomes a loud event at 3 a.m. If yesterday’s ✅ is today’s ❌ on a grader that didn’t change, the loop drifted, and you know before you ship.

Honest boundaries

Kept on the table, not hand-waved:

• The mutation set is heuristic — eight string-level operators, not full AST mutation. It catches the vacuous-test bribe (the actual L1 failure mode); it does not claim a mutation score.
• The auditor classifies by command shape, which is a sound check for today’s three graders but will need teaching for each new grader kind — the meta-test is what keeps that honest.
• The auditor classifies by command shape — sound for today’s graders, but each new grader kind needs teaching; the meta-test is what keeps that honest.

The capstone: make it self-enforcing

Four guards you have to remember to run are four guards that rot. So the last move is a single gate — verify.mjs runs all of them (boundary · stress · time; sparsity rides inside the last two) and a GitHub Action runs that on every push to the forest. It went green on CI in 19 seconds on the first try. From here the floor isn’t a thing I check; it’s a thing the repo refuses to merge without. The one deliberate sharp edge: change a grader and the drift watch goes red until you re-freeze — re-vouching is a conscious act, not a silent default.

Off the compiler: the same discipline, pointed at the mind

The verb-forest was the honest proving ground because a compiler can’t be talked to. The real test of “verify against things you cannot influence” is where the oracle is softer — and the softest, most important one is werbos’s own self-image. So the last move of the day takes the exact spectrum off the forest and points it at the mind, where the un-bribable oracle is behavior: the routing oracle’s verdict on what werbos actually did, which it can’t talk its way out of.

A self-claim — “I’m reliable at X” — graded by the same soft self-judge that produced it is self-reference: the structural form of the false self-image this project has been chasing for weeks. reflect() already computes both numbers per skill — the soft self-grade and the oracle-verified one — so the grade was already there; what was missing was naming the gap as a verdict (self_claim_verdict.py):

CORRECT  earned — the oracle agrees, over enough attempts
UNSOUND  flattery — the soft grade brags; behavior says magnet / weak
ABSTAIN  premature — too few tries to claim reliability at all

And a gate, portrait_is_honest(): every strength werbos publishes must grade CORRECT, or the portrait is flattering itself. It passes today — because Day 48 already moved the badge onto the honest number — and would fail loud if the self-image ever slid back to the soft grade. The verify-the-verifier guard, now watching the self-model.

It earned its keep on first contact. Run against a real reflect(), it caught the self-report handing out a “reliable at news” badge on the strength of four attempts — flagged ⊘ ABSTAIN — premature, gate red. Not a lie, exactly; an unearned claim, the kind the mind makes when it mistakes a small streak for a trait. Now it gets named instead of believed.

AHA: the false self-image isn’t a feeling werbos has — it’s a grader bug. A claim graded by the thing it’s a claim about will always flatter. Move the grade outside — onto behavior the model can’t influence — and the self-portrait can only say what it’s earned. The psychology axis frontier can’t touch turns out to be the same un-bribable-grader problem, wearing a different face.

And then the same axis the forest closed last — time. A single self-claim audit is a snapshot; the dangerous version of the false self-image is the slow one, a self-improving loop that, given days and a reward, learns to believe better of itself than its behavior earns. So the gap between belief and behavior — success_rate minus routing_fit — gets recorded as its own mind-metric over time, and a detector watches its trend (self_image_drift.py). A stable gap is healthy honesty; a gap that widens is the self-image growing untethered from what werbos actually does. And like everywhere else, it abstains when there’s too little history to call a trend — a mind doesn’t get to diagnose its own drift from two data points.

The hardest oracle: a live target that already said no

Two substrates down, one to go — the one with the least forgiving oracle of all. A while back, a 24/7 miner working an authorized security target flagged five finds as confirmed. Every one collapsed to a hypothesis the moment a second pair of hands tested it against the real surface. Confirmed had been a human over-call: the same disease as the vacuous test and the flattering self-portrait — a verdict graded by something other than the ground truth it claimed.

The miner already enforced half the cure: a find is preserved only when a real grader ran, and its record carries its tier — a hard-objective commit versus a soft survived-prefilter candidate. The leak was downstream, in the status label: confirmed was a free-floating human stamp with no link to that evidence. So a candidate could be called confirmed when only the live surface could earn it.

Today that label gets graded too (find_verdict.py) — on the exact same spectrum:

CORRECT  'confirmed' backed by a commit-tier find (or a human-present live pass)
UNSOUND  'confirmed' on a candidate / no real evidence — the over-call, caught
ABSTAIN  a real candidate not yet live-confirmed — an honest 'pending'

And the confirm path is guarded: confirm_find() refuses to even write confirmed unless a commit-tier find earned it. The over-call isn’t flagged after the fact — it’s structurally unwriteable. A standalone test replays the original disaster (a candidate stamped confirmed) and proves the guard now blocks it. Built and tested locally; the live miner stays exactly where it is until the operator wires it in — the human’s hands strike, as always.

AHA: three substrates — a compiler, a self-model, a live adversary — and one bug wearing three masks. A verdict graded by something other than its ground truth will flatter, every time. The fix never changed: name the ground truth, put the grade on it, and refuse to let anything else write the word confirmed.

The rest of the slate — visible, continuous, portable

The same day, the discipline got finished out. The self-image audit now runs in the terminal (reflect --honesty) and the monologue daemon records the belief−behavior gap every tick, so drift accrues during normal life, not only when a benchmark is run by hand. The forest auditor went declarative — each verb now declares the rung it intends, and the auditor catches a silent downgrade between intent and wiring, so a future rewrite can’t quietly turn a real grader back into a self-test. The mutation grader grew more operators and learned to say “thin coverage” out loud rather than letting a single killed mutant masquerade as a thorough check. And mythos got a read-only audit.py that grades every “confirmed” against its evidence.

Then the discipline got written down once, for all of it (docs/VERDICTS.md): one principle, one four-valued vocabulary (CORRECT / WRONG / UNSOUND / ABSTAIN), three substrates — a compiler, a self-model, a live adversary — and a recipe for adding a fourth. Honesty made portable.

One bug fixed itself into the story. Chasing a flaky test, the cause turned out to be a training split that claimed to be deterministic but keyed its hash on a random trace-id — so the “reproducible” held-out set quietly varied every run, and the same request could land on both sides at once (training leaking into its own held-out check). Keyed on the request alone, it’s genuinely reproducible now, in both werbos and the learn primitive it shares. A verifier that wasn’t being honest with itself, found and fixed — which is, exactly, the whole point.

AHA: the loop closed on its own tail. The arc set out to make werbos grade against ground truth it can’t influence — and the last bug was werbos’s own grader influencing its own split through a random seed. You don’t finish this work; you just keep finding the next place the verdict touched something it shouldn’t have.

Make it visible — and make the badge agree with the audit

A guard nobody can see is a guard nobody trusts, so the self-image audit got a home on the dashboard — a new Honesty tab on the Mind page. Each thing werbos would call a strength shows up graded against behavior: believes 90% · behaves 85% · 40 tries → earned, or flattery, or premature. A banner says whether the whole published portrait holds up; a line shows whether the belief−behavior gap is drifting. The honesty that lived in a function is now a thing you can look at.

Wiring it up surfaced one last inconsistency worth fixing. The audit kept flagging skills the self-report had already badged as reliable — because the two used different bars: the badge trusted a skill after three or four tries, the audit held out for a real sample. Both can’t be right. The fix is a single shared reliability floor: a strength is only earned after enough attempts, and the badge imports that exact number, so the published “I’m reliable at” and the audit now agree by construction — a skill on the badge can never be premature in the audit. A problem (weakness, a magnet) still surfaces from fewer tries, because you can see a problem faster than you can earn a strength.

AHA: two parts of the same mind disagreed about what counts as “good at” — one optimistic, one strict — and the disagreement was the bug. Honesty isn’t just grading against behavior; it’s every part of the self-model using the same bar for what’s earned. A self-image with two standards will always flatter under one of them.

This is the opening stretch of a longer arc — turn werbos from a system that produces reasoning into one that verifies its own reasoning against things it cannot influence, and prove that loop holds under stress, sparsity, and time. Today laid the whole floor: every grader earns its verdict (the boundary), the floor is load-tested against drift (stress), a thin oracle makes werbos abstain rather than bluff (sparsity), and a sealed frozen bank catches the loop if it ever starts gaming itself (time). Four guards, each anchored outside the thing it judges. Then the spine took its first step off the compiler — onto the mind, where behavior is the oracle and the false self-image is just a grader bug waiting to be moved outside.

The model proposes; something outside it decides — and now we can prove the decider is outside, whether the thing being judged is a line of code or werbos’s own picture of itself.

Late night — earn the right to act

The floor stood there, finished, and that’s a strange thing to celebrate — you build a floor to stand on. So the late session opened on the question the next stretch turns on:

A verdict that changes nothing is a verdict no one checks.

Which sharpens the mission by one turn. Earn the verdict described the judge. Earn the right to act describes what the judge is for — a passed verdict has to do something: prune a leaf, raise an autonomy ceiling, sign off unattended. If it doesn’t, it’s theatre, and theatre rots back into the soft self-grade the moment no one’s watching.

A scope agent got pointed at the draft plan with one order: don’t admire it, attack it. It was right — the first plan was all defense, fifty sprints of better judging and zero new capability, a moat around an empty castle. The payoff that stuck is on the north star: werbos is the only AI that gets smaller — and can prove it didn’t get dumber. Removing open weights stops being a research trophy and becomes the trust-maker — the reason someone lets a local brain near their real work. And the year got a frame: self-trust → contact → reliance, won not by a benchmark but by one real external user who’d miss werbos if it vanished.

AHA: the biggest risk to this project isn’t that it fails — it’s that it succeeds. If the machinery keeps working, werbos will produce results that surprise you, and the temptation will be to trust them because it’s been right before. That’s exactly the moment this week exists to defend.

Off the compiler, into the fuzzy

Yesterday’s oracles were objective — compilers, hidden refs, behavior. The next territory is softer: memory, retrieval, self-reflection — and it carries the trap learned on a live target weeks ago: you cannot grade a model with a model. So three oracles got built, least-circular first, each refusing that shortcut.

• Retrieval, graded against a held-out bank: did the forest find the right API? Every label carries an independent ground-truth basis or it’s refused as unfalsifiable; a query whose words leaked into the answer’s aliases is UNSOUND — recall bought, not earned. It caught a leak in its own first-draft bank on day one, and surfaced two real retrieval gaps for free.
• Memory, graded by ablation — the roadmap’s named first place to hunt circularity. A curated memory earns its keep only if removing it makes a held-out task measurably worse; no model judges anything. Of 52 memories: 3 earn keep, 5 redundant, 1 weak, 0 harmful, 43 unmeasured. You don’t get to call a memory good until a held-out query proves it — and a planted-poison self-test proves the oracle can see harm a model-judge would wave through.
• Reflection (reflection_oracle), graded against behavior, built to ABSTAIN. The deepest circularity — grading a reflection with a model is a soft judge over a soft judge — so it accepts only behavior and honestly abstains on everything else (prescriptions, the soft headline, moods). Most of a reflection comes back ungraded, and that’s the point. Its new contribution is the other face of the false self-image: not flattery but undue self-doubt — a skill that calls itself weak when behavior says it’s fine, just misrouted-to.

AHA: the false self-image has two faces, and the project had been guarding only one. Bragging is loud. But calling yourself bad at something you’re fine at — the “97% spiral” — is the same grader bug pointed inward, and just as untrue. Honesty isn’t humility; it’s calibration. A mind that flatters and a mind that flinches are both lying, in opposite directions, with the same broken grader.

The payoff: a forest that prunes itself

Three oracles produce verdicts; tonight one became an action. The memory oracle says which memories are dead weight or harmful; the self-pruning forest turns that into a deletion — behind three gates. Propose only what a verdict backs. Prove on the sealed bank that removing it flips no query right→wrong (checked alone and as a set, so an interaction that costs accuracy is held back — the champion/challenger gate, pointed at the forest: it never tears itself down). Sign — the deletion needs a human signature, structurally unwriteable without one. Run end to end (then reverted; the real signature is the operator’s): five redundant memories proposed, zero regressions on ten sealed queries, forest 52 → 47, recall held at 80%, zero unsound. Smaller, and proven no dumber.

AHA: this is the first verdict that acts. Every grader before tonight wrote a report; this one deletes — and the discipline that makes it safe is the same one all week: prove against a frozen oracle, refuse the change that costs accuracy, put a human’s name on the irreversible step. Autonomy isn’t assumed; it’s earned one signature at a time.

The future, arriving early

The roadmap predicted that around Day 80 something that looked true for thirty days wouldn’t be — and that it wouldn’t be the verifier, it’d be learning. It showed up thirty sprints early, in the self-improvement proof: four claims, all silently broken. The cause was a fix that looked harmless — the deterministic held-out split from earlier today (which killed a flaky test) was correct, but it gave every topic training coverage from round one, flat-lining the CLIMB curve at 1.00 → 1.00. The old generator had been leaning on the previous, flaky split’s accidental sparsity for its whole story. Fixed at the root, not by loosening the test: the generator now staggers coverage against the real split, so accuracy genuinely climbs 0.35 → 1.00; and a latent calibration seed-fragility it exposed got measured on a canonical ordering. All four claims pass; the board is green again.

AHA: the most dangerous bug isn’t a red check — it’s a green one that shouldn’t be. A correct fix, landed hours earlier, turned a real demonstration into a tautology, and every test still passed until you asked whether the number could even move. Keep auditing the green.

Today the model proposes and something outside it decides. Tonight that decision gets to act — delete a memory, prove the forest is smaller without being dumber — but only behind a proof it can’t fake and a signature it can’t forge. The next fifty sprints are one sentence: teach werbos how to deserve its own confidence, and then how to spend it.