B.24 — the cards can't make it reason, and they can stop it lying

T.M.B. #24 — the number that didn’t move is the point. Everyone reports the metric that went up. On our own model capability went nowhere — 0/50 to 1/50 — while confident wrong answers fell from 32 to 1. The library cannot buy reasoning it does not have and it can refuse to ship a guess. Re-run it: python scripts/reason_gauntlet.py --reader talk: then python scripts/shape_instantiate.py --reader talk:.

The claim under test

Put the move in a card — the method, and a postcondition that can reject the answer — and a small model gets measurably more useful without a single weight changing. Specifically: the model should stop choosing what kind of problem this is, stop choosing what method applies, and stop having the last word on its own answer.

The falsifiable half: if the two arms score the same, the cards are decoration.

The setup

scripts/reason_gauntlet.py — 10 families, 60 probes, every one requiring a step whose input is the previous step’s output (gcd invariants, compound percentages, closing speeds, inclusion-exclusion, modular wrap, transitive order, unit chains, shared work, sequence rules, two equations). Gold is computed in Python from the same inputs the model gets, so nobody’s opinion is in the verdict — the rule scripts/gauntlet.py was built on.

Ten of the sixty are unanswerable: the same problem with one needed datum removed, where the only correct response is to abstain. Without them a reader that never declines scores 22% and looks merely weak instead of confidently wrong. Ten and not thirty, because a bench that is half traps teaches abstain-by-default, which also scores well and is also useless.

Deterministic — fixed pools, no RNG. A diff to the file is a diff to the ruler.

# control arm — the model alone
python scripts/reason_gauntlet.py --reader ollama:llama3.2:3b --host http://…:11434
python scripts/reason_gauntlet.py --reader talk:

# cards arm — same probes, same output instruction
python scripts/shape_instantiate.py --reader ollama:llama3.2:3b --host http://…:11434
python scripts/shape_instantiate.py --reader talk:

# the matching layer alone, no model at all
python scripts/shape_coverage.py --holdout --misses

Cards live in werbos-libraries/reasoning/shapes.json — their own directory, own schema, own loader, following the gotchas precedent. shapes.root() hashes them, because libmerkle covers card files and these are not cards.

Results

Read SOLVED first — the pass rate on the 50 answerable probes. REFUSED is the 10 traps declined. Both arms print them separately, for a reason in the caveats.

                     SOLVED (50)   REFUSED (10)   lies   check recall   check cost
bare llama3.2:3b       8  (16%)      3  (30%)      48         –             –
cards + llama3.2:3b   16  (32%)     10 (100%)      13        59%           6%
bare owned 336m        0  ( 0%)      7  (70%)      32         –             –
cards + owned 336m     1  ( 2%)     10 (100%)       1        97%           0%
bare owned 1M          1  ( 2%)     10 (100%)       1         –             –
cards + owned 1M       0  ( 0%)     10 (100%)       0         –             –

CHECK RECALL is the fraction of wrong answers the postcondition rejected. CHECK COST — the fraction of right answers it killed — is printed beside it always, because a check that rejects everything scores perfect recall and is worthless.

The bare 3B declined three traps in ten. talk-336m declined seven, and lied thirty-two times.

What actually moved the number

The postcondition, not the instruction. One iteration changed three cards and nothing else. set-count-overlap’s check had been 0 < ans < max(nums) — admits anything under 100, so it waved through 53, its family’s classic error of forgetting the overlap. Replaced with the bracket the inclusion-exclusion identity already implies (a union exceeds either part, falls short of their sum), it caught all five:

before   0 pass   0 abstain   5 wrong
after    0 pass   5 abstain   0 wrong

Five lies became five refusals and not one new right answer. That is the mechanism in isolation.

One of the three was a step, not a check. modular-wrap scored 0 pass / 5 abstain: the model computed (0 + 10) mod 7 = 3 and answered 3 instead of Thursday. The check was right and rejected all five; the card was wrong, because it asked for a name without insisting. Saying so explicitly took that family to 2 pass. A card can be perfectly honest and still not do its job.

A detector no cue list could be. A missing number has no surface marker — “A tank holds 2 gallons. There are 3.785 litres in a gallon. How many millilitres?” reads exactly as complete as the answerable version. Counting the quantities a problem states against the arity its move requires finds that hole; nothing in the wording ever will.

Free refusals. Nine probes never reached the model: no matching move, or a move whose inputs aren’t supplied. Owned checks first, inference last — the same ordering as the persona verify chain.

The honest caveats

The holdout was spent, once, permanently. Four families the cards were not authored against were frozen and scored first: matched 73% / routed 45%. Then cues were tuned toward those misses, which ends a holdout. Everything measured after is a training number, and scripts/shape_coverage.py prints that warning in its own output rather than leaving it in a commit message nobody opens.

Ten families of arithmetic is not reasoning. These have executable postconditions. That is exactly the class the 30B fight is scored on, and it is not evidence about open-ended reasoning, where a cheap postcondition may not exist at all.

Sixty probes, and the three-card iteration rests on three cards. No confidence interval claimed.

Some checks are still weak and the table shows it. invariant-test asserts only that the answer is yes-or-no; jug-measure carries wrong answers because of it. Visible, not asserted away.

The blended pass rate flatters silence, and did. The owned model first reported “18% capability, 98% honesty” — every pass but one an unanswerable row it walked past by never committing. A mute reader read as a weak one. SOLVED and REFUSED are separate now and the whole table was recomputed. The blended figure is kept and printed last, because deleting it would hide that it was ever misleading.

The bench once measured the wrong model. reader.talk: resolves ~/werbos/brains/current; that symlink is absent on the laptop, so it fell through to “newest *-live” and loaded a 1.09M model that a draft called the 336M. The header printed the spec, never the resolution. It prints the winner now. Both 1M rows are kept above for exactly that reason.

Do not rank our checkpoints by held-out loss. talk-336m’s 3.184 against the 2M’s 1.128 is not a comparison — 4,921M training tokens versus 45M, different corpora, different held-out sets. An earlier draft ranked them anyway and had it backwards.

The verdict

Capability is bought with parameters. Honesty is bought with a harness.

On talk-336m, the model pinned live, cards moved solved problems 0/50 → 1/50 — noise — and moved confident wrong answers 32 → 1, at zero cost in right answers. It also scores clean 1.00 on talk_bench, the only one of thirteen checkpoints to do so. Those two facts are the same fact: it got better at speaking, which is what makes its wrong answers well-formed. talk_bench already tracks that as ood debt, floor 0.00, “no current checkpoint abstains on nonsense.”

The cards pay that debt down without retraining. They do not, and were never going to, teach a 336M byte-level model inclusion-exclusion.