Day 81 — if you don't know, invert it

“if you dont know…inverse it”

The ask was this, typed exactly like this:

Create a function that create 2 numbers that was use to create this number: 5

It had always abstained, and the reason looked fundamental. create is not an operation. Five is 2+3, and 1×5, and 10−5. Every attempt to parse that sentence ends in a guess about which one was meant, and a guessed operator answers a factors question with sums — code that runs clean, looks right, and is wrong. Abstaining was honest. It was also useless.

The trick is to stop trying to name the operator. For a target n, run each candidate operation backwards to enumerate what could have produced it, then run it forwards to check:

inverse of +    b = n − a,  a in 0..n   →  (0,5) (1,4) (2,3) ...
verify          a + b === n             →  all hold

Arithmetic decides what the sentence left open. Nothing has to be interpreted, because nothing has to be chosen — addition and multiplication both get enumerated, both get verified, and whatever survives is reported with the operation that made it survive. It is the same discipline as execution-grading JavaScript: don’t reason about whether the output is right, produce it and test it.

The second ask was mean in a different way:

Create a function that output the results:  3 2 1 GO!

No mechanism at all. No “loop”, no “count”, no “from 3 to 1”. There is nothing to parse because there is no description — the sentence contains the output, and the job is the producer. That’s the FizzBuzz shape, and it’s how most beginner exercises are actually phrased.

It also has a property nothing else in the tier has: the ask carries its own oracle. Generate, run it, compare stdout to the exact sequence the user typed, return nothing on a mismatch. Every other generator here emits code and hopes. This one is verified against the user’s own words.

What it declines is the part I’m most pleased with. 1 2 3 Fizz 5 fits “literal at position 4” and “literal when divisible by 3” equally well, and those diverge at 6. Five tokens genuinely do not determine the rule, so it reports a gap instead of guessing an ask whose whole point is the rule.

The weigh-in

Then the question that made the day: how does the 3b do on our quiz set?

Fair question, and the honest one. A 53% that beats nothing is not a number. So the bench got a --model flag that swaps only the answerer — same 568 asks, same gold, same graders — and llama3.2:3b ran the whole set on the same box at the same commit.

Zero parameters, right more often than three billion, wrong four times less. On repair it isn’t a contest — 82% against 50%, one wrong answer against forty-seven. On generation the model is genuinely ahead, 47% to 46%, and that single point is the honest finding: breadth buys something we cannot hand-build our way past, and it costs 94 extra wrong answers to buy.

The 323 is the whole thesis in one figure. It answered every ask and had no way to say which 323 to distrust. Full write-up, caveats included.

The quiet result underneath it was better, though. The owned tier scored 67% / 53% / 81 on SR8 and 67% / 53% / 81 on the laptop — identical, row for row. No sampling, no temperature, no machine dependence. The same question gets the same answer anywhere, which is what makes a number worth arguing about.

The rematch, and the loss

B.10’s honest caveat was that llama3.2:3b is a general chat model, not a code model, and a code-tuned opponent would probably score higher. So we spent the caveat the same evening: qwen2.5-coder:7b, purpose-built for the task, 2.3× the parameters, same 568 asks.

It wins. 58% to 53%, on a fixture frozen before the number existed. Generation went as predicted — 58% against 46%, twelve points, a code model out-generating hand-written patterns by attempting everything. That is the ceiling of the template approach becoming visible, and it’s the thing to aim the next twenty days at.

Repair held exactly as called: 82% against 55%, one wrong answer against forty-two. When the culprit is structurally detectable, a rule that finds it still beats a model that infers it.

The surprise was explain: 46% against 77%. Thirty-one points, our worst category against its best — and it’s the part of the stack we’ve invested most in, retrieval over 121 graded card libraries, cited and model-free. A 7B beats it comfortably. Either the libraries don’t cover those seeds or retrieval is picking the wrong card; those are different problems with different fixes, and finding out which is now top of the queue. Nobody here knew that before tonight.

The prediction logged before the run was that the fight would be decided by whether the opponent’s wrong-count landed nearer 40 or 150. It landed at 234 — it answered everything and shipped 2.9× more bad answers than we did, with no way to say which. A stronger opponent got more right and more wrong at the same time, because it cannot decline.

Correctness and trustworthiness are not the same axis. Tonight the 7B won the first one and nothing moved on the second. B.11, caveats included.

Three checks that couldn’t fail

The rest of the day was one lesson, three times.

git push -q && echo PUSHED printed success for six hours while doing nothing. I’d taken the branch name from a stale snapshot and been committing to main the whole time, so it was pushing a feature branch that was already current. Exit code 0, cheerful message, thirty commits going nowhere. The fix is three characters of actual verification: compare the remote hash to HEAD.

A hand-written hijack guard covered five asks I invented, felt thorough, and missed seven the fixture caught immediately — all textspeak, where the digit in "how 2 make" and "html5" was being read as a target number. Wrong answers went 81 → 88 on a change I’d already convinced myself was safe.

An ETA computed from the ollama request log, which also carries werbos-ab’s traffic on the same box. Every estimate I gave was measurement of somebody else’s telemetry, reported as fact.

And a fourth, for symmetry: both wait-loops deadlocked because pgrep -f "syllabus_bench --model" matched their own command line. The benchmark had been finished for a while behind a watcher staring at itself.

AHA: every one of those was a check in the shape of a check. That’s the same failure as the disabled ternary and the deny-list installed on one of three doors — and it’s why the syllabus bench going blind to three new capabilities is the version that should worry us most. jssig, inverse and byexample claim 0 of 568 asks. An instrument you trust while it’s pointed somewhere else is worse than no instrument.