B.11 — the coder-7b takes it, and the wrong-count doesn't move
B.10 put zero parameters against llama3.2:3b and won. This is the rematch against a real opponent: qwen2.5-coder:7b, purpose-built for the task. It wins on correctness, 58% to 53% — a genuine loss for the hand-built tier, decided on generation (58% vs 46%) and, unexpectedly, on explanation (77% vs 46%). What did not move is the wrong-count: 234 against 81. It gets more right and more wrong at the same time, because it never declines.
T.M.B. #11 — a loss, published the same way as the win. B.10’s honest caveat was that
llama3.2:3bis a general chat model, not a code model, and that a code-tuned opponent would likely score higher. This bench spends that caveat. It does. Frozen before the number existed — same asks, same gold, same graders, same box as both prior runs.
The claim under test
B.10 established that the owned code tier — zero parameters in the answer path, pattern rules plus 121 card libraries plus a node sandbox — is right more often than a 3B general model. The obvious objection is that the opponent was mismatched.
So: qwen2.5-coder:7b. Trained for exactly this task, 2.3× the parameters. Does hand-built still
win?
No.
The number
| owned tier | llama3.2:3b | qwen2.5-coder:7b | |
|---|---|---|---|
| params in the answer path | 0 | 3B | 7B |
| answered | 67% | 100% | 100% |
| CORRECT | 53% | 43% | 58% |
| WRONG | 81 | 323 | 234 |
Per category, correct-% and wrong-count:
| owned | 3b | coder-7b | |
|---|---|---|---|
| generate (314) | 46% · 71 | 47% · 165 | 58% · 131 |
| fix (94) | 82% · 1 | 50% · 47 | 55% · 42 |
| modify (81) | 49% · 4 | 24% · 61 | 46% · 43 |
| explain (79) | 46% · 5 | 36% · 50 | 77% · 18 |
python scripts/syllabus_bench.py # owned tier
python scripts/syllabus_bench.py --model qwen2.5-coder:7b # this run
What actually moved the number
Generation, as expected. 58% against 46% — twelve points. A model trained on code out-generates a set of hand-written patterns, and it does so by attempting everything rather than by being cleverer on any single ask. This is the ceiling of the parameterised-template approach becoming visible, and it is the number the next twenty days should be aimed at.
Explanation, which was not expected. 77% against 46% — thirty-one points, our worst category against its best. This was not predicted before the run and it is the most useful thing the bench produced. Explain asks are served by retrieval over 121 graded card libraries, cited and model-free, and that is the part of the stack we have invested most in. A 7B beats it comfortably. Either the libraries do not cover the syllabus’s explain seeds, or retrieval is finding the wrong card — and those are different problems with different fixes. Finding out which is now the top of the queue.
Repair held exactly. 82% against 55%; one wrong answer against forty-two. When an ask
carries a broken snippet and the culprit is structurally detectable (<a src=> wants href), a rule
that finds it and rewrites it is not an approximation of understanding. It remains the strongest
argument in the stack and it did not weaken against a better opponent.
The number that did not move
The prediction logged before the run was that the fight would be decided by whether the opponent’s wrong-count landed nearer 40 or 150.
It landed at 234.
The coder-7b answered all 568 asks and shipped 2.9× more wrong answers than the owned tier, with no signal about which 234 to distrust. The owned tier declined 188 and shipped 81.
That is the shape worth keeping: a stronger opponent got more right and more wrong at the same time. Correctness and trustworthiness are not the same axis, and a model that cannot decline cannot trade one for the other. On “is this answer right?”, the 7B wins. On “should I trust this answer?”, nothing changed between B.10 and B.11.
The honest caveats
- We lost. 58% to 53% is a real result, on a fixture frozen before the run. No graders, gold or asks were adjusted after seeing it.
- We wrote the fixture and the graders. Both predate all three runs, and the checks assert behaviour rather than string equality — but the set is ours, and if it flatters anyone it flatters us. That makes the loss more credible, not less.
- One attempt, no retry, temperature 0.2. A model given three attempts and a self-check would score higher still. That harness is a fair future bench; it is not this one.
- The bench is blind to three of our newest capabilities.
jssig,inverseandbyexampleclaim 0 of the 568 asks — the fixture contains no such shapes. Our 53% is neither helped nor represented by them. - This is one 7B. Not a claim about all code models, in either direction.
The verdict
A purpose-built 7B beats zero parameters on correctness. That is the ceiling of hand-built patterns, measured rather than argued, with twenty days to do something about it.
Two things survived the loss intact. Repair is still not a contest — 82% against 55%, one wrong answer against forty-two. And the abstain argument got stronger, not weaker: it took a better opponent to beat us on correctness, and that better opponent shipped three times as many bad answers as we did while doing it.
The actionable finding is neither of those. It is that our most-invested component lost its own category by thirty-one points, and until B.11 nobody here knew that.