B.10 — zero parameters vs three billion
The owned code tier has NO model in it — pattern rules, 121 card libraries, and a node sandbox. This bench points it and llama3.2:3b at the same 568 asks, on the same box, at the same commit, graded by the same checkers, and asks who is right more often. Headline: owned 53% correct / 81 wrong, 3b 43% correct / 323 wrong. The 3b answers 100% of asks and cannot tell you which 323 to distrust. It edges us on generation (47% vs 46%) and loses repair 82% to 50%.
T.M.B. #10 — the first head-to-head. Every prior bench measured werbos against itself. A 53% that beats nothing is not a number, so this one puts a 3-billion-parameter model on the same 568 questions and grades both with the same code. The owned side has zero parameters in the answer path. Frozen; re-run it yourself with one flag.
The claim under test
compose/codeask.py answers beginner code asks from parameterised patterns — no model, no
weights, no inference. Explain asks are served by retrieval over 121 card libraries. JavaScript
answers are graded by running them. The whole tier is hand-built, and every capability in it is
one somebody wrote on purpose.
The obvious objection is that hand-built rules are a poor substitute for a model that generalizes. This bench tests that objection directly: is a 3B model right more often than our rules?
The setup
| asks | 568 phrasings over 30 gold-checked syllabus seeds |
| box | SR8, CPU-only, both runs |
| commit | 5436119, both runs |
| graders | identical — execution for JS, structural for HTML/CSS, mention-checks for explain |
| baseline | llama3.2:3b, temperature 0.2, one attempt, no retry |
Only the answerer changes. The baseline is told the language, told to emit code only, handed the same broken snippet the fix/modify seeds hand us, and has its fenced output unwrapped before grading. It gets every advantage the owned tier gets, plus one it does not: it never has to decide whether it knows.
python scripts/syllabus_bench.py # owned tier
python scripts/syllabus_bench.py --model llama3.2:3b # baseline
The number
| owned tier | llama3.2:3b | |
|---|---|---|
| parameters in the answer path | 0 | 3,000,000,000 |
| answered | 67% | 100% |
| CORRECT | 53% | 43% |
| WRONG | 81 | 323 |
Per category, correct-% and wrong-count:
| owned | 3b | |
|---|---|---|
| generate (314) | 46% · 71 | 47% · 165 |
| fix (94) | 82% · 1 | 50% · 47 |
| modify (81) | 49% · 4 | 24% · 61 |
| explain (79) | 46% · 5 | 36% · 50 |
What actually moved the number
Repair is the blowout, and it is not close. 82% against 50%, one wrong answer against
forty-seven. When an ask carries a broken snippet and the culprit is structurally detectable
(<a src=> should be href), a rule that finds it and rewrites it is not an approximation of
understanding — it is the correct answer, every time, for free.
Generation is the honest loss. The 3b edges the owned tier 47% to 46% on the largest category. That is generalization doing what a hand-built set cannot: it will attempt anything typed at it, and on 314 varied generation asks that breadth is worth about one point. It costs 94 extra wrong answers to buy that point, but the point is real and it is the number to chase.
The 323 is the thesis in one figure. The baseline answered every single ask and was wrong on 323 of them with no signal about which. The owned tier declined 188 and was wrong on 81. Honest abstention is not a consolation prize for weak coverage; it is the difference between shipping 81 bad answers and shipping 323.
The honest caveats
llama3.2:3bis a general chat model, not a code model. A code-tuned 3B — or the 7B coder sitting on the same box — would very likely score higher. This is not a claim that zero parameters beat all 3B models. It is one measured pairing, named exactly.- We wrote the fixture and we wrote the graders. Both predate this comparison, and the gold checks assert behaviour rather than string equality, but the set is ours and that is a real thumb on the scale.
- One attempt, no retry, temperature 0.2. A model given three attempts and a self-check would do better. That harness is a fair future bench; it is not this one.
- The bench is blind to three of our newest capabilities.
jssig(four loop constructs),inverse(decomposition) andbyexample(shown-output) claim 0 of the 568 asks — the fixture contains no such shapes. Our 53% neither includes nor is helped by them. - CPU-only inference. Irrelevant to correctness, relevant to anyone re-running it: the baseline took ~90 minutes.
The verdict
Zero parameters, right more often than three billion, wrong four times less. On repair it is not a contest. On generation the model is genuinely ahead by a point, and that point is where the next twenty days of work should go — the fixture needs the shapes it cannot currently see, and the generate tier needs the constructs it currently abstains on.
The result that mattered most, though, is the boring one: the owned tier scored 67% / 53% / 81 on SR8 and 67% / 53% / 81 on the laptop — identical, row for row. No model, no sampling, no temperature. The same question gets the same answer on any machine, which is the property that makes a number like 53% worth arguing about at all.