B.19 — we 'beat' a 14B at coding, then fixed our own grader and it was a tie

Tue, Aug 4 2026

A dry run for the coding weigh-in: the owned coding path — a deterministic composer over card libraries with a node execution sandbox, no LLM in the generation loop — against qwen2.5:14b, a model 7,000× its neural footprint, graded by RUNNING the code. First pass looked like a sweep: 26/26 vs 18/26. Then we read the 14B's actual output and found our grader was failing correct answers on whitespace and unspecified details — it had handed us ~6 wins a skeptic would toss. We fixed the grader (whitespace-insensitive, multiple-valid-methods, lenient on details the ask never gave) and re-ran: 26 vs 21. On the function-building half it's a flat 12–12. The honest verdict is a TIE on capability — the owned edge is speed, offline ownership, and never shipping broken code, not intelligence.

T.M.B. #19 — the win we talked ourselves out of. The first scoreboard said werbos swept a 14B at coding, 100% to 69%. That number was wrong, and it was wrong in our favor — our grader was failing the 14B’s correct answers on spacing. So we fixed the grader against our own interest and re-ran. The honest number is a tie on capability; the moat is reliability and ownership, not IQ. Posting the tie is the whole point of this page.

The setup

Two frozen probe sets, both graded by executing the produced code — line counts, DOM assertions, or (for the reference model) running its own function against the same inputs. No model in the verdict. The reference is qwen2.5:14b over ollama on the homelab box. Same asks, same grader, both sides.

# describe-it — build a function from a description, run it, check the real output
python scripts/describe_battle.py                                              # werbos
python scripts/describe_battle.py ollama:qwen2.5:14b --host http://<box>:11434

# codeask — parameterised generation + HTML/CSS repair, DOM-graded
python scripts/codeask_bench.py                                                # werbos
python scripts/codeask_bench.py ollama:qwen2.5:14b --host http://<box>:11434

Scope is the beginner weigh-in domain: for-loops of an exact length, wrap/fix/modify HTML, small CSS edits, and “build a function that reverses a string / sums an array / returns the nth fibonacci, then run it.” 26 codeask probes (codeask_gold.jsonl) + 12 describe-it probes.

The first number (and why we didn’t trust it)

codeask (26)     werbos 26/26     qwen2.5:14b 18/26      median 0.3ms vs 6.2s
describe-it (12) werbos 12/12     qwen2.5:14b 12/12      median 34ms  vs 12.4s

A 14-billion-parameter model losing a coding bench 100%-to-69% to an owned stack is the kind of number that should make you suspicious of your own bench. So we pulled the 14B’s actual output on all eight of its codeask “failures.” Three of them were correct and our grader was being pedantic:

add an alt to <img> → qwen wrote alt="A cute cat image". Our check demanded the literal string "cat". The ask never said what the alt should be. qwen was right.
center my text → qwen used flexbox (justify-content: center). Our check demanded the literal substring text-align: center. Flexbox centers the text. qwen was right.
hide this → qwen wrote display:none. Our check wanted display: none — a space. qwen was right.

Our grader was rewarding werbos for emitting one canonical string and punishing an equally-correct answer for a space or a synonym. That’s not a benchmark, that’s a home-field.

We fixed the grader against ourselves

codeask_bench.py now judges a solution by whether it does the thing, not by exact string: whitespace- and quote-insensitive matching, a fix_any list of valid alternatives (center via text-align or flexbox or margin: auto), and an attribute value of * meaning “present and non-empty” for details the ask never specified. werbos still scores 26/26 under it — its number was never inflated by strictness; only the 14B was being robbed. Re-run:

codeask (26, fair grader)   werbos 26/26     qwen2.5:14b 21/26 (81%)

What actually survived

Of the 14B’s five remaining misses, only two are genuine capability failures — and both are the same failure mode: it shipped broken output. On “switch the style,” it hallucinated a nonexistent <switch><case> element. On “remove the underline,” it deleted the entire link down to a bare x. The other three were an ambiguous “change my styling” read twice, and returning the sorted result instead of the fixed code once.

werbos’s execution/DOM gate structurally cannot emit either of the two broken answers: output that fails the assertion is never returned. That is the one real edge this benchmark measures — and it’s a reliability edge, not a smarts edge.

Say the model sizes precisely

The honest framing matters more than the headline. There is no LLM in werbos’s coding path. describe-it is deterministic pattern-matching that builds JavaScript and runs it in a node sandbox; codeask is deterministic DOM/CSS surgery. werbos’s entire neural footprint is the ~2M-parameter talk model that fronts the stack — and it is not doing the coding. So this is an owned composer vs a 14B LLM, not “a 2M net beat a 14B net.” That is a cleaner statement of capability = model × library, not a weaker one: the capability lives in the library and the execution, and the generator is nearly free. (And qwen ran on a GPU; werbos ran on CPU and still answered in sub-millisecond-to-34ms.)

The verdict

On capability, the owned composer and qwen2.5:14b are a tie on these beginner tasks. The 14B never once wrote worse-but-working code — its only losses were breakage the sandbox prevents by construction. werbos’s measurable, defensible edges are three, and none of them is intelligence:

speed — 350× to 20,000× faster
owned + offline + zero egress — the 14B needs a GPU server; werbos needs a laptop
never confidently broken — the execution gate

A composer 7,000× smaller in neural terms holds the line against a 14B on the tasks a beginner actually types. That is the werbos bet, measured — and it reads as more credible precisely because it doesn’t claim to be smarter. The next bench (B.20) goes where execution separates from narration — bug-hunts, edge cases, “which input throws?” — to turn the tie into a lead.

Re-run everything above yourself: scripts/describe_battle.py and scripts/codeask_bench.py, plus the full write-up with the per-probe breakdown at benchmarks/v1-owned-composer-vs-qwen14b-coding.md in the repo.