B.14 — coverage was not correctness: 70% answered, 41% right

T.M.B. #14 — the number was real and it was measuring the wrong thing. B.13 asked whether the metric used to CHOOSE models could be trusted. This one asks the same of the metric used to judge answers, and the answer is again no: a bench reporting 70% was reporting how often something came back, not how often it was right. Frozen fixtures; every number below comes from a script in the repo.

The setup

Two fixtures, neither written by the author of the code under test.

568 phrasings across 30 syllabus seeds, minted by qwen2.5:14b with typos and slang requested on purpose (werbos/compose/fixtures/phrasings.jsonl). 1,543 natural-language → Emmet pairs across 28 abbreviations, where the abbreviation is generated from the owned grammar and verified by expand() + validate() before the teacher is ever called, so the LABEL is correct by construction (fixtures/emmet_pairs.jsonl).

python scripts/syllabus_bench.py      # answered AND correct, per kind
python scripts/emmet_bench.py -v      # TREE and SHAPE against the minted pairs
python scripts/emmet_pipeline.py      # where the pipeline loses asks, stage by stage

Results

The bench that existed reported one number, if answer(ask):

                 answered   CORRECT
  coverage_bench      70%   (not measured)

With 30 gold checks — JS graded by RUNNING it, HTML and CSS graded structurally because more than one answer is correct:

              asks  answered  CORRECT   wrong
  generate     314       69%      47%      68
  fix           94       84%      82%       1
  modify        81       58%      49%       7
  explain       79       53%      48%       4
  TOTAL        568       67%      53%      80

The first gold run put TOTAL at 41% correct with 152 wrong; the 53% / 80 above is after a day of fixing what the check exposed.

The number that should have been alarming

The same engine, scored on the Emmet pairs by DOM shape, before any of today’s work:

  answered   317/1543   20%
  CORRECT      4/1543    0%

One ask in five got an answer and four were the right shape. Nothing in a coverage number can show this. The worst single case: “wrap hello world in a header inside a div” returned the entire site navigation bar, because header is how a beginner says heading and nav is a template’s trigger word.

What actually moved the number

Templates cannot compose. div.container — a div with a class — is not buildable from any template, and no quantity of templates changes that. Replacing them with a reader that turns a sentence into one line of Emmet:

                      attempted   TREE   SHAPE
  templates only            20%     3%      0%
  sentence -> Emmet         88%    54%     35%

Parenting comes from containment rank, not word order — "a div containing an h2" and "wrap it in a div" are the same tree in opposite orders, and a <div> contains an <h2> in every phrasing.

Ranked retrieval, not first-hit. The ring ceiling was pinned at NORMALISED because a looser ring with first-hit-wins returned garbage — measured, “difference between == and ===” returned a CSS card about initial vs unset. Weighing every candidate by topic overlap, precision, and the card’s own topic field (which every card has and the index was discarding) took explain from 16% to 48% correct.

Per-feature, the reader’s weakness is legible and it is not slots:

  multiclass 94%   id 85%   attr 73%   class 60%   child 50%
  void 43%   deep 39%   sibling 32%   group 22%

The honest caveats

These fixtures are no longer blind. A full session was spent reading their failures and fixing against them. The fixes are structural rules — containment rank, coordination, implied children — not memorised phrasings, and the two fixtures moved together while being independent. But 54% is an optimistic estimate, not a neutral one, and a fresh mint against the same abbreviations is owed before it is treated as a real number.

Two of the 90 wrong answers were the gold, not the tier. A form template failed for nesting its <input> inside a <div> — correct HTML that the check rejected for demanding a direct parent — and a check demanded four list items from an ask that named no number. tests/test_syllabus_gold.py exists because of this: every check is fed a hand-written correct answer it must accept and a plausible wrong one it must reject. A check that rejects good code is the expensive direction; it sends the next day’s work at a problem that does not exist.

explain improved partly by answering less. Its answered rate fell from 49% to 22% at one point in the day. Those were not newly-correct answers, they were wrong answers becoming abstains — which is the right trade for a blind test and should not be read as capability.

The gold is 30 hand-authored rows. It covers the published beginner syllabus and nothing else.

The verdict

Coverage and correctness are different questions, and only one of them can be read as accuracy. The useful figure from this bench is not 53% correct — it is 80 wrong, the count an abstain would have beaten, because on a test where “I don’t know” is a legal answer a confident wrong one is the only kind that costs anything.