Day 80 — 70% answered was 41% correct

“70% i thought was good.”

It was — as a coverage number. coverage_bench asks one question, if answer(ask), and reports how many came back with something. Nothing in it looks at whether the answer is right.

That distinction stopped being academic the moment it was measured. On the same engine, the template tier scored 20% answered and 0% correct — one ask in five got an answer and essentially none of them were the right shape.

Gold answers, and the number halves

Thirty gold checks, one per syllabus seed, inherited by all 568 phrasings. Gold attaches to the intent, not the wording, which is what makes thirty hand-authored rows cover the whole set.

              asks  answered  CORRECT   wrong
  generate     314       69%      47%      68
  fix           94       84%      82%       1
  modify        81       58%      49%       7
  explain       79       53%      48%       4
  TOTAL        568       67%      53%      80

JavaScript is graded by running it. “Counts to 21” is a claim about behaviour, and a function that computes a value and never returns it looks correct until you call it. HTML and CSS are graded structurally, because more than one answer is right — a table with a header row is correct with or without a <thead>.

And the fix/modify seeds finally got code. The mint was told to emit none, so 175 repair phrasings arrived with nothing to repair and could only ever be answered as generation — scoring as coverage while testing nothing. Each gold row now carries the broken snippet its seed implies. fix is the strongest area at 82%, with a single wrong answer out of 94.

Four bugs surfaced in the first run, all on the most-quoted test in the gauntlet:

"count from 1 to 21"       ->  i <= 1      the first digit is the range START
"counts up to twenty one"  ->  i <= 20     compound word number truncated
"counts to twentyone"      ->  abstain
"cant FIGURE out how to set a range in a for loop"  ->  <figure></figure>

That last one answered a JavaScript question with an HTML element, because the reader knows nothing about loops and everything about nouns that look like tags.

Templates do not compose

div.container — a div with a class — is not buildable from any template here, and no quantity of templates fixes that. Each answers exactly the question it was written for.

Emmet is a finished grammar that already covers nesting, classes, ids, attributes, repetition and text, so the job stopped being “write more templates” and became read the sentence into one line.

Parenting comes from rank, not word order. These describe the same tree:

"a div with class title-wrapper containing an h2 that says hello world"
"wrap hello world in a header inside a div"

Parsing prepositions means enumerating English. What does not vary is that a <div> contains an <h2> and never the reverse — so each element gets a containment rank and attaches to the nearest preceding element of lower rank. Word order only breaks ties among equals, which is exactly where it does carry meaning (img+h3+p really is the order they were said).

1,543 minted asks     TREE 0% -> 54%     SHAPE 0% -> 35%

The failure family throughout was one shape: markup that renders, passes the content-model validator, and describes something nobody asked for. label became the parent of input. A nav with nothing inside it swallowed the heading. id header also built an <h2> until slots learned to consume their words. And the worst of them — given a pasted snippet no modifier handled, the reader read the user’s own tags as a description of themselves and rebuilt them empty:

<ul><li>one</li></ul>              ->  <ul><li></li></ul>
<div class="old"><p>hi</p></div>   ->  <div class="old"><p>old</p></div>

A degraded copy of someone’s own file looks like a careful edit, validates, and silently loses work.

One parser, four slots

Four tiers were each sniffing the raw sentence with a private regex — _IS_EXPLAIN, _ASKING_ABOUT, _JS_INTENT, has_style_intent — four views of one question, disagreeing at the edges.

"how would i set display none for li"
  ${how would i}    intent  SOLUTION -- wants code, not a definition
  ${set}            action
  ${display none}   topic   WHAT to look up
  ${for li}         target  what it applies to

The opener decides the tier, and it is the cheapest signal in the sentence. “how would i set display none” and “what is display none” share every content word and want completely different answers. That mismatch was the single largest source of wrong answers: “when to use a nav instead of a div” returned an entire page header, nineteen times.

The topic is the retrieval key, not the sentence. Six words of scaffolding is how a question about display finds a card about how.

Weighing sixty cards instead of grabbing one

The retrieval ring ceiling was pinned at NORMALISED because a looser ring with first-hit-wins returned garbage — measured, “difference between == and ===” came back with a CSS card about initial vs unset, and div returned a card about <table>.

The fix was not to refuse to look. It was to look at all of them and weigh: topic overlap, then how much of the card is about the ask, then the card’s own topic field — which every card has and the index was discarding, though it is the strongest signal there is. Question text is mostly stopwords; the topic is what the card is filed under.

A close second is only ambiguous if it disagrees. Scoring the margin alone abstained on “what is a const”, where sixty cards were all filed under const and the top five sat within 0.06 of each other — that is abundance, not uncertainty. Real ambiguity is near-ties about different things, which is exactly the shape of the initial vs unset miss.

explain    16% -> 48% correct       wrong answers 90 -> 80

In JS, everything is an object

The operator’s framing, and it was the right unit. A function is not a string assembled left to right — it is {name, params, body, returns}. Fill the slots you know, render what you have.

A string template cannot be partially right: wrong parameters means the whole answer is wrong, and an unworkable body means no answer at all. A slot object with two of three slots filled is still valid, runnable code, and it names the slot that is empty.

The build order then costs nothing — it is the same object rendered at successive fills:

function add() {}                      the shape
function add(a, b) {}                  the inputs
function add(a, b) { return a + b; }   the body

Every stage runs, so a progression is a chain of checked states and not narration. Which adds the answer this tier never had: it could say “here is the code” or say nothing, and now it can say “this much I verified, and here is where I stop.” A structure right in every part but one currently scores as fully wrong and is shown to the user as if it were fully right.

Grading stages by execution caught a defect in the new code immediately: if (n > 0) {} reads as correct and throws ReferenceError. A condition without its subject declared is not a rung, it only looks like one.

What the day is really about

Every number that moved today moved because something got measured differently, not because more was built. Coverage said 70%; correctness said 41%. “Answered” said the Emmet reader worked; DOM shape said it was 0%. A margin rule said “uncertain” where the library was simply rich.

Two of the ninety wrong answers turned out to be the gold being wrong — a form template failing for nesting its input inside a <div>, which is correct HTML, and a check demanding four list items from an ask that named no number. A check that rejects good code is the expensive direction of error: it sends the next day’s work at a problem that does not exist.

21 days. The number worth watching is not 53% correct. It is 80 wrong — the count an abstain would have beaten.