Day 79 — 'if it's on w3schools... it may or may not be fair game'

“I’m not going to tell you the tests ahead of time.”

That one sentence did more for the work than any benchmark did. A test you can see in advance measures how well you tuned for it. A blind one measures whether the thing works.

The rules, such as they are: beginner level, five areas — facts, code gen, fix-my-code, talking, tools — and “if it’s on w3schools… it may or may not be fair game.” Written up properly on the Gauntlet page, because rules recorded before a test are worth something and rules reconstructed after one are worth nothing.

The asks are small, and that is the point

"make a for loop that is 21 long in javascript"
"can you wrap an anchor tag in a div with a class of text-center"
"why doesnt my link work, please fix.. <a src="/about">about</a>"
"please add 2 more items to my list: <ul><li>1</li><li>2</li></ul>"
"can you chnag emy styling on my div? .class1{color:green} .class2{color:blue} <p class="class1">jason</p>"

Not one of those is hard. Every one of them was unanswerable this morning.

werbos owned (no model)   21/21  (100%)   median     0.1ms
llama3.2:3b               13/21  ( 62%)   median 1,661ms

Why the mint didn’t help

The obvious move was to generate more examples. The GPU box spent the night climbing a verified curriculum and produced 40 checked patterns — and they are useless for these asks.

Asked for “for loop counting to 21”, the pattern library scores 0.70 and returns code that sums an array, because TF-IDF matched the word loop. Above the 0.5 threshold, so it would have been served, run, and only then found wanting. Confidently wrong, which is worse than the abstain it replaced.

The reason is structural: stored patterns are fixed strings. A verified “count to 21” snippet can never answer “count to 12”. No quantity of examples fixes that. One template with a slot does.

So the unit became a template plus a rule for reading its slots out of the ask — any N, any class name, six child types. Hand-written, model-free, and it generalises where an example cannot. Third time this week hand-written rules beat minted examples on a closed space: contrastive focus 20/20, bug signatures 10/10, this 21/21.

Which reframes what the GPU is for. Not answers. Questions. My phrasing imagination is the weak point — I invented “wrap an anchor tag in a div”; I did not invent “stick this in a container”, “put the link inside a div”, “can u wrap this w/ class text-center”. A 14B is excellent at that variety. So it is now minting 20 phrasings each of 30 syllabus intents, typos and slang requested on purpose, and every phrasing that makes the tier abstain names the next template to write.

Three shapes, and the third was invisible

generate   no code in the ask                    -> build it from slots
fix        pasted code that is BROKEN            -> repair THEIR snippet
modify     pasted code that is CORRECT           -> transform it

modify is the one I missed entirely. “Add 2 more items to my list” is neither generation (there IS code) nor repair (nothing is broken), and abstaining on it looks like the tier cannot read code at all. Small detail that matters: numeric items continue the sequence. Two more after 1, 2 means 3 and 4, not two more copies of 2 — which is what a beginner means and what a naive duplicate gets wrong.

And misspellings are part of the job, not a bonus round. “can you chnag emy styling” is a real message; an exact keyword match reads it as no intent at all. What makes it survivable isn’t the fuzzy matcher though — it’s that the structure confirms the intent. Two CSS rules defined and only one in use has exactly one sensible reading, whatever the verb looks like.

The tests caught what I didn’t

Three bugs found by writing the negative cases rather than by staring at code:

• let\s+\w+ matched “LET ME know” — every conversational reply containing that phrase was classified as pasted JavaScript and routed into the code tier.
• _child_in matched the article “a” in “wrap AN image in A div” and built an anchor for an ask about an image. English articles collide with short tag names.
• _count_in read the number out of the pasted code instead of the instruction, so “add two more items to <ul><li>1</li></ul>” added one.

The 3B made that last mistake on that exact ask, and produced four items. Same trap. The difference is that ours is now a test.

The part I got wrong

The owned tier abstained on almost everything, because it is patterns-or-nothing and the pattern library holds thirteen entries. So I wired in a local coding model and it worked beautifully: ✓ verified — ran clean, real output, 13 seconds.

It was qwen2.5-coder:7b. Under the Custom model toggle.

“if this toggle is OWN WEIGHT… NEVER ROUTE ME TO ANYTHING BUT OUR TALK MODEL OR CODING MODEL.”

Correct, and the reasoning I’d used to get there was the seductive part: offline means no egress, not no compute. True, and beside the point — an open-weight model is open-weight whether it runs locally or not. I treated the toggle as a networking flag when it is a promise about whose weights answer.

The worst of it was that nothing in the reply said so. [code · offline] ✓ verified — ran clean, with a 7B behind it and no hint. A working answer from the wrong weights is worse than an abstain — the abstain tells you how far the owned library still has to go, and the answer hides it.

Reverted. Forty-one lines deleted. The abstains came back, which is the honest state of a tier with thirteen patterns.

What the abstain was actually saying

“If the coding model is failing, that’s a signal how to fix it. If the talk model is printing AAAAAAA, that’s a signal.”

That reframing is worth more than the code I wrote today. The abstain was never a bug — it was an instrument reading, saying thirteen patterns, fill them. Routing around it didn’t fix anything; it disconnected the warning light.

And every failure mode in this repo names its own fix out loud, if you let it:

"generator returned no code"      -> the pattern library is empty. Fill it.
babble x10 on the 30M            -> data-starved. Shrink it or feed it.
"no <li> directly inside <ul>"    -> it learned the WORD nav, not the shape. Add the rung.
input-sensitive 1.00 on garbage   -> the METRIC cannot see babble. Fix the ruler.

Not one of those was solved by making the red light green.

22 days

Honest coverage against the published syllabus — 24 topics, 58 asks:

covered        3
partial        7
NOT COVERED   14

The weakest area isn’t code, it’s explain. Asked “when to use let vs var”, the libraries return a code snippet, because 10,454 graded JavaScript cards answer “what does this code do?” and the question is “when do I use this?” Different card type. A 3B probably wins that one today, and knowing it now is cheaper than finding out on the day.

The tempting item is the other one — search, learn, then use what was learned:

"i dont know tailwind" -> scan -> learn -> "import the CDN in <head>, and that margin
                                            you wanted is class='mt-3'"

Scoped as Day 150. It is easily the most interesting thing on the list, which is exactly why it is the most dangerous to start. Failing three beginner asks while building frontier-level learning would be the wrong trade.

Coverage of a published syllabus is what a blind beginner test measures.