Day 76 — 'a failure that looks like a result'

Every bug today was a one-line fix. Not one of them was findable by reading.

That’s the whole day. Three benchmarks got built, pointed at working code, and every single thing they surfaced had the same shape: it wasn’t failing loudly, it was succeeding wrongly.

The bucket, again

min_gap = 0.08. If the top two cards score within that, abstain — a tie means ambiguity, and ambiguity means don’t guess. Reasonable. Prudent, even.

Here’s what it actually did:

how do i treat a minor burn?
   0.488  What should you do immediately after a minor burn occurs?   ← right
   0.454  How do you treat a minor sunburn at home?                   ← also right
   gap 0.034 < 0.08  ->  None

Both cards were correct answers. In a well-covered library a tie doesn’t mean ambiguity, it means redundant correct coverage — and the rule punished the library for being thorough. Worse, it got worse every time a library was deepened. Every card added made retrieval slightly more likely to say nothing.

It’s also a relative test making an absolute decision. “Does this card cover this query?” is absolute — that’s what threshold is for. Deleting the relative test and raising the absolute one dominated on both axes, no tradeoff:

threshold .38 / min_gap .08    abstain 21/24    routing  3/21
threshold .50 / min_gap .0     abstain 24/24    routing 11/21

Zero bluffs, and 3.7× the correct answers. One constant.

Sixty-four characters

The owned talk models scored 0.53–0.76 on cleanliness. All seven failed. Truncation 8–14 times per model — far too uniform to be model quality.

def reply(self, prompt: str, max_new: int = 64)

max_new counts tokens. Every shipped checkpoint is byte-level. So it was 64 characters — about ten words.

max_new=64    'I know right? I hope it feels like the planning of the day at a'
max_new=400   'I know right? I hope it feels like the planning of the day at a time.'

The models weren’t failing to finish. They emit their stop token happily a few characters past the cap; we stopped listening. 0/7 shippable became 7/7. Live in the chat the whole time.

It also overturned a read from the previous size sweep: comparing truncated output made the 5M look more fluent than the 30M. Untruncated, the 30M is clearly better. The old verdict was right for a reason it couldn’t see.

The rulers lied too

This is the part worth sitting with.

chat_bench was counting “classified” as “handled.” It filtered out turns the cascade answers upstream, so it wouldn’t measure the library against inputs it never receives. Sensible. But it asked whether speech_act classified the turn — and "you're really good at this" classifies as act='other' while respond() returns None. So it fell straight through to retrieval in production while being excluded from the bench as already answered. Caught by the third bench, built specifically because the second one’s headline rested on an unverified assumption.

And the axis meant to catch filler couldn’t. Exact-match distinctness scored 0.94–0.97 for every checkpoint — no discrimination whatsoever. Why became obvious the moment someone asked what “responsive” meant:

i like corn             -> 'I know that feeling. Any idea what makes you feel like t'
i like trains           -> 'I know that feeling. Any particular memories you want to'
i like corn on the cob  -> 'I know that feeling. Any idea what it was about?'

Three distinct strings. Full marks. One template. A collapsed model varies its tail and repeats its opening, and whole-string comparison can’t see it. Stem diversity spreads 0.62–0.79 over the same models and actually ranks them.

The name was wrong too — the first person to read “RESPONSIVE” assumed it meant how fast it replies. That’s the only usability test that counts. It’s INPUT-SENSITIVE now, and nothing in the suite measures latency.

Eighty-four empty files

except Exception:
    return ""

A campaign to manufacture training corpora for 101 libraries. The teacher died at library 18. The loop ground through the remaining 84, wrote a 0-byte file for each, and reported success.

The evidence that it wasn’t the quality gate is the shape: the populated corpora are the first seventeen libraries alphabetically, and everything from chess onward is empty. A quality filter scatters. This was a cliff.

A transport failure and a rejected take produced byte-identical output, so nothing could tell them apart. Now TeacherUnavailable is a typed exception, transport errors are counted separately from gate drops, and the campaign stops rather than manufacturing more. The regression test caught a hole in the first version of that fix within a minute — a library with fewer than five queries never tripped the counter, so it still wrote an empty file and carried on.

What actually shipped

Pure talk went live. brain='talk' routes the chat to the raw owned 30M — no retrieval, no grounding, no citations, the honest voice of the model with nothing in front of it. The brain service had served that phase all along; nothing could select it. Now four states are reachable: Llama, grounded-and-cited, a grounded take, or the 30M alone.

Fine-tuning became possible. The trainer built a fresh model every run, so a domain mini could only train from scratch — and a subject corpus is a few hundred pairs, which from scratch is noise. --init-from makes adapting a base that already speaks English a real option, which is the precondition for per-domain minis.

And the diversity question got a real answer. Retraining the 30M on 281,829 rows instead of 77,000 moved cleanliness 0.90 → 0.94 and left input-sensitivity flat at 0.74 → 0.72. More of the same data buys fluency, not attention. The corpus is 117,658 rows drawn from fourteen tones — one line fits everything when there are only fourteen ways to sound, and the model found it.

The AHA

Not one of today’s bugs was hard. A constant, an if, a sorted(), a length gate. The engineering was never the weak link — the feedback loops were. And the fix for a missing feedback loop is never clever, it’s just doing it.

The sharper lesson is the second-order one: three of the things found today were bugs in the rulers themselves, discovered only by using them on real questions. A benchmark you don’t question is an unexamined assumption with a number attached — and that’s worse than a hunch, because it looks like evidence.

The changelog you’re reading nearly didn’t survive the day either. It turned out to be living inside a cookietracker repo that had been git init’d one directory too high, never pushed anywhere — fifty commits of history on one laptop with a remote pointing at an unrelated project. It’s in the repo it documents now.