B.12 — the bottleneck was the decoder: 0.65 → 0.94 with no retraining

T.M.B. #12 — measure the data before you blame the model. Every earlier bench asked whether the model was good enough. This one asks whether the number was measuring what we thought, and the answer was no three separate times. Headline: input-sensitivity 0.65 → 0.94 — past a 3B — for a decode-time change, no retraining, no new data. Frozen; the probe set and the corpus statistics are both re-runnable.

The claim under test

The owned 30M talk model scored 0.65 on input-sensitivity against llama3.2:3b’s 0.73. The diagnosis on the table was register monotony: the corpus over-represents one emotional register, so the model has learned one voice, so scale the model and diversify the mint.

That diagnosis had never been checked against the corpus. So:

conversation.jsonl                 117,658 pairs
distinct 4-word openers             85,911
opener entropy               15.84 bits of a possible 16.39   (96.6% of maximum)
top opener share                      0.26%
top-10 opener share                   1.70%

The corpus is at 96.6% of maximum opener entropy. There is no monotony in the data.

Against what the model emits:

The model was amplifying a concentration the data does not have.

Why: greedy decoding

reply() generated with logits.argmax(). Argmax takes the mode of the output distribution at every step, so two inputs landing in nearby hidden states produce byte-identical strings — which is exactly what stem-diversity measures. A model can hold a perfectly well-spread distribution and still emit one opener forever.

Sampling is the fix, and it breaks stable — same input → same reply, 8/8 — which every checkpoint passes at 1.00 and which exists because a chat model that answers differently each time is unusable.

Seed the RNG from the prompt. One input always decodes to one reply (stable holds); different inputs start from different RNG states and explore different regions of the distribution instead of funnelling to the mode. Stem-diversity by construction, not by hope.

blake2b, not hash() — Python salts string hashing per process, so hash() would give replies that are stable within a run and different across runs. That is the exact failure the stable floor exists to catch, and a single-process bench would never see it.

The number

48 frozen probes, five axes, floors not a composite, no model in the verdict — a system wordlist and arithmetic.

k=3, t=0.5 ships. The 0.02 of clean is truncation — babble went to zero.

The reference line

This is the first time the owned model has been scored against the incumbent on identical probes under identical floors, and the result is lopsided in an unexpected direction. llama3.2:3b fails the clean floor at 0.21 — babble ×3, echo ×2, run-on ×33 — and takes 20.6 seconds per reply against our 576ms. It is scored with no allowances: same probes, same floors, same rules.

The honest reading: on four of five axes the owned 30M is already past the 3B for this tier. The axis it genuinely lost — input-sensitivity — is the one this bench just closed. The axis neither model can do is ood, where both score 0.00.

The negative result: 30M → 85M

Tested directly rather than assumed, same corpus and probes:

talk-121k/talk-30m   clean 0.90   input-sens 0.58   stem-div 0.58    534ms
talk-121k/talk-85m   clean 0.85   input-sens 0.58   stem-div 0.58   1158ms

2.8× the parameters bought zero input-sensitivity, cost fluency, and doubled latency. The 85M’s top opener fires 25% of the time against the 30M’s 12.5% — more capacity means sharper mode-seeking and deeper collapse.

This closes the question B.10 left open. That sweep picked its winner on held-out loss, which cannot see filler collapse: a model answering everything with “Sounds like a plan!” has excellent loss (safe, high-frequency continuations) and no conversational value. Scored on generated text instead, the bigger model loses.

What does move it is data: 121k → 282k pairs on the same 30M took stem-diversity 0.58 → 0.65 and clean 0.90 → 0.94. Corpus, not parameters. Third time scaling has lost here.

Composition beats weights: contrastive focus

“I never said he ate my sandwich” is seven words and seven different claims. The only thing that changes is which word carries the pitch accent, and the implicature is mechanically derivable — the accent marks a contrasted constituent, and the sentence asserts not that one, some other member of its alternative set.

compose/focus.py: 20/20. llama3.2:3b: 4/20, which is chance. 2.5µs against 469ms.

The 3B’s failure mode is the informative part: it picks one meaning per sentence and ignores the accent entirely. That is not a knowledge gap a bigger model closes — it is a structural relation a fluent model has no reason to represent. Which makes it a composer’s job. No model, no library, no network: a lexicon and word position.

What this bench does NOT settle

• ood is still 0.00 and it is the honest debt. Neither the owned model nor the 3B signals confusion at nonsense. A first attempt to train it in moved abstention 0.02 → 0.45 and was worthless: false-abstention went 0.00 → 0.31, because the generated nonsense averaged 15 chars against a real corpus at 34, and the model learned that short means nonsense rather than learning what nonsense is. Refused real inputs averaged 16.8 chars; answered ones 27.2. The corpus is now length-matched and the retrain is pending — reported here as unfinished rather than omitted.
• Sampling is a decode-time trick, not comprehension. Higher stem-diversity means replies vary with input. It does not mean they are right. subject-sensitive reaching 1.00 says the reply mentions what the input was about, which is necessary and nowhere near sufficient.
• 48 probes, one operator’s phrasing. They over-represent known failures and under-represent whatever nobody has looked at.
• The floors are still lenient. Chosen before there was a baseline. adjacency was sitting at 0.60 against an observed minimum of 0.92 — unfailable, and therefore decoration — until it was raised this week.

The lesson

Four systems failed the same way in two days. Argmax took the mode because it was cheaper than discriminating. The card grader judged on question text because that was all it was shown. The OOD fine-tune learned length because length was sufficient. The 85M learned the dominant register because that is where the capacity paid.

When a cheap feature separates the classes, the model learns the cheap feature.

None of them broke loudly; each produced a plausible number first. Which is the actual methodological claim of this bench: a result needs a story for why it worked before it can be believed — and twenty minutes counting openers would have saved an 85M training run.