B.29 — compose correctly or abstain, never bluff: the kitchen-sink regression gauntlet

T.M.B. #29 — the receipt for honesty. B.28 proved the brick runtime computes. This proves the router — the English → brick layer — either computes correctly or says “I won’t guess,” and never the confident-wrong thing in between. The number isn’t a score; it’s an invariant: 0 fabrications across the operator’s own kitchen sinks.

The claim under test

The 478/478 gauntlet (B.28) builds brick trees directly ({"brick":"sum_list",...}) — it grades the runtime, and never touches auto_compose, the layer that turns plain English into those trees. A live kitchen-sink test exposed the gap: the router was gauntlet-green but brittle and dishonest off the happy path. “the product of [2,3,4]” missed and got an offline-read essay about premium products; “if x > 5 …” was read by the compare lane as “which is larger” → “10 is larger”; a complex ask got an unrelated sign() template. The honest-abstain moat was leaking confident wrong answers.

Falsifiable claim: after the hardening pass, every prompt in a frozen set of the operator’s real, in-the-wild kitchen-sink prompts resolves to either a correct computed answer or an honest abstain — never a confident wrong answer.

The setup

benchmarks/regression_brick/run.py — 27 prompts the operator actually typed while stress-testing the chat (the ones that caught bugs), each with a frozen expectation: {"contains": "<value>"} for a compute, or {"abstain": true} (⇒ auto_compose returns None) for a capability gap. The grader is mechanical — string containment or is None, no model opinion. Run:

PYTHONPATH=. python benchmarks/regression_brick/run.py

The suite grows on every wild prompt that bites: each new fabrication found becomes a locked row, so the class of bug can’t silently return.

Results

27/27 — 19 compute correctly, 8 honest abstain, 0 confident-wrong.

• Computes: value-ops, char-iteration, filters (even/odd/>N/prime), conditionals (two-way, three-way, divisibility), per-element transforms (multiply every by 3 → a map, not a product), count- occurrences, two-variable arithmetic, multi-step chains ((x−y)×3 → 9), word-length maps, filter→map→reduce pipelines.
• Honest abstains: dual/multi-output (print both X and Y), string-prefix filters (beginning with "a"), cumulative scans (running total), and the deepest multi-filter pipeline — each a genuinely-new shape, not a bug.

Supporting numbers (same session, all re-runnable): brick_gauntlet (the router’s happy-path suite) 39/39, coding_gauntlet_bricks (the runtime) 478/478, and 80 compose/launch/isolation/ocr unit tests green.

What actually moved the number

Not new IQ — conservatism. Three mechanisms:

1. The moat gate — a compute-shaped miss returns an explicit abstain instead of falling to a fabricating lane; the compare lane bails on conditionals; the brick lane composes on the raw query (dequoting had been stripping the literals composers needed).
2. New shapes, honestly — value-ops, filters, conditionals, transforms, multi-step arithmetic all compose now, with predicates emitted inline (a brick-call predicate can’t await inside Array.filter).
3. _beyond_bricks guards — anything the composers can’t fully build (dual-output, prefix filters, scans, three-way-plus) bails to abstain rather than emit a partial. Every guard gap a harder battery found got closed the same day.

The honest caveats

• These are the operator’s own prompts, not an independent adversary. They’re real and they caught real bugs, but a fresh red-teamer will find new ones — that’s the point of a growing suite, not a frozen trophy. Expect the count to climb as new fabrications are found and locked.
• 8 of 27 abstain. This is a win for honesty, not capability — four missing bricks (string-length filter, string-prefix filter, cumulative scan, multi-output) would convert most of them. Until those ship, the honest answer is silence.
• This grades the ROUTER, not coding skill. Like B.28, the underlying tasks are beginner-level data pipelines. The claim is “never lies about what it can build,” not “codes hard problems.”

The verdict

The thesis, made measurable: compose correctly, or say “I won’t guess” — never bluff. 27/27 with 0 confident-wrong across the prompts that were breaking it hours earlier, and a gauntlet that grows teeth every time the wild finds a new one.