B.27 — the rated coding gauntlet: solid 150, a soft 350, a cliff above

Thu, Sep 3 2026

The old coding gauntlet was 478/478 — but every problem was intro-level (reverse a string, factorial), so 100% meant 'aces CS101,' not 'can code.' This is the harder ruler: a difficulty-rated gauntlet, 150→1200, where ~1200 is a frontier model's reach (median of two sorted arrays, minimum window substring, max-profit-with-k-transactions). Same validity contract — execution-graded, gold computed in Python, no model opinion in the verdict. Result: werbos rates ~350 — it aces the intro tier (20/20), scrapes the basic-algorithm tier (12/24), and falls off a cliff on real algorithms (0/24 on classic DP). And the honesty property we sold on the old gauntlet — 'never confidently wrong' — turns out to hold only on easy problems: asked for the indices summing to a target, it returns the array's total; asked for the max subarray, it returns the sum, which coincidentally passes only when the whole array is the answer. A floor-coder, measured honestly.

The claim under test

The retired coding gauntlet scored 478/478 (100%) and beat qwen2.5:14b (471) — a number we leaned on. But every one of its 24 families was intro-level: reverse a string, count vowels, factorial, is-prime, double it. 100% there means aces CS101, and the two things it actually proved — never confidently wrong and 178× faster than a 14B — are an honesty win and a speed win, not a coding win. This asks the harder question: on a difficulty scale where 1200 is a frontier model’s reach, what is werbos’s coding rating? The falsifiable claim: it’s low, and the “never-wrong” property is conditional — it breaks the moment a problem needs a real algorithm.

The setup

scripts/gauntlet_rated.py — same validity contract as the original (scripts/gauntlet.py): execution-graded, gold computed by a Python reference implementation in the file, no model opinion in the verdict. The only additions are a difficulty per family and a rating: werbos’s rating = the hardest tier it still clears ≥50%. Every gold is self-tested (--selftest) before any werbos number is trusted. 140 probes across 24 families and 6 tiers:

tier	what it tests	families
150	intro	is-prime · factorial · sum-of-array
350	basic algorithms	binary-search · gcd · fizzbuzz · two-sum
550	classic DP / parsing	LCS-length · coin-change · valid-parentheses · roman-to-int
750	DP / graphs	edit-distance · 0/1-knapsack · word-break · number-of-islands
950	harder	max-subarray (Kadane) · longest-increasing-subseq · trapping-rain-water · KMP-substring
1200	frontier reach	median-of-two-sorted · largest-rectangle-in-histogram · minimum-window-substring · max-profit-k-transactions

1200 = frontier is definitional: those four are leetcode-hard problems a strong model (Claude, GPT-class) solves reliably. The gauntlet does not yet measure a reference model on this ladder — SR8’s GPU is down (CPU-only inference) and the GPU box is unreachable from the harness, so the 14B/frontier column is pending, not measured. What’s measured here is werbos, honestly.

Results

  tier   solved  abstain  wrong   pass%
   150   20/20         0      0    100%    <- aces intro
   350   12/24         6      6     50%    <- scrapes basic algorithms (borderline)
   550    0/24        24      0      0%    <- collapses on classic DP (LCS, coin change)
   750    1/24        12     11      4%
   950   10/24        12      2     42%    <- INFLATED, see below
  1200    6/24         0     18     25%    <- 0 abstain: it guesses, and mostly misses

  WERBOS CODING RATING: 350 / 1200   (highest tier cleared >=50%)
  totals: solved 43   abstain 54   wrong 43

Rating ≈ 350, and it’s a soft 350 — it clears that tier by exactly one probe, and it’s already producing confident-wrong answers there.

What actually moved the number — two receipts

The valuable part isn’t the 350; it’s why the “never confidently wrong” property doesn’t survive contact with real problems.

1. two-sum: it answers a different question. Asked for the two indices that sum to a target, it returns a number:

solve([2,7,11,15], 9)  ->  got 35   (the array's total)   want [0,1]

It summed the array and ignored the target. A confident wrong answer, not an abstain.

2. max-subarray: a wrong algorithm that hides as a solve. It computes the array sum, not Kadane’s maximum subarray. It “passes” only when the whole array is the best subarray:

solve([1,2,3,4])           got 10  want 10   PASS  (sum == max-subarray, by luck)
solve([-2,1,-3,4,-1,2,...]) got  1  want  6   WRONG (sum != max-subarray)

This is why tier-950 reads 10/24: 4 of those are coincidental matches (sum happening to equal the answer), plus 6 genuine (KMP substring search, which it really can do). Discount the luck and the real tier-950 capability is ~6/24. The gauntlet needs adversarial inputs — cases where the right algorithm and a plausible wrong one diverge — or it credits coincidence as competence. That fix is the next iteration.

The honest caveats

“Never confidently wrong” is conditional. It held on the old (intro) gauntlet because those problems either template-match or abstain. On real algorithms werbos leaks wrong answers (two-sum, knapsack over-filling a bag that doesn’t fit, tier-1200 guessing with 0 abstains). The property is real on easy problems and false on hard ones — and a benchmark that only tested easy ones hid that.
The 350 is soft and partly propped by coincidental matches; the trustworthy floor is the solid 150.
No reference model measured yet. The 14B/frontier column is definitional/pending, not run.

The verdict

werbos is a floor-coder: rock-solid on intro, borderline on basic algorithms, and off a cliff on anything that needs a real algorithm — because the owned composer recognises and instantiates, it does not generate novel algorithms. Against a 1000/1200 goal, the honest gap is ~350 → 1000, and it is a build, not a tune: a library of typed algorithm-shapes, recursive decomposition to verifiable sub-steps, and execution-verify at every leaf — the one edge a frontier model doesn’t have. The number to be proud of isn’t the old 478/478; it’s this one, because it’s the first coding score that cost something to earn.

Update — the leaks, fixed (same day)

B.27 exists to expose problems, and the exposed ones got fixed the same afternoon:

Adversarial inputs. max-subarray’s inputs now mostly diverge from the array sum (5 of 6), so a “just sum it” wrong answer can no longer coincidentally pass. The ruler stopped crediting luck.
An abstain guard in the composer (multistep.py _idiom_fn): the generic sum/max idioms no longer fire on problems they can’t handle. A marker set (subarray, plural indices, sum-to-target, subsequence, histogram, transactions, combined-arrays, …) blocks the fallthrough, so werbos abstains instead of guessing. It uses plural indices, not index, so binary-search’s “the index of” still resolves.

Re-run, same 140 probes:

  tier   solved  abstain  wrong
   150   20/20         0      0    real
   350   12/24        12      0    real (binary-search, gcd)
   550    0/24        24      0    honest abstain
   750    0/24        24      0    honest abstain
   950    6/24        18      0    real (KMP)
  1200    0/24        24      0    honest abstain
  0 CONFIDENT-WRONG across all six tiers   (was 43)
  RATING: 350 / 1200

The rating didn’t move — it was never about the wrong answers — but the “never confidently wrong” property is now real on hard problems, not just easy ones. The 350 is earned: solid intro, real binary-search/gcd, real KMP, and honest silence everywhere else. The original 478/478 intro gauntlet is unaffected (the guard doesn’t touch intro asks). This is the trustworthy baseline the 350→1000 climb starts from — a number that now costs something to move.