B.31 — werbos probe, 571 tests on the live path: 292 pass, and the ruler had a bug too

T.M.B. #31 — the live path, measured end to end. B.30 ran the 500-item gauntlet through the owned brain. Since then the chat became 1chat — the brain’s answer voiced by a small model. This runs the same 500 (plus 71 skill, command and continuity tests) through the path a user actually hits, and reports what that costs. It also reports two wrong numbers we nearly published first, because the way a benchmark lies to you is part of the result.

Browse all 571 tests → — every test, filterable, with werbos’s actual answer on each failure.

The claim under test

Does werbos hold its moat — do the thing correctly, or say “I won’t guess” — on the live chat path, across compute, facts, multi-hop, code, commands and multi-turn continuity? And, just as falsifiable: is the harness measuring werbos, or itself?

The setup

werbos probe — a provider-agnostic describe → it → expect framework (werbos/probe/). 571 tests in 19 suites: the 500-row gauntlet (werbos/probe/data/gauntlet.jsonl, 4 tiers × 125, the same rows as B.30) plus code, facts, multi-step, skills, safety, every help-panel command, and multi-turn continuity arcs. Each test gets its own session. Target: SR8’s live ab server (:8097, phase=onechat) — CPU only, llama3.2:1b voice, 8GB MemoryMax. Re-run it:

WERBOS_PROBE_AB=http://<box>:8097/ WERBOS_GAUNTLET_N=0 WERBOS_PROBE_DELAY=1 \
  werbos probe --html report.html --history "label"

Every run appends a row to benchmarks/probe_history.jsonl — commit, knobs, per-suite scores, transport errors (a dead server, not a wrong answer) and the server’s RSS peak.

Results

Same 571 tests, one night, four runs:

The gauntlet, run 4, next to B.30’s brain-only run of the same 500 rows:

Per tier (run 4): floor 97/125 · ceiling 77/125 · sky 47/125 · space 33/125. Outside the gauntlet: facts recall 10/10, Mexico-trip continuity 4/4, weather continuity 3/3 (+1 skip), code 4/5 — and the commands the chat’s help panel advertises mostly fail on this path (/news and /cooking pass; 2 of 12 non-calculator commands).

What actually moved the number

Nothing in werbos got smarter tonight. Every jump was the measurement getting honest:

1. 159 → 211: the server was dead. Every code turn rebuilt a TF-IDF index over every card (~3.5GB, ~90s) — even a pattern hit — and back-to-back code tests OOM-killed ab every ~10 minutes. A stale duplicate systemd unit fought the real one for the port, stretching each crash into a 12-minute dead window. 126 “failures” were refused connections at 10ms. Fixed: routed + memoised lookups, index caches capped, the duplicate unit disabled.
2. 211 → 292: the harness was wrong. werbos answered [38,46,33,17,10,56]; the gold was [38, 46, 33, 17, 10, 56]; the matcher compared bytes. 62 correct answers were scored as failures. Matchers now ignore formatting (bracket/comma spacing, quote style) but not meaning — 1 2 is still not 12.

What the live path costs

B.30 reported zero confident-wrong on the compute tiers. On the live path, floor + ceiling have 18 answered-but-wrong — three causes, all real:

• The voice misreads the task (7). “Add 7 to every number… map” reaches the 1B voice model as a geography word: “I’m not going to create a map with coordinates like 250,15,75…”, “I’m staring at the map of Jason’s commute route.” The brick computed the right list; the voice buried it in fiction.
• Sort returns nothing (8). Eight sort prompts came back empty — a non-answer, not a lie, but a floor regression.
• A stale verified program (3) — the worst kind. “Count the vowels in river” came back ✓ verified — ran clean with a program that counts the vowels in “education”. Execution-verified, for the wrong input. A confident-wrong answer wearing the moat’s own badge.

And one leak outside compute, on the abstain traps — prompts built so the only honest answer is “no such thing.” The web rung answered them from fan wikis: “Jasonium has an atomic number of 224” (cited to a fandom wiki), “the capital of Wakanda is Birnin Zana” (a Marvel wiki). Cited, labelled, and wrong about which world it’s in. The web rung needs a fiction filter the way it already rejects index pages.

The honest caveats

• Not the same grader as B.30. B.30 graded with its own gauntlet scorer; B.31 uses probe’s containment matchers. The 80% vs 86% honest-rate comparison is directional, not exact.
• “Answered, didn’t match” over-counts confident-wrong. It includes a few answers the matcher can’t see — e.g. the right numbers voiced without brackets. The compute rows were read by hand (above); the other 81 weren’t all read, so treat 99 as a ceiling. At least one is a right answer the gold can’t match (“10 ÷ 0 is undefined” vs the gold ABSTAIN / ERROR — undefined).
• One box, CPU only. SR8’s ollama was itself OOM-killed once during run 2 (an 11GB model). Latency and memory numbers are for this box.
• The teacher lane misroutes. “Let’s learn One…” answered 14 tests in each full run that needed code or an abstain; its guard regex matches none of the “Write and run code:” prompts. Those count as fails here, correctly.

The verdict

292/571 on the live path — and the path costs something the brain alone didn’t. The honest rate holds (80%), facts and continuity are solid, but the voice layer and two compute regressions reopened leaks B.30 had sealed, and one of them lies with a verified badge. The fix list writes itself: never voice a compute result (the weather rule, extended), key cached programs on their input, restore sort, fix the teacher guard, and keep fan wikis out of the web rung. The other lesson is about benchmarks: the first two numbers tonight were wrong for reasons that had nothing to do with the model. A number is only as honest as the harness — so the harness gets tests now too.