B.32 — the probe, one day later: 292 → 403 of 571, 97% honest on the gauntlet
B.31 measured werbos's live chat path at 292/571 and turned the failures into a work list. One day of working it top-down — confident-wrong first, then routing, then coverage — moved the same 571 tests to 403 (110 gained, 0 lost). On the 500-row gauntlet: 355 correct, 131 honest abstains, 14 answered-wrong, a 97% honest rate against B.30's 86% and B.31's 80%. For the first time most remaining failures are honest 'I can't ground that' rather than wrong answers. Includes the intermediate run that lost 20 rows to the day's own changes.
T.M.B. #32 — same ruler, one day apart. Nothing in the test set changed between B.31 and this run: the same 571 tests, the same live path, the same box. What changed is 27 commits of fixes, each aimed at a class of failure the probe named. The number worth reading is not 403 — it is 14: the confident-wrong answers left on the 500-row gauntlet.
Browse all 571 tests, every run →
The claim under test
B.31’s failures clustered by cause, not by question. If the causes were real, fixing a cause should move many rows at once and break none — and the moat metric (answered-and-wrong) should fall much faster than the pass count rises, because the fixes were aimed at wrong answers first.
The setup
Identical to B.31: werbos probe, 571 tests in 19 suites
including the 500-row gauntlet, one fresh session per test, against SR8’s live 1chat endpoint (CPU only,
llama3.2:1b voice). One change to the harness, disclosed below: refusal rows and code rows are now graded
on meaning by a narrow judge. Re-run:
WERBOS_PROBE_AB=http://<box>:8097/ WERBOS_GAUNTLET_N=0 WERBOS_PROBE_DELAY=1 \
werbos probe --html report.html --history "label"
python scripts/probe_page.py <run>.log <site>/static/benchmarks/werbos-spec/index.html
Results
| run | pass | gained / lost vs B.31 | dead-server errors |
|---|---|---|---|
| B.31 (baseline) | 292 / 571 | — | 1 |
| groups 1–5 | 372 / 571 | +99 / −20 | 1 |
| + the 20 fixed | 403 / 571 | +110 / 0 | 0 |
The 500-row gauntlet, three benchmarks on the same rows:
| B.30 (brain only) | B.31 (live, day 1) | B.32 (live, day 2) | |
|---|---|---|---|
| correct | 224 | 254 | 355 |
| honest abstain | 207 | 146 | 131 |
| answered, didn’t match | 31 confident-wrong | ≤ 99 | 14 |
| honest rate | 86% | 80% | 97% |
Per tier: floor 97 → 112, ceiling 77 → 80, sky 47 → 82, space 33 → 81.
What actually moved the number
Fixes aimed at classes, in the order the moat asks for:
- Confident-wrong at the root. Owned code patterns must run on the question’s own inputs (pattern layer on the 500 prompts: 22 wrong → 0). The brick bin declines what it only half covers. Keyword lanes stand aside for named algorithms. Generated code must use the question’s literals and print a result.
- What the voice may touch. Computed results and plain facts ship unvoiced; a short list answer is no longer mistaken for a citation tag (every short sort had shipped empty).
- The web, trusted less. Fan wikis, homework sites and an AI-written encyclopedia are not sources — including answers already cached from them. An inferred answer and a trending headline are not answers.
- Owned facts, then chains. A Wikipedia-infobox source (no model), ports from the OS’s IANA registry, and find-then-compute chains: dependent steps, fact pairs, and the “element with that atomic number” hop.
- Routing. An ordered task is not a lesson request; a number conversion is not a speaking style.
The losses
The first full run of all this lost 20 rows. Every one traced to that day’s changes — a safety list that blocked the ordinary word “divisible”, and a chain that stopped with an honest “I can’t finish” where the web rung used to finish (skipping a rung of the ladder). They were fixed and the whole suite re-run; the number above is that re-run, with nothing lost.
The honest caveats
- The grader changed a little. Refusal rows (“FLAG PREMISE — …”) and code rows are now graded by a narrow LLM judge when the literal match fails; B.31 graded them by substring. It is limited to those two categories. At least three rows in B.31’s wrong column (X-093, X-118, X-119) were correct answers the old grader couldn’t read.
- Some gains came from rented rungs. The code rows now pass through a rented 16B coder checked by execution, and some fact rows through the grounded web reader. Owned-first still holds — they fire only after the owned lanes decline — but the 403 is not 403 owned answers.
- Honest abstains still count as failures. The 131 include facts werbos simply has no source for (piano keys, bone counts, jersey numbers) and computations the bricks can’t compose yet. Counted as failures, which is right.
- One box, one day. SR8, CPU only; a different day’s web can move the web-rung rows either way.
The verdict
Fixing causes moved 110 rows and broke none, and it moved the moat metric hardest: answered-and-wrong went from as many as 99 to 14 on the same 500 rows. For the first time the remaining failures are mostly honest abstains rather than wrong answers — about 45 are still brick coverage, about 60 are facts werbos doesn’t have. That is the line where tweaks stop paying and the library starts to.