B.26 — the ruler turned inward: werbos grades its own skills

The claim under test

The reasoning Gauntlet (B.06, B.14, B.24) measures how far one cursor reasons. It says nothing about whether the dozen-odd owned skills — the deterministic lanes and the composers — still fire. autopilot_reflect reports on skills, but only from whatever traffic happened to arrive: a skill nobody exercised looks neither working nor broken. The claim: werbos can measure whether its own skills still fire on purpose, catch one that silently breaks, and do it without a model grading answer quality.

The falsifiable half is sharp. If a green skill that breaks isn’t caught, there is no ruler. And if the check passes a skill that doesn’t actually fire — or, worse, calls a model to ask “was that a good answer” — it is the false self-image with extra steps: a soft judge grading a soft judge, the exact circularity reflection_oracle (B, “earn the right to act”) exists to refuse.

The setup

python -m werbos.gauntlet.skills_gauntlet — a frozen, content-hashed probe set fired against the real modules, no network, no H200, no model in the check. Each probe runs a skill on a fixed input and asserts a behavioral outcome on one of two axes, deliberately the same two the reasoning Gauntlet scores:

• capability — the skill fires on its own domain (arithmetic_compute("2+2") returns an Answer; empathy.classify reads a crisis message as crisis; shapes.match finds the invariant-test move for a jug problem).
• honesty — it stays quiet off its domain (a false-positive on someone else’s query is the one sin the lanes were built to refuse), and the safety gate holds (empathy.compose on a crisis text returns the referral, never a comfort line).

Eight skills, sixteen probes, each expectation verified against the live module before it was written into the harness. The set is hashed into a bundle_id; every run appends to benchmarks/skills_gauntlet_history.jsonl, so any two runs of the same bundle_id are comparable and a green→red is a caught regression, not drift. An optional --drive adversary rewords each firing probe and checks it still fires; a model may generate those rewordings (the fable seam), but it never scores the output — driver, never judge.

Results

first run — 8 skills · 16 probes · bundle 59a14f8fe3c5
  capability   9/9    every skill fires on its own domain
  honesty      7/7    every skill stays quiet off it / safe on the gate
  total       16/16   the ratchet's FLOOR

teeth test — a lane broken in memory (arithmetic stops firing on "2+2")
  16 → 15    caught under the SAME bundle_id
  weak_at = [lane:arithmetic]   → the self-study miss-list

adversary — --drive, 4 deterministic rewordings per firing probe
  concordance  4/4     units 4/4      shapes:reason 4/4
  arithmetic   3/4     empathy 3/4    cooking 3/4    smalltalk 3/4
  syllogism    2/4   ← weakest: breaks on UPPERCASE / double-spacing

16/16 is not the headline. The headline is the two facts underneath it: a green skill that breaks is caught the instant it does, and the plain-sweep green hides real brittleness the adversary walks straight into.

What actually moved the number

The design decision is the result: no model in the check. Every green here is a deterministic behavioral assertion — did the right lane fire, did the wrong query fall through, did the crisis gate return the exact referral string. That is what makes 16/16 mean something instead of being a model flattering a model. It cost the whole “earn the right to act” arc to learn that a soft judge grading a soft judge measures nothing; this benchmark spends that lesson.

The adversary is what turned a green light into a diagnostic. The plain sweep says syllogism ✓. --drive says syllogism 2/4 — it fires on the clean input and breaks on the same sentence in capitals. That gap is the actual finding: a concrete, re-runnable “here is where I am weak” that the passing score concealed.

The bundle_id is what makes the ratchet a ratchet. Freeze the probes, hash them, log every run, and green→red under the same hash is unambiguous — a regression, not a moved goalpost. The teeth test proves the mechanism, not just the code path: break one lane, watch 16→15, watch the broken skill land in weak_at.

The honest caveats

• Eight skills, not ninety. This probes a representative 8 (three lanes, empathy, cooking, small talk, concordance, the reasoning shapes) out of ~90 composers and 15 lanes. 16/16 is a floor over a sample; it is not a claim that every owned skill fires. The next rung is coverage, and the benchmark does not pretend to have it yet.
• Fires ≠ good. The check is behavioral — a skill can fire on its domain and still produce a mediocre answer, and this says nothing about that. Answer quality is deliberately out of scope: grading it needs a model, and a model grading a model is the one thing the design refuses. So a green means “the plumbing is connected and honest,” never “the output is excellent.”
• One to three probes per skill. Coarse. A skill can pass its single probe and fail on a neighbouring input the set doesn’t contain. More probes per skill tighten this; today it is thin.
• The adversary is cosmetic, not semantic. Four rewordings — uppercase, trailing ?, a prefix, double-spacing. Real robustness is paraphrase and register shift; the model-generated (--drive fable) reworder that would test that is a documented seam, not wired. syllogism 2/4 is a genuine finding, but over a tiny cosmetic set.
• A single run is a floor, not a trend. The value of a ratchet is the series. This is the first datapoint; it means “nothing has regressed yet,” which is only interesting once there is a second run to regress from.

The verdict

As a coverage number, 16/16 over 8 skills is a floor — honest, small, and explicitly not a claim about the other ~80. As a ratchet, it is the thing that matters: a frozen, model-free ruler that catches a green→red silently, turns every miss into a self-study target, refuses to let a model grade a model, and — on its very first run — used its adversary to find a skill (syllogism) that looked green and wasn’t robust. It is the reasoning Gauntlet’s discipline turned inward on werbos’s own skills: werbos checking werbos, an honest doorman on its own front door, built and proven on a laptop while the H200s minted. The number to watch is not this one — it is the second run, and whether anything went red.