Day 140 — the test that tested the tester

A test framework, pointed at everything, found bugs in three places: the product, the server under it, and itself.

rspec for language models

The gauntlets test prompts. HAIRLy tests the UI as a user. werbos probe is the third shape: write one spec — prompt → mock → expect → result — and run it against any target. Every provider maps into the same reply shape, so a spec written for werbos runs unchanged against Claude or a raw open-weights model from ollama. In-process, mocks are real: t.mock('weather').timeout() genuinely breaks the weather API, and the spec checks werbos abstains honestly instead of guessing.

Allowed failures keep the comparison fair: a raw open model is allowed to miss live weather — it has no tool — while werbos must pass the same spec. An LLM-judge matcher ports HAIRLy’s subjective grading, a persona-fuzz layer rephrases every prompt the way a real person types (“so like whats the capital of france man?”), and a Playwright-style HTML report makes a 571-test run browsable. Nine skip placeholders put not-yet-wired surfaces on the map, so the breadth is visible before it’s tested.

It paid for itself on the first run. “Who wrote Romeo and Juliet?” returned a wrestler, then Tchaikovsky’s overture — the title was being cut at “and”. “Tallest mountain” served a List of mountain peaks index page as a fact. A trip opener ending in a question mark skipped the place offer. All fixed.

The trip remembers where it’s going

The overnight carry-over from yesterday’s memory work. “Going to London for 7 days” then “weather while I’m there?” answered for Rocklin, California — home — because the trip was only remembered when one particular lane answered. Now every turn notes it. And a moat catch worth keeping: the weather lane fetched +57°F correctly, then the small voice model restyled it as -57°F and invented a patio. The digits survived the sign flip, so the guard missed it. A precise live number is never voiced now — it ships plain and cited.

The first full run killed the server

571 tests produced a wall of failures at 10 milliseconds each — not timeouts, refused connections. The server was dead. Two causes stacked. Every code question rebuilt a search index over every card in the library — about 3.5GB and 90 seconds — even when an owned pattern answered and the index was never touched; a few in a row blew the 8GB memory cap and the kernel killed it, about every ten minutes. And a forgotten July copy of the service — different config, no memory cap, 33,114 restarts — was fighting the real one for the port, so every crash left a twelve-minute dead window.

Now the lookup indexes only the shelves a question is about, the stale service is off, and the server reports its memory so the probe can pin growth on the test that caused it. On the heaviest tiers: peak memory 7.26GB → 1.14GB, zero restarts. A code question that took 99 seconds takes 6.

Then the probe got caught lying

Reconciling the numbers against last week’s gauntlet — 86% honest on the same 500 rows — turned up the last bug, and it was ours. werbos answered [38,46,33,17,10,56]; the expected answer was [38, 46, 33, 17, 10, 56]; the matcher compared text literally. 62 correct answers were scored as failures. That number was about to be posted. Re-run with the fix, end to end: 292 of 571, not 211. Matchers now ignore formatting but not meaning — 1 2 still isn’t 12 — and every run lands on a timeline (benchmarks/probe_history.jsonl: commit, per-suite scores, dead- server count, memory peak), so the next number is comparable to this one.

Still open

The teacher lane hijacks questions it shouldn’t — “Let’s learn One” answered fourteen tests that needed code or an honest abstain, because its guard misses “Write and run code:” entirely. Slash commands — /stocks, /journal, /story and friends — mostly fail through the unified chat: only /news and /cooking pass, 2 of 12. And “find X, then compute Y” chains abstain at X. Measured, named, next.

A test framework is also a thing that can be wrong. Tonight’s count: product six, server three, test harness one — and the harness bug was the one that would have gone out in public.