Day 25 — 'the harness IS the dataset.'

“the harness IS the dataset.”

Spent the morning trying to figure out why /tarot threw a backend error in the chat UI. The bug was five layers stacked. What came out of the chase wasn’t a one-line fix — it was a 24/7 harness that grades every skill on every change and writes the verdicts to disk while the operator sleeps. Werbos now generates its own quality data.

The /tarot debug saga

Five distinct bugs in one user-facing symptom. In the order they were peeled back:

1. Stale image. The werbos/agent:local container had been “Up 11 days” — git pull had brought tarot into source weeks earlier, but the compose stack never rebuilt. The skill was in the code on disk, not in the running container. (One docker compose up -d --build werbos fixed it.)

2. The chat panel wiped the response after every slash turn. ChatView.tsx’s handleSend() calls getChat(chatId) after the stream completes to refresh from server state. For normal chat turns this is correct — the server archived them. But slash turns are intentionally not archived (per the comment at server/app.py:1512), so the GET returns an empty chat, setChat(fresh) wipes the optimistic user message and the just-streamed slash reply, and the empty “How can I help?” panel re-renders. ( 4abaa90 — slash-form is special-cased, the reply gets appended directly without the refresh.)

3. llama3.1:8b couldn’t reliably hold tarot’s strict format. Even with the prompt explicitly asking for **Spread:** ... headers and named cards, the 8B model emitted generic prose about a third of the time. The prompt was correct; the model was the floor. ( ed3ce91 — model="qwen2.5:14b" override on the tarot skill; 14B holds the structure.)

4. run_prompt_skill hardcoded a 180s timeout. The chat backend’s BellmanBackend already had a per-model size ladder (werbos/backends/__init__.py:64) — 8B gets 240s, 14B gets 600s — but the prompt-skill direct path used a flat 180s. A 14B-on-SER8 tarot reading at ~110s sat inside the budget; a longer reading didn’t. (Fix queued, lower priority once the daemon proved it’s holding.)

5. Markers were over-strict. Three skills (commit, retro, plan) failed the format check the first time the harness ran. Two were false positives — commit’s body separator marker contradicted the prompt’s own “omit body if subject is self-explanatory” rule; plan’s cosmetic **bold** requirement caught a model that produced the right text without the asterisks. One was real — retro’s model interpreted “skip a section if there’s nothing” too liberally and omitted the highest-value section. (1ee8e4b — two markers relaxed, one prompt sharpened.)

SE.1: format markers + smoke_one primitive

PromptSkill gained two append-only fields:

• test_input: str = "" — when set, the skill is part of the eval set. Empty = skipped.
• format_markers: tuple[str, ...] = () — substrings that MUST appear in the response. Empty = length-only grading.

The smoke_one() primitive fires one skill’s test_input at the local model, times it, counts how many markers landed, returns a SkillSmokeResult with `markers_hit / markers_total + duration_s

• response_len. *(c92083eships the primitive;d26b8e8ships thewerbos skill-smoke` subcommand on top.)*

10 skills got markers in the seed pass (9f4f8f3): summarize, eli5, commit, retro, plan, name-gen, tarot, quote, password-gen (the LLM-bypass control), tip-calc, cbt-reframe. Final baseline: 11/11 pass after the calibration round.

SE.1.1: the 24/7 daemon

werbos skill-smoke-loop runs the same harness on a recurring interval (30 min by default), writes a timestamped markdown report per cycle, appends per-skill rows to a long-running trend.csv, and crucially computes regressions against the latest green baseline. A skill that passed yesterday and fails today writes regressions-<stamp>.txt. (c2d7c00)

Launched on the SR8 deploy via docker exec -d inside the agent container. Per-cycle wall-clock: ~4 min. Over the first hours of running:

• 19 cycles overnight
• 209 individual skill smokes
• 203 / 209 = 97.1% pass rate
• 10 of 11 skills at 100% pass over the whole window
• One persistent flake: quote, hitting 2/3 markers about 32% of the time. The third marker (**Why this lands here:**) was the bold wrapper the 8B model sometimes dropped.

The dashboard that didn’t exist yesterday

What this composes into: werbos has its own quality monitoring, running on its own hardware, grading itself. Every 30 minutes a report lands on disk. Every regression writes an alert file. The operator opens tail -n 30 /data/skill-smoke/daemon.log to see the past few cycles’ verdicts. No external service. No paid dashboard. Just a Python loop and a CSV file accumulating behavior data hour by hour.

The harness’s output IS the input to the next sprint: the trend CSV is going to tell us which skills are stable enough to precook, which need a per-skill model override like tarot got, and which are model-floor problems no marker calibration can solve.

Numbers worth keeping

• PromptSkill fields added today: 2 (test_input, format_markers)
• Skills with markers seeded: 10 (+ tarot from the morning)
• SE.1.1 first overnight: 19 cycles, 209 smokes, 6 regression alerts (all quote, same cause)
• Overall pass rate first night: 97.1%
• Container restarts during the chase: at least 5
• Times the operator stopped piling on diagnostic before fixing: 0 (should be more; gets worse before it gets better is the wrong default for a UI bug)

Aphorisms

“the harness IS the dataset — the trend CSV tells you which skills are worth pre-rendering before you ask the question.”

“the symptom was ‘TimeoutError’. The bug was state management. Every layer between them lied a little.”

“if the cure is ‘rebuild the container’, the diagnosis is always ‘I didn’t rebuild the container.’”

Carryover into Day 26

• Fix the 180s prompt-skill timeout. The chat backend has the size ladder; the prompt-skill direct path doesn’t. Skills on qwen2.5:14b that ought to land at 100-200s get the _Couldn't reach the local model_ line instead.
• The quote flake is real model variance, not marker miscalibration. A 32% failure rate on a bold wrapper is the 8B’s instruction-adherence floor. Either tighten the marker further (lose information) or switch the skill’s model.
• Wire SE.1.1 trend data into a “which skills cache well” classifier. A skill that scores 100/100 on format markers across N cycles is a candidate for the precook layer. The trend.csv already has the column shape; just needs a reader.