werbos

← All log entries

Day 34 — 'werbos tells jokes again.'

Day 33 made the homelab box fast; Day 34 made it answer correctly. The symptom: 'tell me a joke' came back as literal {"name": "joke", "parameters": {"args":""}} — the small model naming the SKILL as text instead of calling the dispatch_skill tool. Pitts' salvage parser RIGHTLY refused to lift it (the tool-name allowlist is a load-bearing injection defense), so the JSON leaked verbatim on the bulk path and rendered as an empty bubble on the streaming path. Three stacked fixes, each found by deploying the last and re-running the same three-prompt repro on the real box: (1) a skill-rescue pass that rewrites a hallucinated skill call into a real dispatch and lets the skill output BE the reply; (2) a bare-JSON exemption from restrictive mode, because a turn-1 validation error had left a ToolResult in history that starved the rescue on every later turn; (3) pinning num_ctx on the prompt-skill ollama_chat path — the keepwarm bug class, third occurrence — because the dispatched /joke was 503ing on a mismatched context size. Greeting → prose, joke → an actual joke. 5,530 tests green.

“werbos tells jokes again.”

Yesterday’s hunt got the homelab box answering in ~30s instead of 176. But it was answering wrong. Ask it for a joke and you got back a blob of JSON. The speed was real; the correctness wasn’t there yet.

The symptom

Three prompts against the live box:

"hello there"   → a real greeting ✓
"tell me a joke" → {"name": "joke", "parameters": {"args":""}}
"ping"           → {"name": "small-talk", "parameters": {"args":""}}

The small model was naming the skill as plain text instead of calling the dispatch_skill tool. And pitts’ salvage parser — correctly — refused to lift it: the tool-name allowlist only contains real tools, and that guard is load-bearing against indirect prompt injection. So the JSON survived into the reply: raw on the bulk path, an empty bubble on the streaming path (the filter swallowed it). Same root as the original empty-reply bug.

Fix 1 — rescue the hallucinated call (e7acf96)

The failure is predictable, so encode the recovery. New _skill_rescue module: re-run salvage with the allowlist set to the known skill catalog (so every R2 fence / attribution / restrictive guard still applies), rewrite the lift into the real dispatch_skill call the model meant, and run it. The skill’s output becomes the reply — identical semantics to the model having used the tool channel properly. Plus a step() backstop so an unrescuable JSON blob (naming a tool that doesn’t exist) is stripped and replaced with an honest fallback line, never returned raw. 8 new tests, including the exact repro.

Fix 2 — the restrictive-mode trap (6df3e13)

Deployed Fix 1, re-ran the repro — still empty. The archive showed why: turn 1’s dispatch_skill validation error had left a ToolResult in history, which flips every later turn into restrictive salvage mode, which requires an “I’ll call X:” prelude the bare hallucination doesn’t have. The rescue was starving on provenance grounds. The fix: when the entire message is one bare JSON object, there’s no surrounding prose to attribute from — so exempt that shape from the prelude requirement. Prose-embedded JSON still needs the prelude (R2 defense intact).

Fix 3 — num_ctx, third occurrence (dcbb642)

Re-deployed, re-ran — now the rescue fired, but the dispatched /joke came back “Couldn’t reach the local model.” The prompt-skill path’s ollama_chat sent no num_ctx, so it asked Ollama for the model at a different context size than the warm chat instance — on the RAM-tight CPU box that 503s. This is the same keepwarm bug class (Day 33’s eviction was occurrence #2). Pinned num_ctx (env-overridable, else 8192) on the skill path too.

The payoff

"hello there"   → "I'm doing well, thanks for asking..."
"tell me a joke" → "A man walked into a library and asked for books
                    on Pavlov's dogs and Schrödinger's cat. The
                    librarian said: it rings a bell, but I'm not sure
                    if it's here or not."

Numbers worth keeping

  • Three deploy-and-reverify cycles on the real box, not just unit tests — each fix surfaced the next bug
  • 8 new rescue tests + the num_ctx pin tests
  • 5,530 tests green across the suite

AHAs

  • The same bug class will knock three times. num_ctx mismatch has now bitten keepwarm (ping), then the skill path — and the only reason it was fast to spot the third time is the first two were documented. Write the lesson down or re-debug it.
  • A correct security refusal can still produce a broken UX. Pitts was right to reject the lift; the bug was that nothing caught the fallout downstream. The fix wasn’t to weaken the guard — it was to add a recovery path that inherits the same guard.
  • Deploy-and-reverify on the actual constrained box beats green unit tests. Two of the three bugs were invisible until the real archive showed the turn history.

Tomorrow

The chat surface is fast and correct. Push the README sweep, flip the repo public, and get back to DS.1.