Day 20 — 'the chain moved into code, and the model went PFM.'

“the chain moved into code, and the model went PFM.”

Where Day 19 left off

Day 19 closed with: structural fixes (AO.1 Part 3.1 project-scoped read_file, frontmatter strip, max_iterations bump, persona rewrites for @front-end and @css-wizard). Chain completing end-to-end on the design studio’s @studio → @visionary → @front-end → @css-wizard four-step pipeline. Quality landed but features got dropped under cognitive load.

Day 20’s opening hypothesis: persona discipline + a stronger orchestrator model could close the feature-drop gap.

Day 20’s actual finding: the orchestrator model is the wrong place to put orchestration discipline. Move it to code.

The compounding-bug discovery

Multi-pass chain (visionary → front-end → css-wizard) with persona rules that said “each specialist has one job, preserve what came before.” Sounded clean.

Reality, observed in sequence over the evening:

1. @studio on qwen2.5:14b: dispatched @visionary, got a spec, then emitted the @front-end dispatch call as TEXT in chat content instead of through the structured tool channel. Looked like: {"name": "dispatch_to_agent", "parameters": {"brief":"...","handle":"@front-end","project":"..."}} – a perfectly-formed JSON tool call, narrated instead of invoked. The model UNDERSTOOD the chain; it couldn’t EXECUTE it.
2. Bumped to deepseek-coder-v2:16b. Same failure pattern. Skipped the @front-end step entirely on one run and tried to dispatch @css-wizard with placeholder strings (<-- full html from previous step -->) instead of actually reading the upstream artifact.
3. Multi-pass split (@front-end = bones, @css-wizard = polish) stretched the chain to 5 sequential tool calls. The model understood it; it couldn’t reliably execute 5 steps in a row. Structural-tool-call discipline degrades past depth 3 on local 16b models, period.

This is the kind of failure mode that’s only obvious after you’ve built the architecture relying on the assumption it WOULDN’T fail this way. We had it.

The architectural answer: move the chain into code (AO.1 Part 5)

If the model can’t reliably orchestrate 5 sequential dispatches, don’t make it. Build a new tool that runs the chain internally, in Python, and exposes ONE tool call to the orchestrator.

werbos/tools/build_project.py (d4347be):

build_project(brief)
  ├─ dispatch @visionary  → spec.md
  ├─ read_file("spec.md")
  ├─ dispatch @front-end  → index.html (bones + JS)
  ├─ read_file("index.html")
  └─ dispatch @css-wizard → index.html (styled)

@studio’s persona shrinks to “call build_project(brief) once, then reply done.” No decisions to skip. No dispatch calls to narrate as text. No placeholder strings; the tool reads each upstream artifact from disk and constructs the next brief with the full content pasted in.

This works the first time. Chain completes end-to-end, all three specialist files land on disk, no tool-call hallucination, no truncation.

The pattern generalized to revision passes immediately. AO.1 Part 6 (bac94d6) shipped polish_styling(request) for visual revisions (“make the first column green” → read index.html, read spec.md, dispatch @css-wizard with both pasted in). Same Path B/C orchestration-reliability failure → same fix.

The Personal Finance Manager arc

Between architectural fixes, the visionary kept inventing domains for abstract briefs. The operator typed:

“Build a modern SaaS dashboard: top row 3 columns with data, middle row 2 columns (text + image), bottom row a chart or graph”

That’s a structural brief: rows and columns and shapes, no specific product. The model treated it as a product brief — “what kind of dashboard?” — and consistently chose Personal Finance Manager, three times in a row, with full transaction forms, budget categories, and groceries dropdowns the operator never asked for.

The model “going PFM” became the joke of the night because the project’s own at-rest container format is also called PFM — “Pure Fucking Magic” per its README. The visionary accidentally summoned the project’s own acronym back at us. (README.md of the .pfm sibling literally has “Yes, PFM stands for Pure Fucking Magic” in its “## The Name” section. Triple 100s on socket.dev for the categories that grade your code, not your dependency tree. Live web viewer at getpfm.io. The whitepaper §5.2 I wrote three nights ago calling it “an AI-purpose-built container format” is the scrubbed version; README is the source of truth.)

The PFM-domain-bias took persona iteration + a model bump to break. By 9:50pm: the spec correctly produced “Modern SaaS Dashboard” with all three structural rows and [Metric 1] / [chart placeholder] literal placeholders – no domain invention.

The terse-brief refusal (Parts 5.1 and 6.1)

After qwen2.5:14b stopped inventing domains, a new failure showed up: the model refused to spec terse briefs. A 60-char operator brief got a spec.md containing:

“It seems like you would like me to create a focused product brief based on your request. However, you haven’t provided the specific details or context…”

Two contributing causes, both fixed:

• Part 5.1 (0cebea6): build_project’s brief framing told the model “the chat-turn message may be empty if the operator just said ‘go’ or ‘build it’ — ignore if so.” That sentence PRIMED the 14b to think briefs CAN be too-empty-to-spec, which licensed the refusal. Tightened the framing: intent.md IS the brief, full stop; chat message is supplementary when present, ignored otherwise.
• Part 6.1 (67956e6): the persona’s “don’t invent content” rule (Part 4) was being over-applied. Any uncertainty became license to ask back. Added explicit “you NEVER refuse to produce a spec” section to the persona, listed the canonical refusal phrases the model should NEVER emit, and pointed at “spec it with placeholders
  • an Open questions section” as the right response under uncertainty.

After both: the visionary produced a real spec on the same terse brief that previously refused.

The polish_styling test — the most-satisfying moment of the night

After all the build-path work, the operator typed “can you make the first column Metric 1 green?” on the just-built dashboard.

The orchestrator routed to polish_styling. The tool read index.html + spec.md from disk. Dispatched @css-wizard. The @css-wizard’s output overwrote index.html with one column tinted.

The model made Metric 2 blue. Wrong card, wrong color. But the infrastructure WORKED — first time the revision path didn’t trigger a rebuild, didn’t ask the operator to provide HTML again, didn’t invent a new domain. It painted the wrong apple a wrong color, with the full apple-painting machinery routing correctly in code.

That’s the win. The remaining bug is specialist-discipline, which the eventual screenshot eval Suite would grade as “polish targeted correct element?” — a single binary check, not an architectural puzzle.

AHAs

• Past depth 3, model orchestration is unreliable on every local 16b we have access to. The model UNDERSTANDS the chain (it builds the dispatch call JSON correctly). It can’t EXECUTE the chain reliably (skips steps, narrates calls as text, uses placeholder strings instead of reading artifacts). Move the chain to code; the model makes ONE structured call.
• “Don’t invent content” is one rule away from “refuse to produce content.” Persona rules over-apply when the model is uncertain. Pair every “don’t X” rule with an explicit “DO Y instead” — otherwise refusal becomes the path of least resistance.
• The brief framing IS the prompt. Telling the model “the message may be empty if the operator said ‘go’” primed it to treat empty-ish messages as legitimately unspec-able. Words in the brief construction are part of the contract; choose them like rules.
• Training priors are stronger than persona rules. Three consecutive runs got “Personal Finance Manager” for an abstract dashboard brief because that’s the strongest “SaaS dashboard” association in qwen-family training data. Persona rules can soften the prior, can’t overcome it. The right path forward is bigger models OR code-level brief disambiguation, not more persona iteration.
• Celebrate the small wins or you lose them. Listing remaining bugs in tables after every success buries the signal. The operator named the pattern explicitly tonight; saving it as a feedback memory so future sessions don’t repeat the compound-bug-spiral.

Quote of the day

“sometimes you need to step back and enjoy the small wins vs looking at the componding bugs”

Operator-stated, tonight, after I’d just written a 12-row remaining-bugs table immediately following the polish_styling infrastructure win. The principle is right — and naming it as a principle makes the bias visible. Lead with the win. Name it concretely. Stop. The remaining bugs are obvious from the screenshot; cataloging them buries the architecture win in QA noise.

Carryover

• add_feature tool — Path C (functional revision: “add a delete button”, “wire up edit”) still goes through model-driven read_file + dispatch chaining, which is the same routing-reliability failure build_project and polish_styling fixed. Next sprint: ship add_feature with the same code-level pattern.
• @css-wizard structure-preservation discipline. The styling pass STILL occasionally drops rows from @front-end’s bones output (observed on dashboard rebuild after the @css-wizard pass: 3-row layout collapsed to 2 rows). The persona says “preserve structure verbatim, ADD classes, don’t reshape.” The model doesn’t fully comply. Fix likely involves screenshot eval Suite (catch row-drop programmatically) + persona reinforcement.
• Artifact-extractor fence-header leak. Top of every rendered output: literal \``html path=index.htmltext leaking into the page. Bug is inwerbos.tools._artifact`’s fence-detection logic — falls through to “write whole content” when there’s no closing fence. Small fix, separate from orchestration concerns.
• @css-wizard targeting precision. “Make the first column Metric 1 green” → Metric 2 turned blue. Model interprets revision requests approximately. Either persona reinforcement (CSS-selector specificity hints) or a stronger model. Screenshot eval would catch it.
• Screenshot eval Suite (B0 from yesterday’s Path B plan). All four remaining bugs above (structure preservation, fence leak, targeting precision, feature presence) become measurable with a screenshot grader. This is the highest-leverage follow-up — it turns “model behavior is fuzzy” into a real scoreboard.
• Pull a Day 20 vignette through to the README too when the PFM acronym + getpfm.io credibility markers + socket.dev triple 100 land in the launch-prep narrative.