Day 23 — 'constraint reasoning is upstream of retrieval.'

“constraint reasoning is upstream of retrieval, not downstream.”

First day back from the four-day pause (Days 22-26 didn’t count per the calendar memory; Day 23 resumes today). External input named a gap the operator immediately recognized. One sitting later, the gap is filled — by a primitive the rest of the architecture now stands on.

How the day started

The operator pasted a long-form GPT analysis of a werbos chat trace. The trace: user asked “shore fishing in Cabo San Lucas”; the chat model called web_search("best beach fishing spots cabo san lucas"); retrieval brought back offshore sportfishing content (Gordo Banks, charters, marlin tournaments); the model post-hoc-corrected in prose “these are not necessarily beach fishing spots.”

GPT’s diagnosis, in one line:

“Constraint reasoning is downstream of retrieval, not upstream of it.”

The operator’s reaction:

“once i realized what GPT was talking about i realized we have a gap.”

This is the input-layer-gap insight from Day 22 morning, reapplied. We named the gap for vague-brief → structured-spec (the studio). GPT applied it to vague-user-message → structured-tool-call. Same architectural pattern; different surface.

What landed

werbos/intent_compiler.py (9d75e3a) — the meta-primitive.

• ToolConstraints dataclass with query, must_include, must_exclude, geography, modality, audience, timing, extras. The shape future tools serialize their input space against.
• compile_intent(user_message, tool_name, tool_description, model='') — one LLM call with a strict JSON output template + four worked examples (shore fishing, vegan dinner, crypto news, beginner trail runs) that force the LLM to do the inference step — “shore” implies exclude “offshore” / “charter” / “deep-sea” even though the user didn’t say it. That inference IS the value.
• filter_results(raw, constraints) — downstream bookend. Drops items hitting must_exclude; re-ranks must_include matches first. Handles dicts / dataclasses / to_dict() instances / repr fallback.
• resolve_intent_model(explicit='') — precedence: explicit kwarg > WERBOS_INTENT_MODEL env > user.md ## Intent compiler - model: > chat-session’s default model. Default is the chat model itself (cheap; ~5-15s overhead per tool call). Operator opts up to qwen2.5:14b when stricter parsing is worth +30-60s per call.
• 26 offline regression tests including the canonical shore-fishing trace assertion. JSON-parser robustness (fences, preamble/postamble, missing keys, type drift, invalid JSON), filter behavior (drop + rerank + dataclass handling), model-resolution precedence, graceful fallback when the LLM is unreachable or returns garbage. All pass.

Wired into the first three tools:

• web_search (also 9d75e3a) — _SEARCH_SCHEMA extended with optional user_message. Tool description tells the model to pass it for any query with constraints. Live test: “shore fishing in Cabo San Lucas” → compiled query "shore fishing Cabo San Lucas beach access surf casting public beach entry" + exclude [offshore, charter, deep-sea, sportfishing boat, marlin charter] → DDG returned shore-focused results: “Surf Fishing in Cabo, A Shore Thing” / “Cabo San Lucas Shore Fishing | InCabo Travel” / “Your 2026 Guide to Public Beach Access in Cabo San Lucas”. The canonical trace is fixed.

• fetch_top_news (1067455) — over-fetches 3x when constraints are present, filters by HN title. /news 5 AI news not about crypto → top 5 are all AI-adjacent (LLM training, AI coding, etc.), no crypto stories.

• fetch_stocks (1067455) — upstream merges UPPERCASE ticker-shaped must_include items into the symbol list (intent_compiler does symbol extraction from natural language). Downstream drops symbols whose ticker OR company name matches must_exclude. Live test: /stocks SPY QQQ NVDA "AI tickers, not the ETFs" → intent_compiler EXTRACTED C3.AI as an additional ticker (it’s literally an AI company). ETF post-filter is weak when company names don’t contain the category term — operator can be more explicit (not SPY not QQQ).

/news and /stocks slash handlers now heuristically detect natural-language vs structured args. Leading int on /news is still the count; the rest becomes the constraint phrase. All- uppercase short-alpha tokens on /stocks are still the symbol list; lowercase words trigger natural-language mode.

The architectural insight that hardened

Day 22 evening: “the skill library IS the capability surface.” Day 23 sharpens it:

For local models, the catalog is necessary but not sufficient. The operator’s intent has to compile correctly INTO the catalog or the size of the catalog doesn’t matter.

A 200-skill bundle + tool-use dispatch + curated breadth all degrade if the model picks the WRONG things from the catalog because the user’s constraints aren’t extracted upfront. The intent_compiler is the meta-primitive the rest of the architecture relies on for accuracy.

Saved as memory intent-compiler-for-tools-2026-05-25. The launch line that fell out:

“werbos doesn’t get more out of the model by making it smarter; it gets more out of the model by asking it better questions.”

That’s the intent_compiler in one sentence.

AHAs

• External feedback is signal when the operator immediately recognizes the gap. GPT’s analysis didn’t tell us anything the input-layer-gap memory hadn’t already named for the studio. But applying the same lens to a different surface (tool calls) was the non-obvious move that needed an outside voice to articulate. Worth saving as a meta-pattern: the outside voice often re-applies an insight the inside team has only used in one place.
• Few-shot prompting is the small-model superpower. The first version of the system prompt without worked examples produced conservative constraint extraction — got the obvious stuff, missed the inferential exclusions. Adding four explicit worked examples made llama3.1:8b do the inference step (“shore implies NOT offshore”) consistently. That gap-between- versions is what most “the model is dumb” complaints actually are — prompt design, not model capability.
• Graceful fallback is a feature, not a wart. When the LLM is unreachable / returns garbage / times out, compile_intent returns a ToolConstraints with query = user_message and empty constraint fields. Downstream code can proceed as if the compiler never ran — current behavior preserved. Adding intent_compiler made the pipeline strictly better, not more fragile.

Numbers worth keeping

• 2 commits (intent_compiler primitive + 3-tool wiring)
• 26 new offline regression tests (all pass)
• 3 tools intent-compiler-aware so far (web_search, fetch_top_news, fetch_stocks); each cost ~30 min to wire
• ~5-15s overhead per tool call (default llama3.1:8b compile); operator-configurable upgrade path to qwen2.5:14b
• 0 breaking changes — the user_message kwarg is optional on every wrapped tool; absent it, behavior is identical to before

Afternoon — intent_compiler into 3 more tools (ARCH.3 closure)

(57bf191) The 9am “wire it everywhere” list got executed:

• dispatch_skill — chat model emits dispatch_skill(name="weather", args="95814", user_message=…) and compile_intent enriches args upstream.
• polish — style-edit LLM sees EXTRACTED CONSTRAINTS: block in its user message.
• build_project — closes the ARCH.3 loop the Day 22 morning called out. The vague brief → structured spec gap gets the same upstream-compile treatment.

Four distinct tools now share one constraint-extraction contract. Same operator phrase, same constraints, every surface. The results compose. The audit trail (via the trace_id pipeline below) shows it.

Afternoon — +8 more prompt skills (Track C)

(de4ba27) /timer, /focus-30, /meditate, /tarot, /haiku, /quote, /password-gen, /unit-convert. Skill count: 59 user-facing skills. Two of these — /password-gen and /unit-convert — got reviewed harder by the security loops than any other skill in werbos. See late late night below.

Evening — trace_id primitive + glasshouse sibling

The Day 23 evening’s big architectural move: cross-layer observability becomes a first-class primitive (a42585c).

Operator framed it earlier: “frontier model debugging is opaque; with werbos you ask glasshouse for trace <id> and see the whole chain.” Five hours later that’s a real build.

New in werbos:

• werbos/tracing.py — start_trace() mints a UUIDv7 trace_id, binds it to a contextvars.ContextVar so every awaitable downstream inherits the binding without a kwarg. record_trace_event(layer, event, payload) appends one JSONL row to ~/werbos/traces/<YYYY-MM-DD>.jsonl. Best- effort — every call site is try/except Exception: pass so instrumentation NEVER takes down a turn.
• werbos/server/slash.py records slash/dispatch at the dispatcher boundary.
• werbos/tools/ollama_chat.py records llm/completion or llm/completion_failed with model + host + duration_ms + lengths.
• werbos/intent_compiler.py records intent_compiler/compile_intent with the extracted constraints. The trace shows the operator EXACTLY which constraints werbos derived from their input before any tool fired.

New sibling repo: glasshouse. Private GitHub repo (jasonsutter87/glasshouse). Pure stdlib reader for ~/werbos/traces/. Three subcommands:

• glasshouse trace <id> — markdown timeline render
• glasshouse list [--day] — recent traces, one-line summary
• glasshouse stats — counts per layer / per event / per day

Read-only. The writer (werbos.tracing) and the reader (glasshouse) speak a stable on-disk JSONL contract; either can be upgraded independently. The sibling family is now 8 long, with glasshouse being the diagnostic surface the rest of the family didn’t have.

pitts        the agent loop
mcculloch    the eval
partyparrot  the live wire
pfm          memory at rest
observe      the event layer
watchtower   alerts / triggers
warden       rule enforcement
beacon       signals / state
glasshouse   the trace inspector   ← NEW

Launch line:

“Frontier model debugging is opaque. With werbos you ask glasshouse for trace <id> and see the whole chain.”

Late night — security pipeline Loop 1

Operator invoked the full pipeline against the Day 23 NEW surface: Purple → Black → Red → Archaeologist.

• Purple Loop 1 — 12 findings. 2 HIGH (glasshouse OOM on pathological trace file; trace_id contextvar leaks across sequential turns in the same asyncio.Task), 4 MED, 4 LOW.
• Black Loop 1 — 12 more, 3 fresh HIGHs: glasshouse follows symlinks → arbitrary file read; trace JSONL written mode 0o644 world-readable; a bypass of Purple’s HIGH-P1 fix (get_trace loads full traces dir even for single-trace lookup — the per-line cap doesn’t save you if you have months of data).
• Red Loop 1 — 24/24 unique findings remediated across 72e8fe9 / 03e9f79 / 4661d80 (werbos) + 8b5f2eb / e150147 / 70a546f (glasshouse). werbos +80 tests (4438 → 4518 in target files). glasshouse +23 tests (20 → 43).
• Archaeologist Loop 1 — 15 findings on 25-50yr survival. Three must-fix landed inline: trace JSONL _schema header (glasshouse silently skips rows without trace_id, so the header is invisible to live tooling but legible to a 2050 archaeologist opening the file cold); inline RFC-9562 UUIDv7 fallback so observe.ids disappearing doesn’t kill the trace pipeline; worked ## Intent compiler example in resolve_intent_model docstring. Verdict: DURABLE.

Loop 1 total: 24 security findings + 3 durability fixes shipped across 6 commits in two repos.

Late late night — security pipeline Loop 2

The realization mid-evening:

the fix code is itself new attack surface. wrap_untrusted_text is a new envelope primitive; _mint_trace_id_inline is a new RFC-9562 implementation; _MODEL_NAME_RE is a new regex validator. None of these shipped through Purple/Black review before Loop 1 landed them.

So Loop 2 reviewed the patches that landed in response.

• Purple Loop 2 — 10 patch-quality findings. Headlines: /password-gen uses the LLM to MINT passphrases (HIGH — distribution collapse + training-set memorization; operator treats output as real credentials); UUIDv7 inline minter non-monotonic within a millisecond (violates RFC 9562 §6.2 and defeats glasshouse’s 8-hex-prefix promise).
• Black Loop 2 — 8 more, including the highest-impact Loop-2 finding: _scrub_catalog_text strips Cc but NOT Cf (HIGH — BIDI override, zero-width space, U+2028 line-separator all survive the catalog scrubber and land in the model’s tool description). The Red Loop 1 patch that INTRODUCED _scrub_catalog_text failed to delegate to the canonical strip_format_codepoints primitive that already handles this correctly in werbos/session/untrusted.py.
• Red Loop 2 — 14 unique findings remediated across 795e063 / cbcf30a / 9d3bedf (werbos) + e310d41 / 6c89ea3 (glasshouse). ~534 LOC net. werbos +49 tests (4509 passing), glasshouse +7 tests (50 passing).
• Archaeologist Loop 2 — 8 findings, 3 SIGNIFICANT. All three landed inline: eff_large_wordlist.LICENSE.txt sibling file with EFF / CC-BY 3.0 / SHA-256 / 7776-word attribution; tightened wordlist corruption check from < 1000 to 7776 ± 32 (a 1500-word truncation would have PASSED the old check and silently dropped entropy from 77.5 → 63 bits); externalized ticker blocklist to werbos/util/ticker_blocklist.txt with operator override at ~/werbos/ticker_blocklist.txt. Plus the GOOD-finding’s 1-LOC future-proof: strip_format_codepoints now drops any general-category starting with C except Cn, so Co (private-use codepoints — a smuggling vector) and Cs (surrogates) are stripped without an explicit add. Verdict: DURABLE.

Loop 2 total: 22 findings remediated across 5 commits in both repos.

The Loop-2 fixes worth keeping in mind

• /password-gen no longer asks the LLM to mint passphrases. New werbos/util/passphrase.py uses secrets.choice() over a bundled 7776-word EFF “large” diceware wordlist. Each phrase reports actual entropy in bits; the slash handler bypasses ollama_chat entirely. The “LLM as RNG for security-critical output” anti-pattern is now physically impossible for this skill.
• UUIDv7 inline minter now RFC 9562 §6.2 Method 1 compliant. 12-bit monotonic counter in rand_a keyed on (last_ts_ms, counter), thread-safe via threading.Lock. Same-ms minted IDs sort in mint order. glasshouse’s 8-hex-prefix lookup promise (in the README) is now actually true.
• /unit-convert refuses medication-dose patterns. Operator asks for 50 mg/kg insulin to mL and gets a “consult a pharmacist” refusal instead of a hallucinated number. Software-as-medical-device territory; the refusal is the right posture before EU/UK launch.
• slash dispatcher uses trace_turn context manager. The start_trace + finally clear_trace pattern broke nested scheduler → slash trace handoffs. clear_trace is now DEPRECATED.

The meta-pattern that closed Day 23

A two-loop pipeline turned into a default. Every fix that ships in response to a Purple/Black review introduces NEW code that those reviews never saw. Loop 2 isn’t paranoia — it’s how the fix surface gets reviewed at all. And the second loop found two HIGH findings (/password-gen LLM minting + _scrub_catalog_text Cf-blindness) that Loop 1 couldn’t have caught because the relevant code didn’t exist yet.

Saved as future operating procedure: any security pipeline that ships ≥100 LOC of fix code earns a second pipeline against the fixes themselves. Cost: ~3 hours of agent time. Yield: two HIGH findings the first pass missed by definition.

Quote of the day (late update)

“the fix is the new attack surface.”

The realization mid-Loop-2 when Black flagged _scrub_catalog_text as a HIGH. The function Red INTRODUCED to fix Purple Loop 1’s P7 had a hole bigger than the original finding. The Loop-2 pipeline paid for itself in one finding.

Updated numbers worth keeping

• 22 commits today (3 feat + 6 Loop 1 + 5 Loop 2 + 8 earlier intent_compiler core)
• 2 security loops of Purple → Black → Red → Archaeologist
• 46 unique findings remediated (24 Loop 1 + 22 Loop 2)
• +158 tests across both repos (werbos 4438 → 4509 passing in target files; glasshouse 20 → 50 passing)
• 8th sibling repo shipped (glasshouse v0.1.0)
• 59 user-facing skills + 4 intent-compiler-aware tool surfaces
• 2 Archaeologist verdicts: DURABLE (both loops). All must-fixes landed inline.
• 0 Critical / 0 High security findings outstanding

What’s next (Day 24+)

• Real-LLM smoke from the live chat surface — operator validates the shore-fishing trace + the new tracing pipeline from the dashboard, not just command-line smoke.
• Glasshouse v0.1 → v0.2 — .pfm chat archive join so operator-facing user+assistant turns appear in the timeline next to per-layer trace rows.
• /commit upgrade still queued (Phase B from the May-21 asks).
• Distribution rails (NX.1 + A.5 day 5) — still deferred, still the v1 ship gate. Two security loops have bought enough confidence that the public-install story is closer than it was 24 hours ago.

Loop 3 — final scored Archaeologist verdict: TIMELESS @ 97/100

After the two security loops closed, operator asked for a numeric score. The Archaeologist’s prior verdicts were textual; the path to TIMELESS was named but not walked.

8 outstanding Archaeologist findings landed inline (6950f9c werbos + 6f1a155 glasshouse):

• L1-F2 paths.py cross-repo invariant docstring
• L1-F4 intent_compiler model-coupling block comment + new compile_intent_smoke_test()
• L1-F6 safe_record(layer, event, payload) helper centralizing the try/except Exception: pass pattern
• L1-F7 Row.ts_dt always returns aware-UTC datetime
• L1-F9 _iter_jsonl_paths globs the date-shape pattern
• L2-F4 clear_trace emits DeprecationWarning at call time
• L2-F6 wrap_untrusted_text source escape now lossless backslash
• L2-F7 _NAME_RE security/style comment split

Plus the GOOD-finding 1-LOC future-proof: strip_format_codepoints now drops ANY C-prefix category except Cn (so private-use codepoints Co and surrogates Cs are stripped without explicit listing). 16 new tests verify each fix.

The Archaeologist re-scored

DURABLE → TIMELESS. Score: 97/100.

The 3 missing points:

• R1 (-2pts) intent_compiler system prompt is a load-bearing 110-line English program — unavoidable LLM-as-parser cost
• R2 (-1pt) emoji glyphs in _LAYER_GLYPH are font-dependent in 2050 — presentation choice
• R3 (~-0.5pt rounded in) _NEWLINE_RE / _LINEBREAK_RE regex duplicated within one repo

“The codebase has crossed the threshold where its OWN durability primitives (schema header, safe_record, smoke test, deprecation warning, lossless escape) are now load-bearing for the durability story. That is the mark of TIMELESS code: the code knows what year it is and tells you.”

Full Loop 3 review at security-review/2026-05-27-archaeologist-d23-l3-final-v1.md.

The full Day 23 closure

Three pipelines closed today:

46 security findings + 19 durability findings remediated. +165 tests. 22 commits. Zero Critical / Zero High outstanding.

Day 23 closes with the input layer filled, the observability layer shipped, the security posture re-validated under three adversarial pipelines, AND the 25-50yr-survival score on the record. werbos is the boringest it has ever been — which, for an AI tool in 2026, is the highest praise available.