Day 36 — 'build the eyes.'

“build the eyes.”

Weeks ago, when GPT flagged the four maturity levers, the pick was cross-layer observability — and the note was blunt: “stop adding capabilities until trace_id is plumbed end-to-end.” It sat. Then three straight weeks of debugging-by-archaeology — grepping .pfm archives and docker logs by hand for the latency hunt, the skill-rescue chain, the forgotten third daemon — made the case for itself. Build the eyes before adding anything else to see.

The finding that saved the most work

The reader was already built. glasshouse trace <id> already reads ~/werbos/traces/*.jsonl, groups by trace_id, and renders a timeline. werbos.tracing already had the contextvar id, the trace_turn scope, the JSONL store, the glasshouse-compatible schema. The whole lever was coverage + propagation on werbos’s side — not a greenfield build. So this was a focused fix, not a project.

Phase 1 — propagation (cbcad29)

Threaded trace_id through SessionRegistry + ChatSession.step / step_stream, each scoping its turn under trace_turn(trace_id) so every downstream emit joins ONE trace. The load-bearing subtlety: the streaming generator runs in Starlette’s response context, not the request handler’s — so binding the id only at the entrypoint left the streamed turn’s emits orphaned (trace_id=null). Binding inside the async generator fixes it. CLI/REPL turns get a minted id too.

Phase 2 — light up the agent loop (cbcad29)

chat.py had zero trace emits — a trace showed chat/turn + llm/completion but nothing about which tools fired. Added tool/dispatch (name + arg keys), tool/result (call_id + error), and skill/rescue when the hallucinated-call rescue triggers. Small payloads — the chain stays skimmable; full output stays in the .pfm. 5534 green across 4 new integration tests.

Phase 3 — the glue (ef989be)

The missing piece between “traces exist” and “I can debug this turn”: every chat response now carries an X-Werbos-Trace-Id header. Weird reply → grab the id from the header → glasshouse trace <id>. No more guessing via glasshouse list.

Co-locating the reader

To actually run glasshouse trace on the box, the reader has to reach the traces — which land at /data/traces owned root:root 0700 (a deliberate security choice; traces hold extracted constraints). Host user can’t read them, and there’s no passwordless sudo. So glasshouse got baked into the werbos container (same root context as the writer): docker exec … glasshouse trace <id>. The full loop, verified live on the homelab.

Dogfooding earned its keep on day one

First real trace I pulled: “whats 2+2” took 30 seconds and showed exactly one row — chat/turn. No llm/completion. The primary chat model call (pitts’ BellmanBackend) emits no trace row; only the skill path does. The instrument pointed straight at its own biggest blind spot — which is the whole point of building it first.

Numbers worth keeping

• Reader: already done. The lever was coverage, not a build.
• 5534 tests green (+4 integration)
• The loop: response header → trace_id → glasshouse trace → the chain, on the live box

AHAs

• Build the instrument, and it tells you where to invest. I planned to guess the next coverage gaps; instead the first trace I read named one (the untraced model call). Evidence beats speculation.
• The reader being done changed everything. The temptation was to build a whole observability system. The actual job was three small emits and a propagation fix.
• Security and usability fought, and security won correctly. The 0700-root traces couldn’t be loosened, so the reader moved to where the privilege already was — into the container — instead of weakening the writer.

Tomorrow

Trace the primary model call (at the werbos boundary around agent.run() — keep pitts werbos-agnostic). The eyes are open; now fill in the one thing they can’t yet see.