werbos

← All log entries

Day 28 — 'the trainer is the small piece.'

Day-27 morning shipped the train-daemon orchestration without a real trainer. Day-28 evening built both ENDS of the training pipeline — DATA.1 (corpus ingest, the front door) and SE.9 (eval suite + noise floor, the promotion gate). The B2B reframe locked: werbos is one engine, two entry points (personal / business), same install, same pipeline, setup decides which. Also shipped: NX.1 npm shim, Watchtower-D27 security closeout, SE.6.1 parsetongue path routing, glasshouse MCP server (GH.1 stdio + GH.2 remote HTTP, Purple→Red reviewed), SE.8 keepwarm daemon. Twelve sprints across two repos in one day. The trainer is now the only piece of the loop that's a stub.

“the trainer is the small piece in the middle now.”

Day 27 closed at 1 AM with the orchestration shipping before the trainer — six SE sprints, a stub Modelfile-copy in the trainer slot, and a 2-hour systemd timer armed for overnight cycles. The diagnosis was honest: skill-smoke is monitoring data, not training data; SE.7.0.1 needs to fix the dedup; SE.7.1 needs a real QLoRA trainer; the daemon was working but spinning on an empty plate.

Day 28 was the answer to “what gets built around the trainer while the trainer cooks.” Twelve sprints. Two repos. One framing shift that explains why the whole thing matters.

Morning: distribution rails + security closeout

NPM shim (a1baf7a) shipped as a thin uvx-based wrapper — npm install -g werbos resolves to a Python wheel and chains through to the canonical CLI. Two-line install for engineers who live in Node-land. Pre-launch ready; the actual npm publish waits on the org move.

Watchtower-D27 sweep (2fc5179) closed the security findings that were blocking NX.1’s prereqs. Shell-injection-safe subprocess invocation in the train daemon (shlex.split + per-arg replace), path-traversal in autofix, file-permissions on cache writes, format-string DoS on a log line. Five fixes, no remaining HIGH or MED for the surfaces touched.

Sprint docs landed (b5c9540, ba3675a, 4092395, 6a76b27): CR.1 (code reconnaissance), SE.6.1 (parsetongue path routing) — both queued post-launch. The security-review/ directory got a consistent date-team-pass-vN naming convention applied to the 18 existing session reports.

Afternoon: parsetongue learns paths

SE.6.1 (361d02c, f5a28fd) — Claude-style intent extraction from operator input that includes a filesystem path. Before: "refactor /Users/.../partyparrot/README.md" hit the LLM slow path. After: _try_path_verb_route() sits between the slash fast-path and the LLM, extracts paths via three regex families (absolute Unix, ~/ rel, ./ rel), matches against 9 locked verbs (5 read-only + 4 destructive), routes to a tool handler with confidence = 0.70 + bias.

137 parsetongue tests passing. Five handlers shipped: 4 stubs pointing at CR.1 / CR.1.1 / CR.1.2 / PA.1, plus open_path as the real one (because reading a file is non-destructive and already trivially correct).

Evening: Claude consumes traces

The /flow/ page got Mermaid diagrams. The operator looked at the “debug walk” section and asked: “is this how you trace things now?” Honest answer: no — when something goes sideways, ad-hoc ssh + grep over ~/werbos/traces/*.jsonl. The operator’s correction: “it should be via glasshouse.”

That was the prompt for GH.1 (baba9c6 in glasshouse). Stdio MCP server face for the existing glasshouse Python API. Four tools: get_trace, list_recent_traces, trace_stats, search_traces. Optional [mcp] extra preserves the stdlib-only base install. 94 tests after, 40 new for the MCP layer. Operator adds glasshouse to ~/.claude/mcp.json once; from then on every Claude Code session sees mcp__glasshouse__* tools on the list.

GH.2 (19c9f4b in glasshouse) — the same server over Streamable HTTP for remote consumption. Closes the SR8 trace gap (Claude on Mac can query SR8 traces in real time instead of scp-ing JSONLs down). 36 new tests. Security defaults locked: default bind 127.0.0.1, non-loopback bind refuses to start without GLASSHOUSE_MCP_TOKEN env var or --insecure, hmac.compare_digest on the bearer compare, CORS wildcard rejected at the CLI boundary, no native TLS (caddy in front).

Then the multi-agent pattern: Purple-Team audited GH.1 + GH.2 end-to-end. Verdict: SHIP, with three MEDIUMs (F-1 search-traces DoS budget, F-2 OPTIONS preflight bypass, F-3 session idle timeout). F-3 applied immediately as a one-liner (949b5a1). Red-Team (Crimson) remediated F-1 + F-2 + F-4 + F-7 + a token-entropy floor in one combined commit (3c588a7). 154 tests after Red’s pass, all green. Merged to glasshouse main.

Late evening: keep the model loaded

SE.8 (9f718d3) — keepwarm daemon. Ollama unloads models after 5 min idle; the cold reload of qwen2.5:14b is ~5-15s, and that’s what the operator feels as “first chat turn is slow.” The daemon pings each SE.4 router tier model every 60s with {"keep_alive": "10m"} so the timer never expires. Universal — runs on Mac, SR8, anywhere werbos is hosted, against whatever WERBOS_OLLAMA_HOST resolves to. CLI: werbos keepwarm (foreground) or werbos keepwarm --once (systemd-timer mode). 19 new tests.

Night: the front door

DATA.1 — corpus ingestion primitive — shipped all four days in one session.

Day 1 (6aafa73) — schema, markdown reader, heuristic extractor (heading→body Q&A), JsonlSink with idempotent append/dedup, secret filter (6 patterns). 48 tests. Live smoke against the operator’s own docs/: 57 files → 852 training rows, secret filter caught 2 real leaks.

Day 2 (7992854) — code reader (Python docstrings + JSDoc + Go // comments), plaintext reader (paragraph chunker), werbos data ingest|stats|sample CLI. Live smoke against the werbos codebase: 200 files → 694 training rows. Sample: "What is def build_backend(args) for?" → the actual docstring.

Day 3 (befa700) — LLM Q&A extractor (local Ollama generates Q&A pairs per chunk, cached by chunk_id+model in a sidecar JSON for re-ingest idempotency), 2 hardened secret patterns (operator_path covering ~/.ssh/* / ~/Library/Keychains/* / ~/.gnupg/*, private_key_pem catching -----BEGIN PRIVATE KEY----- headers).

Day 4 (0508638) — docs/data-pipeline.md reference page. The doc a customer reads to figure out what to point werbos at.

97 DATA.1 tests total, all green.

The reframe

Mid-Day-28 the operator typed: “the goal: for a company to feed its corpus of data and to create a custom model on that needs of the company trained on the company data.”

That’s the actual product. werbos is a custom-model factory that produces either a personal AI or a company AI depending on the setup, corpus, and base model. Same install, same pipeline, same primitives. Personal-AI is NOT the wedge — both are first-class. The setup decides which the customer gets.

Two-line video script:

werbos data ingest <corpus>    # ← shipped today
werbos train --corpus X        # ← SE.7.1 (next)
werbos serve --model Y         # ← already exists

Same five lines. Different corpus. Different model. Same engine.

The promotion gate gets a definition

SE.9 (5654354) — per-corpus eval suite. werbos data split deterministically partitions the corpus into train + eval JSONLs using sha256(row_id) as the partition key — same corpus, same fraction, same split, across machines / runs / clones. No --seed flag. werbos data score runs a model against the eval set; --runs 5 produces a NoiseProfile with mean / std / min / max and the 2.5σ promotion threshold that defines “is this candidate REALLY better than baseline, or just noisy?”

Token-F1 scorer in v1 (precision/recall over normalized word multisets — standard QA metric, no LLM dep, well-understood). LLM-judge deferred to v1.1 when paraphrase tolerance becomes load-bearing. 37 new tests.

The first ratchet click is now mathematically defined:

candidate.composite > baseline.mean + 2.5 × baseline.std  ⟹  promote

That’s the bar SE.7.3’s daemon should respect once SE.7.1 ships a real trainer.

Numbers worth keeping

  • 12 sprints / 2 repos / 1 day — NX.1, security closeout, CR.1 + SE.6.1 plans, SE.6.1, GH.1, GH.2 + Purple + Red, SE.8, DATA.1 × 4, SE.9.
  • ~285 new tests today across werbos + glasshouse, all green.
  • 852 training rows from the operator’s own docs/. 694 from werbos’s Python code. Two real-shaped corpora, ten minutes.
  • 154 glasshouse tests after Red-Team’s pass. Down from HIGH/MED findings to zero pre-deploy blockers.
  • 0 → 0 training composite delta. Daemon still spinning on the 11-row corpus until SE.7.0.1 fixes the dedup. Expected.

AHAs

  • “werbos can be business first or personal AI depending on the setup and model” — the reframe that explains the whole product. One engine, two entry points.
  • “the trainer is the small piece in the middle now.” Yesterday the orchestration shipped before the trainer. Today both ENDS of the pipeline (input + eval) shipped before the trainer. SE.7.1 is the only stubbed piece in the loop.
  • The multi-agent pattern works at speed. Purple audited, Red fixed, both in the same session, no manual triage. The pattern scales to any new sibling — same loop, different surface.

Tomorrow

The critical path to the first ratchet click is concrete now:

  1. SE.7.0.1 — fix skill-smoke dedup so the corpus grows
  2. SE.7.1 — real QLoRA trainer (HF PEFT on SR8 AMD CPU)
  3. Run the daemon for ~2 weeks
  4. First promotion → werbos-jason-0.0.1 (or werbos-acme-0.0.1, same engine, different corpus)
  5. The eval-adapter’s baseline rotates to the new model
  6. Every subsequent candidate has to beat THAT
  7. Ratchet engaged.

werbos.ai domain + npm publish + org glow-up still queued; those move when SE.7.1 unblocks the demo video.

The trainer is the small piece in the middle now.

“the trainer is the small piece in the middle now.”