Day 5 — 'I think that is a glass of beer.'

“i think that is a glass of beer”

That sentence — generated by llama3.1:8b on SR8, after moondream described a phone-camera photo, with the description injected as a SystemMessage that the agent reasoned over — closed the multimodal loop on local hardware. Photo on phone → vision module → text model with vision context → streamed reply on phone. End-to-end, on your hardware, with your weights, from a single photo of a beer in a “burning barrel” glass.

The sentence is small, slightly funny, and the actual artifact of the day. Everything else built up to it or fell out of it.

Morning — clearing the post-Day-4 punch list

Day 4 night left several small bugs visible. Day 5 morning closed each:

• Mobile chat scroll (3894fc6). The 100dvh + overflow- wrap: anywhere fix from Day 4 night had landed but mobile chats still wouldn’t scroll. Diagnostic plan in BACKLOG section 7 said “step 2: suspect the flex chain above .chat-scroll is losing min-height: 0 somewhere.” Exactly that. Three CSS lines: min-height: 0, height: 100%, overflow: hidden on .main, plus grid-template-rows: 100dvh on .app. Mobile scroll works.
• End-chat doesn’t persist in history (871598b). CLI’s archive_dir defaulted to None when --archive-dir wasn’t passed; the listing endpoint then skipped the archive directory entirely. Meanwhile ChatSession was happily writing to ~/werbos/chats/. Read and write defaults were silently mismatched. Default both to ~/werbos/chats/.
• New logo (c3b74ed). Sage-green graph-W (a small nod to Paul Werbos, who invented backprop in 1974). Tauri’s npx tauri icon regenerates every platform-specific size from one master; HUGO site picks up the same favicon. Placeholder artwork; the master at werbos/ui/src-tauri/icons/source-1024.png makes future regenerations a one-line command.
• React-key warning + console 404 noise (69b47b0, 07a370a, c77ddb8). Dedup the voice list (macOS exposes multiple “Samantha” entries with the same voiceURI); make the runs-router fall back to active sessions when an in-flight chat hasn’t archived yet (so getRun() doesn’t 404 on every fresh chat); add PUT + PATCH to the CORS allow_methods so labeling endpoints work from localhost:5173 cross-origin.

Afternoon — Sprint A: vision routing, three phases

Per BACKLOG section 3, “vision-as-internal-context” was the locked design: vision model produces a description, ChatSession injects it as a SystemMessage, the text+tools model owns the user-facing turn. User sees one stream of tokens; the vision pass is invisible.

• Phase A.1 — VisionContextProvider + CLI flags (d8e71ed). New werbos/backends/vision.py. Wraps any pitts.Backend whose model accepts images on a UserMessage. One method, describe(). Three new CLI flags on werbos serve: --vision-model, --vision-host, --vision-prompt. Default off when blank. 10 unit tests pin the contract.
• Phase A.2 — ChatSession vision interlude + memo cache (8107283). step and step_stream run the interlude between human turn and AI turn when (a) latest UserMessage carries images and (b) vision_provider is configured. Per-session memo by sha256(sorted-image-b64s + user_text) skips re-running vision for repeat attachments. New KIND_VISION_OBSERVATION recorder event with new ACTOR_VISION (intentionally not ACTOR_SYSTEM — the SFT exporter needs to filter vision events independently). 9 new tests.
• Phase A.3 — Flip the paperclip back on, rebuild APK (7077a18). IMAGE_UPLOAD_ENABLED = true in ChatView.tsx, rebuild Tauri Android, extract APK via bundletool, copy to the Syncthing drop-off path. SR8’s start-werbos.sh updated to include --vision-model llama3.2-vision:11b (later swapped to moondream per the speed story below).

90 minutes of focused work, three commits, working feature on the phone.

Evening — the long sequence of vision bugs

Then four hours of “the demo doesn’t work; here’s why” iterating through layered failure modes:

• Hardware reality check. llama3.2-vision:11b on SR8 (Ryzen 7, no GPU offload) is genuinely slow on CPU — first phone test ran 160s with no output. ollama ps showed both models loaded at 100% CPU. Switched to moondream (~2GB, designed for fast captioning); inference dropped to ~10s. Vision-grade quality vs caption-grade — for “describe this so the text model can answer about it” the trade is fine.
• Phase events + 90s timeout + UI placeholder swap (ad99227). The vision interlude was invisible during the call: the bubble showed “Thinking…” for the entire 30-120s with no signal. Added PhaseEvent("vision", "Looking at the image...") at the start of the interlude and PhaseEvent("thinking") after, plus a hard asyncio.wait_for 90s timeout that emits PhaseEvent("vision_failed", <reason>) and proceeds text-only on overrun. UI swaps the placeholder text from the rotating loading line to the phase message. Failures are now visible even when they happen.
• Directive SystemMessage (5ea587c). First successful vision call with moondream on the phone still produced an empty AI reply. Diagnosis from the .pfm chain: vision succeeded (correct beer description) but the AI turn recorded AI_TURN_START → AI_TURN_END with NO chat content. The SystemMessage shape was meta-flavor ((Vision module saw, attached to user's last image: ...)) — small instruct-tuned models read parenthetical asides as background and don’t act on them. Rephrase as directive: [VISION CONTEXT] The user has attached an image. Vision module describes it as: <X>. Use this description to answer the user's question.
• Message ordering (4199383). Empty AI replies persisted. Live repro against SR8’s Ollama isolated the issue: the chat template (Llama 3.1) breaks when a SystemMessage comes AFTER a UserMessage in the history. Direct curl with system → user → system produced eval_count: 1 and empty content. Same curl with system → system → user produced a coherent reply. _run_vision_interlude was appending the SystemMessage AFTER the user’s UserMessage; fix: insert it BEFORE. One index change in the history-mutation logic.
• Tool-internals leak (88d8d85). Vision now worked; AI produced a real reply. But the model also tried to use run_command to read EXIF from a fake /path/to/image.jpg it imagined existed. The shell tool returned 127 (“identify: not found”) and the model narrated the failure to the user. Right answer, embarrassing narration. Extended the [VISION CONTEXT] SystemMessage with explicit guidance: “The image is NOT available to you as a file or URL. Do NOT attempt to read or inspect the image with run_command, web_fetch, or any other tool — those will fail.”

After the fourth fix landed, the phone test produced “i think that is a glass of beer” on the next try. The dream stopped being theoretical.

Late evening — the voice loop

After vision worked, the next demo was voice: dictate a question into the composer, send to werbos, hear the reply read aloud.

• Mac browser test in Brave: failed. Mic recorded for ~250ms then dropped silently. No console errors, no failed network calls, just a fading red indicator in the tab bar. The hook’s no-speech filter (intended to hide harmless “user said nothing” events) was masking the actual cause: Brave Shields blocks the Web Speech API by default because Chromium’s implementation routes audio to Google’s cloud STT servers.
• Chrome A/B confirmed it. Same Mac, same Vite dev server, same hardware — Chrome’s mic worked end-to-end. Voice input, werbos’s text reply, TTS reading the reply aloud, full loop. Brave Shields was the entire problem.
• Silent-block detection (33daa8c). Made the failure visible. Hook now tracks whether any result event fires during the session; if onend fires within 1s of onstart with no result, populate stt.error with a clear actionable message naming Brave Shields specifically and listing alternatives (Chrome, Safari, disable Shields). ChatView now also RENDERS stt.error in the composer area — previously the error was being set but never displayed to the user, so every failure looked invisible.
• Phone APK still shows “Speech recognition not supported” because Tauri Android WebView genuinely doesn’t expose the Web Speech API. That’s a known WebView limitation, not something werbos shipped wrong; the real fix is a Tauri plugin wrapping native Android SpeechRecognizer. Out of tonight’s scope. Now at least the failure is visible to the user instead of a silent broken mic icon.

End state: voice → text → werbos → voice working end-to-end in Chrome on Mac. Two multimodal pipelines closed in one day.

On hardware honesty

“in 2026 we are just used to INSTANT”

Local-first vision on a CPU-bound box is not instant. moondream takes 8-15 seconds to caption a phone photo. The text model takes another 5-10. That’s a 13-25 second turn for image queries on this hardware tier. The dream isn’t bad, it’s unevenly good: text chat is real-time; tool calls are sub- second; vision is a deliberately-slow lane because that’s where the hardware ceiling actually bites.

Three real paths forward whenever the slowness becomes unacceptable: hardware refresh (used 3090 ≈ $700 turns vision 60s → 2s), Vulkan backend in Ollama (mixed iGPU support, worth a 30-min experiment), or just hide the paperclip again with one flag flip and revisit when the hardware story improves. None of them tonight.

Backlog growth

A long evening of design conversations grew the backlog considerably — continuing to add more ideas that will shape how users control and use werbos. Captured but not detailed here; the BACKLOG file in the repo carries the specifics.

Numbers worth keeping

AHAs

• The 100dvh/overflow chain has many ways to break. Mobile scroll required min-height: 0 AND height: 100% AND overflow: hidden on .main AND grid-template-rows: 100dvh on .app. Any one missing and the scroll silently fails. Captured in BACKLOG section 7 with a 4-step diagnostic plan before the fix landed; the plan accurately predicted step 2 was the issue.
• Read and write defaults must agree. End-chat-doesn’t-show was a one-line fix at the read site (default the archive directory). The bug was invisible because both halves worked in isolation; it only surfaced when the user expected closed chats to appear.
• The vision interlude has four distinct failure modes. Hardware-too-slow, no-UI-feedback, model-doesn’t-read-the- context, model-tries-to-use-tools-on-the-image. Each one required a different fix at a different layer. None were predictable from the design doc; all were obvious in hindsight. The .pfm chain was the indispensable diagnostic tool — recorder events made each turn’s actual behavior inspectable post-hoc.
• System message ordering matters in chat templates. Llama 3.1’s chat template breaks on user → system ordering and produces 1 EOS token. This is not documented anywhere obvious; the bug was found by curl-A/B-testing the same payload with system before vs after the user message. Pin the lesson: when injecting context via SystemMessage, ALWAYS insert before the related UserMessage, never after.
• Caption-grade vision is plenty for “use this in the next agent turn.” moondream’s “the image features a glass of beer with ‘burning barrels’ written on it” is enough for llama3.1:8b to give a coherent answer. The vision model doesn’t need to be the smartest one; it needs to be the fastest reliable describer. Picking a vision model is a speed/quality tradeoff that should default to speed once quality clears a threshold.
• The dogfood loop’s bad label catches narration bugs, not just factual bugs. The “model leaked tool-call internals to user” turn was technically correct but unprofessional. Labeling it bad is exactly the right signal even though no fact was wrong. Pattern failures in voice/style are first-class targets, not also-rans.

Quote of the day

“i think that is a glass of beer”

Said by werbos to Jason, on Jason’s phone, on Jason’s hardware, about Jason’s beer photo. The smallest possible sentence that proves the multimodal pipeline works end-to-end. Everything that took multiple commits to fix made this one sentence possible.

What’s next

• Sprint B candidate: per-move labels in TTT GameView (per BACKLOG section 5.1.2) — extends the dogfood loop’s signal shape from chat preferences to game-move correctness, which has an objective oracle (minimax) for cross-validation.
• The ingestion pipeline for the personal corpus (BACKLOG section 5.1.4) is plausibly the highest-leverage infrastructure to build before any v6+ training run.
• werbos init + .werbos.md (BACKLOG section 9.1) is the smallest-possible memory feature — single sprint, immediate value in any project directory.
• The mirror slider (BACKLOG section 10.7) is testable as an MVP without the full constitution+character infrastructure — even just blending two persona files at a ratio is a working demo of the idea.