werbos

Vision

How werbos sees images. /see slash captures a screenshot; --vision-model wires a vision-capable backend; the description gets injected as a SystemMessage so the text+tools model can reason over it.

werbos can take an image and let the model describe it back. Two moving parts:

  • Capture — get bytes onto disk + into the next turn. The /see slash + werbos.vision.capture package do this.
  • Describe — run the bytes through a vision-capable model and inject the description as a SystemMessage. The VisionContextProvider does this; you wire it via --vision-model.

You can use either piece without the other. /see without --vision-model still saves the image and attaches it to the next turn; the text-only model just can’t see it. --vision-model without /see works fine when the SPA’s image-upload composer sends bytes directly.

Capture — the /see slash

Inside the chat REPL:

werbos> /see                  # interactive region select (drag)
werbos> /see region           # same as above
werbos> /see full             # silent full-screen capture (all displays)
werbos> /see window           # click-to-pick window
werbos> /see clear            # drop any pending images without sending

What happens:

  1. macOS screencapture runs with mode-specific flags (-i -x region, -x full, -W -x window).
  2. The PNG lands at ~/.werbos/captures/cap-<mode>-<ts>-<rand>.png.
  3. The bytes are base64-encoded and attached to the live ChatSession’s pending-images buffer.
  4. No turn is sent. The next message you type becomes the prompt that consumes the image.

ESC during interactive capture cancels cleanly (zero-byte file detected and removed; CaptureCancelled raised).

werbos> /see
captured: /Users/you/.werbos/captures/cap-region-20260509-193045-a1b2c3.png (84 KB, region)
  describe in your next message; type /see clear to drop it

you> what package is on the table?
werbos> [reads the image, replies]

If --vision-model isn’t set, the slash adds a hint:

captured: /Users/you/.werbos/captures/cap-full-20260509-193117-d4e5f6.png (412 KB, full)
  (no --vision-model configured; image will attach but the text model can't see it)
  describe in your next message; type /see clear to drop it

The image still attaches and ships on the next UserMessage — the text-only base model silently ignores image bytes per UserMessage.images’s pre-existing contract. You’re not losing the file; you’re just not getting a description from it.

Multiple images per turn

Each /see appends to the buffer. Send multiple captures in a single turn:

werbos> /see           # captures image A
werbos> /see           # captures image B
you> compare these two

The buffer drains in order onto the next UserMessage’s images tuple. The buffer survives until the next AI turn fires — not until the next text turn. Slash commands don’t drain it.

Describe — VisionContextProvider

The description side wraps a separate vision-capable Backend (typically BellmanBackend pointed at a vision model) and runs a one-shot inference. The result is injected as a SystemMessage in the conversation history; the text+tools model then runs the user-facing turn against that augmented history.

werbos chat \
  --backend ollama \
  --model llama3.1:8b \
  --vision-model llama3.2-vision:11b

# vision model on a different host (e.g. SR8):
werbos chat \
  --backend ollama \
  --model llama3.1:8b \
  --vision-model llama3.2-vision:11b \
  --vision-host http://192.168.1.50:11434

The same flags work on werbos serve so the SPA’s image upload composer routes through the same path:

werbos serve --vision-model llama3.2-vision:11b

Why a separate provider, not a Backend swap?

The originally-proposed approach was per-turn model swap inside BellmanBackend. The actual shipped design wraps a separate backend in VisionContextProvider. Three reasons:

  • Tools still work on image turns. Vision-capable Ollama models reject the tools field; if you swapped the model wholesale for image turns, you’d lose tool use. The provider runs vision as a sidecar and returns text; the main model keeps its tools.
  • Cache by image hash. ChatSession holds a per-session memo on sha256(images || user_text) so re-asking about the same image doesn’t re-call the vision model.
  • Local-first as DATA, not compute. The pixels live on your box; only the description travels to whichever inference backend fits. The vision model and the text model can run on different hosts.

The default vision prompt

You are a vision module describing an image for another AI assistant.
Be precise, factual, and concise (1-3 sentences). Describe what is
visible. Do not speculate, do not interpret intent, do not answer
the user's question. The downstream model will handle that with the
description you provide.

Override with --vision-prompt "..." if you want different framing.

Phase events

When a vision interlude runs, ChatSession emits PhaseEvents that the SPA renders inline:

  • vision — “Looking at image…” while the description runs
  • vision_failed — if the vision backend errors (90s timeout, OOM, model not found); the user-facing turn proceeds without vision context (degraded but not broken)
  • thinking — the regular agent loop streams; the SPA hides the vision label as soon as the first token of the user-facing reply arrives

The CLI’s streaming wrapper renders the same events as [Looking at image...] lines.

Audit shape

Every vision interlude appends a KIND_VISION_OBSERVATION event to the chat’s PFM recorder chain with actor ACTOR_VISION and a payload containing the image hash + the resulting description. The description survives across turns via a post-agent history filter that preserves SystemMessages prefixed with the _VISION_CONTEXT_PREFIX marker.

This means a multi-turn coherence path: image landed turn 3, user asked a follow-up turn 5, the vision context from turn 3 is still in scope. No re-describe per turn.

Platform support

macOS-only at v1. The capture function raises CaptureUnsupportedError on non-Darwin platforms with a message naming Linux/Windows backends as tracked but not shipped. The public surface (capture_screen, the dataclass, the error types) is OS-agnostic so a Linux gnome-screenshot or grim implementation can land later without changing callers.

Pair with the html persona

werbos chat \
  --persona-name html \
  --vision-model llama3.2-vision:11b

Take a screenshot of a page you want to imitate; describe what you want changed; werbos generates the corresponding single-file HTML informed by the visual context. The HTML persona’s style discipline

  • the vision interlude’s description = a sharp single-file deliverable from a screenshot prompt.

See also