Day 54 — 'it's not the model. it's the box.'
A latency hunt on the web chat path, and an honest answer nobody wants: the slowness wasn't the model, it was the machine. On the CPU-only SR8, a turn's cost is dominated by prompt PREFILL (~78 tok/s), and two config defaults made it far worse than a phone — an 8B model that timed out, and a doubled KV cache. Dropping to a 3B at 4096 context with keep-warm makes the box usable again (270-tok prompt = 7s, 2730 = 36s), but the real levers are hardware: the 780M iGPU sits idle (ROCm not offloading) and the prompt itself is ~2700 tokens. Measured, not guessed.
“it’s not the model. it’s the box.”
It started with a good complaint: the chat on my phone is ~80× faster than the one on my computer. That shouldn’t be true. The phone runs a smaller model, sure — but 80×? So we measured instead of guessed, and the answer was the kind you don’t get to argue with: the slowness was never the model. It was the machine underneath it.
The prefill wall
On the CPU-only SR8, a chat turn’s cost is almost entirely prompt prefill — the model reading your prompt before it writes a single token — and on this box that runs about 78 tokens/second. A real turn carries a ~2700-token prompt (persona + recent history). Do the arithmetic: that’s ~35 seconds before the reply starts. No amount of prompt-engineering the reply makes that faster; the model is busy reading.
Two defaults were quietly making it worse than it had to be:
- The 8B model was timing out.
servedefaulted tollama3.1:8b, which on a real prompt takes over two minutes on this box — long enough to trip the timeout entirely. The 3B does the same job and actually finishes. The Dockerfile now readsWERBOS_CHAT_MODEL(defaultllama3.2:3b) (975cb98). - The context window was double what it needed.
num_ctxdefaulted to 8192; the persona plus a few turns of history fit comfortably in 4096. Halving it halves the KV cache — and, just as important, keeps every werbos→Ollama call agreeing on one number, so Ollama doesn’t spin up a second copy of the model and thrash. - Keep it warm.
OLLAMA_KEEP_ALIVE=-1so a sparse, quiet chat doesn’t pay a cold model reload on the first message after idle.
The measured curve
3B at 4096 context, on CPU, prompt-size → time-to-first-token:
| prompt tokens | time |
|---|---|
| ~270 | 7s |
| ~1230 | 18s |
| ~2730 | 36s |
Usable. Not fast. And here’s the honest part: this doesn’t fix the
two levers that actually matter, and pretending it did would be the
kind of self-flattery this project exists to refuse. The 780M iGPU is
sitting idle — ollama ps shows vram=0, meaning ROCm never
offloaded and the whole thing ran on CPU. And the prompt is ~2700
tokens when it could be a third of that. Those two — GPU offload and
prompt trim — are the real fights, tracked separately.
The AHA
The satisfying thing about a latency hunt is that it ends in a number, and the number doesn’t care what you hoped. Framework, not weights has a quieter cousin: hardware, not vibes. The phone wasn’t faster because it was clever. It was faster because it had a GPU doing the reading, and the SR8 didn’t. Sometimes the most useful thing an engineering day produces is permission to stop optimizing the wrong layer.