Day 143 — the conversation, past turn two

A benchmark starts fresh on every question. A conversation doesn’t — it carries everything said so far, so one wrong answer on turn two poisons every turn after it. Tonight was entirely these turn-N bugs: the kind a single-shot score can’t see, found the only way they can be — by talking to it.

Memory across turns

“Make a variable called name = ‘jason’,” then “iterate over it and hash each character,” then “show the final code” — three turns that each used to fall apart. Now the first is composed and remembered, the second actually uses it, and the third shows the exact program that ran. A place you mention one turn (“I went to Mexico”) can be referred to the next (“reverse the place I went”) and resolves correctly.

The leak the tests couldn’t see

Every automated test stamped a session label on its request. The real front door doesn’t — it forwards your turns without one. So a memory bug tied to that label passed every test and still broke live. A new test reproduces the real, label-less path, and the session is now carried end to end, so each conversation is its own — a fresh chat inherits nothing.

The confident-wrong hunt

A borrowed larger model was quietly generating code over an honest “nothing to show yet” — now a deliberate “I can’t” can’t be buried by a fallback. A casual “I changed the oil in my car” dumped a 288-card course off the word “oil” — the teacher no longer fires on a chat line. “The value of string” got reversed into gibberish because a possessive was read as a literal string.

Killing the invented past

“Do you want to vibe?” invented a shared history — “that time we went to the beach, your dad…”. The friendly persona had been turned up too far, and a weak model elaborated the “ride-or-die friend” idea into fiction. The voice got turned down to a short, hard-ruled version, with a guard that rejects any reply claiming a lived history or ownership it doesn’t have.

From chasing phrasings to subtracting them

“Talk about 2JZ cars” grounded; “talk about 2JZ cars with me” went cold. Chasing every phrasing — slang, typos, gen-z — is a losing game. You can’t list every topic, but you can list the wrapper: tag each word as filler, intent, connector or stopword, and whatever’s left is the topic. “Yo vibe about 2JZ fr fr” now grounds — no rule for “vibe,” none for “fr fr,” just subtraction. A bare share with no topic still returns nothing, and is never forced to ground.

The AHA

Beating a chatbot benchmark is nice, but holding a conversation past turn fifteen was always the goal — and it’s a different, harder axis. A benchmark never even reaches turn two. Every fix tonight lived at turn N, and every one was found by talking, not scoring.