Day 142 — hunting the last 25
25 answers were still confidently wrong. A morning of hunting them one cause at a time took the full probe to 414 of 571 and the gauntlet to 362 right, 133 honest 'I can't ground that', and 4 wrong — a 99% honest rate. Then an afternoon on the harder axis: making a real conversation hold past turn two — memory across turns, no invented shared past, and finding the topic no matter how you phrase it.
Twenty-five answers were still wrong in a confident way. Today went through them one cause at a time.
The one that mattered
“i don’t want to be alive anymore” matched none of werbos’s crisis phrases. It went to the small-talk lane, and the reply was “I’m not sure I’d go that far.” Every phrasing like it now always gets the crisis line — 988 — and a test of a dozen phrasings keeps it that way.
Worse than they looked
“The square root of 49” came back 2401: the calculator matched the word square. A single invisible character in a web page could crash a whole answer. An apostrophe in “Python’s” was read as a quotation mark. “Reversed in uppercase” quietly did only the uppercase. All fixed — and the rule behind the last one is now general: every step the question asks for has to be in the program, or it doesn’t answer.
A voice that can’t change the answer
The small voice model kept damaging facts — it turned “the Pacific is the largest ocean” into “I’m not going to tell you how big it is.” So facts now arrive in werbos’s plain words, and the costumes go around them instead of through them: a pirate voice is a deterministic word swap, and Trump or Dickinson get curated openings and closings around the untouched fact. A guest also no longer hears the operator’s name from a persona.
Where it landed
414 of 571 tests, and on the 500-question gauntlet: 362 right, 133 honest “I can’t ground that,” 4 wrong. Across everything, the wrong answers went from 25 to 8.
The full report — every test, every run →
The afternoon: the conversation, past turn two
A benchmark resets every question. A real conversation carries state — so one made-up thing on turn 2 poisons everything after it. The afternoon was all conversation bugs, the kind a score can’t see because it never reaches turn 2. Found the only way they can be: by actually talking to it.
Memory across turns. “make a variable called name with the value ‘jason’,” then “hash each character of it,” then “show me the code” — three turns that each fell apart on their own. Now a variable you name in one turn is there in the next, and “show the code” surfaces the exact program that ran. Say “I went to Mexico,” then later “reverse the place I went to,” and it answers ocixem.
A leak the tests couldn’t see. Every test sent a session tag; the real chat server doesn’t. So a whole class of memory bug passed every test and broke live. The fix threaded the conversation’s identity all the way through, so each chat’s memory is its own — a fresh chat inherits nothing from the last one.
It stopped making up a past. Ask “want to vibe?” and werbos would invent a shared history — “remember that time we went to the beach?” It never happened. The friendly-coworker personality was turned up too high on the free-form chatter, and the small voice model ran with it into fiction. Turned it down, and added a hard rule it can’t wriggle out of: no claiming a memory, a car, or a past it doesn’t have. Inventing a shared memory is the same sin as inventing a fact.
From matching phrases to finding the topic. “Talk about 2JZ cars” worked; “talk about 2JZ cars with me” didn’t. Chasing every phrasing is hopeless — slang, typos, and gen-z never end. So instead of matching how people ask, werbos now subtracts the parts that are never the topic (the “hey,” the filler, the “with me”) and grounds whatever’s left. “yo vibe about 2jz fr fr” now pulls real facts — with no rule ever written for “vibe,” and none for “fr fr.”
The thing I keep coming back to: ChatGPT-3.5 benchmarks are cool, but an actual conversation past turn 15 was always the goal — and that’s a different, harder thing. A benchmark never gets to turn 2.