Day 146 — the conversation that passes
The goal was never a benchmark — it was a fluid conversation that doesn't feel like a fact machine. So I built a judge where Claude reads a whole chat the way a human would and scores how human it feels, not whether it's correct. It opened at 4 out of 10 and said the facts read like a search snippet. Five rounds of fixes didn't move it. The thing that did: writing the facts in a warm, spoken voice at the source instead of letting a small model restyle them live — which it turns out can't do it honestly. That took a real 15-turn chat to 7 out of 10 and the first PASS.
The goal was never a benchmark score. It was a real conversation — fifteen turns and still fluid, grounded in facts, without feeling like a vending machine that spits a fact when you put a question in.
So the first thing I built was a way to know. A judge, where Claude reads a whole transcript the way a person in the chat would, and scores how human it felt — not whether the facts were right, but whether you’d want to keep talking. It gives a mark out of ten and a PASS or FAIL.
What it said
Four out of ten. And it put its finger on exactly the problem: the small-talk turns sounded like a person, and the fact turns sounded like a search result pasted in. Two different personalities in one chat.
Five rounds that didn’t work
I turned off the citation clutter. I stopped it repeating the same canned lines. I trimmed the encyclopedia dumps. I made it react to “that’s wild” and notice when you’d asked the same thing twice. Each round, I re-ran the judge. Each round it held at four — and eventually it told me why: the little frame I was bolting onto every fact was the robot.
The thing that worked
I tried the obvious fix — let the model rewrite each fact in a warm voice on the fly. It turns out a small model can’t do that honestly: three times out of four it either dropped a key detail or invented a little lived experience (“I was reading about this the other day,” “my 2JZ”) to sound casual — and werbos’s own honesty guard, correctly, threw it away. A bigger model would be faster at the same mistake. And loosening the guard was never going to happen.
So the answer was the werbos-shaped one: don’t rewrite facts while you’re talking — write them warm and exact up front, the way a card should read, and just say them. The fact already is the friendly sentence. Nothing to restyle, nothing to invent, nothing for the guard to catch.
To prove it, I kept every greeting, share and honest “I can’t ground that” the live engine already produces, and swapped only the dry fact turns for warm-written ones. The judge went four, then five, then seven out of ten — the first PASS. “I’d happily keep chatting.” And the facts it used to hate, it now named as the most human part of the chat.
The AHA
The wall that held through five rounds of clever fixes wasn’t about a smarter model or a weaker honesty rule. It was about how the facts were written. The one thing still standing between a seven and a nine is that werbos won’t make up a number it can’t back — and that’s not a flaw to patch. That’s the whole point.