B.33 — the human-likeness judge: 4 → 7 of 10, FAIL → PASS, from warm-authored facts
The gauntlets measure correctness; this measures the actual goal — does a 15-turn conversation feel human? A new judge (Claude reads the whole transcript as a person in the chat, scores 1-10 + PASS/FAIL) put the live engine at 4/10 and named the cause: the social turns read human, the fact turns read like a search snippet. Five waves of deterministic polish held it at 4. The thing that moved it: writing the facts in a warm, exact voice at the source. An A/B on the same conversation — dry engine facts vs warm-authored facts — went 4 → 5 → 7, FAIL → PASS. The warm facts are hand-authored (the mint-time pipeline isn't built), so 7 is the TARGET the approach reaches, not what ships today.
T.M.B. #33 — a different ruler, on purpose. Every prior benchmark measured whether werbos is right. This one measures whether it’s human — a thing a correctness score can’t see and that was always the real goal past turn fifteen. The number that matters isn’t 7. It’s the gap between 4 (what ships) and 7 (what warm-authored facts reach), because that gap is a to-do list, not a mystery.
The claim under test
The engine stuck at 4/10 for five straight waves of fixes, and the judge kept giving the same reason: the small-talk turns sound like a person, the fact turns sound like an encyclopedia. If that diagnosis is real, then changing only how the facts are written — nothing else — should move the score, and the social turns (already liked) should be left untouched.
The setup
One scripted 15-turn car conversation, one session against SR8’s live 1chat endpoint, graded by
claude -p reading the whole transcript as the person in the chat (not an examiner): per-turn flags, the
bot-tells that broke the illusion, a 1-10 score and PASS/FAIL. The judge never sees a right answer.
- Baseline (live engine, every turn as served):
scripts/convo_judge.py - A/B (same convo, dry facts vs warm-authored facts):
scripts/warm_poc.py— keeps every social and abstain turn the engine produces, swaps only the dry fact turns for warm-authored-but-faithful prose.
WERBOS_PROBE_AB=http://<box>:8097/ python scripts/convo_judge.py --convo car # baseline
WERBOS_PROBE_AB=http://<box>:8097/ python scripts/warm_poc.py # warm A/B
Results
| run | what changed | score | verdict |
|---|---|---|---|
| baseline | live engine, dry fact turns | 4 / 10 | FAIL |
| + 5 waves | receipts off · anti-repeat · trimmed frames · reactions · repeat-detect | 4 / 10 | FAIL |
| warm facts | only the fact turns rewritten warm + exact | 5 / 10 | FAIL |
| warm facts + 3 fixes | + non-dead-end abstain, fresh repeat answer, specific share reaction | 7 / 10 | PASS |
What actually moved the number
Nothing cosmetic did. Receipts, repetition, trimming, reaction-acks — all landed, none moved the 4. The judge eventually said the deterministic frame itself read as a template. The single change that moved it was rewriting the fact prose: “the 2JZ-GTE is an inline-layout, six-cylinder…” became “heck yes — the 2JZ-GTE’s the legend, Toyota’s twin-turbo inline-six, cast-iron block.” At 7/10 the judge listed those warm facts as the most human part of the chat — the exact turns it had hated at 4.
The honest caveats — read these before quoting the 7
- 7 is the target, not what ships. The live engine still scores 4. The warm facts in the A/B are hand-authored to simulate a minted warm card; the serve-side pipeline (author warm at mint → grade for meaning → store → serve plain) is not built yet. This measures the ceiling the approach reaches, not today’s product.
- One conversation, one domain (car). A single transcript, not a suite — the food/emotional/rapid-fire convos aren’t averaged in yet. Treat 7 as “this approach can PASS,” not “werbos scores 7.”
- The judge is a single
sonnetgrader and is noisy; ±1 between runs is plausible. It was run from a neutral cwd via stdin (the repo’s security context trips the model’s safeguard otherwise). - The ceiling is the moat, by design. The #1 complaint still standing at 7 is werbos refusing to make up a horsepower number it can’t source. A labeled ballpark would buy a point or two and dent the honest-abstain moat — explicitly off the table. So on this rubric werbos tops out below a model that will cheerfully guess, and that is the intended trade.
- Warm serving reaches owned facts only. The warm card replaces a library fact; web-sourced facts have no card and stay dry until one is minted.
The verdict
The prose ceiling that held through five waves is an authoring problem, not a model-size or moat problem: a bigger runtime mouth failed werbos’s own faithfulness guard 3 times in 4 (its “personality” is confabulation), and relaxing the guard was never on the table. Warm-authored canonical facts took the same conversation from a 4/FAIL to a 7/PASS with zero moat cost and no bigger model. The next entry worth writing is the one where that 7 comes from the engine, not the A/B rig.