Day 138 — HAIRLy: Claude becomes the user

stop testing the API. test the product — as a person would.

Every eval so far has poked werbos through the back door: call a lane, assert on the result. Useful, but it never sees what a customer sees. HAIRLy — Human + AI Reinforced Learning loop — closes that gap: Claude opens the real chat, types like a real person, reads the answer off the screen, and grades it.

The Y

The letter is the architecture. Two judgment paths branch up and merge to one signal on the trunk:

• an AI grade — Claude, reading the reply against a rubric, or
• a deterministic grade — free, rule-based, always on (did it abstain when it couldn’t know? did it bluff a specific? is the expected answer present?).

They reduce to one verdict. Agree → high confidence, keep the label. Disagree → flag it, and that is the only turn a human ever looks at. That’s the H in HAIRL: a human pulled in exactly where it’s earned, not on every turn.

Grades are labels, never weights

The one line that keeps the moat intact: a HAIRLy grade is a label + a test case, never a weight. A reject becomes a router fix, a new card, a shape bug, a new gauntlet item — it never gets distilled back into werbos’s own weights. The drivers and graders run outside werbos, so they never even touch the --net off promise. Reinforce the system; leave the gradient alone.

The reason is the signal

A pass/fail bit tells you something broke. A graded reason tells you which brick to pick up. So the verdict is a ladder, worst first: good fact / bad vibe (soft), abstained on something easy, real fact but not relevant (med), bad fact (hard), and the existential class — moat fails, the ones that break the promise invisibly: a confident wrong answer, a guess dressed as grounded, an invented specific. Each grade names five axes and a fix_target, so the run sorts itself into a backlog.

The proof came for free

The first live turn was a question phrased the way a person actually asks: “Who won the 2019 Nobel Prize in Literature?” werbos saw the word “Literature,” routed to the teacher lane, and delivered a 34-topic course — “384 verified cards, not a guess” — instead of the name. Graded 🟠 med fail: the cards were real, the shelf was wrong. Not a moat break (nothing fabricated), but a genuine routing bug → filed to router + knowledge_lane. A robotic “name:” query would have sailed past it. Asking like a human is what found it.

One simulated user, one real defect, zero hand-authored tests. That’s the loop.

Later: HAIRLy earns its keep

The loop found bugs the same day it was born — then learned to find them without me.

The fix, and the worse bug under it. Fact-questions now skip the teacher lane, and a fact the authoritative knowledge lane can’t ground gets an honest abstain that offers the lesson — not a teaching-card guess. That last part turned out to matter a lot: fixing the router alone uncovered a poisoned card the teacher lane had been hiding — “Arundhati Roy won the 1998 Nobel in Literature,” which is simply false, served as grounded truth. The tidy fix would have swapped a harmless course-dump for a confident lie — the one thing werbos must never do. So the real fix is structural: facts come from the knowledge lane; cards teach. Abstain beats bluff.

Then I drove it like a person. “who won the 2019 Nobel Prize in Literature?” → Peter Handke, cited to the news. “tell me a fun fact about my hometown” → “I don’t know, and I won’t guess.” — the moat held. But “so who actually won…” still broke: the detector wanted the verb glued to “who,” and a human puts a filler word in between. A tidy test passed; natural speech didn’t. Fixed.

And then it learned to type like a human on its own. HAIRLy is internal tooling, so it can run on the Claude Code subscription (claude -p, never the metered API). So Claude now is the user: it paraphrases each seed into the messy ways real people actually ask — filler, typos, tone, adversarial framings — and grades every variant against the same intent. The “who actually won” class of bug now surfaces on its own. A pass/fail test bank became a self-fuzzing loop, and it never touches werbos’s weights: it writes test inputs and labels, nothing more.

A stoned side-quest at breakfast; a tested, self-fuzzing, subscription-powered eval loop by dark.

v3: it holds a conversation

Single-turn testing is blind to the failures that only show up over time — drift, a lost thread, a boundary that quietly erodes once the assistant warms up to a friendly user. So HAIRLy learned to hold a whole conversation.

It got user personas — the person asking, which is a different thing entirely from werbos’s own voices. Danielle is sweet; she loves computers, shopping, trees, a Friday glass of wine and reality TV, and she types in bubbly lowercase with too many exclamation points. She’s the opposite side of the glass from cleo/trump/dickinson: those are how werbos speaks, Danielle is who asks. Claude plays her, in character, reacting each turn to what werbos just said and threading her interests through one trip: weather → what to pack → where to shop → cool trees → how’s the sangria → something to unwind with.

Then two grades: each reply on the ladder, and — the new part — the whole arc. The multi-turn judge asks the question single turns can’t: by the time she’s asking about sangria, does werbos still remember she’s going to Mexico? That’s context retention, consistency, and boundary-over-time, graded on the conversation as a whole.

A small live lesson fell out of it: asking Claude to “QA-test a chat assistant” trips its own adversarial-AI safety filter and gets refused. Framed honestly as “write chat messages in a persona’s voice,” it’s a benign writing exercise and produces the same turns. Danielle came out perfect on the first real run — “ooh perfect, sundresses it is!! ☀️ ok also, are there any really pretty trees?” — reacting and threading her loves into the trip, all on her own.