Days 148–159 — the ruler that reads structure
Twelve days and 246 commits turned fill-in-the-blank bricks into a conversation, and then built a scoreboard that grades a reply's shape instead of who won a blind contest. Three long runs moved it hour by hour, until it stopped moving. The numbers went up on conversations werbos never saw. The worst bugs of the stretch didn't show up on any number at all.
On Day 147 werbos learned to answer with bricks: small fill-in-the-blank sentences, filled from what you said and what it actually knows. Twelve days later those bricks hold a conversation. Most of the work, though, went into the ruler.
Stop asking who won
For weeks the scoreboard was a blind contest: show a judge a werbos reply and a ChatGPT reply, ask which is better. That number bounced around and taught almost nothing. A loss didn’t tell you why.
So the question changed. Instead of “did it win?”, ask “does the reply have the right shape?” Three rungs, each one only counted if the one below passes:
- Did it give an answer? Not a dodge, not just a question back.
- Is it readable? No garbled blanks, no wall of pasted card text.
- Is it conversational? Does it fit what was just said, in the tone it was said?
Every conversation the operator has saved runs through werbos and gets graded on that ladder. Twenty of them are set aside as a held-out set: the bricks are never built from them, and nobody reads the replies, only the numbers. If werbos gets better there, it’s getting better at conversations it’s never seen, not at memorizing the test.
The bricks plan, the mouth writes
A brick knows the shape of a reply: react, then one question; or answer first, then two specifics. A small open model (the “tongue”) fills in the wording, and only the wording. It isn’t allowed to answer facts from its own memory. When a question needs a fact and the bricks have none, the tongue gets one narrow job: librarian. Turn the question into search words, look at what the card library turns up, and say yes or no on whether a card fits. If one does, the reply quotes the card. Of 176 questions werbos had previously answered with an honest “not sure”, 42 now get a real answer this way.
What the hour-by-hour runs found
Three long runs (5, 10 and 12 hours) re-measured every hour. Some of what turned up:
- Calculators nobody called. werbos has had exact unit conversion, arithmetic and date math for months. Chat never used them. “How many cups in a gallon?” got a beer-brewing card. Now it gets 16.
- A safety floor with holes. A swallowed battery got no reply at all. Now poisoning goes to Poison Control or the ER, a running engine in a garage gets a carbon-monoxide warning, and a medicine dose is never guessed.
- The worst one: cheerful replies to terrible news. A brick that reacts to “I saw a deer!” with “Oh nice, a deer! How was it?” turned out to react to a cancer diagnosis the same way. Another read “I got into a fight” as an achievement. Grief, bad news and good news now each get their own tone. No score caught any of this, because saved conversations rarely contain those lines. It took feeding werbos a list of negative sentences and reading every reply.
The numbers
| start | end | |
|---|---|---|
| Held-out conversations: conversational | 59.8% | 70.1% (peak 71.1%) |
| Saved conversations, strict hand read | 38% | 66% |
Both rose by about the same amount, so the gains carry over to new conversations. Then, for the last ten hours, nothing moved. The bricks alone score 29% on conversations they were built from and 4% on ones they weren’t. The fixes left are either missing knowledge (questions the library can’t ground yet) or bricks that only fit the conversations they came from. More patching won’t fix either one.
The AHA
A ruler only sees what you feed it. The held-out set proved the gains were real. The tone sweep proved the scoreboard was blind to the bugs that matter most. Next: fill the library from the 112 queued “not sure” topics, make bricks that generalize, and put a frozen set of safety and tone lines on the ruler so that work finally counts.