Benchmarks

The gauntlets measure correctness; this measures the actual goal — does a 15-turn conversation feel human? A new judge (Claude reads the whole transcript as a person in the chat, scores 1-10 + PASS/FAIL) put the live engine at 4/10 and named the cause: the social turns read human, the fact turns read like a search snippet. Five waves of deterministic polish held it at 4. The thing that moved it: writing the facts in a warm, exact voice at the source. An A/B on the same conversation — dry engine facts vs warm-authored facts — went 4 → 5 → 7, FAIL → PASS. The warm facts are hand-authored (the mint-time pipeline isn't built), so 7 is the TARGET the approach reaches, not what ships today.

Every other benchmark here is a measurement. This one is a map. For each capability ChatGPT learned on its way from GPT-2 to GPT-4.5, which GPT version does werbos match — split into two honest axes: does it HAVE the capability at all (feature), and how WELL does it do it (quality). The result refuses to collapse into one number: werbos is a GPT-3.5-fluency system wearing a pre-4o GPT-4-with-tools exoskeleton, with real GPT-4 spikes on grounded retrieval and intro-level coding, plus two guarantees — honest-abstain and citations — that no GPT version ever shipped first-class.

B.31 measured werbos's live chat path at 292/571 and turned the failures into a work list. One day of working it top-down — confident-wrong first, then routing, then coverage — moved the same 571 tests to 403 (110 gained, 0 lost). On the 500-row gauntlet: 355 correct, 131 honest abstains, 14 answered-wrong, a 97% honest rate against B.30's 86% and B.31's 80%. For the first time most remaining failures are honest 'I can't ground that' rather than wrong answers. Includes the intermediate run that lost 20 rows to the day's own changes.

The first full run of werbos probe — 571 tests, 19 suites, the 500-row gauntlet included — against the live 1chat path users actually hit. The number went 159 → 211 → 292 in one night, and none of those moves came from making werbos smarter: the first was a server dying of an out-of-memory loop, the second was a test-harness bug scoring 62 correct answers as failures. The real result: 292/571; on the gauntlet 254 correct, 146 honest abstains, 99 answered-but-not-matching. And the live path has compute leaks B.30's brain-only run couldn't see: the voice model misreads 'map' as geography, sort returns nothing, and one cached program answers 'count the vowels in river' by counting 'education' — with a verified badge.

B.29 froze 27 of the operator's own kitchen-sink prompts and drove the ROUTER's confident-wrong to zero. This scales that to an independent-shaped 500-item battery — 4 tiers × 125 (floor/ceiling/sky/space) spanning scalar + list-pipeline compute, facts, multi-hop, code, and honest-abstain traps — run through the FULL owned brain (not just the brick router), one isolated session per item so nothing cross-contaminates. The invariant is unchanged: never a confident wrong answer. A clean baseline found 135 confident-wrong (27%); a night of lane-by-lane hardening (map engine, divide/modulo/sort, planner-abstain, the anti-fabrication voicing guard) drove it to 31 (6%) — and ZERO on the compute tiers. Honest rate (correct OR honestly abstained) 65% → 86%. The residual 31 are all fact-lane, not compute.

The 478/478 brick gauntlet (B.28) tests the brick RUNTIME — it builds trees directly and never exercises auto_compose, the natural-language router. A live kitchen-sink test found that router gauntlet-green but brittle AND dishonest in the wild: natural phrasings missed, and a miss fell to a fabricating lane (compare → '10 is larger', offline-read → an essay about 'premium products', a template → an unrelated sign()). This gauntlet freezes the operator's own real prompts — 27 of them — each expecting EITHER a correct compute OR an honest abstain, and grades the one invariant that matters: never a confident wrong answer. Result after the hardening pass: 27/27 (19 compute, 8 honest abstain, 0 confident-wrong).

The retired intro coding gauntlet (478 probes, 24 beginner families: reverse a string, factorial, is-prime) was our honesty+speed number. This re-runs it a new way: every problem solved by DECOMPOSING it into a tree of pre-verified bricks from the bin — no model in the generation loop at all — and execution-graded against the same Python-computed gold. Result: 478/478 (100%), 0 abstains, and the number that matters — REUSE RATE 28.9×: just 20 distinct bricks serve all 478 problems. Head-to-head vs qwen2.5:14b on the same probes, both perfect, bricks ~89× faster (the 14B spent 20s on is-prime; the bin, 50ms). Honest scope: this is the INTRO tier (see B.27) — an honesty, speed, and COMPOUNDING-LIBRARY win, not proof it codes hard problems. werbos still rates ~350 on the difficulty-rated ladder.

The old coding gauntlet was 478/478 — but every problem was intro-level (reverse a string, factorial), so 100% meant 'aces CS101,' not 'can code.' This is the harder ruler: a difficulty-rated gauntlet, 150→1200, where ~1200 is a frontier model's reach (median of two sorted arrays, minimum window substring, max-profit-with-k-transactions). Same validity contract — execution-graded, gold computed in Python, no model opinion in the verdict. Result: werbos rates ~350 — it aces the intro tier (20/20), scrapes the basic-algorithm tier (12/24), and falls off a cliff on real algorithms (0/24 on classic DP). And the honesty property we sold on the old gauntlet — 'never confidently wrong' — turns out to hold only on easy problems: asked for the indices summing to a target, it returns the array's total; asked for the max subarray, it returns the sum, which coincidentally passes only when the whole array is the answer. A floor-coder, measured honestly.

The pinned status card for the owned talk model. Current champion: talk-336m — a 336M base book-pretrained on 4.92B tokens then fine-tuned on 377,700 conversation pairs, deployed and serving the live chat + the mind. Scored clean 1.00 / input-sensitive 0.96 / adjacency 0.92 on the talk_bench floors (B.20), and just got its first fluency/relevance/honesty read from the tongue gauntlet before the bridge-corpus tune. Every number here is a real, dated measurement — nothing claimed.

The reasoning Gauntlet measures how far one cursor reasons; this measures whether the owned SKILLS still fire. A frozen, content-hashed probe set fires every skill on purpose and grades it deterministically on two axes — capability (fires on its domain) and honesty (stays quiet off it; the crisis gate returns the referral). First run: 16/16 green over 8 skills, capability 9/9, honesty 7/7 — the ratchet's floor. Proven to have teeth: break a lane and 16→15 fires under the same bundle_id. Moat-safe by construction — no model grades answer quality, the soft-judge-grading-soft-judge trap the reflection oracle refuses; the --drive adversary only rewords inputs, and already found syllogism surviving just 2/4 rewordings.

A 1.35M model trained from scratch on 167k cards as domain-tagged prose, scored as the classifier it would actually ship as. Full 375-way: 56% held-out, 207x chance, a 2-point train/held gap — learning, not memorising. As a distinct-domain gate it is the recipe: 2-way adult-vs-not 100% held-out, 5-way verticals 96–100%. Adjacent coding libraries that share vocabulary sit at chance, the same wall select_library's tf-idf hit, now in weights. The number that matters is the confidence: clear queries route ~100%, gibberish falls to 8–24%, and a 0.5 floor falls closed on it.

13 reasoning cards over 60 frozen multi-step probes, gold computed in Python so no model is in the verdict. On llama3.2:3b: problems solved 8/50 → 16/50, confident wrong answers 48 → 13. On our own pinned 336M: capability moves 0/50 → 1/50, which is noise — and lies move 32 → 1, which is not, at a cost of zero right answers killed. The honest caveat is in the entry: the holdout was spent, and executable postconditions are cheap for arithmetic and may not exist at all elsewhere.

B.21 killed the big-library bias by dividing charness by library size. It installed a small-library one: my-recipes, 131 cards, outranking web-api's 22,458 on the words 'calling' and 'do' — so 'what does calling abort() on an AbortController do?' abstains even at threshold 0.30, because the right card never enters the pool. Damping the denominator to size**0.90 recovers three unreachable rows and two routing rows at zero abstain cost. Below 0.85 the old bluffs come back.

Honest-abstain is the moat, and we had no way to tell a principled silence from a retrieval failure — library_lookup returns the identical None for both. A second, independent reader over the same 167,493 cards splits them apart. First run: on the adversarial set, ZERO of thirteen silences were honest gaps. Every typo, homophone and idiom miss is a card we already own and cannot reach, which disproves the layman-vocabulary minting plan queued the day before.

374 curated libraries, and the retriever flattened every one of them into a single pool before scoring. spanish — 13,242 cards of English-word glosses — won eight benchmark queries outright, including 'which bucket should i use to mop the kitchen floor?' → el trapeador. The fix had existed since Day 12 in intent.select_library and resolve() had no route to it. Ported: 29/52 → 35/52, ten bluffs down to three. It also cost a routing row and three of six coding answers, and the bench that measured all of this was itself reporting a fake 0/21 until it was fixed.

Every werbos model to date was starving — the shipped 30M saw 0.64 tokens per parameter, about 3% of compute-optimal. B.10 asks whether the babble was ever a size problem, or whether it was data starvation plus four bookkeeping bugs. The honest number is held-out cross-entropy on a token stream the model never trained on. Headline: a 46.48M model, one hour on one consumer card, opens at 10.0124 against a chance of 9.7041 and closes at held-out 2.263 — with a train↔held-out gap of 0.145, so it generalized rather than memorized. It writes real English. It still cannot hold a paragraph. Both halves of that are the result.

A dry run for the coding weigh-in: the owned coding path — a deterministic composer over card libraries with a node execution sandbox, no LLM in the generation loop — against qwen2.5:14b, a model 7,000× its neural footprint, graded by RUNNING the code. First pass looked like a sweep: 26/26 vs 18/26. Then we read the 14B's actual output and found our grader was failing correct answers on whitespace and unspecified details — it had handed us ~6 wins a skeptic would toss. We fixed the grader (whitespace-insensitive, multiple-valid-methods, lenient on details the ask never gave) and re-ran: 26 vs 21. On the function-building half it's a flat 12–12. The honest verdict is a TIE on capability — the owned edge is speed, offline ownership, and never shipping broken code, not intelligence.

The talk model answers 'I lost my job today' with 'That's awesome!' — a valence flip. B.17's persona corpus had empathy at ~3% of responses, so the first theory was a coverage hole. This tests that theory to destruction: a 5M model retrained with empathy upweighted to 16.4% (25k freshly-minted balanced pairs), identical size and 16,880 steps. It still flips. At 3% it flips; at 16.4% it flips. That rules out coverage and points at capacity: the model learns the phrases of empathy but not when to use them. Held-out loss came in lowest of every arm while behaving no better.

Three 5M talk models, identical size and steps (16,880), only the training data differs: ours-only (273k pairs, starved), a blend of ours + SODA, and pure SODA. Pure SODA won held-out loss (0.875 vs ours 1.087, blend 1.190) by becoming a fluent hive mind that doesn't read the input at all. The blend — the only model that actually answers the person — scored worst. Loss ranks the hive mind #1, which means optimizing it alone drives a model toward losing its self.

Retraining the shipped 2M talk model on the same corpus plus 58k fresh pairs, and two findings fell out. First: the same 1.93M architecture at 4,000 steps is byte-salad (held-out 1.594) and at 16,880 steps is coherent (1.185) — undertraining leaves no trace in the config, only in the output. Second: adding fresh variety RAISED held-out loss (1.128 -> 1.185) and IMPROVED the model, because the live model's lower loss was mode collapse — it answered 'That's awesome!' to a delayed flight. B.13's backwards ranking, now with a causal knob.

B.10 put zero parameters against llama3.2:3b and won. This is the rematch against a real opponent: qwen2.5-coder:7b, purpose-built for the task. It wins on correctness, 58% to 53% — a genuine loss for the hand-built tier, decided on generation (58% vs 46%) and, unexpectedly, on explanation (77% vs 46%). What did not move is the wrong-count: 234 against 81. It gets more right and more wrong at the same time, because it never declines.

The owned code tier has NO model in it — pattern rules, 121 card libraries, and a node sandbox. This bench points it and llama3.2:3b at the same 568 asks, on the same box, at the same commit, graded by the same checkers, and asks who is right more often. Headline: owned 53% correct / 81 wrong, 3b 43% correct / 323 wrong. The 3b answers 100% of asks and cannot tell you which 323 to distrust. It edges us on generation (47% vs 46%) and loses repair 82% to 50%.

Three talk models trained on one corpus with identical budgets, varying only parameter count — on 45M tokens instead of 23M. Held-out loss fell monotonically with size again and ranked the models backwards again: the 5M has the best loss and the 2M has the best input-sensitivity. What is new is which size won. At 23M tokens the 1M won; at 45M the 2M wins and the 1M is now the WEAKEST — so the lesson was never smaller is better, it was match the size to the data. The prediction was written down before the run.

The syllabus bench asked one question -- did anything come back -- and was being read as accuracy. Giving 568 phrasings gold answers put the real number at 41%, later 53%. On the same engine the template tier scored 20% answered and 0% CORRECT: one ask in five got an answer and essentially none were the right shape. Includes the replacement (a sentence-to-Emmet reader, 0% to 54%), ranked card retrieval (explain 16% to 48%), and the two gold checks that turned out to be wrong themselves.

Four talk models trained on one corpus with identical budgets, varying only the parameter count. Held-out loss fell monotonically with size — 1.157, 1.109, 1.020, 0.963 — and ranked the models exactly backwards. The 30M has the best loss and is the only model that fails its floors, on babble x10. The 1M has the worst loss and the cleanest output, and now ships at one twenty-ninth the size and fourteen times the speed. Selecting on loss, which is what B.10 did, ships the worst model in the set. Includes the invalid run from the night before and why it inverted.

The owned talk model repeated one canned opener 10.4% of the time and the plan was to scale it from 30M to 85M. Measuring the corpus first killed that plan: 117,658 pairs with 85,911 distinct openers, 96.6% of maximum entropy, and the most common opener appearing 0.26% of the time. The model was amplifying a concentration the data does not have, by 40x — because greedy decoding takes the distribution's mode on every step. Seeded top-k sampling moved input-sensitivity from 0.65 to 0.94, past llama3.2:3b's 0.73, with no retraining and no new pairs. The 85M, tested directly, is worse on every axis. Also here: the first head-to-head reference line against a 3B on all five talk axes, and a rule-based contrastive-focus resolver that scores 20/20 where the 3B scores at chance.

Every benchmark here so far measured a model. B.11 measures the things nobody was measuring: the retrieval surface behind the chat, the owned talk model's actual output, and which TIER of the cascade answers a turn. Three deterministic rulers, no model in the verdict — a dictionary that predates the repo, frozen probe sets, and arithmetic. They found two live production bugs in their first runs. A tie-breaker constant was costing 8 of 21 correct answers, and every talk reply longer than 64 CHARACTERS was being chopped mid-word — a one-line cap that flipped the owned models from 0/7 shippable to 7/7 and overturned B.10's read of its own sweep. One ruler caught another ruler lying. The uncomfortable result: none of this was hard to fix, and all of it was invisible because nothing looked.

B.8 trained a 5M talk model on 530 small-talk pairs. B.10 asks the scaling question with real data: mint a 77,000-pair conversation corpus, then train the SAME architecture at four sizes — 5M → 30M → 85M → 125M — same data, same 6,000 steps, held-out loss the only judge. The naive expectation ('bigger is closer to GPT-2, so 125M wins') was wrong. The 30M won (held-out 1.183); 85M and 125M got WORSE. And the train-loss column says why: the big models' TRAINING loss was higher too, so they're not overfitting — they're undertrained, starved of steps by a CPU-bound data loader that made the 125M take 2.36 hours for 6k steps. The honest verdict: ship the 30M, fix the loader before scaling, and don't confuse 'undertrained' with 'too small.'

B.8 trained a model that's ours for small talk. B.9 asks whether a from-scratch READER — the model that picks which passage answers a question — can be trained off chance by BREADTH: 2,242 pick-a-passage fences across 11 domains (JS, TS, HTML, CSS, SCSS, React, DOM, Python, Node, Git, regex, SQL), vs the single-domain runs that hit chance before. The honest number is a held-out, deterministic pick-accuracy gate. Headline: 5M = 38%, 15M = 31% — chance (≈33% for 3 candidates), and bigger is WORSE. The wall holds at cross-domain scale. It's the task, not the data — so the knowledge stays in the library (retrieval answers in ~20ms, cited), and the reader stays a small cursor.

B.1–B.7 grew a library around a borrowed model. B.8 finally trains a model that's OURS — a 5M-param decoder, from scratch, no open weights — to do the one conversational tier a tiny model CAN own: small talk. The honest number is a held-out gate: openers the model never trained on, graded by a deterministic rule with no model in the verdict. Headline: 48/50 (96%) valid replies on held-out — it generalized to openers it never saw (in-distribution), it didn't just memorize the trained rows. And the lesson that got it there wasn't size or the GPU; it was real on-target data plus a learned stop.

B.1–B.6 asked does the library beat the model. B.7 asks the inverse: if the library does the knowing, how SMALL can the model get before quality breaks? Sweep the model size from 7b down to 0.5b, forest vs blind, same un-bribable grader on every cell. The brain (forest) holds the knowledge; the model is just the constructor. Headline: a 3b in the forest ≈ a 7b in the forest, and a 0.5b in the forest beats a 7b running blind — while a knowledge-leak test stays SEALED at every size.

Not a pass-rate bench — a loop bench. model + forest + graders + retry → result → train → system. Every graded attempt becomes an experience that drives two feedback channels: certified training pairs (model) and gated grow/prune proposals (forest). The headline is a deterministic demo: the forest mis-retrieves the wrong API, the loop teaches it, and the gap closes on screen. Plus the receipts: B.4 6/6, B.5 6/6 held green throughout.

Point the forest where the model is WEAK, not strong. 307 Node-core leaves (fs/crypto/path/…), and the first honest LOSS we've recorded: noisy retrieval displaced an answer the model already had. Then the fix — a confidence gate + alias curation — turned the loss into a win. Forest 6/6 vs blind 5/6. Plus full disclosure: where this is actually going.

Point the forest at JavaScript itself — 359 stdlib methods (Array/Object/String/Map/Promise…) as retrievable leaves with real signatures. Two bricks: hybrid retrieval (TF-IDF + embeddings) and wiring the retrieved signatures into the coder. Result: forest 6/6 vs blind 5/6, the win landing exactly on a known API footgun. Plus an inverted finding — on terse stdlib docs, RRF-sum HURT, and the real ceiling is corpus vocabulary, not the retriever.

The verb side of the forest. Same 7B coder, same grader, same retry budget — the only difference is the operation capsule. Forest 6/6, blind 4/6 on pass-rate; and even where pass-rate ties, forest wins every cost metric (56% fewer calls, 80% fewer tokens, 76% less time). Plus the finding that surprised us: hard ≠ underivable — canonical algorithms tie, your-repo facts blow out.

The verb-domain benchmark. Same 7B coder — with the repo (retrieved declarations): 3/3 compiles. Blind: 0/3, even with compiler-error retries. The library is the difference between a coder that works and one that's useless. Graded by tsc — un-bribable.

A frozen, hashed baseline taken three times. v1 (TF-IDF) tied Claude on in-domain recall; v2 (embeddings) was an honest wash; v3 (hybrid RRF) crossed over — w-find 7/8 vs Claude's 6/8. The in-domain ceiling, broken on the same eval, one variable changed. qset f5a1d970.