gapmarks — where werbos ranks on the GPT-2 → 4.5 ladder
Every other benchmark here is a measurement. This one is a map. For each capability ChatGPT learned on its way from GPT-2 to GPT-4.5, which GPT version does werbos match — split into two honest axes: does it HAVE the capability at all (feature), and how WELL does it do it (quality). The result refuses to collapse into one number: werbos is a GPT-3.5-fluency system wearing a pre-4o GPT-4-with-tools exoskeleton, with real GPT-4 spikes on grounded retrieval and intro-level coding, plus two guarantees — honest-abstain and citations — that no GPT version ever shipped first-class.
T.M.B. — the honest exception. Every other page in this folder earns its number by executing something. This one does not. It is a map, not a measurement — an architectural judgment placing werbos on the ladder ChatGPT climbed. It is here to make the strategic claim legible and to give every future tune a frozen cell to move. Two rows are execution-backed and linked; the rest are mapped, and labelled as such. A judgment shown as a judgment, not dressed up as a run.
The claim under test
The instinct is to ask “is werbos GPT-4 yet?” and want one number back. That number would lie. ChatGPT never got better by making the model smarter alone — it stacked capabilities (tools, memory, vision, voice, search) around the model. So the honest comparison is not one number but a ledger with two columns:
- Feature — does werbos have this capability, wired and first-class? Which GPT version first had it?
- Quality — how well does it do it, expressed as the GPT version it feels like?
The falsifiable claim: on feature surface werbos is a pre-4o GPT-4-with-tools system (≈2023–early 2024) — it can do what that ChatGPT could, minus the 4o senses. On quality it is GPT-3.5 with two genuine GPT-4 spikes. And it carries one axis the ladder never had.
The ledger
| Lane | Feature = GPT version | Quality = GPT version | Note |
|---|---|---|---|
| Text generation | GPT-2 | GPT-2 | completes; can’t reliably free-generate (the owned-gen ceiling) |
| Prompt / task following | GPT-3 | GPT-3 | composes over a library, doesn’t invent |
| Conversation | GPT-3.5 | GPT-3.5 | multi-turn works; fluency is 3.5-tier |
| Reasoning | GPT-4 (structure) | GPT-3.5 | reason-tree gives GPT-4-shaped decomposition, 3.5 wording |
| Coding | GPT-4 + tools | GPT-4 on intro, floor-coder above | 478/478 vs qwen2.5:14b on CS101 (execution-graded); but the harder ruler rates it ~350 and the “never-wrong” property holds only on easy problems |
| Tools / lanes | GPT-4 + tools (2023) | GPT-3.5–4 | compute lanes, resolve(), doc→cards ingest, hunt harness; narrower coverage than OpenAI’s |
| Memory | 4o-era feature, present | GPT-4 | user.md + cards; conceptually as good, less automatic |
| Search / retrieval | Search-era feature, present | GPT-4 (narrow) | LSA + a 167k-card hub; owned corpus, so narrower — but cited |
| Vision | ❌ absent | — | a 4o feature; not a demerit below 4o — every GPT ≤ mid-2023 lacked it too |
| Voice | ❌ absent | — | a 4o feature; caps werbos below 4o only |
| Honest-abstain | — (no GPT ships it) | beyond GPT-4.5 | the rung they never built: “I don’t know” as a contract, not a failure |
| Citations / grounding | — (no GPT ships it) | beyond GPT-4.5 | the answer grounds in a cited card or it doesn’t ship |
Two rows are real numbers, not judgment: coding is execution-graded (the code runs, gold is computed in Python, no model opinion in the verdict) and conversation / honesty is blind-graded on a frozen floor set. The other rows say “GPT-4” to mean has the capability category and it works — not measured at GPT-4 level.
The verdict
Read the two columns together, and refuse to collapse them:
Feature: GPT-4-with-tools (pre-4o, ~2023–early 2024). werbos does what that ChatGPT did — conversation, tools, memory, retrieval — minus the 4o senses, whose absence caps it below 4o but not below 4.
Quality: GPT-3.5 baseline, with GPT-4 spikes on grounded retrieval/memory and intro-level coding, and off-the-chart on abstain/citations — because the chart has no such axis.
Then the footnote that flips the ranking: werbos does all of this on 730M owned parameters, running locally, with honest-abstain and citations as contracts — two guarantees no GPT version in the entire ladder ever shipped first-class. On the ladder they climbed, werbos sits at the GPT-4-tools rung. On the rung werbos added, they are at zero.
The point isn’t “a small model magically became GPT-4.5.” It’s that “the AI” was never just the model — and measured that way, a 730M model plus a library is further up the ladder than its parameter count has any right to be.
How to move a cell
This is a scoreboard with open cells. Each tune or ship should aim at exactly one, and cite the run that moved it — no cell moves without a linked benchmark, same rule as every other page here:
- bridge-corpus tune → move Conversation quality 3.5 → 4, and put a real abstain number behind honest-abstain
- reason gauntlet green → promote Reasoning quality 3.5 → 4
- the rated coding ladder climbing past tier 350 → move the Coding quality cell up honestly
- a measured retrieval run → put a number behind Search / retrieval instead of “narrow”
A moved cell needs a run behind it. Until then, it stays exactly where the map puts it.