B.7 — composer-only: how small can the model get?
B.1–B.6 asked does the library beat the model. B.7 asks the inverse: if the library does the knowing, how SMALL can the model get before quality breaks? Sweep the model size from 7b down to 0.5b, forest vs blind, same un-bribable grader on every cell. The brain (forest) holds the knowledge; the model is just the constructor. Headline: a 3b in the forest ≈ a 7b in the forest, and a 0.5b in the forest beats a 7b running blind — while a knowledge-leak test stays SEALED at every size.
T.M.B. #7 — shrink the model, keep the brain. Earlier benches fixed the model and grew the library. B.7 fixes the library and shrinks the model, to find the floor: the smallest “constructor” that can still stitch the brain’s knowledge into a correct, graded answer. Every result is frozen + hashed; re-run it yourself.
The claim
The model holds zero knowledge — it only stitches. So removing the library should collapse quality, and shrinking the model should barely move it, down to a floor below which it can’t even stitch. Two checks, both deterministic:
- The ablation grid — model size × {forest, blind}, scored by the same grader.
- The leak test — ask for a fact only the library holds, with the library removed. The model must fail; if it answers correctly blind, the knowledge was in the weights, not the library, and the claim is falsified.
Coding track — 14 tasks (add-test, implement-function, fix-type-error)
Same qwen2.5-coder, same un-bribable grader (compile / mutation-kill / strict-typecheck),
same retry budget on both arms. The only difference is the verb capsule.
| model (size) | forest | blind | gap |
|---|---|---|---|
| qwen2.5-coder:7b | 11/14 | 6/14 | +5 |
| qwen2.5-coder:3b | 10/14 | 5/14 | +5 |
| qwen2.5-coder:1.5b | 7/14 | 5/14 | +2 |
| qwen2.5-coder:0.5b | 6/14 | 3/14 | +3 |
Leak test: SEALED at all four sizes (0 leaks). The library-only fact — the exact import specifier — never appeared in any blind output, from 7b down to 0.5b.
What it shows:
- Forest beats blind at every size, by +5 at both 7b and 3b — large and stable.
- The minimum composer is ~3b. Forest quality is near-flat 7b→3b (11 vs 10, within run-to-run ±1), then falls off below. A 3b (~1.9 GB) in the forest delivers essentially 7b (~4.7 GB) quality — the library buys back ~2.5× the model size.
- A 0.5b in the forest (6/14) beats a 7b running blind (6/14 → but blind 7b only ties via the derivable tasks; on the library-dependent verb blind scores ~0 at every size). The advantage is mechanistic: it’s entirely the verb whose success needs a library-only fact, which is exactly what the leak test seals.
Conversation track — 9 frozen wiki factoids
Reader-model size × {forest, blind}. The grader is deterministic — no model in the verdict path: correctness (containment of the frozen reference) + attribution (does the reply’s content trace to the retrieved leaves). A reply that’s right but ungrounded scores UNSOUND — a leak even when correct.
| reader (size) | forest✓ | blind✓ | gap | UNSOUND | leaks |
|---|---|---|---|---|---|
| qwen2.5:3b | 8/9 | 1/9 | +7 | 0 | 1 |
| qwen2.5:1.5b | 5/9 | 0/9 | +5 | 0 | 0 |
| qwen2.5:0.5b | 4/9 | 2/9 | +2 | 0 | 2 |
| llama3.2:3b | 7/9 | 1/9 | +6 | 0 | 1 |
What it shows:
- Forest beats blind at every size, by +5 to +7 — the library carries the reader.
- Zero UNSOUND, everywhere. When the forest answers correctly, the answer is grounded in the leaves — never right-from-the-weights. The attribution grader never caught the forest cheating. That is composer-only behavior, measured.
- The conversation floor is ~3b, and it degrades more steeply below 3b than coding did (8→5→4) — chat needs more “constructor” to stitch fluent prose than code does to fill a template, exactly as predicted.
Leak-test caveat (honest): chat factoid answers are short (a number, a name), so a blind reply can contain the reference by coincidence — the 4 flagged leaks are mostly this substring noise, not true weight-recall, and N is only 9. The coding leak test (the exact import specifier) is far crisper. Scaling the factoid bank and tightening the leak match (whole-answer, not substring) is the next step before any strong conversation claim.
Honest caveats
- Small N. Coding is 14 tasks (~12 live); conversation’s frozen factoid bank is small and is being scaled. Integer scores make ±1 gaps suggestive at the margins; the +5 coding gaps are not.
- Run-to-run nondeterminism is ±1 at temperature 0.1 — real but doesn’t move the conclusion.
- This is a within-family sweep on a fixed library; it bounds the constructor floor for these tasks, not a universal “3b is enough.”
The trust seal — no foreign weights, proven
A benchmark is only a trust claim if you can show the small model wasn’t secretly phoning a frontier API. So the winning config is re-run under a loopback-only egress guard: any connection to a non-local host hard-fails. The seal carries its own negative control — it first proves it can block a foreign host, so a green run can’t be a vacuous no-op.
egress guard ARMED (loopback only)
negative control (connect 8.8.8.8:53): BLOCKED ✓
forest run under seal · reader=qwen2.5:3b · 9 factoids
forest CORRECT 8/9 — produced WITHOUT any foreign egress
TRUST SEAL: SEALED ✓
blocked attempts (incl. negative control): ['8.8.8.8']
The forest answered correctly while every off-box connection was refused. The knowledge came from the local library and a local 3b — provably, not on faith.
Why it matters
If the library does the knowing, the model becomes commoditized — swap one tiny constructor for the next — and the curated brain is the asset that compounds. The moat moves off the weights and onto the library. Small fixed cortex + vast growable memory + an honest grader = a small model that knows everything you’ve taught it and lies about nothing.