B.2 — code forest: a small coder + your repo, graded by the compiler
The verb-domain benchmark. Same 7B coder — with the repo (retrieved declarations): 3/3 compiles. Blind: 0/3, even with compiler-error retries. The library is the difference between a coder that works and one that's useless. Graded by tsc — un-bribable.
T.M.B. #2 — but the judge is
tsc. No fragment-match, no vibes. The code either typechecks against the real repo or it doesn’t. The most un-bribable scoreboard there is.
The setup
Three coding tasks on the werbos dashboard (real TypeScript, 446 declaration
leaves). Each task requires a real repo shape a model can’t guess — e.g.
getMindMetrics() returns { dimensions: MindDimension[] } (not an array), and
MindDimension.score is number | null. Three contestants, same coder model
where it matters:
- forest —
qwen2.5-coder:7b+ the retrieved declarations from your repo - coder — the same
qwen2.5-coder:7b, blind (no repo context) - claude — frontier, blind
Everyone gets the compile-feedback loop: generate → tsc → if it fails, the
error is fed back → retry (×3). Grade = does it compile, and in how many tries.
The results
| compiles /3 | avg tries | |
|---|---|---|
| forest (coder + your repo) | 3 | 1.7 |
| claude (frontier, blind) | 3 | 1.7 |
| coder (same model, blind) | 0 | 3.0 |
What it says
Same 7B model. With your repo: 3/3. Without it: 0/3. That row is the whole thesis — the library is the difference between a coder that works and one that’s useless.
The blind 7B couldn’t recover even with compiler feedback. It got tsc
errors that literally spell out the real type — Property 'dimensions' does not exist on 'MindDimension[]' — and still failed all three, all three tries. A
small model can’t reverse-engineer your repo from error messages. Retrieval
isn’t a nicety; for a small model it’s load-bearing.
forest tied Claude — didn’t beat it (both 3/3, both 1.7 tries). Frontier reasoning can decode the tsc errors and learn your shapes over retries, so the loop let it match. So the honest claim isn’t “beats frontier” — it’s: a small local coder + your repo matches frontier on your codebase, and is infinitely better than the same model without the library. On a domain where “knows your code” is the entire game, with the only judge that can’t be sweet-talked.
Honest caveats: N = 3 (small — more tasks would harden it); forest needed one
retry on strict-null even with the right shapes (a stronger coder closes that).
Grader: tsc --noEmit on a clean baseline. Run it yourself.