B.4 — JS-API forest: teaching the language foundation

T.M.B. #4 — the judge runs the function. Every task here is graded by importing the generated function and running it against assertions. The lazy wrong answer doesn’t just look wrong — it throws or returns the wrong value. No vibes.

The idea

B.2 and B.3 pointed the forest at your repo. B.4 points it at the language itself. We parsed the TypeScript standard-library .d.ts files into 359 foundation leaves — every Array, Object, String, Number, Math, Map, Set, Promise, Date method — each carrying its real signature + JSDoc. Ask in plain English, retrieve the exact method the coder should use, instead of letting a 7B half-remember it.

Two bricks, in order.

Brick #1 — hybrid retrieval (and an inverted finding)

The lexical retriever (TF-IDF) nails direct terms but whiffs on synonyms: “round a number down” wants Math.floor, but the word “floor” isn’t in the query. So we added an embedding arm (nomic-embed-text) and went to fuse the two rankings — expecting the B.1 wiki result, where RRF + embeddings beat everyone.

It inverted. On terse stdlib docs, RRF-sum made things worse: the embedder is noisy enough here that “agreement between the two arms” stopped meaning “quality.” RRF rewards leaves ranked mid-pack in both lists — and those mediocre-in-both leaves buried the confident hits (Array.map was the #1 lexical result for “transform each element,” and RRF dropped it off the page).

The fix is interleave, not sum: round-robin merge the two rankings (lex#1, sem#1, lex#2, sem#2, …), dedup. Each arm contributes its best picks directly, so a confident #1 is never dragged down by a committee.

A small, real lift — embeddings surface Math.max for “find the largest number” (lexical ranked it #134). But the honest headline is the ceiling: 3 of 10 queries miss even at recall@8, and not because the retriever is dumb — because the word isn’t in the doc. Set’s docstring never says “duplicate,” so no lexical or semantic match exists to find. The bottleneck is corpus vocabulary, not the retriever. The fix is curation — task-language aliases per leaf — which is the MMM thesis itself (the library is hand-built), and the next lever.

Why the opposite of B.1? Wikipedia prose is synonym-rich, so embeddings + RRF-sum thrive. Stdlib signatures are terse and near-duplicate, so the embedder is noisy and “agreement” is a bad signal. Same machinery, opposite corpus, opposite answer — which is exactly why you run the bench instead of assuming.

Brick #2 — wire the signatures into the coder

The payoff. Two arms, the same qwen2.5-coder:7b, one shot, no feedback loop — deliberately no grade-retry, so we isolate one variable: does injecting retrieved real signatures reduce first-try API hallucination?

• forest — top-6 retrieved JS-API signatures injected into the coder’s context, then write the function.
• blind — the same model, the same task, nothing injected.

Six tasks favoring modern / less-common stdlib (toSorted, toReversed, findLast, .at(), replaceAll, flatMap). Several assertions check non-mutation — the lazy .sort() / .reverse() answer mutates its input and fails the test, while the right non-mutating method passes.

What it says

The win is one task — and it’s exactly the predicted one. “Second-to-last element by negative index”: the blind coder wrote arr[-2], which in JavaScript silently returns undefined — a classic footgun. Handed the real at(index: number) signature, the forest coder wrote .at(-2) and passed. Stable across three runs.

Everything else ties — and that’s the honest, expected shape. qwen2.5-coder:7b already knows toSorted, replaceAll, flatMap; the signature in its context is one it could derive from memory, so it doesn’t move the needle. This is the underivable-information law again, now on the language’s own surface: the library helps exactly where the model half-remembers an API (a footgun, a brand-new ES method), ties where the model already knows it cold, and never loses.

So B.4’s claim is narrow and true: feeding a small coder the real stdlib signature rescues it on the APIs it gets subtly wrong, and costs nothing on the ones it knows. The edge is thin on a strong 7B’s home turf (common stdlib) and widens on footguns and modern methods — and the biggest lever from here isn’t the retriever, it’s alias curation to make the corpus speak the language of tasks, which lifts recall and the coder’s hit-rate together.

Honest caveats: N = 6, one differentiator — small, and qwen-coder:7b is strong on stdlib so most tasks tie by design. 1-shot, no retry (isolates the retrieval effect; the grade-retry loop would let blind recover the footgun too). Retrieval eval N = 10. Both arms: qwen2.5-coder:7b, temp 0.1. Embeddings: nomic-embed-text. Corpus: 359 leaves from the TypeScript stdlib .d.ts. Graded by importing the function and running assertions. Code (ingest_js_api.mjs, embed_js_api.mjs, js_api.mjs, battle_jsapi.mjs) in the mmm-poc repo — run it yourself.