Eval suite

A corpus is also the source of the eval suite that measures whether training made the model better. Sprint SE.9 turns the same JSONL produced by werbos data ingest into:

• A held-out eval set the trainer never sees
• A composite score of how a model handles that set
• A noise floor measurement that tells you when the composite delta is real vs. random

Without this, every promotion decision is “did training help? maybe? feels like it?” With it, the answer is a number with a threshold.

TL;DR

# Carve a held-out eval set
werbos data split corpus.jsonl --eval-fraction 0.10

# Score a model against the eval set
werbos data score corpus.eval.jsonl --model llama3.1:8b

# Measure noise floor with N repeats (the "first ratchet click" gate)
werbos data score corpus.eval.jsonl --model llama3.1:8b --runs 5

1. Deterministic split

werbos data split corpus.jsonl --eval-fraction 0.10 --out-prefix corpus
# → corpus.train.jsonl  (90% of rows)
# → corpus.eval.jsonl   (10%)

The split key is sha256(row_id)[:8] — same row_id always lands at the same partition position. No --seed flag. The hash IS the seed.

What this buys you:

• Reproducible across machines. Two operators with the same corpus get the same eval set without coordinating.
• Stable under corpus growth. Adding new rows to the corpus tomorrow changes the partition by exactly the new rows’ positions — existing train/eval rows stay where they are. No re-randomization.
• No “lucky seed” tuning. Whatever the hash hands you is what you eval against. If the partition seems unrepresentative, fix the corpus, not the split.

Hard floor: minimum 5 rows in the eval set (smaller is just statistical noise). Hard ceiling: 50% eval fraction (above which you probably typo’d the flag).

2. Scoring

werbos data score corpus.eval.jsonl --model llama3.1:8b

Output:

SE.9 SCORE — llama3.1:8b  scorer=token_f1
  composite:        0.4561
  rows scored:      104 / 104
  errors:           0
  duration:         21.3s

The score command:

1. Reads each eval row from JSONL.
2. Sends the prompt to the named Ollama model.
3. Compares the response to the gold answer with the selected scorer.
4. Aggregates: composite = mean per-row score.

Failed model calls (Ollama down, timeout, empty response) count as n_errors and don’t pull the composite down. The composite stays a fair mean over rows where the model actually answered. The error count surfaces separately so “0.45 composite over 80/100 rows” is distinguishable from “0.36 composite over 100 rows when 20 errored out.”

token_f1 scorer

The v1 default. Standard QA metric: precision/recall over normalized word multisets.

normalize: lowercase + strip punctuation + split on whitespace
precision = |pred ∩ gold| / |pred|
recall    = |pred ∩ gold| / |gold|
F1        = 2 * P * R / (P + R)

Multiset (not set) means a model that over-generates the same token pays in precision; a model that under-generates pays in recall. Both bounded [0.0, 1.0].

Edge cases:

• Both predicted + gold empty → 1.0 (vacuous match, saves the runner from a divide-by-zero in error-only runs)
• Either empty (but not both) → 0.0
• Zero token overlap → 0.0
• Identical (after normalization) → 1.0

No NLTK / spaCy dependency. Stdlib only. Standard SQuAD/TriviaQA normalization rules.

What v1 doesn’t ship

• llm_judge scorer — local Ollama grades “are these answers equivalent? 0-10”. Queued for v1.1 when paraphrase tolerance becomes load-bearing. Token-F1 underrates a model that says “Yes, X is the answer” vs. gold “X is the answer”; llm_judge would catch the equivalence.
• Custom scorers — closed scorer set in v1 to keep the surface tight.

3. Noise floor — the “first ratchet click”

Single-run composite isn’t enough. Sampling noise can make any one run swing 1-3 percentage points either way. You need to know the variance before you trust a delta.

werbos data score corpus.eval.jsonl --model llama3.1:8b --runs 5

Output:

SE.9 NOISE PROFILE — llama3.1:8b  scorer=token_f1
  runs:               5
  rows per run:       104
  errors (total):     0
  mean composite:     0.4523
  std:                0.0102
  min / max:          0.4391 / 0.4612
  promotion threshold (2.5σ): 0.4778
  composites:         [0.448, 0.461, 0.450, 0.456, 0.447]

The promotion threshold is the mathematical definition of “the training actually worked”:

candidate.composite  >  baseline.mean + 2.5 × baseline.std
                                              ⟹  promote

2.5σ is a one-sided 99% confidence interval — there’s only a ~0.5% chance a candidate composite this high above baseline is sampling noise.

The defaults:

4. The full ratchet loop

# Step 1: split the corpus once
werbos data split corpus.jsonl --eval-fraction 0.10 --out-prefix corpus

# Step 2: measure baseline noise floor
werbos data score corpus.eval.jsonl --model llama3.1:8b --runs 5 \
    --out baseline.json
# baseline.json carries: mean / std / threshold_2_5sigma

# Step 3: train (see /docs/training/)
werbos train --corpus corpus.train.jsonl --base llama3.1:8b \
             --out werbos-0.0.1

# Step 4: score the candidate
werbos data score corpus.eval.jsonl --model werbos-0.0.1 \
    --out candidate.json
# candidate.json carries: composite / n_rows / n_errors / per_row

# Step 5: compare
jq '.composite' candidate.json
jq '.threshold_2_5sigma' baseline.json
# if first > second → promote; werbos-0.0.1 becomes the new baseline

After promotion, re-measure the noise floor on the new baseline. The next training cycle has to beat werbos-0.0.1’s threshold, not raw llama3.1:8b’s.

werbos data score corpus.eval.jsonl --model werbos-0.0.1 --runs 5 \
    --out baseline.json  # overwrites: werbos-0.0.1 is the new floor

That’s the ratchet click. Each promotion locks in a new minimum.

CLI reference

werbos data split <corpus.jsonl> [options]

werbos data score <eval.jsonl> [options]

Output schemas

Score report (eval-suite/1):

{
  "schema_version": "eval-suite/1",
  "model": "llama3.1:8b",
  "scorer": "token_f1",
  "composite": 0.4561,
  "n_rows": 104,
  "n_errors": 0,
  "duration_s": 21.3,
  "per_row": [
    {
      "row_id": "sha256:...",
      "prompt": "...",
      "gold": "...",
      "predicted": "...",
      "score": 0.82,
      "scorer": "token_f1",
      "error": ""
    }
  ]
}

Noise profile (noise-profile/1):

{
  "schema_version": "noise-profile/1",
  "model": "llama3.1:8b",
  "scorer": "token_f1",
  "runs": 5,
  "mean": 0.4523,
  "std": 0.0102,
  "min": 0.4391,
  "max": 0.4612,
  "composites": [0.4480, 0.4611, 0.4502, 0.4564, 0.4456],
  "n_rows_per_run": 104,
  "n_errors_total": 0,
  "threshold_2sigma":   0.4727,
  "threshold_2_5sigma": 0.4778
}

Same versioning discipline as the corpus schema — additive fields safe within v1.x; rename/retype/remove ships eval-suite/2 as a sibling.

What does NOT ship in v1

• llm_judge scorer (paraphrase tolerance via local LLM) — v1.1 when token-F1’s strictness becomes load-bearing.
• Multi-scorer composite weighting (e.g. 0.7 × F1 + 0.3 × LLM-judge) — v1.2.
• Per-skill stratification (separate composites for /weather vs /joke vs /tarot eval rows) — v1.3.
• Significance testing beyond 2σ / 2.5σ (t-test, bootstrap CI) — overkill for the single-operator case.

Architecture notes

werbos.eval.corpus_suite
├── schema.py         EvalRow / RowScore / ScoreReport / NoiseProfile
├── split.py          deterministic split + split_file
├── scorers/
│   └── token_f1.py   word-token F1 over normalized multisets
├── runner.py         async pass: read → predict → score → aggregate
└── noise.py          N-run variance + 2.5σ threshold math

Sprint doc: see the architecture page for the full design record. 37 tests, all green on commit.

Next

• Data pipeline — the corpus side: how the JSONL this evals against gets built
• How to train — the integrated story