Day 29 — 'the trainer ships.'
Day 28 shipped both ENDS of the pipeline (DATA.1 + SE.9) and called the trainer 'the small piece in the middle.' Day 29 was building that small piece. SE.7.1 — real QLoRA trainer wrapping mlx_lm.lora, formatter that hash-splits DATA.1 corpus into train/valid, bring-up smoke that trains 50 rows × 5 iters and verifies the model produces a real 22 MB adapter. Then the end-to-end MLX scorer wire (--backend mlx + auto-detect via training_meta.json) and the first ratchet click on a bring-up signal (50-row Floogle → purple unicorn corpus): baseline 0.17 → candidate 0.74. THEN the real-world run: ingest /Users/.../werbosLLM, get 7664 training rows from 1424 files, split 7281/383, train 1B for 300 iters in 13 min, candidate composite 0.1136 vs baseline 0.0814 (+39.6%). Public docs rewrote training.md around the new 6-step DATA.1+SE.9 flow plus deep-dive pages for data-pipeline and eval-suite.
“the trainer ships.”
Day 28 closed with “the trainer is the small piece in the middle” after shipping both ends of the pipeline (DATA.1 input
- SE.9 eval gate). Day 29 was building that small piece.
The shape of the day: skeleton + bring-up + real run, in that order. Each step a verification that the previous one didn’t lie about working.
Public docs (morning)
Three pages rewrote how werbos-site/content/docs/ describes
the training loop. The legacy werbos import chatgpt /
werbos export sft flow had been the canonical path for months;
DATA.1 + SE.9 changed it overnight. Site commit 61ffc8b:
training.md(rewritten) — six-step loop: ingest → split → baseline noise → train → score → promote. Same five lines whether the operator’s feeding the engine their own notes or a company docs export. Legacy chat-archive importers kept as an aside.data-pipeline.md(new, weight 15) — DATA.1 deep dive: every reader, every secret pattern, the two extractor strategies, the schema versioning rules.eval-suite.md(new, weight 16) — SE.9 deep dive: the determ-by-hash split semantics, token-F1 math, noise-floor measurement, the 2.5σ promotion threshold derivation.
The launch-page videos can now point at these pages without hand-waving over which CLI commands are real.
SE.7.1 — the trainer (5e9d2e2)
Werbos already had werbos train lora --data <sft.jsonl> as a
thin wrapper around mlx_lm.lora. What it didn’t have: the
DATA.1-compatible path that takes the new chat-format JSONL
corpus directly + a werbos/training/ package that records
what got trained on what.
The shape:
werbos/training/
schema.py TrainingConfig + TrainingResult, training/1 lock
formatter.py DATA.1 corpus → mlx-lm {train,valid}.jsonl
with deterministic sha256(row_id) split (same
philosophy as SE.9; same code-shape, different
purpose — this split is the trainer's internal
loss reporting, not the held-out eval)
mlx_runner.py subprocess wrapper around `python -m mlx_lm lora`
parses Train/Val loss lines from stdout
persists training_meta.json next to the adapter
bring_up.py synthetic 50-row corpus with a deliberate signal
(Floogle → purple unicorn) so the operator can
smoke-test in under a minute
The CLI surface: werbos train --corpus X --base Y --out Z
matches what the docs promised. Existing werbos train lora ...
preserved.
Bring-up: 50 rows × 5 iters on Llama-3.2-1B in 5.9 s, produces
a 22 MB adapters.safetensors + adapter_config.json +
training_meta.json. The trainer works.
MLX scorer wire-up (965e4cf)
SE.9’s werbos data score was Ollama-only. To score a trained
LoRA adapter, the runner needed to route through MLX. Three
fixes:
- Use proper
pitts.core.SystemMessage/UserMessagetypes — MLXBackend rejects dicts as “unknown message type.” - Await
backend.generate— it’s async; was being called synchronously and silently dropping the coroutine (the test suite caught this because the next test errored on “coroutine was never awaited,” but the live path just silently scored 0.0). - When
--backend mlx --base X --model X, treat as “score the bare base, no adapter” instead of trying to load X as an adapter directory.
Plus a --backend {auto, ollama, mlx} + --base CLI surface.
auto (the default) sniffs the --model for a werbos-trained
adapter dir (training_meta.json present) and routes
accordingly — zero flag threading for the common case.
First ratchet click (bring-up corpus)
The full mini-cycle on the synthetic Floogle data:
werbos data ingest → 50-row bring-up corpus
werbos data split → 40 train / 10 eval (deterministic)
werbos data score → baseline 0.1707 (bare Llama-3.2-1B)
werbos train → 33.9 s, 22 MB adapter, loss 0.18/0.29
werbos data score → candidate 0.7354 (+0.5647)
Delta +0.5647 — 4.3× baseline. The qualitative wins were brutal:
- “Describe Floogle.” → base: “fictional character from a children’s book series” → candidate: “In short, Floogle is a purple unicorn.”
- “Capital of France?” (distractor) → both: “The capital of France is Paris.” (no overfit)
The model genuinely learned the signal AND kept general knowledge. That’s a real LoRA training cycle on a real model through werbos’s own CLI.
First ratchet click — the operator’s actual data (496160f)
Bring-up validated the pipeline. The real test was werbos’s own dogfood corpus.
werbos data ingest /Users/.../werbosLLM \
--include '*.md' --include '*.py' --include '*.ts' --include '*.tsx' \
--exclude '*/node_modules/*' --exclude '*/.venv/*' …
→ 1424 files, 7664 training rows, 19 secrets filtered
werbos data split --eval-fraction 0.05
→ 7281 train / 383 eval (then sampled to 50 for speed)
werbos data score --backend mlx --base 1B --model 1B
→ baseline composite 0.0814
werbos train --base 1B --iters 300 --batch-size 2
→ 12 min 56 sec, final loss 2.77, 22 MB adapter
werbos data score --model /tmp/werbos-0.0.1-1b
→ candidate composite 0.1136
Delta +0.0322 — 1.40× baseline. The token-F1 number understates what happened. Three per-row examples:
| prompt | base | werbos-0.0.1 |
|---|---|---|
class PFMStreamWriter | “Portable Font Metrics format” (hallucination) | “Write a PFM file to disk.” (your actual term) |
def _clear_cache() | rambles about Python decorators | “Clear the cache.” |
def test_pipeline_all_complete() | hallucinates pytest framework details | “All 5 tasks complete.” |
The model learned werbos’s terse declarative style — exactly the operator’s voice. PFM is “Pure Fucking Magic” in werbos; the base model invented “Portable Font Metrics.” The trained model knew the right answer because it had seen the operator’s docs.
That’s the operator’s voice in the LoRA weights.
Verification script (scripts/train-werbos-self.sh)
The whole flow as a parameterized one-shot:
scripts/train-werbos-self.sh # 1B / 300 iters
BASE=mlx-community/Llama-3.2-3B-Instruct-4bit \
ITERS=500 EVAL_N=100 \
scripts/train-werbos-self.sh # 3B / 500 iters (Day 30)
Six steps in sequence: ingest → split → baseline → train → candidate → ratchet report. Same code path for any operator on any directory tree.
Numbers worth keeping
- Day 29 commits: 3 (SE.7.1 trainer + MLX scorer + dogfood click) + the site repo doc rewrite
- Tests: 36 new training tests; 160 total green across DATA.1 + SE.9 + SE.7.1
- First trained model: 22 MB at
/tmp/werbos-0.0.1-1b - Composite delta: +0.0322 on the operator’s own corpus
- Dogfood corpus: 1424 files / 7664 rows / 19 secrets filtered / median response 241 chars
AHA
“the model learned werbos’s terse declarative style — that’s the operator’s own voice in the LoRA weights.”
The token-F1 number is modest. The qualitative change is huge. A 1B model that was confidently bullshitting about werbos became a 1B model that knows when to say “Clear the cache.” in three words instead of three paragraphs.
That’s not memorization. That’s voice.
Tomorrow
Day 30 = the 3B run + the dashboard surface + the launch UX. The 1B click was the proof; 3B is the demo. The TR.1 backend needs to land so the operator dashboard can show “current baseline / recent cycles / per-row drill-down” without shell-tailing JSON files.
The trainer ships.
“the trainer ships.”