How to train

Training werbos is the same engine, two entry points. Same install. Same pipeline. The corpus you feed in decides whether the output is a model that knows you, or a model that knows your company.

ingest  →  split  →  measure baseline  →  train  →  score  →  promote

Six commands. Each is one line. None of them ship your data anywhere except the box you’re running them on.

1. Ingest your corpus

Point werbos data ingest at a directory (or several). It walks files, extracts training rows by format, filters secrets, and writes JSONL.

# Personal AI — your notes, code, docs
werbos data ingest ~/notes ~/code ~/docs --out me.jsonl

# Business AI — same engine, different corpus
werbos data ingest ./company-docs ./tickets --out company.jsonl

Out of the box, the ingester reads:

Add --strategy llm_qa to also generate Q&A pairs from unstructured prose (uses your local Ollama; never leaves the box).

Eight secret patterns are stripped by default — GitHub PATs, OpenAI/Anthropic keys, AWS keys, JWTs, env-style KEY=value, operator filesystem paths (~/.ssh/*, Keychain, ~/.gnupg/*), and -----BEGIN PRIVATE KEY----- headers. Disable with --no-filter-secrets only after you’ve verified.

Re-running on the same corpus is idempotent — same content hashes to the same row_id, sink dedups on row_id, file unchanged on the second run.

→ Full reference: data pipeline.

2. Split into train + eval

Carve off a held-out eval set the trainer never sees. Default 10%.

werbos data split me.jsonl --eval-fraction 0.10 --out-prefix me
# → me.train.jsonl  (90% of rows)
# → me.eval.jsonl   (10%, never touched during training)

The split is deterministic by sha256(row_id) — same corpus, same fraction, same split across machines, runs, clones. No --seed flag. The hash IS the seed. Add new rows to the corpus later and the existing partition stays stable; only the new rows shuffle into position.

Why this matters: reproducibility. Two operators with the same corpus get the same eval set without coordinating.

3. Measure the baseline noise floor

Before training, find out how much the base model’s eval score varies just from sampling noise. Without this, you can’t tell whether a candidate’s improvement is real or random.

werbos data score me.eval.jsonl --model llama3.1:8b --runs 5

Output:

SE.9 NOISE PROFILE — llama3.1:8b  scorer=token_f1
  runs:               5
  rows per run:       104
  mean composite:     0.4523
  std:                0.0102
  min / max:          0.4391 / 0.4612
  promotion threshold (2.5σ): 0.4778

Save the threshold. Any candidate needs to clear it to count as a real improvement.

→ Full reference: eval suite.

4. Train

werbos train --corpus me.train.jsonl \
             --base llama3.1:8b \
             --out werbos-me-0.0.1 \
             --epochs 3

Defaults are tuned for the M1 + 3B/8B base model case:

• LoRA rank 8, alpha 16 (HF PEFT standard)
• 3 epochs (sweet spot for a 5k-row corpus)
• Batch size 1 (raise on bigger machines)
• Output: a LoRA adapter + an Ollama Modelfile that bundles it with the base model so werbos serve --model werbos-me-0.0.1 just works

Per-iter wall time on M1 / 8B base: ~30 seconds. So a 5k-row corpus × 3 epochs ≈ 4-5 hours. Plan for overnight.

The trainer writes a checkpoint per epoch. Keep all of them. They’re your A/B fodder if epoch 3 overfits and epoch 2 wins.

5. Score the candidate

werbos data score me.eval.jsonl --model werbos-me-0.0.1

Output:

SE.9 SCORE — werbos-me-0.0.1  scorer=token_f1
  composite:        0.5121
  rows scored:      104 / 104
  errors:           0
  duration:         18.4s

Compare against the baseline noise profile from Step 3:

candidate.composite (0.5121)  >  baseline.threshold_2.5σ (0.4778)
                                                    ✓ PROMOTE

That’s the first ratchet click: a candidate clears the noise floor by a meaningful margin, and we declare the training worked.

→ The math: candidate > baseline.mean + 2.5 × baseline.std. The 2.5σ is a one-sided 99% confidence the delta isn’t noise.

6. Promote

# The trained adapter becomes the new baseline. Any further
# training has to beat THIS, not raw llama3.1:8b.
werbos data score me.eval.jsonl --model werbos-me-0.0.1 --runs 5
# → save THIS noise profile as the new baseline

Use the model immediately:

werbos serve --model werbos-me-0.0.1
# Now ask werbos about your own life / code / docs. It knows.

The ratchet is engaged. Every subsequent training cycle starts from the new floor. Old corpus + new corpus + retrain → werbos-me-0.0.2. And so on.

The one-shot script

The whole loop, copy-pasteable:

# 1. Ingest your corpus
werbos data ingest ~/notes ~/code ~/docs --out me.jsonl

# 2. Split
werbos data split me.jsonl --eval-fraction 0.10 --out-prefix me

# 3. Baseline noise floor
werbos data score me.eval.jsonl --model llama3.1:8b --runs 5 \
    --out baseline.json

# 4. Train
werbos train --corpus me.train.jsonl \
             --base llama3.1:8b \
             --out werbos-me-0.0.1

# 5. Score candidate
werbos data score me.eval.jsonl --model werbos-me-0.0.1 \
    --out candidate.json

# 6. Compare + promote if delta > 2.5σ
werbos serve --model werbos-me-0.0.1

Adapter stacking (the kung-fu cable)

You can stack adapters at inference for personality + specialty:

werbos serve --backend mlx \
  --model llama3.1:8b \
  --adapter ~/werbos/adapters/personality-v1/ \
  --adapter ~/werbos/adapters/werbos-me-0.0.1/

Order matters. Each adapter applies on top of the previous. Performance cost ~5-10% per stack member.

Importing from chat archives (legacy path)

If you’re starting from ChatGPT exports or Claude Code transcripts instead of a directory of files, use the chat-archive importers:

# ChatGPT export
werbos import chatgpt --src ~/Desktop/chatgpt-export --dst ~/werbos/corpus-chatgpt/

# Claude Code transcripts
werbos import claude-code --dst ~/werbos/corpus-claude-code/ \
    --include-subagents --min-tool-calls 3

Then export to the same JSONL shape werbos data ingest produces:

werbos export sft --src ~/werbos/corpus-claude-code/ \
    --out ~/werbos/sft-claude-code.jsonl --label good

From there, steps 2–6 above are identical.

The Claude-Code importer is particularly high-yield because it has tool-use sequences — exchanges where the model called real tools and acted on the result. Plain chat data won’t make werbos a better coding agent; tool-use traces will.

Rule of thumb

• 2000+ rows → training has signal to work with.
• Mix structured and unstructured. A corpus that’s only markdown trains a model that’s only good at markdown questions.
• Always run the noise floor before declaring a win. A composite that’s 0.005 above baseline could just be a lucky sample.
• One adapter per specialty. Stack at inference rather than training a generalist.

Next

• Data pipeline — DATA.1 deep dive (readers, extractors, secret filters, schema)
• Eval suite — SE.9 deep dive (split math, token-F1 scorer, noise floor semantics, promotion math)
• How to nudge — labeling loop that filters which rows enter the next training cycle
• How to rate — GUI for batch labeling