Data pipeline

werbos data is the front door of the training story. Same code path whether the input is a personal notes folder, a company docs export, or a 10-GB GitHub repo clone.

The output is chat-format JSONL ready for HF PEFT / TRL / any QLoRA trainer that expects {"messages": [...]}.

Sprint DATA.1 shipped this end-to-end on 2026-05-31. The short-form version is in How to train; this page is the deep dive.

TL;DR

# Walk a directory, write training rows
werbos data ingest ~/docs --out corpus.jsonl

# What's in there?
werbos data stats corpus.jsonl

# Eyeball a few rows
werbos data sample corpus.jsonl --n 5

What gets read

Files with no registered reader are walked but skipped (PDF, HTML, Slack exports, Notion exports — those land in DATA.1.1+ when customer demand appears).

The markdown reader strips YAML frontmatter, walks the heading tree, and emits one chunk per (heading, body) pair. Bodies under 40 characters are skipped (noise). Bodies over 4000 characters split on paragraph boundaries so the resulting chunks fit the trainer’s context window.

The code readers extract documented entities only:

• Python: top-level + class-level def / class with a triple-quoted docstring → "What is def foo() for?" → the docstring
• JSDoc: /** ... */ function|class|const foo → docs as body, declaration as heading
• Go: // comment runs preceding a func / type / var declaration → comments as body, identifier as heading

Undocumented code is intentionally skipped. Training on code without comments produces noisy rows the model can’t learn from.

Strategies

heuristic (default — fast, no LLM)

Turns structured input into Q&A pairs deterministically. The prompt template is language-aware:

• Markdown: heading becomes a question; body becomes the answer. Template chosen by content-hash so re-ingesting the same chunk produces the same prompt (idempotency).
• Code: signature is the heading; docstring is the body. Asks “what is def foo() for?” / “describe the behavior of def bar()” etc.
• Plain text: first sentence is a pseudo-heading; falls back to “summarize the following” for headingless bodies.

No LLM call. Idempotent on re-ingest. Suitable when the input data is already shaped like a knowledge base.

llm_qa (slow, broader coverage)

For unstructured prose where heuristic has nothing to grip on (personal notes, slack threads, meeting notes, legacy docs without headings):

1. Hash the chunk + model + prompt version.
2. Check ~/werbos/llm_qa_cache.json.
3. On miss, send the chunk to your local Ollama with a fixed system prompt asking for N=3 Q/A pairs.
4. Parse the response.
5. Cache the result.
6. Emit one TrainingRow per Q&A pair.

Privacy invariant: chunk text never leaves the box. Same SSRF guard as every other werbos LLM call. The cache and the output JSONL stay local.

Idempotency invariant: same chunk + same model + same prompt version = same rows on re-ingest. Blow away ~/werbos/llm_qa_cache.json to force regeneration.

both

Runs heuristic and llm_qa over every chunk. Useful when you want the deterministic baseline plus the broader coverage — sink dedup means redundant rows collapse via row_id.

Filtering

Secret detection

Eight patterns ship in v1, all dropping the whole row when matched (prompt OR response):

werbos data stats reports the per-pattern drop count so you can see what got filtered.

Disable with --no-filter-secrets (verified-clean corpora only).

Length cap

Rows whose prompt + response > 16,000 characters are dropped. The trainer’s tokenizer caps at ~4096 tokens; rows past this just get truncated mid-content anyway.

Dedup

Two layers:

1. In-memory row_id set during a single run — same content produced by the same strategy hashes to the same row_id.
2. Pre-existing-file scan at sink open time. Re-running werbos data ingest over the same corpus loads the existing row_id values from the output JSONL and skips them. Result: re-ingest is a true no-op even after a process restart.

CLI reference

werbos data ingest <path> [options]

werbos data stats <corpus.jsonl>

Prints row count, per-strategy histogram, prompt/response length distribution (min/median/max/mean), top 10 sources by row count.

werbos data sample <corpus.jsonl> [--n N]

Pretty-prints N sample rows for eyeball verification.

→ Split + score commands are documented in eval suite.

Output schema

Locked v1 contract (schema_version: "corpus/1"):

{
  "messages": [
    {"role": "user",      "content": "What is X?"},
    {"role": "assistant", "content": "X is Y."}
  ],
  "_meta": {
    "schema_version": "corpus/1",
    "source":         "/path/to/source.md",
    "strategy":       "heuristic",
    "chunk_id":       "sha256:...",
    "row_id":         "sha256:..."
  }
}

Pass --strip-meta to omit _meta (some trainers reject extra fields). Keep meta by default so werbos data stats and re-ingest dedup work.

Schema versioning

CORPUS_SCHEMA_VERSION = "corpus/1" is stamped on every row. Bumping it would break v1 readers, so the versioning rules are:

• Additive optional fields are safe within v1.x. A new _meta.tags field appended tomorrow still loads in v1 readers — they just ignore it.
• Renaming, retyping, removing, or changing semantics ships a sibling corpus/2 module. v1 stays untouched. A future trainer dispatches on _meta.schema_version to pick the right reader.
• The IntentKind of the extractor (strategy field) is open-ended on purpose. New strategies land as new values without bumping the schema.

Same discipline as parsetongue, precook, glasshouse.

What does NOT ship in v1

• PDF / HTML / Slack / Notion / GitHub-clone readers — DATA.1.1+ if customer demand appears.
• Embedding / semantic dedup — exact-hash only in v1.
• Quality scoring — flagging boilerplate / legal headers / license blocks is DATA.3.
• Streaming / resumable ingest — runs to completion in one process; DATA.4 if corpus size demands it.
• PII detection beyond secrets — names / emails / phone numbers in personal corpus. v1 trusts the operator’s judgment on what to feed in.

Architecture notes

werbos.data
├── schema.py         TrainingRow / Chunk / IngestStats
├── ingest.py         walk → read → extract → filter → sink
├── readers/
│   ├── markdown.py   heading-tree chunker
│   ├── code.py       Python docstring + JSDoc + Go doc-comment
│   └── plaintext.py  paragraph chunker
├── extractors/
│   ├── heuristic.py  language-aware Q&A template (deterministic)
│   └── llm_qa.py     local Ollama Q&A pair generation
├── filters.py        8 secret regexes + length cap + dedup
└── sink.py           JsonlSink (append, dedup, idempotent reload)

Sprint doc: docs/sprints/data1-corpus-ingest.md in the werbos repo. 97 tests, all green on commit.

Next

• Eval suite — the other side of the pipeline: split corpus, score model, measure noise floor
• How to train — the integrated story