`knowledge-layer/references/compile-protocol.md`

97 lines
# Compile Protocol

This is the exact procedure Claude follows to compile `sources/` into wiki
entries. It is the LLM half of the skill — the scripts prepare and verify; the
reasoning here is what turns raw material into a durable, cross-referenced,
contradiction-aware knowledge layer.

This is the Karpathy pattern: **read the sources once, compile a structured
wiki** (summaries, cross-references, contradictions) that the LLM maintains —
instead of re-reading everything through RAG on every question.

## Preconditions
1. The layer exists (`kl_init.py` has run).
2. New/changed material is in `sources/`.
3. You have run `kl_scan.py <root>` and have the NEW / CHANGED / REMOVED lists.

Only work the sources in that work-list. Do **not** recompile untouched
entries — incremental compilation is what keeps a large layer maintainable.

## The pass

### 1. Read the work-list sources fully
Read each NEW/CHANGED source end to end before writing anything. For large
sources, read in sections but hold the whole thing in mind before routing —
scattered half-reads produce duplicate, shallow entries.

### 2. Extract atomic units
From each source pull: entities (people, companies), concepts/terms, decisions,
claims/facts, open questions. Keep each unit atomic — one assertion — so it can
be provenance-tagged and later contradicted independently.

### 3. Route to typed entries (one entity per file)
For each entity/concept, find or create `<folder>/<slug>.md`:
- **Exists?** Merge — update in place, append new facts, refresh `**Summary:**`
  and `updated`. Never fork a second entry for the same entity (see dedup below).
- **New?** Create it from the SCHEMA format: frontmatter (`id, type, tags,
  aliases, sources, updated`) + `**Summary:**` + `## Facts` + `## Open questions`
  + `## Related`.

### 4. Provenance on every claim
Every fact carries an inline `<!-- src: <source-stem> -->` and the source path
appears in the entry's `sources:` frontmatter list. A claim with no source is
not allowed — `kl_status.py` flags entries missing provenance.

### 5. Cross-reference
Link related entries with `[[slug]]` in the `## Related` section and inline
where natural. Cross-references are the graph that makes retrieval work without
vectors — a person links to their company, a project to its decisions.

### 6. Detect contradictions — never silently overwrite
Before overwriting a fact, check whether the new source *disagrees* with the
existing one. If so:
- Keep **both** claims on the entry, each with its own `<!-- src -->`.
- Add an **open** item to `CONTRADICTIONS.md` citing both sources.
- Do not pick a winner — a human resolves it. Stating false certainty is the
  failure mode this whole pattern exists to prevent.

### 7. Handle REMOVED sources
For each removed source, find entries citing it. Drop claims that depended on it,
or mark them stale if still plausible, and update `sources:` lists. An entry left
with zero sources should be deleted or explicitly marked speculative in `ideas/`.

### 8. Roll up
- Update `INDEX.md`: one line per entry, grouped by folder, with the summary.
- Update `SOURCES.md` roster with any new source rows.
- Update `SCHEMA.md` only if a genuinely new typed folder was needed.

### 9. Commit the manifest
Run `kl_scan.py <root> --commit`. This marks the current sources as compiled so
the next scan shows a clean slate until something changes. Do this ONLY after
the entries are actually written — the manifest is a promise that the work is done.

### 10. Verify
Run `kl_status.py <root>`. Resolve what it surfaces:
- orphan `[[links]]` → create the missing entry or fix the link,
- no-provenance entries → add sources,
- open contradictions → leave for human, but make sure they're logged.

## Dedup discipline
Before creating an entry, search existing slugs and aliases for the same
entity under other names ("ACME" vs "Acme Inc." vs "acme-corp"). Merge into one
canonical entry and add the alternates to `aliases:` — duplicate entities are
the #1 way a layer rots.

## Cost
This pass is **$0** beyond the model tokens you're already spending as Claude.
For bulk, low-stakes enrichment (e.g. tagging hundreds of media notes), a local
Ollama model can pre-draft entries for you to verify — keep frontier-model
tokens for the judgment steps (routing, contradiction detection, summaries).

## What NOT to do
- Don't build a vector store or embed anything — the structure is the index.
- Don't recompile the whole layer for a one-source change.
- Don't dump a source's raw text into an entry — compile it into atomic,
  cited facts.
- Don't resolve contradictions on the model's own authority.