← back to Knowledge Layer Skill
knowledge-layer skill: Karpathy/gBrain LLM-compiled wiki (init, scan, query, status + compile protocol)
0f05fc4db754f81323aa22d34b1cfc41e02fced5 · 2026-07-16 20:54:13 -0700 · Steve Abrams
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Files touched
A .gitignoreA LICENSEA README.mdA knowledge-layer/SKILL.mdA knowledge-layer/assets/template/CONTRADICTIONS.mdA knowledge-layer/assets/template/INDEX.mdA knowledge-layer/assets/template/SCHEMA.mdA knowledge-layer/assets/template/SOURCES.mdA knowledge-layer/references/compile-protocol.mdA knowledge-layer/references/wiki-principles.mdA knowledge-layer/scripts/kl_init.pyA knowledge-layer/scripts/kl_query.pyA knowledge-layer/scripts/kl_scan.pyA knowledge-layer/scripts/kl_status.py
Diff
commit 0f05fc4db754f81323aa22d34b1cfc41e02fced5
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Thu Jul 16 20:54:13 2026 -0700
knowledge-layer skill: Karpathy/gBrain LLM-compiled wiki (init, scan, query, status + compile protocol)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
.gitignore | 18 +++
LICENSE | 21 +++
README.md | 53 +++++++
knowledge-layer/SKILL.md | 100 +++++++++++++
knowledge-layer/assets/template/CONTRADICTIONS.md | 24 +++
knowledge-layer/assets/template/INDEX.md | 25 ++++
knowledge-layer/assets/template/SCHEMA.md | 64 ++++++++
knowledge-layer/assets/template/SOURCES.md | 22 +++
knowledge-layer/references/compile-protocol.md | 96 ++++++++++++
knowledge-layer/references/wiki-principles.md | 64 ++++++++
knowledge-layer/scripts/kl_init.py | 119 +++++++++++++++
knowledge-layer/scripts/kl_query.py | 174 ++++++++++++++++++++++
knowledge-layer/scripts/kl_scan.py | 131 ++++++++++++++++
knowledge-layer/scripts/kl_status.py | 161 ++++++++++++++++++++
14 files changed, 1072 insertions(+)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..8fb49c8
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,18 @@
+# Steve fleet standard
+node_modules/
+.env*
+tmp/
+*.log
+.DS_Store
+dist/
+build/
+.next/
+
+# Python
+__pycache__/
+*.pyc
+.venv/
+
+# Knowledge-layer runtime state (skill scaffolds these into user dirs, never here)
+knowledge-layer/knowledge-layer/
+*/.kl/
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000..7181d72
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Steve Abrams
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..38a110a
--- /dev/null
+++ b/README.md
@@ -0,0 +1,53 @@
+# knowledge-layer-skill
+
+A Claude Code **skill** that builds and maintains an **AI Knowledge Layer** — a
+plain-directory, LLM-compiled wiki (the Karpathy / "gBrain" pattern) that agents
+read *first* before any task. No RAG, no vector database: the LLM reads curated
+sources once and compiles them into typed, cross-referenced, contradiction-aware
+entries. The **structure is the index**.
+
+> Pattern credit: Andrej Karpathy's LLM knowledge-base workflow, and Shann
+> Holmberg ([@shannholmberg](https://x.com/shannholmberg)), who popularized the
+> "gBrain" / AI-Knowledge-Layer framing.
+
+## Layout
+
+```
+knowledge-layer-skill/ ← this git repo
+└── knowledge-layer/ ← the skill package (symlinked into ~/.claude/skills/)
+ ├── SKILL.md ← metadata + workflow
+ ├── scripts/ ← stdlib-only Python 3, $0 to run
+ │ ├── kl_init.py ← scaffold a new knowledge layer
+ │ ├── kl_scan.py ← incremental compile work-list (content-hash diff)
+ │ ├── kl_query.py ← read-first retrieval (keyword/frontmatter, no vectors)
+ │ └── kl_status.py ← health report (drift / contradictions / orphans)
+ ├── references/
+ │ ├── compile-protocol.md ← the exact LLM compile procedure
+ │ └── wiki-principles.md ← "a layer you keep vs. one you abandon"
+ └── assets/template/ ← seed docs copied into each new layer
+```
+
+## Quick start
+
+```bash
+cd knowledge-layer
+python3 scripts/kl_init.py mybrain # scaffold
+# … drop sources into mybrain/sources/ …
+python3 scripts/kl_scan.py mybrain # what changed?
+# … compile per references/compile-protocol.md, then:
+python3 scripts/kl_scan.py mybrain --commit
+python3 scripts/kl_query.py "some question" mybrain --print
+python3 scripts/kl_status.py mybrain # health
+```
+
+## Install as a skill
+
+The package is symlinked into `~/.claude/skills/knowledge-layer` so Claude Code
+discovers it. To install elsewhere, symlink or copy the `knowledge-layer/`
+folder into any `skills/` directory, or run
+`skill-creator/scripts/package_skill.py knowledge-layer` to produce a
+distributable `.zip`.
+
+## License
+
+MIT — see `LICENSE`.
diff --git a/knowledge-layer/SKILL.md b/knowledge-layer/SKILL.md
new file mode 100644
index 0000000..97faa4f
--- /dev/null
+++ b/knowledge-layer/SKILL.md
@@ -0,0 +1,100 @@
+---
+name: knowledge-layer
+description: "Build and maintain an AI Knowledge Layer — a plain-directory, LLM-compiled wiki (the Karpathy / gBrain pattern) that agents read FIRST before any task. Instead of RAG or a vector database, the LLM reads curated sources once and compiles them into typed entry folders (people, companies, concepts, ideas, projects, operations, media) with per-fact provenance, cross-references, and a contradictions log, then refreshes incrementally as sources change. Use this when the user wants a persistent, read-first shared brain / knowledge base / memory layer for a project, company, domain, or fleet of agents; when onboarding agents that must wake up with full context; when turning transcripts, chats, docs, campaigns, threads, or research into a durable queryable wiki without vector infrastructure; or when they say knowledge layer, gBrain, second brain for my agents, compile my notes into a wiki, Karpathy knowledge base, read-first memory, or /knowledge-layer. Not for one-off Q&A over a single file, and not a RAG or embeddings pipeline."
+---
+
+# Knowledge Layer
+
+## Overview
+
+A Knowledge Layer turns curated raw material into a structured wiki that a fleet
+of agents reads before acting and writes durable context back to. It implements
+Andrej Karpathy's LLM-knowledge-base idea (popularized as "gBrain"): **read the
+sources once, compile a maintained wiki** — summaries, cross-references, and
+contradictions handled by the LLM — instead of re-retrieving raw chunks through
+RAG on every question. There is no vector database; the directory structure is
+the index.
+
+## When to use
+
+Use this skill to **create** a new knowledge layer, **compile** sources into it,
+**query** it read-first, or **maintain** its health. Reach for it whenever
+someone wants a persistent shared brain for a project/company/domain or agent
+fleet — not for single-file Q&A and not to stand up an embeddings pipeline.
+
+## Tools in this skill
+
+All scripts are stdlib-only Python 3 (no installs) and cost **$0** to run.
+Run any with `-h` for full flags.
+
+| Script | Purpose |
+|--------|---------|
+| `scripts/kl_init.py [path]` | Scaffold a new layer (typed folders + SCHEMA/INDEX/CONTRADICTIONS/SOURCES + manifest). |
+| `scripts/kl_scan.py [root]` | Content-hash `sources/` vs. the manifest → the incremental compile work-list (NEW/CHANGED/REMOVED). `--commit` after compiling. |
+| `scripts/kl_query.py "<q>" [root]` | Read-first retrieval (keyword + frontmatter + cross-ref neighbours, no vectors) → the entries an agent should load. `--print` to dump bodies. |
+| `scripts/kl_status.py [root]` | Health report: pending sources, open contradictions, orphan cross-refs, entries missing provenance. |
+
+## Workflow
+
+### 1. Create the layer
+```bash
+python3 scripts/kl_init.py <path> # default ./knowledge-layer
+```
+This drops the typed folders and the four control docs (`SCHEMA.md`, `INDEX.md`,
+`CONTRADICTIONS.md`, `SOURCES.md`). Read `SCHEMA.md` — it defines the entry
+format everything else depends on.
+
+### 2. Add sources
+A human curates what goes in `sources/` (transcripts, chats, docs, contracts,
+research, saved threads). Quality in, quality out — the LLM compiles; it does not
+decide what is worth ingesting.
+
+### 3. Compile
+```bash
+python3 scripts/kl_scan.py <root> # shows NEW / CHANGED / REMOVED
+```
+Then follow **`references/compile-protocol.md`** exactly: read the work-list
+sources, extract atomic facts, route them into one-entity-per-file typed entries
+with per-fact provenance, cross-reference with `[[wikilinks]]`, and — critically
+— **log contradictions instead of silently overwriting**. Finish with
+`kl_scan.py <root> --commit`, then `kl_status.py <root>` and clear what it flags.
+
+Only work the changed sources — incremental compilation is what keeps a large
+layer maintainable.
+
+### 4. Use it (read-first / write-back)
+Before an agent does a task, load context:
+```bash
+python3 scripts/kl_query.py "renewal terms for our SaaS vendors" <root> --print
+```
+The agent reads the returned entries first, acts, then writes durable new
+decisions back as fresh source notes → recompile. That read-first / write-back
+loop is what makes the layer compound.
+
+### 5. Keep it healthy
+Run `kl_status.py <root>` on a cadence. When it flags drift (orphan links,
+missing provenance, a growing contradiction backlog), do a consolidation pass
+per **`references/wiki-principles.md`** — merge duplicates, prune dead
+cross-refs, re-summarize.
+
+## Core rules (see references for detail)
+
+- **No RAG, no vectors** — the structure is the index.
+- **Provenance on every claim** — an unsourced fact is not allowed.
+- **Surface contradictions, never resolve them silently** — a human decides.
+- **One entity per entry; narrow beats broad.**
+- **Incremental compile** — recompile only what changed.
+- **Summary-first, read-first, write-back.**
+
+## References
+
+- `references/compile-protocol.md` — the exact LLM compile procedure. Read it
+ every compile pass.
+- `references/wiki-principles.md` — the "layer you keep vs. abandon" disciplines;
+ read before a consolidation pass.
+
+## Attribution
+
+Pattern credit: Andrej Karpathy's LLM knowledge-base workflow, and Shann
+Holmberg (@shannholmberg), who popularized the "gBrain" / AI-Knowledge-Layer
+framing this skill is built from.
diff --git a/knowledge-layer/assets/template/CONTRADICTIONS.md b/knowledge-layer/assets/template/CONTRADICTIONS.md
new file mode 100644
index 0000000..731a88c
--- /dev/null
+++ b/knowledge-layer/assets/template/CONTRADICTIONS.md
@@ -0,0 +1,24 @@
+# Contradictions
+
+When a new source disagrees with an existing entry, **do not silently
+overwrite**. Record both claims here with their sources and leave the item
+**open** (`- [ ]`) until a human resolves it. `kl_status.py` counts open items.
+
+This log is what makes the wiki trustworthy: a knowledge layer that quietly
+picks a winner teaches agents to state false certainty.
+
+_Last updated: {{DATE}}_
+
+## Open
+
+<!-- template:
+- [ ] **[[acme-corp]] renewal date** — `discovery-call-2026-05-01` says "renews
+ in Q3"; `contract.pdf` says "12-month term from 2026-04-01" (→ Q1 2027).
+ Both cited on the entry; needs human confirmation.
+-->
+
+_(none)_
+
+## Resolved
+
+<!-- Move items here as `- [x]` with the resolution + who decided. -->
diff --git a/knowledge-layer/assets/template/INDEX.md b/knowledge-layer/assets/template/INDEX.md
new file mode 100644
index 0000000..618b34e
--- /dev/null
+++ b/knowledge-layer/assets/template/INDEX.md
@@ -0,0 +1,25 @@
+# Index
+
+The rolled-up map of this Knowledge Layer. The compile agent keeps this current
+so a reader (human or agent) can orient in one screen before drilling into
+entries. Keep it terse — one line per entry, grouped by folder.
+
+_Last updated: {{DATE}} (freshly initialized — empty until first compile)._
+
+## How to read this
+- Each line: `[[slug]]` — one-line summary.
+- To load context for a task, prefer `kl_query.py "<topic>"`; use this index
+ for a birds-eye scan.
+
+## Entries by folder
+
+_(none yet — drop material into `sources/`, run `kl_scan.py`, then compile per
+`references/compile-protocol.md`.)_
+
+<!-- Example once populated:
+### companies/
+- [[acme-corp]] — SaaS vendor onboarded 2026-05; contact [[jane-doe]].
+
+### people/
+- [[jane-doe]] — CEO of [[acme-corp]]; owns renewal decision.
+-->
diff --git a/knowledge-layer/assets/template/SCHEMA.md b/knowledge-layer/assets/template/SCHEMA.md
new file mode 100644
index 0000000..d7aa053
--- /dev/null
+++ b/knowledge-layer/assets/template/SCHEMA.md
@@ -0,0 +1,64 @@
+# Schema
+
+This Knowledge Layer is a plain-directory, LLM-compiled wiki. The **structure is
+the index** — there is no vector database. Agents read the relevant entries
+first, then act.
+
+_Initialized: {{DATE}}_
+
+## Typed folders
+
+Every entry lives in exactly one typed folder: **{{FOLDERS}}**.
+
+One entity/concept per file. Narrow scope beats a sprawling page — a small,
+sharp entry is easier to keep correct than a giant one.
+
+## Entry file format
+
+`<folder>/<slug>.md`, kebab-case slug, YAML frontmatter + markdown body:
+
+```markdown
+---
+id: acme-corp
+type: company
+tags: [vendor, saas, active]
+aliases: ["ACME", "Acme Inc."]
+sources: [sources/discovery-call-2026-05-01.md, sources/contract.pdf]
+updated: 2026-05-02
+---
+# ACME Corp
+
+**Summary:** SaaS vendor we onboarded May 2026; primary contact [[jane-doe]].
+
+## Facts
+- Founded 2019, HQ Lisbon. <!-- src: discovery-call-2026-05-01 -->
+- Net-30 terms, 15% partner discount. <!-- src: contract -->
+
+## Open questions
+- Renewal date not yet confirmed.
+
+## Related
+- [[jane-doe]] — CEO / our contact
+- [[onboarding-playbook]]
+```
+
+### Frontmatter fields
+| field | required | meaning |
+|-------|----------|---------|
+| `id` | yes | stable slug, matches filename |
+| `type` | yes | one of the typed folders |
+| `tags` | recommended | lowercase facets for retrieval/filtering |
+| `aliases` | recommended | alternate names → boosts `kl_query.py` recall |
+| `sources` | **yes** | provenance: which `sources/` files back this entry |
+| `updated` | yes | ISO date of last compile touch |
+
+### Body conventions
+- `**Summary:**` one line — agents read this first, drill down on demand.
+- Per-fact provenance with an inline `<!-- src: <source-stem> -->` comment.
+- Cross-reference related entries with `[[slug]]` wikilinks.
+- Contradictions do NOT get resolved silently here — log them in
+ `CONTRADICTIONS.md` and cite both sources.
+
+## Evolving the schema
+Add a new typed folder only when a real category emerges (not speculatively).
+Record the addition here so the compile agent routes future material correctly.
diff --git a/knowledge-layer/assets/template/SOURCES.md b/knowledge-layer/assets/template/SOURCES.md
new file mode 100644
index 0000000..69f1283
--- /dev/null
+++ b/knowledge-layer/assets/template/SOURCES.md
@@ -0,0 +1,22 @@
+# Sources
+
+The ledger of raw material this wiki was compiled from. Everything under
+`sources/` is an input; every entry cites the sources it derives from. This
+file is a human-readable roster — the authoritative compile state lives in
+`.kl/manifest.json` (managed by `kl_scan.py`).
+
+_Last updated: {{DATE}}_
+
+## Roster
+
+| source file | what it is | added |
+|-------------|------------|-------|
+| _(none yet)_ | | |
+
+## Curation notes
+- Sources are curated by a human — quality in, quality out. The LLM compiles;
+ it does not decide what is worth ingesting.
+- Prefer primary material (transcripts, docs, contracts, real chats) over
+ second-hand summaries.
+- Never delete a source that entries still cite without running a compile pass
+ to re-home or retire those claims (`kl_scan.py` flags REMOVED sources).
diff --git a/knowledge-layer/references/compile-protocol.md b/knowledge-layer/references/compile-protocol.md
new file mode 100644
index 0000000..23509a3
--- /dev/null
+++ b/knowledge-layer/references/compile-protocol.md
@@ -0,0 +1,96 @@
+# Compile Protocol
+
+This is the exact procedure Claude follows to compile `sources/` into wiki
+entries. It is the LLM half of the skill — the scripts prepare and verify; the
+reasoning here is what turns raw material into a durable, cross-referenced,
+contradiction-aware knowledge layer.
+
+This is the Karpathy pattern: **read the sources once, compile a structured
+wiki** (summaries, cross-references, contradictions) that the LLM maintains —
+instead of re-reading everything through RAG on every question.
+
+## Preconditions
+1. The layer exists (`kl_init.py` has run).
+2. New/changed material is in `sources/`.
+3. You have run `kl_scan.py <root>` and have the NEW / CHANGED / REMOVED lists.
+
+Only work the sources in that work-list. Do **not** recompile untouched
+entries — incremental compilation is what keeps a large layer maintainable.
+
+## The pass
+
+### 1. Read the work-list sources fully
+Read each NEW/CHANGED source end to end before writing anything. For large
+sources, read in sections but hold the whole thing in mind before routing —
+scattered half-reads produce duplicate, shallow entries.
+
+### 2. Extract atomic units
+From each source pull: entities (people, companies), concepts/terms, decisions,
+claims/facts, open questions. Keep each unit atomic — one assertion — so it can
+be provenance-tagged and later contradicted independently.
+
+### 3. Route to typed entries (one entity per file)
+For each entity/concept, find or create `<folder>/<slug>.md`:
+- **Exists?** Merge — update in place, append new facts, refresh `**Summary:**`
+ and `updated`. Never fork a second entry for the same entity (see dedup below).
+- **New?** Create it from the SCHEMA format: frontmatter (`id, type, tags,
+ aliases, sources, updated`) + `**Summary:**` + `## Facts` + `## Open questions`
+ + `## Related`.
+
+### 4. Provenance on every claim
+Every fact carries an inline `<!-- src: <source-stem> -->` and the source path
+appears in the entry's `sources:` frontmatter list. A claim with no source is
+not allowed — `kl_status.py` flags entries missing provenance.
+
+### 5. Cross-reference
+Link related entries with `[[slug]]` in the `## Related` section and inline
+where natural. Cross-references are the graph that makes retrieval work without
+vectors — a person links to their company, a project to its decisions.
+
+### 6. Detect contradictions — never silently overwrite
+Before overwriting a fact, check whether the new source *disagrees* with the
+existing one. If so:
+- Keep **both** claims on the entry, each with its own `<!-- src -->`.
+- Add an **open** item to `CONTRADICTIONS.md` citing both sources.
+- Do not pick a winner — a human resolves it. Stating false certainty is the
+ failure mode this whole pattern exists to prevent.
+
+### 7. Handle REMOVED sources
+For each removed source, find entries citing it. Drop claims that depended on it,
+or mark them stale if still plausible, and update `sources:` lists. An entry left
+with zero sources should be deleted or explicitly marked speculative in `ideas/`.
+
+### 8. Roll up
+- Update `INDEX.md`: one line per entry, grouped by folder, with the summary.
+- Update `SOURCES.md` roster with any new source rows.
+- Update `SCHEMA.md` only if a genuinely new typed folder was needed.
+
+### 9. Commit the manifest
+Run `kl_scan.py <root> --commit`. This marks the current sources as compiled so
+the next scan shows a clean slate until something changes. Do this ONLY after
+the entries are actually written — the manifest is a promise that the work is done.
+
+### 10. Verify
+Run `kl_status.py <root>`. Resolve what it surfaces:
+- orphan `[[links]]` → create the missing entry or fix the link,
+- no-provenance entries → add sources,
+- open contradictions → leave for human, but make sure they're logged.
+
+## Dedup discipline
+Before creating an entry, search existing slugs and aliases for the same
+entity under other names ("ACME" vs "Acme Inc." vs "acme-corp"). Merge into one
+canonical entry and add the alternates to `aliases:` — duplicate entities are
+the #1 way a layer rots.
+
+## Cost
+This pass is **$0** beyond the model tokens you're already spending as Claude.
+For bulk, low-stakes enrichment (e.g. tagging hundreds of media notes), a local
+Ollama model can pre-draft entries for you to verify — keep frontier-model
+tokens for the judgment steps (routing, contradiction detection, summaries).
+
+## What NOT to do
+- Don't build a vector store or embed anything — the structure is the index.
+- Don't recompile the whole layer for a one-source change.
+- Don't dump a source's raw text into an entry — compile it into atomic,
+ cited facts.
+- Don't resolve contradictions on the model's own authority.
diff --git a/knowledge-layer/references/wiki-principles.md b/knowledge-layer/references/wiki-principles.md
new file mode 100644
index 0000000..bb03f6b
--- /dev/null
+++ b/knowledge-layer/references/wiki-principles.md
@@ -0,0 +1,64 @@
+# Wiki Principles — a layer you keep vs. one you abandon
+
+A knowledge layer earns its keep only if it stays trustworthy as it grows. These
+are the disciplines that separate a wiki agents rely on from one that rots into
+noise and gets abandoned. Read this before a consolidation pass or when the
+layer feels stale.
+
+## The principles
+
+1. **Provenance or it didn't happen.** Every claim links to a source. An
+ unfalsifiable fact can't be checked, updated, or retired — it becomes a
+ liability. `kl_status.py` flags entries with no `sources:`.
+
+2. **Surface contradictions, never resolve them silently.** When sources
+ disagree, keep both and log it. A layer that quietly picks a winner trains
+ agents to speak with false certainty — the exact failure this pattern exists
+ to prevent.
+
+3. **Compile incrementally.** Recompile only what changed (`kl_scan.py` tells
+ you what). Full recompiles are for schema migrations, not routine updates —
+ they're expensive and they churn stable entries.
+
+4. **One entity per entry; narrow beats broad.** A sharp, small entry is easy to
+ keep correct. A sprawling page accumulates drift and duplication. Just as a
+ narrow-scope *agent* outperforms a vague one, a narrow-scope *entry* stays
+ truer than a catch-all.
+
+5. **Structure is the index.** Typed folders + tags + `[[wikilinks]]` are the
+ retrieval mechanism. No vectors, no RAG — the shape of the layer is what makes
+ `kl_query.py` work. Invest in the schema; let it evolve only when a real new
+ category appears.
+
+6. **Summary-first.** Every entry opens with a one-line `**Summary:**`. Agents
+ read summaries to orient and drill into detail only when needed — this keeps
+ context loads small and cheap.
+
+7. **Read-first, write-back.** Agents load the relevant entries BEFORE acting
+ (so they wake up with full context) and write durable decisions BACK after
+ (so the layer compounds). A layer that's only ever read goes stale; one that's
+ only ever written becomes a junk drawer. Both directions, every task.
+
+8. **Consolidate on a cadence.** Periodically: merge duplicate entities, prune
+ dead cross-refs, re-summarize bloated entries, and clear resolved
+ contradictions. Run `kl_status.py`; treat every flag as a to-do. Curation is
+ the maintenance cost — pay it or the layer decays.
+
+## Curation is a human job
+The LLM compiles; a human decides *what is worth ingesting*. Garbage sources
+produce a confident garbage wiki. Keep `sources/` curated — primary material
+over second-hand summaries, signal over volume.
+
+## When to do a consolidation pass
+- `kl_status.py` shows orphan cross-refs, no-provenance entries, or a growing
+ contradiction backlog.
+- Entry count in one folder balloons (likely duplicates or too-broad entries).
+- Before onboarding a new agent to the layer — a clean layer teaches clean habits.
+
+## Why no RAG / vector DB
+The Karpathy insight: for a curated, human-sized knowledge base, compiling
+sources once into a structured wiki the LLM maintains beats re-retrieving raw
+chunks on every query. You get summaries, cross-references, and contradiction
+handling that a similarity search can't give you — and zero infrastructure. If a
+layer genuinely outgrows this (hundreds of thousands of entries), that's the
+signal to add retrieval tooling on top, not to abandon the structure.
diff --git a/knowledge-layer/scripts/kl_init.py b/knowledge-layer/scripts/kl_init.py
new file mode 100755
index 0000000..4220187
--- /dev/null
+++ b/knowledge-layer/scripts/kl_init.py
@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+"""
+kl_init.py — scaffold a new Knowledge Layer instance.
+
+A Knowledge Layer is a plain-directory, LLM-compiled wiki (the Karpathy /
+gBrain pattern): raw sources go in `sources/`, the LLM compiles them into
+typed entry folders, and agents read the wiki FIRST before any task. No RAG,
+no vector DB — the structure IS the index.
+
+Usage:
+ kl_init.py [<path>] [--force] [--folders people,companies,concepts,...]
+
+Defaults:
+ <path> ./knowledge-layer
+ folders people,companies,concepts,ideas,projects,operations,media
+
+Creates:
+ <path>/
+ SCHEMA.md INDEX.md CONTRADICTIONS.md SOURCES.md
+ .kl/manifest.json (source content-hash state; do not edit by hand)
+ sources/ (drop raw material here)
+ <each typed folder>/_about.md
+
+Exit codes: 0 ok, 1 refused (exists, no --force), 2 bad args.
+"""
+import argparse
+import json
+import shutil
+import sys
+from datetime import date
+from pathlib import Path
+
+DEFAULT_FOLDERS = ["people", "companies", "concepts", "ideas",
+ "projects", "operations", "media"]
+
+FOLDER_ABOUT = {
+ "people": "One entry per person. Aliases, role, affiliations, quotes, decisions they made.",
+ "companies": "One entry per organization/vendor. What they do, our relationship, key facts.",
+ "concepts": "One entry per idea/term/framework worth a durable definition.",
+ "ideas": "Proposals and hypotheses (not yet decided). Link to the project once acted on.",
+ "projects": "One entry per initiative. Goal, status, decisions, open questions.",
+ "operations": "How-we-do-it: workflows, playbooks, standing rules, gotchas.",
+ "media": "Notes on assets/links/videos/threads — the durable takeaway, not the raw file.",
+}
+
+
+def template_dir() -> Path:
+ return Path(__file__).resolve().parent.parent / "assets" / "template"
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description="Scaffold a Knowledge Layer instance.")
+ ap.add_argument("path", nargs="?", default="knowledge-layer",
+ help="target directory (default: ./knowledge-layer)")
+ ap.add_argument("--force", action="store_true",
+ help="proceed even if the directory already exists")
+ ap.add_argument("--folders", default=",".join(DEFAULT_FOLDERS),
+ help="comma-separated typed folder names")
+ args = ap.parse_args()
+
+ root = Path(args.path).resolve()
+ folders = [f.strip() for f in args.folders.split(",") if f.strip()]
+
+ if root.exists() and any(root.iterdir()) and not args.force:
+ print(f"REFUSED: {root} exists and is not empty. Re-run with --force to add "
+ f"missing scaffolding without overwriting your entries.")
+ return 1
+
+ tdir = template_dir()
+ if not tdir.is_dir():
+ print(f"ERROR: template dir missing at {tdir}", file=sys.stderr)
+ return 2
+
+ root.mkdir(parents=True, exist_ok=True)
+ (root / ".kl").mkdir(exist_ok=True)
+ (root / "sources").mkdir(exist_ok=True)
+
+ today = date.today().isoformat()
+
+ # Seed the four root docs (never clobber an existing one).
+ for name in ("SCHEMA.md", "INDEX.md", "CONTRADICTIONS.md", "SOURCES.md"):
+ dst = root / name
+ if dst.exists():
+ print(f"kept {name} (already present)")
+ continue
+ text = (tdir / name).read_text()
+ text = text.replace("{{DATE}}", today).replace(
+ "{{FOLDERS}}", ", ".join(folders))
+ dst.write_text(text)
+ print(f"create {name}")
+
+ # Typed folders, each with an _about.md so git tracks the empty folder and
+ # the compile agent knows what belongs where.
+ for f in folders:
+ d = root / f
+ d.mkdir(exist_ok=True)
+ about = d / "_about.md"
+ if not about.exists():
+ desc = FOLDER_ABOUT.get(f, "One entry per item in this category.")
+ about.write_text(f"# {f}/\n\n{desc}\n\n"
+ f"Entry files here are `<slug>.md` with YAML "
+ f"frontmatter (see ../SCHEMA.md).\n")
+ print(f"folder {f}/")
+
+ # Fresh manifest if none.
+ manifest = root / ".kl" / "manifest.json"
+ if not manifest.exists():
+ manifest.write_text(json.dumps(
+ {"version": 1, "compiled_at": None, "sources": {}}, indent=2) + "\n")
+ print("create .kl/manifest.json")
+
+ print(f"\nKnowledge Layer ready at: {root}")
+ print("Next: drop raw material into sources/, then run kl_scan.py to see the "
+ "compile work-list, then follow references/compile-protocol.md.")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/knowledge-layer/scripts/kl_query.py b/knowledge-layer/scripts/kl_query.py
new file mode 100755
index 0000000..3c0f2f0
--- /dev/null
+++ b/knowledge-layer/scripts/kl_query.py
@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+"""
+kl_query.py — read-first retrieval for a Knowledge Layer (NO vectors, NO RAG).
+
+Given a question or topic, return the handful of wiki entry files an agent
+should LOAD before doing its task — so it "wakes up with full context." Ranking
+is deterministic keyword overlap over each entry's title, aliases, tags, and
+summary (the frontmatter + first lines), plus a small bonus for [[wikilink]]
+neighbours of top hits. It prints paths (and optionally the entries), never a
+guess — you read the files it names.
+
+Usage:
+ kl_query.py "<question or topic>" [<kl-root>] [-n N] [--print] [--json]
+
+Options:
+ -n N max entries to return (default 6)
+ --print also print each entry's body (for direct context loading)
+ --json machine-readable {path, score, why}[]
+
+Exit codes: 0 = hits, 1 = no hits, 2 = error.
+"""
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+
+STOP = set("the a an of to and or for in on at by with from is are be as it "
+ "this that these those we you i our your their his her its what how "
+ "why when who whom which about into over under than then".split())
+ENTRY_DIRS_SKIP = {".kl", "sources"}
+FM_RE = re.compile(r"^---\s*$")
+
+
+def tokenize(s: str):
+ return [t for t in re.findall(r"[a-z0-9]+", s.lower()) if t and t not in STOP]
+
+
+def parse_entry(p: Path):
+ """Return (title, aliases[], tags[], summary, body, wikilinks[])."""
+ text = p.read_text(errors="replace")
+ lines = text.splitlines()
+ fm = {}
+ body_start = 0
+ if lines and FM_RE.match(lines[0]):
+ for i in range(1, len(lines)):
+ if FM_RE.match(lines[i]):
+ body_start = i + 1
+ break
+ m = re.match(r"([A-Za-z_]+):\s*(.*)$", lines[i])
+ if m:
+ fm[m.group(1).lower()] = m.group(2).strip()
+ body = "\n".join(lines[body_start:])
+
+ def listify(v):
+ if not v:
+ return []
+ v = v.strip().strip("[]")
+ return [x.strip().strip('"').strip("'") for x in v.split(",") if x.strip()]
+
+ title = ""
+ mtitle = re.search(r"^#\s+(.+)$", body, re.M)
+ if mtitle:
+ title = mtitle.group(1).strip()
+ if not title:
+ title = p.stem.replace("-", " ")
+
+ summ = ""
+ msum = re.search(r"\*\*Summary:\*\*\s*(.+)", body)
+ if msum:
+ summ = msum.group(1).strip()
+
+ wikilinks = re.findall(r"\[\[([^\]]+)\]\]", body)
+ return {
+ "title": title,
+ "aliases": listify(fm.get("aliases", "")),
+ "tags": listify(fm.get("tags", "")),
+ "type": fm.get("type", p.parent.name),
+ "summary": summ,
+ "body": body,
+ "links": [w.strip().lower() for w in wikilinks],
+ }
+
+
+def collect_entries(root: Path):
+ out = {}
+ for p in sorted(root.rglob("*.md")):
+ rel = p.relative_to(root)
+ top = rel.parts[0]
+ if top in ENTRY_DIRS_SKIP:
+ continue
+ if p.name in ("SCHEMA.md", "INDEX.md", "CONTRADICTIONS.md",
+ "SOURCES.md", "_about.md"):
+ continue
+ out[p] = parse_entry(p)
+ return out
+
+
+def score(qtokens, e):
+ hay_strong = " ".join([e["title"]] + e["aliases"] + e["tags"]).lower()
+ strong = set(tokenize(hay_strong))
+ weak = set(tokenize(e["summary"] + " " + e["body"][:600]))
+ s, why = 0, []
+ for t in qtokens:
+ if t in strong:
+ s += 3
+ why.append(t)
+ elif t in weak:
+ s += 1
+ return s, why
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description="Read-first retrieval (no vectors).")
+ ap.add_argument("query")
+ ap.add_argument("root", nargs="?", default="knowledge-layer")
+ ap.add_argument("-n", type=int, default=6)
+ ap.add_argument("--print", dest="show", action="store_true")
+ ap.add_argument("--json", action="store_true")
+ args = ap.parse_args()
+
+ root = Path(args.root).resolve()
+ if not (root / ".kl").is_dir():
+ print(f"ERROR: {root} is not a Knowledge Layer (run kl_init.py first).",
+ file=sys.stderr)
+ return 2
+
+ qtokens = tokenize(args.query)
+ entries = collect_entries(root)
+ scored = []
+ stem_to_path = {p.stem.lower(): p for p in entries}
+ for p, e in entries.items():
+ s, why = score(qtokens, e)
+ if s > 0:
+ scored.append([p, e, s, why])
+
+ # neighbour bonus: entries linked from a top hit get a small nudge.
+ top_paths = {p for p, *_ in sorted(scored, key=lambda x: -x[2])[:args.n]}
+ for p, e, s, why in scored:
+ for lk in e["links"]:
+ if lk in stem_to_path and stem_to_path[lk] in top_paths:
+ s += 1
+ scored.sort(key=lambda x: -x[2])
+ scored = scored[:args.n]
+
+ if not scored:
+ print("No matching entries. The knowledge layer may not cover this yet — "
+ "add a source to sources/ and compile.", file=sys.stderr)
+ return 1
+
+ if args.json:
+ print(json.dumps([{
+ "path": str(p.relative_to(root)),
+ "score": s, "why": why, "summary": e["summary"]
+ } for p, e, s, why in scored], indent=2))
+ return 0
+
+ print(f"Read these {len(scored)} entries first (query: {args.query!r}):\n")
+ for p, e, s, why in scored:
+ rel = p.relative_to(root)
+ tag = f"[{e['type']}]"
+ print(f" {s:>3} {rel} {tag}")
+ if e["summary"]:
+ print(f" {e['summary']}")
+ if args.show:
+ print("\n" + "=" * 60)
+ for p, e, s, why in scored:
+ print(f"\n### {p.relative_to(root)}\n")
+ print(e["body"].strip())
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/knowledge-layer/scripts/kl_scan.py b/knowledge-layer/scripts/kl_scan.py
new file mode 100755
index 0000000..dbed3b2
--- /dev/null
+++ b/knowledge-layer/scripts/kl_scan.py
@@ -0,0 +1,131 @@
+#!/usr/bin/env python3
+"""
+kl_scan.py — incremental compile work-list for a Knowledge Layer.
+
+The whole point of the Karpathy/gBrain pattern is that you DON'T recompile the
+whole wiki every time. This script content-hashes everything under `sources/`,
+diffs against `.kl/manifest.json`, and prints exactly what changed so the
+compile agent only touches NEW / CHANGED sources (and knows which entries a
+REMOVED source used to support).
+
+Usage:
+ kl_scan.py [<kl-root>] # show the work-list (read-only)
+ kl_scan.py [<kl-root>] --json # same, machine-readable
+ kl_scan.py [<kl-root>] --commit # mark current sources as compiled
+
+Run WITHOUT --commit before a compile pass (to get the work-list) and WITH
+--commit only after you have finished writing the wiki entries for those
+sources — otherwise the manifest will claim work is done that isn't.
+
+Exit codes: 0 = nothing to do, 10 = there is pending work, 2 = error.
+"""
+import argparse
+import hashlib
+import json
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+TEXT_SUFFIXES = {".md", ".markdown", ".txt", ".rst", ".org", ".json",
+ ".yaml", ".yml", ".csv", ".html", ".htm", ".vtt", ".srt"}
+
+
+def sha256(p: Path) -> str:
+ h = hashlib.sha256()
+ with p.open("rb") as fh:
+ for chunk in iter(lambda: fh.read(65536), b""):
+ h.update(chunk)
+ return h.hexdigest()
+
+
+def scan_sources(root: Path) -> dict:
+ src = root / "sources"
+ out = {}
+ if not src.is_dir():
+ return out
+ for p in sorted(src.rglob("*")):
+ if not p.is_file() or p.name.startswith("."):
+ continue
+ if p.suffix.lower() not in TEXT_SUFFIXES:
+ # still track binaries by hash so their removal is noticed
+ pass
+ rel = p.relative_to(root).as_posix()
+ st = p.stat()
+ out[rel] = {"hash": sha256(p), "bytes": st.st_size}
+ return out
+
+
+def load_manifest(root: Path) -> dict:
+ m = root / ".kl" / "manifest.json"
+ if m.exists():
+ try:
+ return json.loads(m.read_text())
+ except json.JSONDecodeError:
+ pass
+ return {"version": 1, "compiled_at": None, "sources": {}}
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description="Incremental compile work-list.")
+ ap.add_argument("root", nargs="?", default="knowledge-layer")
+ ap.add_argument("--json", action="store_true", help="machine-readable output")
+ ap.add_argument("--commit", action="store_true",
+ help="record current sources as compiled")
+ args = ap.parse_args()
+
+ root = Path(args.root).resolve()
+ if not (root / ".kl").is_dir():
+ print(f"ERROR: {root} is not a Knowledge Layer (run kl_init.py first).",
+ file=sys.stderr)
+ return 2
+
+ manifest = load_manifest(root)
+ prev = manifest.get("sources", {})
+ now = scan_sources(root)
+
+ new = [k for k in now if k not in prev]
+ changed = [k for k in now if k in prev and now[k]["hash"] != prev[k]["hash"]]
+ removed = [k for k in prev if k not in now]
+
+ if args.commit:
+ manifest["sources"] = now
+ manifest["compiled_at"] = datetime.now(timezone.utc).isoformat()
+ (root / ".kl" / "manifest.json").write_text(
+ json.dumps(manifest, indent=2) + "\n")
+ print(f"committed: {len(now)} sources marked compiled at "
+ f"{manifest['compiled_at']}")
+ return 0
+
+ pending = len(new) + len(changed) + len(removed)
+ if args.json:
+ print(json.dumps({"new": new, "changed": changed, "removed": removed,
+ "total_sources": len(now),
+ "last_compiled": manifest.get("compiled_at")}, indent=2))
+ else:
+ print(f"Knowledge Layer: {root}")
+ print(f"last compiled: {manifest.get('compiled_at') or 'never'}")
+ print(f"sources: {len(now)} tracked\n")
+ if not pending:
+ print("Up to date — no NEW/CHANGED/REMOVED sources.")
+ else:
+ if new:
+ print(f"NEW ({len(new)}) — compile into typed entries:")
+ for k in new:
+ print(f" + {k}")
+ if changed:
+ print(f"CHANGED ({len(changed)}) — re-read & merge into existing "
+ f"entries (surface contradictions, don't silently overwrite):")
+ for k in changed:
+ print(f" ~ {k}")
+ if removed:
+ print(f"REMOVED ({len(removed)}) — review entries that cited these; "
+ f"drop unsupported claims or mark them stale:")
+ for k in removed:
+ print(f" - {k}")
+ print("\nAfter you finish writing entries, run with --commit.")
+
+ return 10 if pending else 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/knowledge-layer/scripts/kl_status.py b/knowledge-layer/scripts/kl_status.py
new file mode 100755
index 0000000..ce3c710
--- /dev/null
+++ b/knowledge-layer/scripts/kl_status.py
@@ -0,0 +1,161 @@
+#!/usr/bin/env python3
+"""
+kl_status.py — health report for a Knowledge Layer.
+
+Surfaces the drift that turns "a wiki you keep" into "a wiki you abandon":
+ - pending sources (not yet compiled — delegates the diff to kl_scan logic)
+ - unresolved contradictions (open items in CONTRADICTIONS.md)
+ - orphan cross-refs ([[links]] pointing at entries that don't exist)
+ - entries with no source provenance (unfalsifiable claims)
+ - entry / source counts per typed folder
+
+Usage:
+ kl_status.py [<kl-root>] [--json]
+
+Exit codes: 0 = healthy, 10 = attention needed, 2 = error.
+"""
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+
+SKIP_TOP = {".kl", "sources"}
+ROOT_DOCS = {"SCHEMA.md", "INDEX.md", "CONTRADICTIONS.md", "SOURCES.md"}
+
+
+def load_manifest(root: Path) -> dict:
+ m = root / ".kl" / "manifest.json"
+ if m.exists():
+ try:
+ return json.loads(m.read_text())
+ except json.JSONDecodeError:
+ pass
+ return {"sources": {}, "compiled_at": None}
+
+
+def count_sources(root: Path) -> int:
+ src = root / "sources"
+ if not src.is_dir():
+ return 0
+ return sum(1 for p in src.rglob("*") if p.is_file() and not p.name.startswith("."))
+
+
+def entries(root: Path):
+ for p in sorted(root.rglob("*.md")):
+ rel = p.relative_to(root)
+ if rel.parts[0] in SKIP_TOP:
+ continue
+ if p.name in ROOT_DOCS or p.name == "_about.md":
+ continue
+ yield p
+
+
+def has_provenance(text: str) -> bool:
+ m = re.search(r"^sources:\s*(.*)$", text, re.M)
+ if not m:
+ return False
+ v = m.group(1).strip().strip("[]").strip()
+ return bool(v)
+
+
+def open_contradictions(root: Path) -> int:
+ """Count unchecked task-list items (`- [ ]`), skipping anything inside an
+ HTML comment (the template ships commented-out examples)."""
+ f = root / "CONTRADICTIONS.md"
+ if not f.exists():
+ return 0
+ text = f.read_text()
+ # strip <!-- ... --> comment regions (incl. multi-line) before counting
+ text = re.sub(r"<!--.*?-->", "", text, flags=re.S)
+ n = 0
+ for line in text.splitlines():
+ if line.strip().lower().startswith("- [ ]"):
+ n += 1
+ return n
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description="Knowledge Layer health report.")
+ ap.add_argument("root", nargs="?", default="knowledge-layer")
+ ap.add_argument("--json", action="store_true")
+ args = ap.parse_args()
+
+ root = Path(args.root).resolve()
+ if not (root / ".kl").is_dir():
+ print(f"ERROR: {root} is not a Knowledge Layer (run kl_init.py first).",
+ file=sys.stderr)
+ return 2
+
+ manifest = load_manifest(root)
+ tracked = set(manifest.get("sources", {}))
+ src_now = set()
+ src_dir = root / "sources"
+ if src_dir.is_dir():
+ for p in src_dir.rglob("*"):
+ if p.is_file() and not p.name.startswith("."):
+ src_now.add(p.relative_to(root).as_posix())
+ pending = len((src_now - tracked) | (tracked - src_now))
+
+ stems = set()
+ per_folder = {}
+ no_prov = []
+ all_links = []
+ for p in entries(root):
+ stems.add(p.stem.lower())
+ folder = p.relative_to(root).parts[0]
+ per_folder[folder] = per_folder.get(folder, 0) + 1
+ text = p.read_text(errors="replace")
+ if not has_provenance(text):
+ no_prov.append(p.relative_to(root).as_posix())
+ for lk in re.findall(r"\[\[([^\]]+)\]\]", text):
+ all_links.append((p.relative_to(root).as_posix(), lk.strip().lower()))
+
+ orphans = sorted({f"{src} -> [[{lk}]]" for src, lk in all_links
+ if lk not in stems})
+ contradictions = open_contradictions(root)
+
+ report = {
+ "root": str(root),
+ "last_compiled": manifest.get("compiled_at"),
+ "sources_tracked": len(tracked),
+ "sources_present": len(src_now),
+ "sources_pending": pending,
+ "entries_total": len(stems),
+ "entries_by_folder": per_folder,
+ "open_contradictions": contradictions,
+ "orphan_crossrefs": orphans,
+ "entries_without_provenance": no_prov,
+ }
+ attention = bool(pending or contradictions or orphans or no_prov)
+
+ if args.json:
+ report["attention"] = attention
+ print(json.dumps(report, indent=2))
+ return 10 if attention else 0
+
+ print(f"Knowledge Layer: {root}")
+ print(f"last compiled : {manifest.get('compiled_at') or 'never'}")
+ print(f"sources : {len(src_now)} present, {pending} pending compile")
+ print(f"entries : {len(stems)} total")
+ for f, n in sorted(per_folder.items()):
+ print(f" {n:>4} {f}/")
+ print(f"contradictions: {contradictions} open")
+ print(f"orphan links : {len(orphans)}")
+ for o in orphans[:20]:
+ print(f" {o}")
+ if len(orphans) > 20:
+ print(f" … and {len(orphans) - 20} more")
+ print(f"no provenance : {len(no_prov)} entries")
+ for o in no_prov[:20]:
+ print(f" {o}")
+ if len(no_prov) > 20:
+ print(f" … and {len(no_prov) - 20} more")
+ print()
+ print("ATTENTION NEEDED — run a consolidation pass (see references/"
+ "wiki-principles.md)." if attention else "Healthy.")
+ return 10 if attention else 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
(oldest)
·
back to Knowledge Layer Skill
·
(newest)