fix(skills): inline dogfood protocol steps to prevent skipping

a1096f4152450ab16197e6883cb907302a60d0d0 · 2026-04-04 20:43:03 -0700 · Trevin Chow

The dogfood phase was delegated to a reference file with "Read and
follow references/dogfood-testing.md" — easy to skip. The model
shortcut the process with ad-hoc smoke tests instead of following the
structured protocol, missing mutation testing bugs.

Now the critical steps are inline in SKILL.md:
- Step 1: Ask user for depth (mandatory AskUserQuestion, not optional)
- Step 2: Structured test list with exact format ([1/N] name / $ cmd / PASS)
- Step 3: Fix inline with CLI/machine tagging
- Step 4: Report format

The reference file is now supplementary (failure patterns, what NOT to
test) rather than the primary protocol. The model can't skip what's
inline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Files touched

M skills/printing-press/SKILL.md
M skills/printing-press/references/dogfood-testing.md

Diff

commit a1096f4152450ab16197e6883cb907302a60d0d0
Author: Trevin Chow <trevin@trevinchow.com>
Date:   Sat Apr 4 20:43:03 2026 -0700

    fix(skills): inline dogfood protocol steps to prevent skipping
    
    The dogfood phase was delegated to a reference file with "Read and
    follow references/dogfood-testing.md" — easy to skip. The model
    shortcut the process with ad-hoc smoke tests instead of following the
    structured protocol, missing mutation testing bugs.
    
    Now the critical steps are inline in SKILL.md:
    - Step 1: Ask user for depth (mandatory AskUserQuestion, not optional)
    - Step 2: Structured test list with exact format ([1/N] name / $ cmd / PASS)
    - Step 3: Fix inline with CLI/machine tagging
    - Step 4: Report format
    
    The reference file is now supplementary (failure patterns, what NOT to
    test) rather than the primary protocol. The model can't skip what's
    inline.
    
    Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 skills/printing-press/SKILL.md                     |  74 ++++++++--
 .../printing-press/references/dogfood-testing.md   | 157 ++-------------------
 2 files changed, 75 insertions(+), 156 deletions(-)

diff --git a/skills/printing-press/SKILL.md b/skills/printing-press/SKILL.md
index 86d89e5b..1d400468 100644
--- a/skills/printing-press/SKILL.md
+++ b/skills/printing-press/SKILL.md
@@ -1296,21 +1296,71 @@ The working copy remains in `$CLI_WORK_DIR` for potential future retry. Proceed
 
 ## Phase 5: Dogfood Testing
 
-**MANDATORY when an API key is available.** This is where you use the CLI as an
-actual user would — not just "does it start" but "does it produce correct, useful
-output for real workflows."
+**MANDATORY when an API key is available. Do NOT skip or shortcut this phase.**
 
-Shipcheck verified the CLI builds and commands respond. Dogfood verifies it
-actually works.
+Shipcheck verified commands start and return exit codes. Dogfood verifies the CLI
+produces correct, useful output for real workflows. These are different checks.
 
-Read and follow [references/dogfood-testing.md](references/dogfood-testing.md) for
-the complete dogfood protocol: depth selection (quick check vs full lifecycle),
-test plan execution, inline fix workflow, and reporting.
+### Step 1: Ask the user for depth
 
-**Key rules:**
-- Fix issues as you find them, not at the end. The value is discovering bugs in
-  context.
-- Note whether each fix is CLI-specific or a machine issue (feeds the retro).
+Present via `AskUserQuestion`:
+
+> "Shipcheck passed. How thoroughly should I test against the live API?"
+>
+> 1. **Quick check** — Read-only: doctor, list, sync, search, output modes (~5 min)
+> 2. **Full dogfood** — Complete lifecycle with mutations: create, modify, cancel, sync verification (~15-30 min). I'll create test entities on your account.
+> 3. **Skip testing** — Proceed to polish
+
+Do NOT proceed without asking. Do NOT substitute an ad-hoc smoke test.
+
+### Step 2: Run structured tests
+
+For each test, print the command and result in this format:
+```
+[1/N] <test name>
+  $ <command>
+  <result summary>
+  PASS / FAIL
+```
+
+**Quick check tests (always run unless skipped):**
+1. `doctor` — auth valid, API reachable
+2. 3-5 list commands (`bookings`, `event-types`, `schedules`, `slots`, `me`) — return data, not empty
+3. `sync --full` → `sql "SELECT count(*) FROM user_bookings"` — count > 0
+4. `search "<term from synced data>"` — finds results
+5. One list command with `--json`, `--select <fields>`, `--csv` — all produce correct output
+6. One transcendence command with synced data (`health`, `today --date <date>`) — produces output
+
+**Full dogfood tests (if user selected):**
+7. Create a test entity via the API (booking, record, etc.) with obviously-test data
+8. Verify creation in list command + get-by-ID
+9. Re-sync → search for test entity → verify in transcendence commands
+10. Test 2-3 mutation subcommands (reschedule, cancel, etc.) — verify via `--help` that the command path is discoverable, then execute
+11. Re-sync → verify status change in SQL and health
+12. Output fidelity: `--json` produces valid JSON, `--select` returns only those fields, `--csv` has header row
+13. Error paths: non-existent ID → meaningful error, missing required flag → help text
+
+### Step 3: Fix issues inline
+
+When a test fails, fix it immediately — do not accumulate failures. Tag each fix:
+- **CLI fix** — specific to this printed CLI
+- **Machine issue** — should be fixed in the generator (note for retro)
+
+### Step 4: Report
+
+```
+Dogfood Results: <cli-name>
+  Level: Quick Check / Full Dogfood
+  Tests: N/N passed
+  Fixes applied: M
+    - [each fix]
+  Machine issues: K
+    - [each issue for retro]
+```
+
+After dogfood, Phase 5.5 (Polish) runs automatically. See
+[references/dogfood-testing.md](references/dogfood-testing.md) for additional
+guidance on common failure patterns and what NOT to test.
 
 Write:
 
diff --git a/skills/printing-press/references/dogfood-testing.md b/skills/printing-press/references/dogfood-testing.md
index aaf429ca..274542a9 100644
--- a/skills/printing-press/references/dogfood-testing.md
+++ b/skills/printing-press/references/dogfood-testing.md
@@ -1,149 +1,10 @@
 # Dogfood Testing Reference
 
-> **When to read:** Phase 4.5 of the printing-press skill, after shipcheck passes.
-> This replaces the old Phase 5 (Optional Live Smoke) with structured testing.
+> **Supplementary reference for Phase 5.** The test protocol (steps 1-4, test
+> lists, reporting format) is inline in the main skill. This file contains
+> additional guidance: common failure patterns and what NOT to test.
 
-## Overview
-
-Dogfood testing runs the CLI against the real API as an actual user would.
-Shipcheck verifies that commands start and return exit codes. Dogfood verifies
-that they produce correct, useful output for real workflows.
-
-The user chooses the depth before testing begins. Both levels require an API key.
-
-## Depth Levels
-
-### Quick Check (5-10 min)
-
-Verifies core commands produce correct output against the live API. No mutations.
-
-**What it tests:**
-1. **Auth**: `doctor` shows valid credentials
-2. **List endpoints**: The 3-5 highest-traffic list commands return data (not empty, not error)
-3. **Sync round-trip**: `sync` → `sql "SELECT count(*) FROM user_bookings"` → count > 0
-4. **Search**: `search "<term from synced data>"` → finds the result
-5. **Output modes**: Pick one list command and test `--json`, `--select <2 fields>`, `--compact`, `--csv`
-6. **Transcendence**: At least one transcendence command produces output from synced data (e.g., `health`, `today --date <date-with-data>`)
-
-**What it skips:**
-- Mutations (create, update, delete, cancel)
-- Full booking/entity lifecycle
-- Edge cases and error paths
-
-**Pass criteria:** All 6 checks produce non-empty, correctly-formatted output. Any failure → investigate and fix before proceeding.
-
-### Full Dogfood (15-30 min)
-
-Tests a complete entity lifecycle including mutations, sync verification, and
-all output modes. Requires explicit user permission since mutations affect their account.
-
-**What it tests:**
-
-Everything in Quick Check, plus:
-
-7. **Create**: Create a test entity via the API (booking, issue, record — whatever
-   the primary mutable entity is). Use obviously-test data ("Dogfood Test User",
-   "dogfood@test.example.com").
-
-8. **Verify creation**: List command shows the new entity. Get-by-ID returns full details.
-
-9. **Sync + search**: Re-sync. Search for the test entity by name. Verify it appears
-   in transcendence commands (`today --date`, `health`, `sql`).
-
-10. **Mutate**: Test 2-3 mutation subcommands on the test entity:
-    - Update/modify (reschedule, edit, etc.)
-    - Re-sync and verify the change is reflected
-    - Cancel/delete
-    - Re-sync and verify status change
-
-11. **Output fidelity**: For each tested command, verify:
-    - `--json` produces valid JSON (pipe through `python3 -c "import json,sys; json.load(sys.stdin)"`)
-    - `--select <fields>` returns only those fields
-    - `--csv` produces CSV with header row
-    - `--compact` returns fewer fields than full output
-    - Table output (default terminal) is readable
-
-12. **Error paths**: Test 2-3 expected error cases:
-    - Get a non-existent ID → meaningful error message (not stack trace)
-    - Missing required flag → help text shown
-    - Invalid filter value → clear error
-
-13. **Incremental sync**: Run `sync` twice. Second run should be faster (fewer items fetched).
-    Verify with: `sync` → note count → create entity via API → `sync` → count increased by 1.
-
-**Pass criteria:** All 13 checks pass. Mutations successfully execute against the real API.
-Entity lifecycle completes: create → verify → mutate → sync → verify change → delete → verify deletion.
-
-## Running the Dogfood
-
-### Before starting
-
-1. API key must be available (from the API Key Gate in Phase 0.5)
-2. Shipcheck must have passed (at least `ship-with-gaps`)
-3. CLI binary must be built and working
-
-### Step 1: Ask the user
-
-Present via `AskUserQuestion`:
-
-> "Shipcheck passed. How thoroughly should I test against the live API?"
->
-> 1. **Quick check** — Read-only tests: list, sync, search, output modes (~5 min)
-> 2. **Full dogfood** — Complete lifecycle with mutations: create, modify, cancel, sync verification (~15-30 min). I'll create test entities on your account.
-> 3. **Skip testing** — Proceed to publish
-
-If the user selects "Full dogfood", confirm: "I'll create test data on your account (test bookings, test records, etc.) and clean up by cancelling/deleting them. OK to proceed?"
-
-### Step 2: Execute the test plan
-
-For each test, print the command being run and its result. Use a clear format:
-
-```
-[1/6] doctor
-  $ cal-com-pp-cli doctor
-  PASS: Config ok, API reachable, credentials valid
-
-[2/6] bookings list
-  $ cal-com-pp-cli bookings --no-cache --json | head -3
-  PASS: 3 bookings returned
-
-[3/6] sync + sql
-  $ cal-com-pp-cli sync --full
-  $ cal-com-pp-cli sql "SELECT count(*) as n FROM user_bookings"
-  PASS: 3 rows synced
-```
-
-### Step 3: Fix issues inline
-
-When a test fails:
-1. Diagnose the root cause immediately
-2. Fix it in the printed CLI
-3. Re-run the failing test
-4. Note whether the fix is CLI-specific or a machine issue (for the retro)
-
-Do NOT accumulate failures and fix them later. Fix each one as you find it.
-This is the value of dogfood — you discover issues in context and fix them
-while you understand the API's behavior.
-
-### Step 4: Report results
-
-After all tests complete, print a summary:
-
-```
-Dogfood Results: <cli-name>
-  Level: Quick Check / Full Dogfood
-  Tests: N/N passed
-  Fixes applied: M
-    - [list each fix with 1-line description]
-  Machine issues found: K
-    - [list issues for the retro]
-```
-
-After dogfood completes, the main skill's Phase 5.5 (Polish) runs
-automatically — it re-runs verify + scorecard and fixes regressions.
-Dogfood does not need its own polish step.
-
-### Common failure patterns
+## Common Failure Patterns
 
 | Symptom | Likely cause | Fix location |
 |---------|-------------|--------------|
@@ -155,11 +16,19 @@ Dogfood does not need its own polish step.
 | Mutation command requires ugly name | operationId not cleaned up | Command Use: field |
 | `<cmd> --help` shows wrong example | Example field has placeholder values | Command Example: field |
 | `me` shows "0 results" | Provenance counter assumes array | Count single objects as 1 |
+| Cancel/confirm path not discoverable | Subcommand buried under operationId group | Check `bookings --help` for Available Commands |
 
-## What NOT to test
+## What NOT to Test
 
 - Internal implementation details (store schema, migration order)
 - Performance benchmarks
 - Concurrent access
 - Edge cases that require specific account setup (team features, org hierarchy)
 - Endpoints the user doesn't have access to (org-level when user is individual)
+
+## Full Dogfood Confirmation
+
+When the user selects "Full dogfood", confirm before creating test data:
+
+> "I'll create test data on your account (test bookings, test records, etc.)
+> and clean up by cancelling/deleting them. OK to proceed?"

← 75ad9cdb feat(cli): infer API auth from spec description when securit · back to Cli Printing Press · feat(cli): non-skippable dogfood gate and deeper data pipeli 8ec4fc7e →