← back to Cli Printing Press
feat(pipeline): add autonomous dogfood phase with 3-tier test system
aae780175eff65c9f44573f5bb81891c81b0ffad · 2026-03-24 09:03:39 -0700 · Matt Van Horn
Review phase (Phase 4) now includes autonomous dogfooding alongside
static quality checks:
- Tier 1 (always): version, doctor, help, dry-run, output modes
- Tier 2 (if creds): list/get, auth error handling, rate limits
- Tier 3 (sandbox only): create/delete roundtrip with cleanup
Also adds:
- KnownSpec struct with SandboxSafe field (petstore marked safe)
- IsSandboxSafe() function for querying sandbox status
- DogfoodTimeout field on PipelineState (default 10 min)
- Combined quality score: static (0-50) + dogfood (0-50) = 0-100
Co-Authored-By: GPT-5.4 <noreply@openai.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Files touched
A docs/plans/2026-03-24-feat-autonomous-dogfood-phase-plan.mdM internal/pipeline/discover.goM internal/pipeline/seeds.goM internal/pipeline/state.goM skills/printing-press/SKILL.md
Diff
commit aae780175eff65c9f44573f5bb81891c81b0ffad
Author: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Date: Tue Mar 24 09:03:39 2026 -0700
feat(pipeline): add autonomous dogfood phase with 3-tier test system
Review phase (Phase 4) now includes autonomous dogfooding alongside
static quality checks:
- Tier 1 (always): version, doctor, help, dry-run, output modes
- Tier 2 (if creds): list/get, auth error handling, rate limits
- Tier 3 (sandbox only): create/delete roundtrip with cleanup
Also adds:
- KnownSpec struct with SandboxSafe field (petstore marked safe)
- IsSandboxSafe() function for querying sandbox status
- DogfoodTimeout field on PipelineState (default 10 min)
- Combined quality score: static (0-50) + dogfood (0-50) = 0-100
Co-Authored-By: GPT-5.4 <noreply@openai.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
...026-03-24-feat-autonomous-dogfood-phase-plan.md | 320 +++++++++++++++++++++
internal/pipeline/discover.go | 106 +++++--
internal/pipeline/seeds.go | 72 ++++-
internal/pipeline/state.go | 25 +-
skills/printing-press/SKILL.md | 16 ++
5 files changed, 496 insertions(+), 43 deletions(-)
diff --git a/docs/plans/2026-03-24-feat-autonomous-dogfood-phase-plan.md b/docs/plans/2026-03-24-feat-autonomous-dogfood-phase-plan.md
new file mode 100644
index 00000000..0a95c953
--- /dev/null
+++ b/docs/plans/2026-03-24-feat-autonomous-dogfood-phase-plan.md
@@ -0,0 +1,320 @@
+---
+title: "Autonomous Dogfood Phase - Test Generated CLIs Against Real APIs"
+type: feat
+status: completed
+date: 2026-03-24
+---
+
+# Autonomous Dogfood Phase - Test Generated CLIs Against Real APIs
+
+## Overview
+
+The printing press pipeline generates CLIs that compile and pass 7 quality gates - but it never actually uses them. The Review phase (Phase 4) only does static analysis: help output, name quality, description checks. It never calls an API. It never discovers that auth is misconfigured, pagination doesn't work, or table output chokes on real response shapes.
+
+This plan adds autonomous dogfooding to the pipeline. After the CLI is generated and enriched, the press builds it, runs it against the real API (or a sandboxed version), captures what works and what breaks, and writes a structured dogfood report that feeds the Review phase's quality score.
+
+Modeled on the OSC dogfooding patterns (osc-newfeature pre-plan scoring, osc-work post-build gates, evidence checkpoints) adapted for unattended pipeline execution.
+
+## Problem Statement
+
+Today's pipeline:
+```
+Preflight -> Scaffold -> Enrich -> Regenerate -> Review (static) -> Ship
+```
+
+The Review phase checks syntax, not behavior. It scores names and descriptions but can't answer:
+- Does `doctor` actually report the API as reachable?
+- Does `list` return a table or crash?
+- Does auth setup work?
+- Do pagination flags fetch all pages?
+- Are exit codes correct for 401/404/429 responses?
+
+These are the bugs that dogfooding always finds (ghcrawl #19, Homebrew #21781, Discord CLI, Gmail CLI). Static analysis can't catch them. Only running the CLI as a real user would catches them.
+
+## Proposed Solution
+
+Add a **Dogfood step** inside the Review phase (Phase 4) that:
+
+1. Builds the generated CLI binary
+2. Runs a tiered test suite against the API
+3. Captures structured results (pass/fail/skip per test, real output, latency)
+4. Writes `dogfood-results.json` and appends findings to `review.md`
+5. Feeds a combined quality score (static 0-50 + dogfood 0-50 = total 0-100)
+
+This happens autonomously inside the pipeline - no human interaction needed. The test suite adapts based on what the API supports (public vs auth-required, read-only vs read-write).
+
+## Acceptance Criteria
+
+- [ ] Review phase plan seed includes dogfood units alongside static units
+- [ ] Dogfood tier system: Tier 1 (always safe), Tier 2 (read-only with auth), Tier 3 (write with test data)
+- [ ] Tier 1 tests run without any API credentials (doctor, help, version, compile)
+- [ ] Tier 2 tests run with credentials from env vars or config (list, get on safe resources)
+- [ ] Tier 3 tests run only on sandboxed/test APIs (create, update, delete with cleanup)
+- [ ] `dogfood-results.json` written with structured per-test results
+- [ ] `review.md` includes combined score (static + dogfood)
+- [ ] Dogfood results include real terminal output (not fake examples)
+- [ ] Failed dogfood tests don't block the pipeline - they lower the score and get reported
+- [ ] Budget-aware: dogfood phase has a 10-minute timeout per API
+
+## Implementation Units
+
+### Unit 1: Update Review Phase Seed Template
+
+**Files:** `internal/pipeline/seeds.go`
+
+**Approach:**
+Update the `reviewSeedTemplate` to include dogfood units after the existing static units. The seed becomes:
+
+```markdown
+## Implementation Units
+
+### Unit 1: Build the Generated CLI
+[existing - compile and check binary exists]
+
+### Unit 2: Static Quality Checks
+[existing - help, names, descriptions, scoring]
+
+### Unit 3: Dogfood Tier 1 - No Credentials Required
+Run tests that need zero configuration:
+
+a. Build the binary:
+ cd {{.OutputDir}} && go build -o {{.APIName}}-cli ./cmd/{{.APIName}}-cli
+
+b. Version check:
+ ./{{.APIName}}-cli version
+ Expected: prints version string, exit code 0
+
+c. Doctor check:
+ ./{{.APIName}}-cli doctor
+ Expected: runs without crash
+ Record: which checks pass/fail, base_url, config_path
+
+d. Help completeness:
+ ./{{.APIName}}-cli --help
+ For each top-level resource: ./{{.APIName}}-cli <resource> --help
+ Expected: non-empty output for every resource
+
+e. Dry-run a mutation:
+ Pick the first POST/PUT endpoint found. Run with --dry-run flag.
+ Expected: shows request preview, does NOT send
+
+f. Output mode flags:
+ Run any list command with --json, --plain, --quiet
+ Expected: each produces output, no crashes
+
+Record all results to dogfood-results.json.
+
+### Unit 4: Dogfood Tier 2 - Read-Only API Calls (if credentials available)
+
+Check for API credentials:
+ - Env vars matching the CLI's auth config (e.g., PETSTORE_API_KEY, STRIPE_API_KEY)
+ - Config file at ~/.config/{{.APIName}}-cli/config.toml
+
+If no credentials found:
+ - Print "Tier 2 skipped: no credentials available"
+ - Score Tier 2 as N/A (don't penalize)
+
+If credentials found:
+a. List command on the first safe resource:
+ ./{{.APIName}}-cli <resource> list --limit 5
+ Expected: returns data or empty list, not error
+ Record: response shape, latency, exit code
+
+b. Get command on a known ID (if list returned results):
+ ./{{.APIName}}-cli <resource> get <first-id-from-list>
+ Expected: returns single record
+ Record: fields present, latency
+
+c. Auth error handling:
+ Run with intentionally wrong credentials (set env var to "invalid")
+ Expected: exit code 4, not a crash or stack trace
+
+d. Rate limit behavior (if API allows):
+ Rapid-fire 5 identical requests
+ Record: any 429 responses, retry behavior
+
+### Unit 5: Dogfood Tier 3 - Write Operations (sandboxed APIs only)
+
+Only run on APIs known to have test/sandbox modes:
+ - Petstore (public test server, free writes)
+ - Stripe (test mode with sk_test_ keys)
+ - Stytch (test project)
+
+If API is not in the sandbox-safe list: skip Tier 3 entirely.
+
+If sandboxed:
+a. Create a test resource:
+ ./{{.APIName}}-cli <resource> create --name "printing-press-dogfood-test" ...
+ Expected: returns created resource with ID
+ Record: response, latency
+
+b. Read it back:
+ ./{{.APIName}}-cli <resource> get <created-id>
+ Expected: matches what was created
+
+c. Delete it (cleanup):
+ ./{{.APIName}}-cli <resource> delete <created-id>
+ Expected: exit code 0
+
+d. Verify deletion:
+ ./{{.APIName}}-cli <resource> get <created-id>
+ Expected: exit code 3 (not found)
+
+### Unit 6: Write Dogfood Results
+
+Write dogfood-results.json:
+{
+ "api": "{{.APIName}}",
+ "timestamp": "ISO8601",
+ "tiers_run": [1, 2],
+ "tiers_skipped": [3],
+ "results": [
+ {"test": "version", "tier": 1, "pass": true, "output": "petstore-cli 1.0.27", "latency_ms": 12},
+ {"test": "doctor", "tier": 1, "pass": true, "output": "...", "latency_ms": 45},
+ {"test": "list-pets", "tier": 2, "pass": true, "output": "...", "latency_ms": 230},
+ {"test": "auth-error", "tier": 2, "pass": false, "output": "panic: ...", "issue": "crashes on invalid auth instead of exit code 4"}
+ ],
+ "score": {
+ "tier1": 30,
+ "tier2": 15,
+ "tier3": 0,
+ "total": 45,
+ "max_possible": 50
+ }
+}
+
+### Unit 7: Combined Review Score
+
+Update review.md scoring to include dogfood results:
+
+Static score (0-50):
+ +10: compiles cleanly
+ +10: all help commands work
+ +10: no name quality issues
+ +10: no empty descriptions
+ +5: doctor works
+ +5: binary < 50MB
+
+Dogfood score (0-50):
+ +10: version prints correctly
+ +10: doctor reports API status
+ +10: list/get returns data (Tier 2)
+ +10: create/delete roundtrip (Tier 3)
+ +5: --dry-run works
+ +5: auth error handling correct
+
+Combined: static + dogfood = total (0-100)
+
+Grade: A (90+), B (75+), C (60+), D (40+), F (<40)
+```
+
+**Patterns to follow:** osc-newfeature dogfood scoring (complexity tiers), osc-work post-build checkpoints (gate tracker)
+
+### Unit 2: Add Sandbox-Safe API Registry
+
+**Files:** `internal/pipeline/seeds.go` or `internal/pipeline/discover.go`
+
+**Approach:**
+Add a `SandboxSafe` field to the known specs that indicates Tier 3 dogfooding is safe:
+
+```go
+type KnownSpec struct {
+ URL string
+ Source string
+ SandboxSafe bool // true = Tier 3 write tests are safe
+ TestEnvVar string // e.g., "STRIPE_TEST_KEY" for sandbox auth
+}
+```
+
+Known sandbox-safe APIs:
+- Petstore (public test server, no auth needed for writes)
+- Stripe (test mode with sk_test_ prefixed keys)
+- Stytch (test project environment)
+
+All other APIs default to Tier 2 max (read-only dogfooding).
+
+### Unit 3: Add Dogfood Timeout to Pipeline State
+
+**Files:** `internal/pipeline/state.go`
+
+**Approach:**
+Add a `DogfoodTimeout` field to `PipelineState` (default: 10 minutes). The dogfood step in Review should respect this timeout:
+
+```go
+type PipelineState struct {
+ // ... existing fields
+ DogfoodTimeout time.Duration `json:"dogfood_timeout,omitempty"` // default 10m
+}
+```
+
+The budget gate already caps the overall pipeline at 3 hours. The dogfood timeout prevents a single API test from hanging the pipeline (e.g., slow DNS, unreachable servers).
+
+### Unit 4: Update SKILL.md Workflow 4 Documentation
+
+**Files:** `skills/printing-press/SKILL.md`
+
+**Approach:**
+Update the pipeline phase table in Workflow 4 to reflect the enhanced Review phase:
+
+```markdown
+| Phase | What Happens |
+|-------|-------------|
+| 0. Preflight | Verify Go, download spec, cache conventions |
+| 1. Scaffold | Generate CLI, pass 7 quality gates |
+| 2. Enrich | Research API docs, discover missing endpoints, auth flows |
+| 3. Regenerate | Merge enrichments, regenerate, re-validate |
+| 4. Review | Static quality checks + **autonomous dogfooding against real API** |
+| 5. Ship | Build, tag, generate release notes |
+```
+
+Add a note under the table:
+```
+The Review phase dogfoods the generated CLI in three tiers:
+- Tier 1 (always): version, doctor, help, dry-run, output modes
+- Tier 2 (if credentials available): list, get, auth error handling
+- Tier 3 (sandbox APIs only): create/delete roundtrip with cleanup
+```
+
+## Scope Boundaries
+
+- Don't add a new pipeline phase (keep 6 phases) - dogfooding enhances Review, doesn't replace it
+- Don't require API credentials for Tier 1 - the pipeline must produce useful results with zero config
+- Don't make write calls to production APIs - Tier 3 is sandbox-only, explicitly gated
+- Don't block the pipeline on dogfood failures - failures lower the score, they don't stop shipping
+- Don't change the Go CLI templates - this is about testing what's generated, not changing the generator
+- Don't implement the full pipeline execution loop (that's already in the nightnight chaining plan)
+
+## Technical Considerations
+
+### Safety
+- Tier 3 writes are limited to known sandbox APIs with explicit opt-in
+- Delete/cleanup always runs even if create fails (idempotent cleanup)
+- Auth credentials are read from env vars, never stored in state.json or plans
+- dogfood-results.json may contain API response data - mark as sensitive in .gitignore
+
+### Budget
+- Dogfood timeout (10 min) prevents hanging on slow/unreachable APIs
+- Overall pipeline budget gate (3h) still applies
+- Tier 2/3 tests are skipped if no credentials - don't penalize the score for missing config
+
+### Reliability
+- Network failures during dogfooding are recorded as "skip" not "fail"
+- Timeout is per-test (30 seconds default), not per-tier
+- If the generated CLI crashes during dogfooding, that's a valid finding (recorded as fail)
+
+## Dependencies
+
+- Nightnight chaining plan (completed) - provides the pipeline execution loop
+- Compound Engineering plugin - ce:plan and ce:work run the Review phase
+- Known specs registry - provides sandbox-safe metadata per API
+
+## Sources
+
+- OSC dogfooding patterns: `~/.claude/skills/osc-newfeature/SKILL.md` (Phase 1b.5, complexity scoring)
+- OSC post-build dogfooding: `~/.claude/skills/osc-work/SKILL.md` (Step 5e, evidence capture)
+- Dogfood before PR feedback: `~/.claude/projects/-Users-mvanhorn/memory/feedback_dogfood_before_pr.md`
+- Current Review seed: `internal/pipeline/seeds.go` (reviewSeedTemplate)
+- Pipeline state: `internal/pipeline/state.go` (PipelineState, PhaseOrder)
+- Existing dogfood plans: `docs/plans/2026-03-23-fix-press-dogfood-until-steinberger-quality-plan.md`
+- Gmail dogfood gaps: `docs/plans/2026-03-24-fix-gmail-cli-dogfood-gaps-plan.md`
+- Agent-browser research: web-only tool, not applicable to CLI testing
diff --git a/internal/pipeline/discover.go b/internal/pipeline/discover.go
index 907703f7..2ebea347 100644
--- a/internal/pipeline/discover.go
+++ b/internal/pipeline/discover.go
@@ -5,25 +5,82 @@ import (
"strings"
)
+// KnownSpec holds metadata about a known API spec.
+type KnownSpec struct {
+ URL string
+ SandboxSafe bool
+}
+
// KnownSpecs maps common API names to their OpenAPI spec URLs.
-var KnownSpecs = map[string]string{
- "petstore": "https://petstore3.swagger.io/api/v3/openapi.json",
- "gmail": "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/googleapis.com/gmail/v1/openapi.yaml",
- "calendar": "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/googleapis.com/calendar/v3/openapi.yaml",
- "drive": "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/googleapis.com/drive/v3/openapi.yaml",
- "sheets": "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/googleapis.com/sheets/v4/openapi.yaml",
- "youtube": "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/googleapis.com/youtube/v3/openapi.yaml",
- "stripe": "https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json",
- "twilio": "https://raw.githubusercontent.com/twilio/twilio-oai/main/spec/json/twilio_api_v2010.json",
- "sendgrid": "https://raw.githubusercontent.com/sendgrid/sendgrid-oai/main/oai_stoplight.json",
- "github": "https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json",
- "discord": "https://raw.githubusercontent.com/discord/discord-api-spec/main/specs/openapi.json",
- "digitalocean": "https://api-engineering.nyc3.cdn.digitaloceanspaces.com/spec-ci/DigitalOcean-public.v2.yaml",
- "slack": "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/slack.com/1.7.0/openapi.yaml",
- "asana": "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/asana.com/1.0/openapi.yaml",
- "hubspot": "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/hubspot.com/crm/v3/openapi.yaml",
- "openai": "https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml",
- "anthropic": "https://raw.githubusercontent.com/anthropics/anthropic-cookbook/main/misc/anthropic.openapi.yaml",
+var KnownSpecs = map[string]KnownSpec{
+ "petstore": {
+ URL: "https://petstore3.swagger.io/api/v3/openapi.json",
+ SandboxSafe: true,
+ },
+ "gmail": {
+ URL: "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/googleapis.com/gmail/v1/openapi.yaml",
+ SandboxSafe: false,
+ },
+ "calendar": {
+ URL: "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/googleapis.com/calendar/v3/openapi.yaml",
+ SandboxSafe: false,
+ },
+ "drive": {
+ URL: "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/googleapis.com/drive/v3/openapi.yaml",
+ SandboxSafe: false,
+ },
+ "sheets": {
+ URL: "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/googleapis.com/sheets/v4/openapi.yaml",
+ SandboxSafe: false,
+ },
+ "youtube": {
+ URL: "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/googleapis.com/youtube/v3/openapi.yaml",
+ SandboxSafe: false,
+ },
+ "stripe": {
+ URL: "https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json",
+ SandboxSafe: false,
+ },
+ "twilio": {
+ URL: "https://raw.githubusercontent.com/twilio/twilio-oai/main/spec/json/twilio_api_v2010.json",
+ SandboxSafe: false,
+ },
+ "sendgrid": {
+ URL: "https://raw.githubusercontent.com/sendgrid/sendgrid-oai/main/oai_stoplight.json",
+ SandboxSafe: false,
+ },
+ "github": {
+ URL: "https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json",
+ SandboxSafe: false,
+ },
+ "discord": {
+ URL: "https://raw.githubusercontent.com/discord/discord-api-spec/main/specs/openapi.json",
+ SandboxSafe: false,
+ },
+ "digitalocean": {
+ URL: "https://api-engineering.nyc3.cdn.digitaloceanspaces.com/spec-ci/DigitalOcean-public.v2.yaml",
+ SandboxSafe: false,
+ },
+ "slack": {
+ URL: "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/slack.com/1.7.0/openapi.yaml",
+ SandboxSafe: false,
+ },
+ "asana": {
+ URL: "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/asana.com/1.0/openapi.yaml",
+ SandboxSafe: false,
+ },
+ "hubspot": {
+ URL: "https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/hubspot.com/crm/v3/openapi.yaml",
+ SandboxSafe: false,
+ },
+ "openai": {
+ URL: "https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml",
+ SandboxSafe: false,
+ },
+ "anthropic": {
+ URL: "https://raw.githubusercontent.com/anthropics/anthropic-cookbook/main/misc/anthropic.openapi.yaml",
+ SandboxSafe: false,
+ },
}
// ApisGuruPattern builds an apis-guru URL for a provider and version.
@@ -37,8 +94,8 @@ func DiscoverSpec(apiName string) (string, string, error) {
normalized := strings.ToLower(strings.TrimSpace(apiName))
// Check known specs first
- if url, ok := KnownSpecs[normalized]; ok {
- return url, "known-specs registry", nil
+ if spec, ok := KnownSpecs[normalized]; ok {
+ return spec.URL, "known-specs registry", nil
}
// Try apis-guru with common version patterns
@@ -49,3 +106,12 @@ func DiscoverSpec(apiName string) (string, string, error) {
return "", "", fmt.Errorf("could not find OpenAPI spec for %q - try providing a URL with --spec", apiName)
}
+
+// IsSandboxSafe returns true if the API is known to have a safe test/sandbox environment.
+func IsSandboxSafe(apiName string) bool {
+ normalized := strings.ToLower(strings.TrimSpace(apiName))
+ if spec, ok := KnownSpecs[normalized]; ok {
+ return spec.SandboxSafe
+ }
+ return false
+}
diff --git a/internal/pipeline/seeds.go b/internal/pipeline/seeds.go
index 6fd1fdfe..3e71afe9 100644
--- a/internal/pipeline/seeds.go
+++ b/internal/pipeline/seeds.go
@@ -171,12 +171,14 @@ date: {{now}}
# Review: {{.APIName}} CLI Quality
## Goal
-Static quality analysis of the generated CLI. No API calls.
+Static quality analysis and autonomous dogfooding of the generated CLI.
## Acceptance Criteria
-- [ ] Quality score calculated (0-100)
-- [ ] All critical checks pass
-- [ ] Issue list written to review.md
+- [ ] Static quality score calculated (0-50)
+- [ ] Dogfood score calculated (0-50)
+- [ ] Combined score written to review.md (0-100)
+- [ ] dogfood-results.json written with per-test results
+- [ ] All critical static checks pass
## Implementation Units
@@ -194,14 +196,60 @@ Static quality analysis of the generated CLI. No API calls.
- No empty descriptions on top-level resources
- No descriptions that just repeat the command name
-### Unit 4: Scoring
-- +20 points for compiles cleanly
-- +20 points for all help commands work
-- +20 points for no name quality issues
-- +20 points for no empty descriptions
-- +10 points for doctor works
-- +10 points for binary < 50MB
-- Write score and issues to {{.PipelineDir}}/review.md
+### Unit 4: Static Scoring
+- +10 compiles cleanly
+- +10 all help commands work
+- +10 no name quality issues
+- +10 no empty descriptions
+- +5 doctor works
+- +5 binary < 50MB
+
+### Unit 5: Dogfood Tier 1 - No Credentials Required
+Build the binary and run zero-config tests:
+
+a. Build: ` + "`cd {{.OutputDir}} && go build -o {{.APIName}}-cli ./cmd/{{.APIName}}-cli`" + `
+b. Version: ` + "`./{{.APIName}}-cli version`" + ` - expect exit code 0
+c. Doctor: ` + "`./{{.APIName}}-cli doctor`" + ` - expect runs without crash
+d. Dry-run: pick first POST/PUT endpoint, run with --dry-run
+e. Output modes: run any list command with --json, --plain, --quiet
+
+Record all results to dogfood-results.json.
+
+### Unit 6: Dogfood Tier 2 - Read-Only API Calls
+Check for API credentials in env vars or config file.
+
+If no credentials: print "Tier 2 skipped: no credentials" and score as N/A.
+
+If credentials found:
+a. List: ` + "`./{{.APIName}}-cli <resource> list --limit 5`" + `
+b. Get: ` + "`./{{.APIName}}-cli <resource> get <id-from-list>`" + `
+c. Auth error: run with invalid credentials, expect exit code 4
+d. Record response shape, latency, exit codes
+
+### Unit 7: Dogfood Tier 3 - Write Operations (Sandbox APIs Only)
+Only run on sandbox-safe APIs (petstore, stripe test mode, stytch test).
+If not sandbox-safe: skip entirely.
+
+If sandbox-safe:
+a. Create test resource
+b. Read it back
+c. Delete it (cleanup)
+d. Verify deletion returns exit code 3
+
+### Unit 8: Combined Scoring
+Write dogfood-results.json with per-test results.
+Write review.md with combined score:
+
+Static (0-50): from Unit 4
+Dogfood (0-50):
+ +10 version prints correctly
+ +10 doctor reports API status
+ +10 list/get returns data (Tier 2, 0 if skipped)
+ +10 create/delete roundtrip (Tier 3, 0 if skipped)
+ +5 dry-run works
+ +5 auth error handling correct
+
+Grade: A (90+), B (75+), C (60+), D (40+), F (<40)
`,
PhaseShip: `---
title: "{{.APIName}} CLI Pipeline - Phase 5: Ship"
diff --git a/internal/pipeline/state.go b/internal/pipeline/state.go
index 4e2fd79b..fee655b1 100644
--- a/internal/pipeline/state.go
+++ b/internal/pipeline/state.go
@@ -38,12 +38,13 @@ const (
// PipelineState tracks which phases are done across sessions.
type PipelineState struct {
- APIName string `json:"api_name"`
- OutputDir string `json:"output_dir"`
- StartedAt time.Time `json:"started_at"`
- Phases map[string]PhaseState `json:"phases"`
- SpecPath string `json:"spec_path,omitempty"`
- SpecURL string `json:"spec_url,omitempty"`
+ APIName string `json:"api_name"`
+ OutputDir string `json:"output_dir"`
+ StartedAt time.Time `json:"started_at"`
+ Phases map[string]PhaseState `json:"phases"`
+ SpecPath string `json:"spec_path,omitempty"`
+ SpecURL string `json:"spec_url,omitempty"`
+ DogfoodTimeout int `json:"dogfood_timeout_seconds,omitempty"` // default 600 (10 min)
}
// PhaseState tracks a single phase.
@@ -71,12 +72,14 @@ func NewState(apiName, outputDir string) *PipelineState {
PlanPath: filepath.Join(PipelineDir(apiName), fmt.Sprintf("%02d-%s-plan.md", i, name)),
}
}
- return &PipelineState{
- APIName: apiName,
- OutputDir: outputDir,
- StartedAt: time.Now(),
- Phases: phases,
+ state := &PipelineState{
+ APIName: apiName,
+ OutputDir: outputDir,
+ StartedAt: time.Now(),
+ Phases: phases,
+ DogfoodTimeout: 600, // 10 minutes default
}
+ return state
}
// Save writes state to disk.
diff --git a/skills/printing-press/SKILL.md b/skills/printing-press/SKILL.md
index cec6210b..e6cd9f14 100644
--- a/skills/printing-press/SKILL.md
+++ b/skills/printing-press/SKILL.md
@@ -208,6 +208,22 @@ Show the user the PR link and note that CI will validate the entry.
### Workflow 4: Autonomous Pipeline
+| Phase | Purpose |
+|-------|---------|
+| 0. Preflight | Validate environment and discover the OpenAPI spec |
+| 1. Scaffold | Generate the initial CLI and verify quality gates |
+| 2. Enrich | Deep-read the spec and research missing hints |
+| 3. Regenerate | Merge enrichments and re-generate the CLI |
+| 4. Review | Static quality checks + autonomous dogfooding against real API |
+| 5. Ship | Initialize git repo, commit, write report |
+
+The Review phase dogfoods the generated CLI in three tiers:
+- **Tier 1** (always): version, doctor, help, dry-run, output modes - no credentials needed
+- **Tier 2** (if credentials available): list, get, auth error handling - read-only API calls
+- **Tier 3** (sandbox APIs only): create/delete roundtrip with cleanup - write operations on safe test servers
+
+Results feed a combined quality score (static 0-50 + dogfood 0-50 = total 0-100).
+
When the user says "print <api-name>":
**Step 1: Initialize**
← 59ae5340 docs: rewrite README with full feature coverage and Steinber
·
back to Cli Printing Press
·
feat(pipeline): add 10 new APIs to known specs registry for d81e961a →