← back to Cli Printing Press
feat(cli): non-skippable dogfood gate and deeper data pipeline validation (#127)
8ec4fc7e86c3627f7410d1f5c7b0f3fa0d210a35 · 2026-04-04 22:19:34 -0700 · Trevin Chow
* feat(cli): non-skippable dogfood gate and deeper data pipeline validation
Make Phase 5 dogfood testing non-skippable when an API key is available
(removes contradictory "Skip testing" option), add structured acceptance
report format with pass/fail thresholds, and connect gate failure to the
existing hold mechanism.
Extend runDataPipelineTest() to discover domain tables via sqlite_master
and validate row counts in live mode. Adds runCLIWithOutput() for parsing
sql command output. Data pipeline now reports "PASS: 54 domain tables,
resources has 3 rows" instead of just "PASS".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): WARN states fail pipeline gate, parse boxed SQL count output
Address Codex review findings:
- P1: "0 rows in live mode" WARN now returns false so the pipeline gate
fails instead of silently passing. "No domain tables" stays true
(ambiguous — could be minimal CLI or unusual naming).
- P2: parseCountOutput now strips box-drawing characters (│42│) before
parsing, matching parseSQLOutput's existing handling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Files touched
A docs/plans/2026-04-04-004-feat-layered-verification-plan.mdM internal/cli/verify.goM internal/pipeline/runtime.goM internal/pipeline/runtime_test.goM skills/printing-press/SKILL.md
Diff
commit 8ec4fc7e86c3627f7410d1f5c7b0f3fa0d210a35
Author: Trevin Chow <trevin@trevinchow.com>
Date: Sat Apr 4 22:19:34 2026 -0700
feat(cli): non-skippable dogfood gate and deeper data pipeline validation (#127)
* feat(cli): non-skippable dogfood gate and deeper data pipeline validation
Make Phase 5 dogfood testing non-skippable when an API key is available
(removes contradictory "Skip testing" option), add structured acceptance
report format with pass/fail thresholds, and connect gate failure to the
existing hold mechanism.
Extend runDataPipelineTest() to discover domain tables via sqlite_master
and validate row counts in live mode. Adds runCLIWithOutput() for parsing
sql command output. Data pipeline now reports "PASS: 54 domain tables,
resources has 3 rows" instead of just "PASS".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): WARN states fail pipeline gate, parse boxed SQL count output
Address Codex review findings:
- P1: "0 rows in live mode" WARN now returns false so the pipeline gate
fails instead of silently passing. "No domain tables" stays true
(ambiguous — could be minimal CLI or unusual naming).
- P2: parseCountOutput now strips box-drawing characters (│42│) before
parsing, matching parseSQLOutput's existing handling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
...026-04-04-004-feat-layered-verification-plan.md | 213 +++++++++++++++++++++
internal/cli/verify.go | 6 +-
internal/pipeline/runtime.go | 142 ++++++++++++--
internal/pipeline/runtime_test.go | 90 +++++++++
skills/printing-press/SKILL.md | 53 ++++-
5 files changed, 476 insertions(+), 28 deletions(-)
diff --git a/docs/plans/2026-04-04-004-feat-layered-verification-plan.md b/docs/plans/2026-04-04-004-feat-layered-verification-plan.md
new file mode 100644
index 00000000..bd85b8d4
--- /dev/null
+++ b/docs/plans/2026-04-04-004-feat-layered-verification-plan.md
@@ -0,0 +1,213 @@
+---
+title: "feat: Deep verification for printed CLIs"
+type: feat
+status: completed
+date: 2026-04-04
+---
+
+# feat: Deep verification for printed CLIs
+
+## Overview
+
+Formalize the skill's dogfood testing as a non-skippable acceptance gate with structured pass/fail criteria, and extend the data pipeline test to validate that sync actually writes rows. These are the two remaining verification gaps — HTTP error detection is already handled by `classifyAPIError()` in the generated templates.
+
+## Problem Frame
+
+The skill's Phase 5 dogfood testing catches real problems (broken auth, missing data, bad response quality) but it's optional — the user can select "Skip testing." When skipped, the CLI ships with only mechanical verification (exit codes, dry-run, help text). Exit codes catch HTTP errors (401/403/404/5xx are already mapped to non-zero exit codes by `classifyAPIError()` in `helpers.go.tmpl`), but they can't validate:
+
+- **Response quality**: Does the data look correct? Does a list return actual items?
+- **Data pipeline integrity**: Did sync write rows to SQLite? Does search find results?
+- **End-to-end workflows**: Does sync → sql → search → health chain actually produce meaningful output?
+
+These require agent judgment (picking the right commands, understanding the API's data) — not mechanical parsing.
+
+### Why not stdout parsing?
+
+During planning, document review revealed that the generated CLI templates already have comprehensive HTTP error handling:
+
+- `classifyAPIError()` in `helpers.go.tmpl` maps HTTP 401→exit 4, 403→exit 4, 404→exit 3, 429→exit 7, 5xx→exit 5
+- `main.go.tmpl` calls `os.Exit(cli.ExitCode(err))` on any error
+- `sanitizeJSONResponse()` strips JSONP/XSSI prefixes (the root cause of the Redfin false-pass scenario)
+
+Verify's exit-code checking already catches these cases. Building a parallel stdout-parsing layer (regex-matching for error codes in CLI output) would be fragile, coupled to template versions, and redundant with the structured exit code mechanism the machine already controls.
+
+### Evidence from retros
+
+- **Redfin (52% verify)**: The false-pass was caused by JSONP prefix `{}&&` corrupting JSON parsing *before* the status code check ran. This has been fixed — `sanitizeJSONResponse()` now strips these prefixes.
+- **Cal.com (89% verify)**: Missing `cal-api-version` header caused 404s. Fixed by required header detection (#125). With the header present, `classifyAPIError()` correctly maps 404→exit 3.
+- **Steam (75% verify)**: False negatives from verify not passing auth env vars. Fixed in verify harness, not a template issue.
+
+### What's still missing
+
+The data pipeline test (`runDataPipelineTest()` in `runtime.go:570-597`) only checks "sync doesn't crash" — it doesn't validate that tables were created or rows were written. A sync that exits 0 with 0 rows is a silent failure that exit codes can't detect.
+
+## Requirements Trace
+
+### Dogfood Gate (Skill)
+
+- R1. The skill's Phase 5 dogfood testing becomes a formal gate — not skippable when an API key is available
+- R2. Dogfood produces a structured, machine-readable report that can be archived and compared across runs
+- R3. When the dogfood gate fails and the agent exhausts fix-loop attempts, the CLI goes on hold (connects to existing hold mechanism)
+
+### Data Pipeline Validation (Verify Command)
+
+- R4. After sync, verify checks that domain tables were created in SQLite
+- R5. In live mode, verify checks that sync wrote >0 rows to at least one domain table
+- R6. Data pipeline result reported as PASS/WARN/SKIP with detail, not just boolean
+
+## Scope Boundaries
+
+- **In scope**: Non-skippable dogfood gate, structured acceptance report, deeper data pipeline validation in verify
+- **Not in scope**: Stdout parsing / response body analysis in verify — `classifyAPIError()` already handles HTTP error detection via exit codes
+- **Not in scope**: OpenAPI schema validation of response fields
+- **Not in scope**: Mock server intelligence
+- **Not in scope**: Mutation testing in verify — mutations stay in the agent-driven dogfood
+- **Dropped from original plan**: Units 1-3 (runCLIWithOutput, analyzeOutput, response quality reporting) — invalidated by discovery that `classifyAPIError()` + `sanitizeJSONResponse()` already produce structured non-zero exit codes for HTTP errors
+
+## Context & Research
+
+### Relevant Code and Patterns
+
+- `internal/pipeline/runtime.go:570-597` — `runDataPipelineTest()`: tests sync→health chain but only checks "sync doesn't crash," not "sync wrote rows"
+- `internal/pipeline/runtime.go:600-611` — `runCLI()`: returns only error. `runCLIWithOutput()` variant needed for Unit 2 to parse sql command output
+- `internal/generator/templates/helpers.go.tmpl:141-237` — `classifyAPIError()`: comprehensive HTTP error→exit code mapping. Covers 401, 403, 404, 409, 429, 5xx.
+- `internal/generator/templates/client.go.tmpl:362` — `sanitizeJSONResponse()`: strips JSONP/XSSI prefixes before parsing (fixes the Redfin root cause)
+- `internal/generator/templates/main.go.tmpl:14-16` — `os.Exit(cli.ExitCode(err))`: ensures non-zero exit on any classified error
+- `skills/printing-press/SKILL.md:1297-1367` — Phase 5 dogfood protocol: structured test list with quick check (6 tests) and full dogfood (13 tests). Currently optional despite being marked "MANDATORY."
+
+### Institutional Learnings
+
+- **Redfin retro finding #4**: JSONP prefix caused JSON parse failure before status check. Root cause fixed by `sanitizeJSONResponse()`. Exit codes now work correctly for this class of bug.
+- **Cal.com retro finding #5**: Missing required headers fixed by header detection (#125). With headers present, `classifyAPIError()` catches 404s.
+- **Prior plans**: Two prior verification plans (2026-03-26 proof-of-behavior, 2026-03-27 runtime verification) created the current verify command. This plan narrows to the remaining gaps rather than adding another parsing layer.
+
+## Key Technical Decisions
+
+- **Don't parse stdout for error codes**: The generated CLI templates already map HTTP errors to structured exit codes. Building a parallel regex-based detection path would be fragile, coupled to template versions, and redundant. Trust the machine's own error handling.
+- **Dogfood gate is skill-enforced, not binary-enforced**: The skill already has the Phase 5 protocol. Making it non-skippable is a skill text change. The binary doesn't know if an API key was available during generation.
+- **Connect to existing hold mechanism**: When the dogfood gate fails and fix loops are exhausted, the CLI goes on hold via the existing `printing-press lock` mechanism (SKILL.md:1293-1295), not a new enforcement path.
+- **`runCLIWithOutput()` for data pipeline only**: A new function that returns stdout for parsing, but scoped to Unit 2's data pipeline validation (parsing `sql` command output), not for general response analysis.
+
+## Open Questions
+
+### Resolved During Planning
+
+- **Q: Should verify parse stdout for HTTP error codes?** No. `classifyAPIError()` in the templates already maps HTTP errors to non-zero exit codes. The Redfin false-pass was caused by JSONP prefix corruption, now fixed by `sanitizeJSONResponse()`. Stdout parsing would be redundant.
+- **Q: Should Layer 2 be a new printing-press subcommand or stay in the skill?** Stay in the skill. The value is agent judgment — picking which commands to test deeply based on the API.
+- **Q: What happens when dogfood gate fails and fix loops are exhausted?** CLI goes on hold via existing mechanism. User can retry or release the lock.
+
+### Deferred to Implementation
+
+- **Q: How should verify discover domain table names for row-count validation?** Options: parse `sqlite_master`, use the `health` command output, or query via `sql "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite%' AND name NOT LIKE '%_fts%'"`. Implementation should test which approach is most reliable across CLIs.
+
+## Implementation Units
+
+- [ ] **Unit 1: Formalize dogfood gate in skill**
+
+**Goal:** Make Phase 5 dogfood testing non-skippable when an API key is available, require a structured acceptance report, and connect to the existing hold mechanism for gate failures.
+
+**Requirements:** R1, R2, R3
+
+**Dependencies:** None
+
+**Files:**
+- Modify: `skills/printing-press/SKILL.md`
+- Modify: `skills/printing-press/references/dogfood-testing.md`
+
+**Approach:**
+- SKILL.md:1299 already declares Phase 5 "MANDATORY when an API key is available." The actual change is removing the contradictory option 3 ("Skip testing") at line 1312 and adding the structured report format.
+- Replace the three-option prompt with: "Quick check (recommended)" / "Full dogfood" — no skip option. When no API key is available, skip is automatic and documented.
+- Add a structured report format requirement:
+ ```
+ Acceptance Report: <cli-name>
+ Level: Quick Check / Full Dogfood
+ Tests: N/M passed
+ Failures: [list with command, expected, actual]
+ Gate: PASS / FAIL
+ ```
+- Define acceptance threshold: Quick Check must pass 5/6 core tests. Full Dogfood must pass 10/13. Any auth or sync failure is an automatic gate FAIL.
+- The report artifact goes to `$PROOFS_DIR/<stamp>-fix-<api>-pp-cli-acceptance.md`
+- Add gate failure path: when dogfood gate=FAIL and agent exhausts fix loops (max 2), the CLI goes on hold via existing `printing-press lock hold` mechanism. Document that the user can retry or release.
+- Phase 5.6 (Promote) checks for the acceptance artifact — if it doesn't exist and an API key was available, promotion is blocked.
+
+**Patterns to follow:**
+- `skills/printing-press/SKILL.md:1297-1367` — existing Phase 5 structure
+- `skills/printing-press/SKILL.md:1293-1295` — existing hold/lock mechanism
+
+**Test scenarios:**
+- Happy path: API key available → user sees "Quick check (recommended)" / "Full dogfood" with no skip option → tests run → acceptance report written
+- Edge case: no API key available → Phase 5 auto-skips with documented reason → no acceptance artifact → promotion still allowed
+- Error path: Quick Check fails 2/6 tests → gate=FAIL → agent fixes and retries → still fails → CLI goes on hold
+- Integration: Phase 5.6 checks for acceptance artifact presence when API key was known to be available
+
+**Verification:**
+- Reading the modified SKILL.md shows no skip option when API key is available
+- The acceptance report format is clearly specified with machine-parseable structure
+- Gate failure path connects to existing hold mechanism
+
+---
+
+- [ ] **Unit 2: Data pipeline deep test in verify**
+
+**Goal:** Extend `runDataPipelineTest()` to validate that sync actually creates tables and writes rows — not just "sync doesn't crash."
+
+**Requirements:** R4, R5, R6
+
+**Dependencies:** None
+
+**Files:**
+- Modify: `internal/pipeline/runtime.go` (the `runDataPipelineTest` function)
+- Test: `internal/pipeline/runtime_test.go`
+
+**Approach:**
+- Add `runCLIWithOutput()` — a variant of `runCLI()` that returns `(stdout []byte, error)` using `cmd.CombinedOutput()` (no need to separate stdout/stderr for this use case — we just need to parse the sql command's text output)
+- After sync completes, run `sql "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite%' AND name NOT LIKE '%_fts%' AND name != 'sync_state'"` to discover domain tables
+- Parse the output to get table names. If 0 domain tables exist, DataPipeline=WARN
+- In live mode with API key: run `sql "SELECT count(*) FROM <first-domain-table>"` for the first discovered table. If count is 0 after a full sync against a live API, DataPipeline=WARN
+- In mock mode: skip the row-count check (mock data is minimal, 0 rows expected)
+- Run `health` and check for non-empty output (exit 0 + non-empty = health works)
+- Report data pipeline result as `PASS` (tables created + rows written in live), `WARN` (tables created but 0 rows in live), `SKIP` (mock mode or no sync command), or `FAIL` (sync crashes — existing behavior preserved)
+- Update `DataPipeline` field in `VerifyReport` from `bool` to a string with the PASS/WARN/SKIP/FAIL detail
+
+**Patterns to follow:**
+- `internal/pipeline/runtime.go:570-597` — existing `runDataPipelineTest()` structure
+- `internal/pipeline/runtime.go:600-611` — existing `runCLI()` as base for `runCLIWithOutput()`
+
+**Test scenarios:**
+- Happy path (live mode): sync writes rows → sql count > 0 → DataPipeline=PASS
+- Error path (live mode): sync exits 0 but writes 0 rows → DataPipeline=WARN
+- Edge case (mock mode): sync exits 0, tables created but 0 rows → DataPipeline=PASS (expected in mock)
+- Edge case: CLI has no sync command → DataPipeline=SKIP
+- Edge case: sql command not available or fails → fall back to existing boolean check (sync doesn't crash = PASS)
+- Edge case: domain table names contain SQL reserved words (quoted identifiers) → query still works
+- Error path: sync crashes → DataPipeline=FAIL (existing behavior preserved)
+
+**Verification:**
+- `go test ./internal/pipeline/...` passes
+- Running verify on cal-com-pp-cli in live mode shows DataPipeline=PASS with table names and row count
+- Running verify in mock mode shows DataPipeline=PASS (no row-count check)
+- `VerifyReport.DataPipeline` field change is backward-compatible (consumers checking truthiness still work if the string is non-empty for PASS)
+
+## System-Wide Impact
+
+- **Interaction graph:** Unit 1 changes skill text only — no binary consumers affected. Unit 2 changes `VerifyReport.DataPipeline` from `bool` to `string` — the scorecard reads this field. The string values are truthy/falsy compatible ("PASS"/"WARN" are truthy, "" is falsy), so existing boolean consumers survive.
+- **Error propagation:** The dogfood gate adds a new blocking path — if dogfood fails and fix loops are exhausted, the CLI goes on hold. This is intentional and connects to the existing hold mechanism.
+- **State lifecycle risks:** None for Unit 2 (verify is stateless). Unit 1's acceptance artifact is a new file in the manuscripts directory — follows existing proof artifact patterns.
+- **Unchanged invariants:** Verify's per-command pass/fail scoring is unchanged. `classifyAPIError()` exit codes continue to work as before. The `VerifyReport` struct's `Total`, `Passed`, `Failed`, `PassRate`, `Verdict` fields keep their current semantics.
+
+## Risks & Dependencies
+
+| Risk | Mitigation |
+|------|------------|
+| Dogfood gate blocks shipping on API-specific issues (rate limits, permissions) | Quick Check tests are read-only and lightweight. Agent distinguishes "API limitation" vs "CLI bug" in the report. Exhausted fix loops → hold, not permanent block. |
+| `sql` command interface varies across CLIs (flags, positional args) | Use the simplest invocation: `sql "SELECT ..."` as a positional arg. Fall back to existing boolean check if sql fails. |
+| `DataPipeline` field type change (bool → string) breaks consumers | String values are truthy/falsy compatible. Add `DataPipelineDetail` as a new field if backward compatibility requires keeping `DataPipeline` as bool. |
+| Domain table discovery query may miss tables with unusual naming | The `sqlite_master` query excludes only known system tables (sqlite%, %_fts%, sync_state). Any other table is treated as a domain table. |
+
+## Sources & References
+
+- Redfin retro: `docs/retros/2026-03-30-redfin-retro.md` — finding #4 (JSONP prefix root cause, now fixed)
+- Cal.com retro: `docs/retros/2026-04-04-cal-com-retro.md` — finding #5 (required headers, now fixed)
+- Key code: `internal/pipeline/runtime.go:570-597` (`runDataPipelineTest`), `internal/generator/templates/helpers.go.tmpl:141-237` (`classifyAPIError`)
+- Existing mechanism: `skills/printing-press/SKILL.md:1293-1295` (hold/lock for gate failures)
+- Document review: 5-persona review on 2026-04-04 revealed `classifyAPIError()` invalidates stdout parsing approach, leading to this narrowed scope
diff --git a/internal/cli/verify.go b/internal/cli/verify.go
index d3ae045d..ee2ebb3c 100644
--- a/internal/cli/verify.go
+++ b/internal/cli/verify.go
@@ -142,7 +142,11 @@ func printVerifyReport(report *pipeline.VerifyReport) {
}
fmt.Println()
- fmt.Printf("Data Pipeline: %s\n", passFail(report.DataPipeline))
+ if report.DataPipelineDetail != "" {
+ fmt.Printf("Data Pipeline: %s\n", report.DataPipelineDetail)
+ } else {
+ fmt.Printf("Data Pipeline: %s\n", passFail(report.DataPipeline))
+ }
fmt.Printf("Pass Rate: %.0f%% (%d/%d passed, %d critical)\n",
report.PassRate, report.Passed, report.Total, report.Critical)
fmt.Printf("Verdict: %s\n", report.Verdict)
diff --git a/internal/pipeline/runtime.go b/internal/pipeline/runtime.go
index 0a055560..af140af9 100644
--- a/internal/pipeline/runtime.go
+++ b/internal/pipeline/runtime.go
@@ -28,16 +28,17 @@ type VerifyConfig struct {
// VerifyReport is the output of a runtime verification run.
type VerifyReport struct {
- Mode string `json:"mode"` // "live" or "mock"
- Total int `json:"total"`
- Passed int `json:"passed"`
- Failed int `json:"failed"`
- Critical int `json:"critical"`
- PassRate float64 `json:"pass_rate"`
- DataPipeline bool `json:"data_pipeline"`
- Verdict string `json:"verdict"` // PASS, WARN, FAIL
- Results []CommandResult `json:"results"`
- Binary string `json:"binary"`
+ Mode string `json:"mode"` // "live" or "mock"
+ Total int `json:"total"`
+ Passed int `json:"passed"`
+ Failed int `json:"failed"`
+ Critical int `json:"critical"`
+ PassRate float64 `json:"pass_rate"`
+ DataPipeline bool `json:"data_pipeline"`
+ DataPipelineDetail string `json:"data_pipeline_detail,omitempty"` // PASS, WARN, SKIP, FAIL with context
+ Verdict string `json:"verdict"` // PASS, WARN, FAIL
+ Results []CommandResult `json:"results"`
+ Binary string `json:"binary"`
}
// CommandResult is the test result for a single command.
@@ -180,7 +181,7 @@ func RunVerify(cfg VerifyConfig) (*VerifyReport, error) {
}
// 8. Data pipeline test
- report.DataPipeline = runDataPipelineTest(binaryPath, report.Mode, buildEnv)
+ report.DataPipeline, report.DataPipelineDetail = runDataPipelineTest(binaryPath, report.Mode, buildEnv)
// 9. Compute aggregate
for _, r := range report.Results {
@@ -568,13 +569,14 @@ func runCommandTests(binary string, cmd discoveredCommand, mode string, env []st
}
// runDataPipelineTest tests the sync -> sql -> search -> health chain.
-func runDataPipelineTest(binary, mode string, envFn func() []string) bool {
+// Returns (pass bool, detail string) where detail gives PASS/WARN/SKIP/FAIL context.
+func runDataPipelineTest(binary, mode string, envFn func() []string) (bool, string) {
env := envFn()
// Create a temp dir for the test database
tmpDir, err := os.MkdirTemp("", "verify-db-*")
if err != nil {
- return false
+ return false, "FAIL: could not create temp dir"
}
defer func() { _ = os.RemoveAll(tmpDir) }()
@@ -587,13 +589,47 @@ func runDataPipelineTest(binary, mode string, envFn func() []string) bool {
// Sync might not accept --db flag - try without
syncErr = runCLI(binary, []string{"sync", "--full"}, env, 30*time.Second)
}
+ if syncErr != nil {
+ return false, "FAIL: sync crashed"
+ }
// Test health (if available)
- healthErr := runCLI(binary, []string{"health", "--db", dbPath}, env, 10*time.Second)
- _ = healthErr
+ _ = runCLI(binary, []string{"health", "--db", dbPath}, env, 10*time.Second)
+
+ // Discover domain tables via sql command
+ tableQuery := `SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite%' AND name NOT LIKE '%_fts%' AND name != 'sync_state'`
+ tablesOut, sqlErr := runCLIWithOutput(binary, []string{"sql", tableQuery}, env, 10*time.Second)
+ if sqlErr != nil {
+ // sql command may not exist or may not accept positional args — fall back to basic check
+ return true, "PASS: sync completed (table validation skipped — sql command unavailable)"
+ }
+
+ // Parse table names from output (one per line, skip empty lines and header noise)
+ tables := parseSQLOutput(tablesOut)
+ if len(tables) == 0 {
+ // No domain tables found — ambiguous (could be minimal CLI or unusual naming).
+ // Don't fail the pipeline gate; report for human review.
+ return true, "WARN: sync completed but no domain tables found in sqlite_master"
+ }
+
+ // In live mode, check that at least one table has rows
+ if mode == "live" {
+ for _, table := range tables {
+ countQuery := fmt.Sprintf("SELECT count(*) FROM \"%s\"", table)
+ countOut, countErr := runCLIWithOutput(binary, []string{"sql", countQuery}, env, 10*time.Second)
+ if countErr != nil {
+ continue
+ }
+ count := parseCountOutput(countOut)
+ if count > 0 {
+ return true, fmt.Sprintf("PASS: %d domain tables, %s has %d rows", len(tables), table, count)
+ }
+ }
+ return false, fmt.Sprintf("WARN: %d domain tables created but 0 rows after sync (live mode)", len(tables))
+ }
- // The pipeline passes if sync doesn't crash (even if it syncs 0 rows from mock)
- return syncErr == nil
+ // Mock mode: tables created is sufficient (mock data is minimal)
+ return true, fmt.Sprintf("PASS: %d domain tables created", len(tables))
}
// runCLI executes the CLI binary with the given args and returns any error.
@@ -610,6 +646,78 @@ func runCLI(binary string, args []string, env []string, timeout time.Duration) e
return nil
}
+// runCLIWithOutput executes the CLI binary and returns its combined output.
+func runCLIWithOutput(binary string, args []string, env []string, timeout time.Duration) ([]byte, error) {
+ ctx, cancel := context.WithTimeout(context.Background(), timeout)
+ defer cancel()
+
+ cmd := exec.CommandContext(ctx, binary, args...)
+ cmd.Env = env
+ out, err := cmd.CombinedOutput()
+ if err != nil {
+ return out, fmt.Errorf("exit %v: %s", err, string(out))
+ }
+ return out, nil
+}
+
+// parseSQLOutput extracts non-empty, non-header lines from sql command output.
+func parseSQLOutput(out []byte) []string {
+ var tables []string
+ for _, line := range strings.Split(string(out), "\n") {
+ line = strings.TrimSpace(line)
+ if line == "" || line == "name" || strings.HasPrefix(line, "---") {
+ continue
+ }
+ // Skip box-drawing borders and separators
+ if strings.HasPrefix(line, "┌") || strings.HasPrefix(line, "└") || strings.HasPrefix(line, "├") {
+ continue
+ }
+ if strings.Contains(line, "───") || strings.Contains(line, "===") {
+ continue
+ }
+ // Strip box-drawing pipe characters from cell content
+ if strings.HasPrefix(line, "│") {
+ line = strings.Trim(line, "│")
+ line = strings.TrimSpace(line)
+ if line == "" || line == "name" {
+ continue
+ }
+ }
+ tables = append(tables, line)
+ }
+ return tables
+}
+
+// parseCountOutput extracts a numeric count from sql command output.
+func parseCountOutput(out []byte) int {
+ for _, line := range strings.Split(string(out), "\n") {
+ line = strings.TrimSpace(line)
+ if line == "" || line == "count(*)" || strings.HasPrefix(line, "---") {
+ continue
+ }
+ // Skip box-drawing borders and separators
+ if strings.HasPrefix(line, "┌") || strings.HasPrefix(line, "└") || strings.HasPrefix(line, "├") {
+ continue
+ }
+ if strings.Contains(line, "───") || strings.Contains(line, "===") {
+ continue
+ }
+ // Strip box-drawing pipe characters from cell content
+ if strings.HasPrefix(line, "│") {
+ line = strings.Trim(line, "│")
+ line = strings.TrimSpace(line)
+ if line == "" || line == "count(*)" {
+ continue
+ }
+ }
+ var n int
+ if _, err := fmt.Sscanf(line, "%d", &n); err == nil {
+ return n
+ }
+ }
+ return 0
+}
+
// startMockServer creates an httptest.Server from the OpenAPI spec.
func startMockServer(spec *openAPISpec) (*httptest.Server, string) {
mux := http.NewServeMux()
diff --git a/internal/pipeline/runtime_test.go b/internal/pipeline/runtime_test.go
index 1805143f..22769c74 100644
--- a/internal/pipeline/runtime_test.go
+++ b/internal/pipeline/runtime_test.go
@@ -244,3 +244,93 @@ func TestSyntheticArgValue(t *testing.T) {
})
}
}
+
+func TestParseSQLOutput(t *testing.T) {
+ tests := []struct {
+ name string
+ input string
+ expected []string
+ }{
+ {
+ name: "simple table names one per line",
+ input: "bookings\nevent_types\nschedules\n",
+ expected: []string{"bookings", "event_types", "schedules"},
+ },
+ {
+ name: "with header and separator",
+ input: "name\n---\nbookings\nevent_types\n",
+ expected: []string{"bookings", "event_types"},
+ },
+ {
+ name: "empty output",
+ input: "",
+ expected: nil,
+ },
+ {
+ name: "only whitespace and empty lines",
+ input: "\n \n\n",
+ expected: nil,
+ },
+ {
+ name: "with box-drawing characters",
+ input: "┌────────┐\n│ name │\n├────────┤\n│bookings│\n└────────┘\n",
+ expected: []string{"bookings"},
+ },
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ result := parseSQLOutput([]byte(tt.input))
+ assert.Equal(t, tt.expected, result)
+ })
+ }
+}
+
+func TestParseCountOutput(t *testing.T) {
+ tests := []struct {
+ name string
+ input string
+ expected int
+ }{
+ {
+ name: "simple count",
+ input: "42\n",
+ expected: 42,
+ },
+ {
+ name: "count with header",
+ input: "count(*)\n---\n15\n",
+ expected: 15,
+ },
+ {
+ name: "zero count",
+ input: "0\n",
+ expected: 0,
+ },
+ {
+ name: "empty output",
+ input: "",
+ expected: 0,
+ },
+ {
+ name: "non-numeric output",
+ input: "error: no such table\n",
+ expected: 0,
+ },
+ {
+ name: "box-drawn count",
+ input: "┌──────────┐\n│ count(*) │\n├──────────┤\n│ 42 │\n└──────────┘\n",
+ expected: 42,
+ },
+ {
+ name: "pipe-wrapped count no spaces",
+ input: "│15│\n",
+ expected: 15,
+ },
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ result := parseCountOutput([]byte(tt.input))
+ assert.Equal(t, tt.expected, result)
+ })
+ }
+}
diff --git a/skills/printing-press/SKILL.md b/skills/printing-press/SKILL.md
index 1d400468..8caff14a 100644
--- a/skills/printing-press/SKILL.md
+++ b/skills/printing-press/SKILL.md
@@ -1307,9 +1307,12 @@ Present via `AskUserQuestion`:
> "Shipcheck passed. How thoroughly should I test against the live API?"
>
-> 1. **Quick check** — Read-only: doctor, list, sync, search, output modes (~5 min)
+> 1. **Quick check (recommended)** — Read-only: doctor, list, sync, search, output modes (~5 min)
> 2. **Full dogfood** — Complete lifecycle with mutations: create, modify, cancel, sync verification (~15-30 min). I'll create test entities on your account.
-> 3. **Skip testing** — Proceed to polish
+
+There is no skip option when an API key is available. When no API key is
+available, Phase 5 auto-skips: display "No API key available — skipping live
+dogfood testing. The CLI was verified against exit codes and dry-run only."
Do NOT proceed without asking. Do NOT substitute an ad-hoc smoke test.
@@ -1346,25 +1349,45 @@ When a test fails, fix it immediately — do not accumulate failures. Tag each f
- **CLI fix** — specific to this printed CLI
- **Machine issue** — should be fixed in the generator (note for retro)
-### Step 4: Report
+### Step 4: Report and gate
+
+Write a structured acceptance report. This report is **required** — Phase 5.6
+checks for it before promoting.
```
-Dogfood Results: <cli-name>
+Acceptance Report: <cli-name>
Level: Quick Check / Full Dogfood
- Tests: N/N passed
- Fixes applied: M
+ Tests: N/M passed
+ Failures:
+ - [command]: expected [X], got [Y]
+ Fixes applied: K
- [each fix]
- Machine issues: K
+ Machine issues: J
- [each issue for retro]
+ Gate: PASS / FAIL
```
-After dogfood, Phase 5.5 (Polish) runs automatically. See
-[references/dogfood-testing.md](references/dogfood-testing.md) for additional
+**Acceptance threshold:**
+- Quick Check: 5/6 core tests must pass. Auth (`doctor`) or sync failure is automatic FAIL.
+- Full Dogfood: 10/13 tests must pass. Auth or sync failure is automatic FAIL.
+
+**Gate = PASS:** proceed to Phase 5.5 (Polish).
+
+**Gate = FAIL:** fix issues inline (Step 3) and re-run failing tests, up to
+2 fix loops. If the gate still fails after 2 loops, put the CLI on hold:
+```bash
+printing-press lock release --cli <api>-pp-cli
+```
+The working copy remains in `$CLI_WORK_DIR`. Proceed to Phase 5.6 to archive
+manuscripts (archiving still happens on hold). Tag the failure reason in the
+acceptance report so the next run can learn from it.
+
+See [references/dogfood-testing.md](references/dogfood-testing.md) for additional
guidance on common failure patterns and what NOT to test.
Write:
-`$PROOFS_DIR/<stamp>-fix-<api>-pp-cli-dogfood.md`
+`$PROOFS_DIR/<stamp>-fix-<api>-pp-cli-acceptance.md`
## Phase 5.5: Polish
@@ -1405,6 +1428,16 @@ Write the agent's full response to:
## Phase 5.6: Promote and Archive
+### Acceptance gate check
+
+Before promoting, verify the acceptance artifact exists when an API key was
+available during this run:
+
+- If `$PROOFS_DIR/*-acceptance.md` exists with `Gate: PASS` → proceed to promote.
+- If `$PROOFS_DIR/*-acceptance.md` exists with `Gate: FAIL` → CLI is on hold. Do NOT promote. Proceed to Archive Manuscripts.
+- If no acceptance artifact exists AND an API key was available → Phase 5 was skipped. Go back and run it. Do NOT promote without it.
+- If no acceptance artifact exists AND no API key was available → acceptable. Proceed to promote (the CLI was verified mechanically only).
+
### Promote to Library
If the shipcheck verdict is `ship` or `ship-with-gaps`, promote the verified CLI from the working directory to the library. This must happen BEFORE archiving — the CLI in the library is the primary deliverable.
← a1096f41 fix(skills): inline dogfood protocol steps to prevent skippi
·
back to Cli Printing Press
·
fix(cli): auto-award OAuth2 auth points until generator supp 81716728 →