← back to Cli Printing Press
feat(pipeline): full press run with MakeBestCLI, scorecard fixes, and comparison table
99de67ec0ab380b9531dd3e57212a6485a43ca2c · 2026-03-25 07:49:00 -0700 · Matt Van Horn
- Fix scorecard file paths: all 8 dimensions now score correctly
(was scoring 0 on everything except README due to wrong paths)
- Add MakeBestCLI() - one function does everything: research,
generate, count commands, dogfood, scorecard, fix plans
- Add PrintComparisonTable() - side-by-side results for all 3 APIs
- Add GenerateLearningsPlan() - auto-writes fix plan from results
- Add fullrun_test.go with FULL_RUN=1 gated integration test
- Petstore now scores 51/80 (63%, Grade C) - realistic assessment
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Files touched
A docs/plans/2026-03-25-test-full-press-run-easy-medium-hard-plan.mdA internal/pipeline/fullrun.goA internal/pipeline/fullrun_test.goM internal/pipeline/scorecard.goA internal/pipeline/scorecard_run_test.go
Diff
commit 99de67ec0ab380b9531dd3e57212a6485a43ca2c
Author: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Date: Wed Mar 25 07:49:00 2026 -0700
feat(pipeline): full press run with MakeBestCLI, scorecard fixes, and comparison table
- Fix scorecard file paths: all 8 dimensions now score correctly
(was scoring 0 on everything except README due to wrong paths)
- Add MakeBestCLI() - one function does everything: research,
generate, count commands, dogfood, scorecard, fix plans
- Add PrintComparisonTable() - side-by-side results for all 3 APIs
- Add GenerateLearningsPlan() - auto-writes fix plan from results
- Add fullrun_test.go with FULL_RUN=1 gated integration test
- Petstore now scores 51/80 (63%, Grade C) - realistic assessment
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
...25-test-full-press-run-easy-medium-hard-plan.md | 352 +++++++++++++++
internal/pipeline/fullrun.go | 492 +++++++++++++++++++++
internal/pipeline/fullrun_test.go | 77 ++++
internal/pipeline/scorecard.go | 26 +-
internal/pipeline/scorecard_run_test.go | 42 ++
5 files changed, 975 insertions(+), 14 deletions(-)
diff --git a/docs/plans/2026-03-25-test-full-press-run-easy-medium-hard-plan.md b/docs/plans/2026-03-25-test-full-press-run-easy-medium-hard-plan.md
new file mode 100644
index 00000000..a87d7352
--- /dev/null
+++ b/docs/plans/2026-03-25-test-full-press-run-easy-medium-hard-plan.md
@@ -0,0 +1,352 @@
+---
+title: "Full Press Run: Easy/Medium/Hard with Steinberger Scorecard and Learnings Loop"
+type: test
+status: active
+date: 2026-03-25
+---
+
+# Full Press Run: Easy/Medium/Hard
+
+## What This Is
+
+Run the printing press at full power on 3 APIs. Clean room - fresh /tmp directories, no prior CLIs visible. The press uses every capability it has: research competitors, generate the CLI, dogfood it, score it against Steinberger AND the competitors it found, measure API coverage, then write a learnings plan for how to do it better next time.
+
+## The 3 APIs
+
+| Level | API | Why | Spec Source | Known Competitor |
+|-------|-----|-----|-------------|-----------------|
+| EASY | Petstore | Simple, 3 resources, demo API | OpenAPI JSON | None |
+| MEDIUM | Plaid | Real fintech API, 50+ resources, official spec | OpenAPI YAML | landakram/plaid-cli (57 stars, abandoned) |
+| HARD | Notion | No OpenAPI spec, must use doc-to-spec | Docs URL | 4ier/notion-cli (87 stars, active) |
+
+## What "Full Power" Means
+
+For each API, the press runs this sequence:
+
+```
+Step 1: RESEARCH
+ - Search GitHub for competing CLIs
+ - For each competitor: fetch issues, README, PRs
+ - Produce research.json with novelty score + competitor insights
+ - Measure: how many competitors? what's the command target?
+
+Step 2: GENERATE
+ - Easy/Medium: generate from OpenAPI spec
+ - Hard: generate from docs URL (doc-to-spec)
+ - Run 7 quality gates
+ - Measure: did it compile? how many commands? how many resources?
+
+Step 3: API COVERAGE
+ - Count endpoints in the source spec (or extracted from docs)
+ - Count commands in the generated CLI
+ - Calculate: commands / spec_endpoints = % API coverage
+ - Measure: what % of the API did we cover?
+
+Step 4: DOGFOOD
+ - Run Tier 1: --help, version, doctor, <resource> --help for each resource
+ - Capture all output to evidence files
+ - Measure: how many commands passed? any crashes?
+
+Step 5: SCORECARD
+ - Score on 8 Steinberger dimensions (0-80, shown as %)
+ - Compare vs each competitor found in research
+ - Assign grade (A/B/C/D/F)
+ - Measure: what's our % vs Steinberger? do we beat competitors?
+
+Step 6: LEARNINGS
+ - What broke? What scored low? What could the press do better?
+ - Write a learnings plan (ce:plan format) for press improvements
+ - Measure: how many fix plans generated?
+```
+
+## Implementation Units
+
+### Unit 1: Fix Scorecard File Paths (Blocker)
+
+**Problem:** The scorecard (`internal/pipeline/scorecard.go`) looks for files at wrong paths. Every dimension except README scores 0 because it can't find the code.
+
+**File:** `internal/pipeline/scorecard.go`
+
+**Fixes (the scoring functions grep files at these wrong paths):**
+
+| Function | Currently Looks At | Should Look At |
+|----------|-------------------|----------------|
+| `scoreOutputModes` | `cmd/root.go` | `internal/cli/root.go` |
+| `scoreAuth` | `internal/config.go` | `internal/config/config.go` |
+| `scoreAuth` | `internal/auth.go` | `internal/cli/auth.go` |
+| `scoreErrorHandling` | `internal/helpers.go` | `internal/cli/helpers.go` |
+| `scoreErrorHandling` | counts `os.Exit(` | should count `code:` in cliError structs |
+| `scoreTerminalUX` | `internal/helpers.go` | `internal/cli/helpers.go` |
+| `scoreDoctor` | check path | `internal/cli/doctor.go` |
+| `scoreAgentNative` | `cmd/root.go` | `internal/cli/root.go` |
+
+**Verification:** Run scorecard on an existing generated CLI. Output modes should score 6+ (has json, plain, select). Error handling should score 5+ (has hint: messages). Terminal UX should score 8+ (has colorEnabled, NO_COLOR, isatty).
+
+### Unit 2: Add API Coverage Metric
+
+**Problem:** We don't track what % of the API we covered. If Plaid has 200 endpoints and we generated 51 commands, that's ~25% coverage.
+
+**File:** `internal/pipeline/scorecard.go` (extend)
+
+**Add to Scorecard struct:**
+```go
+type Scorecard struct {
+ // ... existing fields ...
+ APICoverage APICoverageMetric `json:"api_coverage"`
+}
+
+type APICoverageMetric struct {
+ SpecEndpoints int `json:"spec_endpoints"` // total endpoints in the source spec
+ GeneratedCmds int `json:"generated_commands"` // commands in the CLI
+ CoveragePercent int `json:"coverage_percent"` // generated/spec * 100
+}
+```
+
+**How to count:**
+- Spec endpoints: count paths in the parsed OpenAPI spec (already available from the generator). For doc-to-spec, count extracted endpoints.
+- Generated commands: count `.go` files in `internal/cli/` minus helpers.go, root.go, doctor.go, auth.go (those are infrastructure, not API commands). Or grep for `func new.*Cmd` in the generated files.
+- Coverage = generated / spec * 100
+
+**Verification:** Petstore should show ~100% (small spec, all endpoints covered). Plaid should show < 50% (large spec, 50-endpoint-per-resource limit truncates).
+
+### Unit 3: Build MakeBestCLI Function
+
+**Goal:** One function that runs the entire sequence. No pipeline state files, no plan files, no agent execution. Just: API name in, graded CLI out.
+
+**File:** `internal/pipeline/fullrun.go` (new)
+
+```go
+// FullRunResult holds everything the press produced for one API.
+type FullRunResult struct {
+ APIName string
+ Level string // "easy", "medium", "hard"
+
+ // Step 1: Research
+ Research *ResearchResult
+
+ // Step 2: Generate
+ OutputDir string
+ GatesPassed int // out of 7
+ GatesFailed int
+ CommandCount int
+ ResourceCount int
+
+ // Step 3: API Coverage
+ Coverage APICoverageMetric
+
+ // Step 4: Dogfood
+ Dogfood *DogfoodResults
+
+ // Step 5: Scorecard
+ Scorecard *Scorecard
+
+ // Step 6: Learnings
+ FixPlans []string // paths to generated fix plans
+
+ // Errors
+ Errors []string
+}
+
+func MakeBestCLI(apiName, level, specFlag, specURL, outputDir string) (*FullRunResult, error) {
+ result := &FullRunResult{APIName: apiName, Level: level, OutputDir: outputDir}
+ pipelineDir := filepath.Join(outputDir, ".pipeline")
+ os.MkdirAll(pipelineDir, 0755)
+
+ // Step 1: Research
+ result.Research, _ = RunResearch(apiName, "catalog", pipelineDir)
+
+ // Step 2: Generate (shell out to printing-press binary)
+ // Use --spec or --docs based on specFlag
+ // Parse output for PASS/FAIL gates
+
+ // Step 3: API Coverage
+ // Count commands in generated CLI vs endpoints in spec
+
+ // Step 4: Dogfood
+ // Find the built binary, run Tier 1
+
+ // Step 5: Scorecard
+ result.Scorecard, _ = RunScorecard(outputDir, pipelineDir)
+
+ // Step 6: Learnings
+ // If scorecard exists, generate fix plans for gaps
+
+ return result, nil
+}
+```
+
+**Verification:** `MakeBestCLI("petstore", "easy", "--spec", "https://petstore3.swagger.io/api/v3/openapi.json", "/tmp/fullrun-petstore")` returns a complete FullRunResult with all 6 steps populated.
+
+### Unit 4: Build the Comparison Table Printer
+
+**File:** `internal/pipeline/fullrun.go` (extend)
+
+```go
+func PrintComparisonTable(results []*FullRunResult) string
+```
+
+Produces the final output:
+
+```
+╔═══════════════════════════════════════════════════════════════════════╗
+║ PRINTING PRESS FULL RUN RESULTS ║
+╠═══════════════════════════════════════════════════════════════════════╣
+║ │ Petstore (EASY) │ Plaid (MED) │ Notion (HARD) ║
+╠═════════════════════╪═════════════════╪══════════════╪═══════════════╣
+║ GENERATION │ │ │ ║
+║ Quality Gates │ 7/7 PASS │ 7/7 PASS │ 7/7 PASS ║
+║ Commands │ 8 │ 51 │ ??? ║
+║ Resources │ 3 │ 51 │ ??? ║
+║ API Coverage │ 100% │ 25% │ ??? ║
+╠═════════════════════╪═════════════════╪══════════════╪═══════════════╣
+║ STEINBERGER (0-80) │ │ │ ║
+║ Output Modes │ 8/10 │ 8/10 │ 8/10 ║
+║ Auth │ 0/10 │ 8/10 │ 5/10 ║
+║ Error Handling │ 8/10 │ 8/10 │ 8/10 ║
+║ Terminal UX │ 10/10 │ 10/10 │ 10/10 ║
+║ README │ 8/10 │ 8/10 │ 8/10 ║
+║ Doctor │ 4/10 │ 4/10 │ 4/10 ║
+║ Agent Native │ 9/10 │ 9/10 │ 9/10 ║
+║ Local Cache │ 0/10 │ 0/10 │ 0/10 ║
+║ TOTAL │ 47/80 (59%) │ 55/80 (69%) │ 52/80 (65%) ║
+║ GRADE │ C │ B │ B ║
+╠═════════════════════╪═════════════════╪══════════════╪═══════════════╣
+║ vs COMPETITORS │ │ │ ║
+║ Competitors Found │ 0 │ 1 │ 1 ║
+║ Best Competitor │ N/A │ plaid-cli │ notion-cli ║
+║ Their Commands │ N/A │ 8 │ 87 stars ║
+║ WE WIN? │ N/A │ YES (51>8) │ ??? ║
+╠═════════════════════╪═════════════════╪══════════════╪═══════════════╣
+║ DOGFOOD │ │ │ ║
+║ Tier 1 Pass Rate │ 100% │ 100% │ ??? ║
+║ Dogfood Score │ 40/50 │ 40/50 │ ??? ║
+╠═════════════════════╪═════════════════╪══════════════╪═══════════════╣
+║ FIX PLANS GENERATED │ 2 │ 2 │ 2 ║
+╚═══════════════════════════════════════════════════════════════════════╝
+```
+
+### Unit 5: Build the Test Runner
+
+**File:** `internal/pipeline/fullrun_test.go` (new)
+
+```go
+func TestFullRun(t *testing.T) {
+ if os.Getenv("FULL_RUN") == "" {
+ t.Skip("Set FULL_RUN=1 to run full press test (takes 2-5 min)")
+ }
+
+ // Clean room: use /tmp, no access to prior CLIs
+ baseDir := filepath.Join(os.TempDir(), "press-fullrun-"+time.Now().Format("20060102-150405"))
+ os.MkdirAll(baseDir, 0755)
+ defer os.RemoveAll(baseDir)
+
+ apis := []struct {
+ name, level, flag, url string
+ }{
+ {"petstore", "easy", "--spec", "https://petstore3.swagger.io/api/v3/openapi.json"},
+ {"plaid", "medium", "--spec", "https://raw.githubusercontent.com/plaid/plaid-openapi/master/2020-09-14.yml"},
+ {"notion", "hard", "--docs", "https://developers.notion.com/reference"},
+ }
+
+ var results []*FullRunResult
+ for _, api := range apis {
+ t.Run(api.name, func(t *testing.T) {
+ outputDir := filepath.Join(baseDir, api.name+"-cli")
+ result, err := MakeBestCLI(api.name, api.level, api.flag, api.url, outputDir)
+ require.NoError(t, err)
+ results = append(results, result)
+
+ // Basic assertions
+ assert.Equal(t, 7, result.GatesPassed, "all 7 gates should pass")
+ assert.True(t, result.CommandCount > 0, "should generate at least 1 command")
+ assert.NotNil(t, result.Scorecard, "scorecard should be produced")
+ })
+ }
+
+ // Print the comparison table
+ fmt.Println(PrintComparisonTable(results))
+
+ // Write results to a file for review
+ resultsPath := filepath.Join(baseDir, "full-run-results.md")
+ os.WriteFile(resultsPath, []byte(PrintComparisonTable(results)), 0644)
+ fmt.Printf("\nResults written to: %s\n", resultsPath)
+}
+```
+
+**How to run:**
+```bash
+FULL_RUN=1 go test ./internal/pipeline/ -run TestFullRun -v -timeout 10m
+```
+
+### Unit 6: Learnings Plan Generator
+
+**Goal:** After the full run, auto-generate a ce:plan for improving the press based on what went wrong.
+
+**File:** `internal/pipeline/fullrun.go` (extend)
+
+```go
+func GenerateLearningsPlan(results []*FullRunResult, outputPath string) error
+```
+
+Reads all 3 results and produces a markdown plan:
+
+```markdown
+---
+title: "Learnings from Full Run 2026-03-25"
+type: fix
+status: active
+date: 2026-03-25
+---
+
+# Learnings from Full Press Run
+
+## Run Summary
+- Easy (Petstore): Grade C, 59% Steinberger, 100% API coverage
+- Medium (Plaid): Grade B, 69% Steinberger, 25% API coverage
+- Hard (Notion): Grade C, 65% Steinberger, 15% API coverage
+
+## Consistent Gaps (same problem across all 3)
+- Local Cache: 0/10 on all 3. Need SQLite caching template.
+- Doctor: 4/10 on all 3. Need more health check patterns.
+
+## Level-Specific Issues
+- HARD: doc-to-spec only extracted N endpoints from Notion docs.
+ Need multi-page crawling.
+- MEDIUM: API coverage only 25% due to 50-endpoint limit per resource.
+ Need smarter resource grouping.
+
+## Recommended Fixes (prioritized by impact)
+1. [fix] Bump endpoint-per-resource limit from 50 to 100
+2. [fix] Doc-to-spec: follow links to find all endpoint pages
+3. [feat] Add SQLite caching template for local-first CLI pattern
+4. [fix] Doctor: add more health check endpoint patterns
+
+## Next Run
+After implementing fixes, re-run:
+FULL_RUN=1 go test ./internal/pipeline/ -run TestFullRun -v -timeout 10m
+
+Expected improvement: Grade C -> B on easy, Grade B -> B+ on medium.
+```
+
+**The loop:** Run full press -> get scores -> generate learnings plan -> execute fixes -> run full press again -> scores improve.
+
+## Acceptance Criteria
+
+- [ ] Scorecard file paths fixed - all 8 dimensions score correctly on real CLIs
+- [ ] API coverage metric added - shows % of spec endpoints covered
+- [ ] `MakeBestCLI()` runs the full sequence: research, generate, dogfood, score
+- [ ] Comparison table prints all 3 APIs side by side with scores
+- [ ] Petstore (easy): 7/7 gates, grade C or better, 80%+ API coverage
+- [ ] Plaid (medium): 7/7 gates, grade B or better, beats landakram/plaid-cli on commands
+- [ ] Notion (hard): 7/7 gates from doc-to-spec, produces a grade (any grade - it compiled from docs)
+- [ ] Learnings plan auto-generated identifying consistent gaps and recommended fixes
+- [ ] Clean room: all 3 runs in /tmp, no access to prior generated CLIs
+- [ ] One command runs everything: `FULL_RUN=1 go test ./internal/pipeline/ -run TestFullRun -v -timeout 10m`
+
+## Scope Boundaries
+
+- Do NOT fix the press based on results (that's the learnings plan's job)
+- Do NOT ship any CLIs
+- Do NOT create Homebrew tap
+- Do NOT run Tier 2/3 dogfood (no API credentials)
+- The learnings plan is WRITTEN but not EXECUTED in this plan
diff --git a/internal/pipeline/fullrun.go b/internal/pipeline/fullrun.go
new file mode 100644
index 00000000..8ea46877
--- /dev/null
+++ b/internal/pipeline/fullrun.go
@@ -0,0 +1,492 @@
+package pipeline
+
+import (
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "time"
+)
+
+// FullRunResult holds everything the press produced for one API.
+type FullRunResult struct {
+ APIName string
+ Level string // "EASY", "MEDIUM", "HARD"
+
+ // Step 1: Research
+ Research *ResearchResult
+ ResearchError string
+
+ // Step 2: Generate
+ OutputDir string
+ GatesPassed int
+ GatesFailed int
+ GatesOutput string
+ CommandCount int
+ ResourceCount int
+
+ // Step 3: Coverage
+ SpecEndpoints int
+ CoveragePercent int
+
+ // Step 4: Dogfood
+ Dogfood *DogfoodResults
+ DogfoodError string
+
+ // Step 5: Scorecard
+ Scorecard *Scorecard
+ ScorecardError string
+
+ // Step 6: Learnings
+ FixPlans []string
+
+ Errors []string
+ Duration time.Duration
+}
+
+// MakeBestCLI runs the full printing press pipeline for a single API and
+// returns a result summarizing every phase.
+func MakeBestCLI(apiName, level, specFlag, specURL, outputDir, pressBinary string) *FullRunResult {
+ start := time.Now()
+ result := &FullRunResult{
+ APIName: apiName,
+ Level: level,
+ OutputDir: outputDir,
+ }
+
+ pipelineDir := PipelineDir(apiName)
+ os.MkdirAll(pipelineDir, 0o755)
+
+ // Step 1: Research
+ research, err := RunResearch(apiName, "catalog", pipelineDir)
+ if err != nil {
+ result.ResearchError = err.Error()
+ result.Errors = append(result.Errors, fmt.Sprintf("research: %v", err))
+ }
+ result.Research = research
+
+ // Step 2: Generate
+ repoRoot := findRepoRootFrom(pressBinary)
+ var genArgs []string
+ if specFlag == "--docs" {
+ genArgs = []string{"generate", "--docs", specURL, "--name", apiName, "--output", outputDir, "--force"}
+ } else {
+ genArgs = []string{"generate", "--spec", specURL, "--output", outputDir, "--force", "--lenient"}
+ }
+
+ cmd := exec.Command(pressBinary, genArgs...)
+ cmd.Dir = repoRoot
+ genOut, genErr := cmd.CombinedOutput()
+ result.GatesOutput = string(genOut)
+
+ for _, line := range strings.Split(result.GatesOutput, "\n") {
+ trimmed := strings.TrimSpace(line)
+ if strings.Contains(trimmed, "PASS") {
+ result.GatesPassed++
+ }
+ if strings.Contains(trimmed, "FAIL") {
+ result.GatesFailed++
+ }
+ }
+
+ if genErr != nil {
+ result.Errors = append(result.Errors, fmt.Sprintf("generate: %v", genErr))
+ result.Duration = time.Since(start)
+ return result
+ }
+
+ // Step 3: Count commands and resources
+ resources := listResources(outputDir)
+ result.ResourceCount = len(resources)
+ result.CommandCount = len(resources) + 4 // +4 for root, help, version, doctor
+
+ // Step 4: API Coverage estimate
+ skippedCount := strings.Count(result.GatesOutput, "skipping") + strings.Count(result.GatesOutput, "Skipping")
+ result.SpecEndpoints = result.ResourceCount + skippedCount
+ if result.SpecEndpoints > 0 {
+ result.CoveragePercent = (result.ResourceCount * 100) / result.SpecEndpoints
+ }
+
+ // Step 5: Dogfood
+ cliBinaryPath := filepath.Join(outputDir, apiName+"-cli")
+ buildCmd := exec.Command("go", "build", "-o", cliBinaryPath, "./cmd/...")
+ buildCmd.Dir = outputDir
+ if buildErr := buildCmd.Run(); buildErr != nil {
+ result.DogfoodError = fmt.Sprintf("build failed: %v", buildErr)
+ result.Errors = append(result.Errors, fmt.Sprintf("dogfood build: %v", buildErr))
+ } else {
+ cfg := DogfoodConfig{
+ BinaryPath: cliBinaryPath,
+ PipelineDir: pipelineDir,
+ MaxTier: 1,
+ CmdTimeout: 15 * time.Second,
+ Resources: resources,
+ }
+ dogfood, dogErr := RunDogfood(cfg)
+ if dogErr != nil {
+ result.DogfoodError = dogErr.Error()
+ result.Errors = append(result.Errors, fmt.Sprintf("dogfood: %v", dogErr))
+ }
+ result.Dogfood = dogfood
+ }
+
+ // Step 6: Scorecard
+ scorecard, scErr := RunScorecard(outputDir, pipelineDir)
+ if scErr != nil {
+ result.ScorecardError = scErr.Error()
+ result.Errors = append(result.Errors, fmt.Sprintf("scorecard: %v", scErr))
+ }
+ result.Scorecard = scorecard
+
+ // Step 7: Fix Plans
+ if scorecard != nil {
+ plans, planErr := GenerateFixPlans(scorecard, pipelineDir)
+ if planErr != nil {
+ result.Errors = append(result.Errors, fmt.Sprintf("fix plans: %v", planErr))
+ }
+ result.FixPlans = plans
+ }
+
+ result.Duration = time.Since(start)
+ return result
+}
+
+// listResources returns the resource names found in the generated CLI's
+// internal/cli directory, excluding infrastructure files.
+func listResources(outputDir string) []string {
+ cliDir := filepath.Join(outputDir, "internal", "cli")
+ entries, err := os.ReadDir(cliDir)
+ if err != nil {
+ return nil
+ }
+
+ infraFiles := map[string]bool{
+ "helpers.go": true,
+ "root.go": true,
+ "doctor.go": true,
+ "auth.go": true,
+ }
+
+ var resources []string
+ for _, e := range entries {
+ name := e.Name()
+ if e.IsDir() || !strings.HasSuffix(name, ".go") {
+ continue
+ }
+ if strings.HasSuffix(name, "_test.go") {
+ continue
+ }
+ if infraFiles[name] {
+ continue
+ }
+ // Resource name is the filename without .go
+ resources = append(resources, strings.TrimSuffix(name, ".go"))
+ }
+ return resources
+}
+
+// findRepoRootFrom walks up from the binary path (or cwd) to find go.mod.
+func findRepoRootFrom(binaryPath string) string {
+ dir := filepath.Dir(binaryPath)
+ for {
+ if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {
+ return dir
+ }
+ parent := filepath.Dir(dir)
+ if parent == dir {
+ // Fallback: try cwd
+ cwd, _ := os.Getwd()
+ return cwd
+ }
+ dir = parent
+ }
+}
+
+// PrintComparisonTable produces a formatted comparison table showing all
+// FullRunResults side by side with fixed-width columns.
+func PrintComparisonTable(results []*FullRunResult) string {
+ if len(results) == 0 {
+ return "(no results)\n"
+ }
+
+ var b strings.Builder
+ b.WriteString("\n")
+ b.WriteString("=== CLI Printing Press - Full Run Comparison ===\n\n")
+
+ // Header
+ b.WriteString(fmt.Sprintf("%-25s", "Metric"))
+ for _, r := range results {
+ b.WriteString(fmt.Sprintf("| %-18s", r.APIName+" ("+r.Level+")"))
+ }
+ b.WriteString("|\n")
+
+ // Separator
+ b.WriteString(strings.Repeat("-", 25))
+ for range results {
+ b.WriteString("|" + strings.Repeat("-", 19))
+ }
+ b.WriteString("|\n")
+
+ // Quality Gates
+ writeRow(&b, "Quality Gates", results, func(r *FullRunResult) string {
+ return fmt.Sprintf("%d/7 PASS", r.GatesPassed)
+ })
+
+ // Commands
+ writeRow(&b, "Commands", results, func(r *FullRunResult) string {
+ return fmt.Sprintf("%d", r.CommandCount)
+ })
+
+ // Resources
+ writeRow(&b, "Resources", results, func(r *FullRunResult) string {
+ return fmt.Sprintf("%d", r.ResourceCount)
+ })
+
+ // API Coverage
+ writeRow(&b, "API Coverage", results, func(r *FullRunResult) string {
+ return fmt.Sprintf("%d%%", r.CoveragePercent)
+ })
+
+ // Steinberger dimensions
+ steinDimensions := []struct {
+ label string
+ get func(*Scorecard) int
+ }{
+ {"Output Modes", func(s *Scorecard) int { return s.Steinberger.OutputModes }},
+ {"Auth", func(s *Scorecard) int { return s.Steinberger.Auth }},
+ {"Error Handling", func(s *Scorecard) int { return s.Steinberger.ErrorHandling }},
+ {"Terminal UX", func(s *Scorecard) int { return s.Steinberger.TerminalUX }},
+ {"README", func(s *Scorecard) int { return s.Steinberger.README }},
+ {"Doctor", func(s *Scorecard) int { return s.Steinberger.Doctor }},
+ {"Agent Native", func(s *Scorecard) int { return s.Steinberger.AgentNative }},
+ {"Local Cache", func(s *Scorecard) int { return s.Steinberger.LocalCache }},
+ }
+
+ for _, dim := range steinDimensions {
+ writeRow(&b, " "+dim.label, results, func(r *FullRunResult) string {
+ if r.Scorecard == nil {
+ return "n/a"
+ }
+ return fmt.Sprintf("%d/10", dim.get(r.Scorecard))
+ })
+ }
+
+ // Steinberger total + %
+ writeRow(&b, "Steinberger Total", results, func(r *FullRunResult) string {
+ if r.Scorecard == nil {
+ return "n/a"
+ }
+ return fmt.Sprintf("%d/80 (%d%%)", r.Scorecard.Steinberger.Total, r.Scorecard.Steinberger.Percentage)
+ })
+
+ // Grade
+ writeRow(&b, "Grade", results, func(r *FullRunResult) string {
+ if r.Scorecard == nil {
+ return "n/a"
+ }
+ return r.Scorecard.OverallGrade
+ })
+
+ // Competitors found
+ writeRow(&b, "Competitors Found", results, func(r *FullRunResult) string {
+ if r.Research == nil {
+ return "0"
+ }
+ return fmt.Sprintf("%d", len(r.Research.Alternatives))
+ })
+
+ // We Win?
+ writeRow(&b, "We Win?", results, func(r *FullRunResult) string {
+ if r.Scorecard == nil || len(r.Scorecard.CompetitorScores) == 0 {
+ return "n/a"
+ }
+ wins := 0
+ for _, cs := range r.Scorecard.CompetitorScores {
+ if cs.WeWin {
+ wins++
+ }
+ }
+ return fmt.Sprintf("%d/%d", wins, len(r.Scorecard.CompetitorScores))
+ })
+
+ // Dogfood pass rate
+ writeRow(&b, "Dogfood Pass Rate", results, func(r *FullRunResult) string {
+ if r.Dogfood == nil {
+ return "n/a"
+ }
+ if r.Dogfood.TotalCommands == 0 {
+ return "0/0"
+ }
+ pct := (r.Dogfood.PassedCommands * 100) / r.Dogfood.TotalCommands
+ return fmt.Sprintf("%d/%d (%d%%)", r.Dogfood.PassedCommands, r.Dogfood.TotalCommands, pct)
+ })
+
+ // Fix plans generated
+ writeRow(&b, "Fix Plans", results, func(r *FullRunResult) string {
+ return fmt.Sprintf("%d", len(r.FixPlans))
+ })
+
+ // Duration
+ writeRow(&b, "Duration", results, func(r *FullRunResult) string {
+ return r.Duration.Round(time.Second).String()
+ })
+
+ // Errors
+ writeRow(&b, "Errors", results, func(r *FullRunResult) string {
+ return fmt.Sprintf("%d", len(r.Errors))
+ })
+
+ b.WriteString("\n")
+ return b.String()
+}
+
+func writeRow(b *strings.Builder, label string, results []*FullRunResult, fn func(*FullRunResult) string) {
+ b.WriteString(fmt.Sprintf("%-25s", label))
+ for _, r := range results {
+ b.WriteString(fmt.Sprintf("| %-18s", fn(r)))
+ }
+ b.WriteString("|\n")
+}
+
+// GenerateLearningsPlan writes a markdown plan summarizing consistent gaps
+// and recommended fixes across all runs.
+func GenerateLearningsPlan(results []*FullRunResult, outputPath string) error {
+ if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
+ return fmt.Errorf("creating output dir: %w", err)
+ }
+
+ var b strings.Builder
+ b.WriteString("# Learnings Plan - CLI Printing Press Full Run\n\n")
+ b.WriteString(fmt.Sprintf("Generated: %s\n\n", time.Now().Format("2006-01-02 15:04:05")))
+
+ // Summarize runs
+ b.WriteString("## Runs\n\n")
+ for _, r := range results {
+ status := "OK"
+ if len(r.Errors) > 0 {
+ status = fmt.Sprintf("%d errors", len(r.Errors))
+ }
+ b.WriteString(fmt.Sprintf("- **%s** (%s) - Gates %d/7, Duration %s, Status: %s\n",
+ r.APIName, r.Level, r.GatesPassed, r.Duration.Round(time.Second), status))
+ }
+ b.WriteString("\n")
+
+ // Find consistent gaps (dimensions scoring <5 across multiple runs)
+ type dimTally struct {
+ lowCount int
+ total int
+ sum int
+ }
+ dimNames := []string{"output_modes", "auth", "error_handling", "terminal_ux", "readme", "doctor", "agent_native", "local_cache"}
+ dimGetters := []func(*SteinerScore) int{
+ func(s *SteinerScore) int { return s.OutputModes },
+ func(s *SteinerScore) int { return s.Auth },
+ func(s *SteinerScore) int { return s.ErrorHandling },
+ func(s *SteinerScore) int { return s.TerminalUX },
+ func(s *SteinerScore) int { return s.README },
+ func(s *SteinerScore) int { return s.Doctor },
+ func(s *SteinerScore) int { return s.AgentNative },
+ func(s *SteinerScore) int { return s.LocalCache },
+ }
+
+ tallies := make([]dimTally, len(dimNames))
+ for _, r := range results {
+ if r.Scorecard == nil {
+ continue
+ }
+ for i, getter := range dimGetters {
+ score := getter(&r.Scorecard.Steinberger)
+ tallies[i].total++
+ tallies[i].sum += score
+ if score < 5 {
+ tallies[i].lowCount++
+ }
+ }
+ }
+
+ b.WriteString("## Consistent Gaps\n\n")
+ b.WriteString("Dimensions scoring below 5/10 across multiple runs:\n\n")
+ hasGaps := false
+ for i, name := range dimNames {
+ t := tallies[i]
+ if t.total == 0 {
+ continue
+ }
+ avg := t.sum / t.total
+ if t.lowCount > 0 {
+ hasGaps = true
+ b.WriteString(fmt.Sprintf("- **%s** - low in %d/%d runs (avg %d/10)\n", name, t.lowCount, t.total, avg))
+ }
+ }
+ if !hasGaps {
+ b.WriteString("No consistent gaps found - all dimensions scored 5+ across runs.\n")
+ }
+ b.WriteString("\n")
+
+ // Recommended fixes
+ b.WriteString("## Recommended Fixes\n\n")
+ b.WriteString("Priority order (most impactful first):\n\n")
+ priority := 1
+ for i, name := range dimNames {
+ t := tallies[i]
+ if t.total == 0 || t.lowCount == 0 {
+ continue
+ }
+ b.WriteString(fmt.Sprintf("%d. **Improve %s templates** - affects %d/%d APIs tested\n",
+ priority, name, t.lowCount, t.total))
+ if advice, ok := dimensionAdvice[name]; ok {
+ // Include first line of advice
+ lines := strings.SplitN(advice, "\n", 2)
+ b.WriteString(fmt.Sprintf(" - %s\n", strings.TrimSpace(lines[0])))
+ }
+ priority++
+ }
+ if priority == 1 {
+ b.WriteString("No template fixes needed - all dimensions healthy.\n")
+ }
+ b.WriteString("\n")
+
+ // Gate failures
+ b.WriteString("## Gate Failures\n\n")
+ for _, r := range results {
+ if r.GatesFailed > 0 {
+ b.WriteString(fmt.Sprintf("- **%s** - %d gates failed\n", r.APIName, r.GatesFailed))
+ }
+ }
+ allPassed := true
+ for _, r := range results {
+ if r.GatesFailed > 0 {
+ allPassed = false
+ break
+ }
+ }
+ if allPassed {
+ b.WriteString("All gates passed across all runs.\n")
+ }
+ b.WriteString("\n")
+
+ // Dogfood summary
+ b.WriteString("## Dogfood Summary\n\n")
+ for _, r := range results {
+ if r.Dogfood != nil {
+ pct := 0
+ if r.Dogfood.TotalCommands > 0 {
+ pct = (r.Dogfood.PassedCommands * 100) / r.Dogfood.TotalCommands
+ }
+ b.WriteString(fmt.Sprintf("- **%s** - %d/%d passed (%d%%)\n",
+ r.APIName, r.Dogfood.PassedCommands, r.Dogfood.TotalCommands, pct))
+ } else {
+ b.WriteString(fmt.Sprintf("- **%s** - dogfood not run (%s)\n", r.APIName, r.DogfoodError))
+ }
+ }
+ b.WriteString("\n")
+
+ // Next steps
+ b.WriteString("## Next Steps\n\n")
+ b.WriteString("1. Fix the highest-priority template gaps listed above\n")
+ b.WriteString("2. Re-run this full comparison to verify improvements\n")
+ b.WriteString("3. Add more APIs at each difficulty level to broaden coverage\n")
+
+ return os.WriteFile(outputPath, []byte(b.String()), 0o644)
+}
diff --git a/internal/pipeline/fullrun_test.go b/internal/pipeline/fullrun_test.go
new file mode 100644
index 00000000..997433bb
--- /dev/null
+++ b/internal/pipeline/fullrun_test.go
@@ -0,0 +1,77 @@
+package pipeline
+
+import (
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+func TestFullRun(t *testing.T) {
+ if os.Getenv("FULL_RUN") == "" {
+ t.Skip("Set FULL_RUN=1 to run full press test")
+ }
+
+ // Build the press binary first
+ pressBinary := filepath.Join(t.TempDir(), "printing-press")
+ repoRoot := findRepoRoot()
+ cmd := exec.Command("go", "build", "-o", pressBinary, "./cmd/printing-press")
+ cmd.Dir = repoRoot
+ require.NoError(t, cmd.Run(), "failed to build printing-press")
+
+ baseDir := filepath.Join(os.TempDir(), "press-fullrun-"+time.Now().Format("150405"))
+ os.MkdirAll(baseDir, 0755)
+
+ apis := []struct {
+ name, level, flag, url string
+ }{
+ {"petstore", "EASY", "--spec", "https://petstore3.swagger.io/api/v3/openapi.json"},
+ {"plaid", "MEDIUM", "--spec", "https://raw.githubusercontent.com/plaid/plaid-openapi/master/2020-09-14.yml"},
+ {"notion", "HARD", "--docs", "https://developers.notion.com/reference"},
+ }
+
+ var results []*FullRunResult
+ for _, api := range apis {
+ t.Run(api.name, func(t *testing.T) {
+ outputDir := filepath.Join(baseDir, api.name+"-cli")
+ result := MakeBestCLI(api.name, api.level, api.flag, api.url, outputDir, pressBinary)
+ results = append(results, result)
+
+ assert.Equal(t, 7, result.GatesPassed, "%s: all 7 gates should pass", api.name)
+ assert.True(t, result.CommandCount > 0, "%s: should have commands", api.name)
+ assert.NotNil(t, result.Scorecard, "%s: should have scorecard", api.name)
+ })
+ }
+
+ // Print comparison table
+ table := PrintComparisonTable(results)
+ fmt.Println(table)
+
+ // Write learnings plan
+ learningsPath := filepath.Join(baseDir, "learnings-plan.md")
+ GenerateLearningsPlan(results, learningsPath)
+ fmt.Printf("Learnings plan: %s\n", learningsPath)
+
+ // Also write results to file
+ os.WriteFile(filepath.Join(baseDir, "comparison-table.txt"), []byte(table), 0644)
+ fmt.Printf("Full results at: %s\n", baseDir)
+}
+
+func findRepoRoot() string {
+ dir, _ := os.Getwd()
+ for {
+ if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {
+ return dir
+ }
+ parent := filepath.Dir(dir)
+ if parent == dir {
+ return "."
+ }
+ dir = parent
+ }
+}
diff --git a/internal/pipeline/scorecard.go b/internal/pipeline/scorecard.go
index a7292c9c..07995064 100644
--- a/internal/pipeline/scorecard.go
+++ b/internal/pipeline/scorecard.go
@@ -87,7 +87,7 @@ func RunScorecard(outputDir, pipelineDir string) (*Scorecard, error) {
}
func scoreOutputModes(dir string) int {
- content := readFileContent(filepath.Join(dir, "cmd", "root.go"))
+ content := readFileContent(filepath.Join(dir, "internal", "cli", "root.go"))
score := 0
if strings.Contains(content, "json") {
score += 2
@@ -112,14 +112,14 @@ func scoreOutputModes(dir string) int {
func scoreAuth(dir string) int {
score := 0
- configContent := readFileContent(filepath.Join(dir, "internal", "config.go"))
+ configContent := readFileContent(filepath.Join(dir, "internal", "config", "config.go"))
envCount := strings.Count(configContent, "os.Getenv")
if envCount >= 2 {
score += 8
} else if envCount >= 1 {
score += 5
}
- if fileExists(filepath.Join(dir, "internal", "auth.go")) {
+ if fileExists(filepath.Join(dir, "internal", "cli", "auth.go")) {
score += 2
}
if score > 10 {
@@ -129,13 +129,13 @@ func scoreAuth(dir string) int {
}
func scoreErrorHandling(dir string) int {
- content := readFileContent(filepath.Join(dir, "internal", "helpers.go"))
+ content := readFileContent(filepath.Join(dir, "internal", "cli", "helpers.go"))
score := 0
if strings.Contains(content, "hint:") || strings.Contains(content, "Hint:") {
score += 5
}
- // Count distinct exit codes (os.Exit calls with different values)
- exitCount := strings.Count(content, "os.Exit(")
+ // Count typed exit codes (cliError with code:)
+ exitCount := strings.Count(content, "code:")
if exitCount > 5 {
exitCount = 5
}
@@ -147,7 +147,7 @@ func scoreErrorHandling(dir string) int {
}
func scoreTerminalUX(dir string) int {
- content := readFileContent(filepath.Join(dir, "internal", "helpers.go"))
+ content := readFileContent(filepath.Join(dir, "internal", "cli", "helpers.go"))
score := 0
if strings.Contains(content, "colorEnabled") {
score += 5
@@ -186,11 +186,9 @@ func scoreREADME(dir string) int {
}
func scoreDoctor(dir string) int {
- content := readFileContent(filepath.Join(dir, "cmd", "doctor.go"))
- if content == "" {
- content = readFileContent(filepath.Join(dir, "internal", "doctor.go"))
- }
- healthChecks := strings.Count(content, "http.Get")
+ content := readFileContent(filepath.Join(dir, "internal", "cli", "doctor.go"))
+ // Count health check patterns
+ healthChecks := strings.Count(content, "http.Get") + strings.Count(content, "http.Head") + strings.Count(content, "http.NewRequest") + strings.Count(content, "http.Client")
score := healthChecks * 2
if score > 10 {
score = 10
@@ -199,8 +197,8 @@ func scoreDoctor(dir string) int {
}
func scoreAgentNative(dir string) int {
- rootContent := readFileContent(filepath.Join(dir, "cmd", "root.go"))
- helpersContent := readFileContent(filepath.Join(dir, "internal", "helpers.go"))
+ rootContent := readFileContent(filepath.Join(dir, "internal", "cli", "root.go"))
+ helpersContent := readFileContent(filepath.Join(dir, "internal", "cli", "helpers.go"))
combined := rootContent + helpersContent
score := 0
diff --git a/internal/pipeline/scorecard_run_test.go b/internal/pipeline/scorecard_run_test.go
new file mode 100644
index 00000000..f4b4949f
--- /dev/null
+++ b/internal/pipeline/scorecard_run_test.go
@@ -0,0 +1,42 @@
+package pipeline
+
+import (
+ "fmt"
+ "os"
+ "testing"
+)
+
+func TestScorecardOnRealCLI(t *testing.T) {
+ outputDir := os.Getenv("SCORECARD_CLI_DIR")
+ if outputDir == "" {
+ t.Skip("Set SCORECARD_CLI_DIR to run scorecard on a real CLI")
+ }
+ pipelineDir := os.Getenv("SCORECARD_PIPELINE_DIR")
+ if pipelineDir == "" {
+ pipelineDir = t.TempDir()
+ }
+
+ sc, err := RunScorecard(outputDir, pipelineDir)
+ if err != nil {
+ t.Fatalf("RunScorecard: %v", err)
+ }
+
+ fmt.Printf("\n=== STEINBERGER SCORECARD ===\n")
+ fmt.Printf("Score: %d/80 (%d%%)\n", sc.Steinberger.Total, sc.Steinberger.Percentage)
+ fmt.Printf("Grade: %s\n\n", sc.OverallGrade)
+ fmt.Printf(" Output Modes: %d/10\n", sc.Steinberger.OutputModes)
+ fmt.Printf(" Auth: %d/10\n", sc.Steinberger.Auth)
+ fmt.Printf(" Error Handling: %d/10\n", sc.Steinberger.ErrorHandling)
+ fmt.Printf(" Terminal UX: %d/10\n", sc.Steinberger.TerminalUX)
+ fmt.Printf(" README: %d/10\n", sc.Steinberger.README)
+ fmt.Printf(" Doctor: %d/10\n", sc.Steinberger.Doctor)
+ fmt.Printf(" Agent Native: %d/10\n", sc.Steinberger.AgentNative)
+ fmt.Printf(" Local Cache: %d/10\n", sc.Steinberger.LocalCache)
+ if len(sc.GapReport) > 0 {
+ fmt.Println("\nGaps:")
+ for _, g := range sc.GapReport {
+ fmt.Printf(" - %s\n", g)
+ }
+ }
+ fmt.Println()
+}
← 731b0ce7 feat(pipeline): press intelligence engine - dynamic plans, c
·
back to Cli Printing Press
·
fix(templates): improve doctor health checks and add HTTP re 3b4f89db →