← back to Cli Printing Press
feat(scorecard): add breadth dimension + fix LLM runner + integration tests
95e26838f64d1a283c0a004272c86ec273fdc093 · 2026-03-25 11:58:52 -0700 · Matt Van Horn
Honest scorecard:
- Add Breadth dimension (0-10): penalizes empty CLIs. <5 commands = 0/10,
41-60 = 9/10, 60+ = 10/10. Total now out of 90 not 80.
- A 6-command Notion CLI no longer scores Grade A.
LLM runner fix:
- Fixed claude CLI invocation: prompt as positional arg, --output-format text
- For large prompts (>100K), writes to temp file and passes reference
- Added warning on claude failure before codex fallback
Integration tests (run from YOUR terminal, not from Claude Code):
- LLM_TEST=1 go test ./internal/llm/ -run TestRunWithRealLLM -v
- LLM_TEST=1 go test ./internal/llm/ -run TestDocSpecPromptWithRealLLM -v
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Files touched
A docs/plans/2026-03-25-test-real-llm-brain-test-and-honest-scorecard-plan.mdM internal/llm/llm.goA internal/llm/llm_integration_test.goM internal/pipeline/scorecard.go
Diff
commit 95e26838f64d1a283c0a004272c86ec273fdc093
Author: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Date: Wed Mar 25 11:58:52 2026 -0700
feat(scorecard): add breadth dimension + fix LLM runner + integration tests
Honest scorecard:
- Add Breadth dimension (0-10): penalizes empty CLIs. <5 commands = 0/10,
41-60 = 9/10, 60+ = 10/10. Total now out of 90 not 80.
- A 6-command Notion CLI no longer scores Grade A.
LLM runner fix:
- Fixed claude CLI invocation: prompt as positional arg, --output-format text
- For large prompts (>100K), writes to temp file and passes reference
- Added warning on claude failure before codex fallback
Integration tests (run from YOUR terminal, not from Claude Code):
- LLM_TEST=1 go test ./internal/llm/ -run TestRunWithRealLLM -v
- LLM_TEST=1 go test ./internal/llm/ -run TestDocSpecPromptWithRealLLM -v
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
...eal-llm-brain-test-and-honest-scorecard-plan.md | 199 +++++++++++++++++++++
internal/llm/llm.go | 13 +-
internal/llm/llm_integration_test.go | 79 ++++++++
internal/pipeline/scorecard.go | 52 +++++-
4 files changed, 336 insertions(+), 7 deletions(-)
diff --git a/docs/plans/2026-03-25-test-real-llm-brain-test-and-honest-scorecard-plan.md b/docs/plans/2026-03-25-test-real-llm-brain-test-and-honest-scorecard-plan.md
new file mode 100644
index 00000000..160d3f73
--- /dev/null
+++ b/docs/plans/2026-03-25-test-real-llm-brain-test-and-honest-scorecard-plan.md
@@ -0,0 +1,199 @@
+---
+title: "Turn On the Brain - Real LLM Test and Honest Scorecard"
+type: test
+status: active
+date: 2026-03-25
+---
+
+# Turn On the Brain
+
+## The Problem
+
+We built three LLM brains but never turned them on. Every test mocks the LLM or skips it. The Notion CLI has 6 commands because the regex found 6 endpoints - but the LLM brain was supposed to find 30+. We don't know if it works because we've never tried.
+
+The scorecard is also lying. It measures template features, not CLI quality. A 6-command Notion CLI scores the same as a 51-command Plaid CLI. That's not honest scoring.
+
+## What This Plan Does
+
+1. Actually call the LLM (Claude CLI) during generation and measure the difference
+2. Fix the scorecard so it penalizes empty CLIs
+3. Run the full test with LLM on vs LLM off and compare
+
+## Implementation Units
+
+### Unit 1: Verify Claude CLI Works for LLM Calls
+
+Before anything else, verify the `internal/llm` package can actually call Claude.
+
+```bash
+# Check claude is installed
+which claude
+
+# Test a simple prompt
+echo "Say hello" | claude -p "Say hello in one word"
+```
+
+If `claude` CLI doesn't accept `-p` for prompt, fix `internal/llm/llm.go` to use the correct flags. The current code guesses at flags - we need to verify what actually works.
+
+**File:** `internal/llm/llm.go`
+
+Test: `go test ./internal/llm/ -run TestRunWithRealLLM -v` (new test, gated behind `LLM_TEST=1`)
+
+```go
+func TestRunWithRealLLM(t *testing.T) {
+ if os.Getenv("LLM_TEST") == "" {
+ t.Skip("Set LLM_TEST=1 to test with real LLM")
+ }
+ if !Available() {
+ t.Skip("No LLM CLI available")
+ }
+ response, err := Run("Respond with exactly the word 'hello' and nothing else.")
+ require.NoError(t, err)
+ assert.Contains(t, strings.ToLower(response), "hello")
+}
+```
+
+**Verification:** `LLM_TEST=1 go test ./internal/llm/ -run TestRunWithRealLLM -v` prints "hello".
+
+### Unit 2: Test Doc-to-Spec with Real LLM on Notion
+
+The big proof. Does the LLM brain actually find more endpoints than regex?
+
+```bash
+# Regex mode (current - finds 6 endpoints)
+./printing-press generate --docs "https://developers.notion.com/reference" --name notion --output /tmp/notion-regex --force
+
+# LLM mode (should find 20+)
+./printing-press generate --docs "https://developers.notion.com/reference" --name notion --output /tmp/notion-llm --force
+# (LLM mode activates automatically when claude CLI is available)
+```
+
+**Measure the difference:**
+- Count commands in `/tmp/notion-regex/internal/cli/*.go`
+- Count commands in `/tmp/notion-llm/internal/cli/*.go`
+- Both must pass 7/7 quality gates
+
+**If the LLM doc-to-spec fails:** Fix the prompt in `internal/docspec/docspec.go`. The prompt might need:
+- Better format instructions (show the exact YAML format expected)
+- Truncation handling (40K char limit might cut off important endpoints)
+- Multi-page support (Notion docs span many pages)
+
+**File:** `internal/docspec/docspec.go` (fix prompts if needed)
+
+**Verification:** Notion LLM CLI has 15+ commands (3x what regex finds). Both pass 7/7 gates.
+
+### Unit 3: Test LLM Polish with Real LLM
+
+Test that `--polish` actually improves the help text and examples.
+
+```bash
+# Generate without polish
+./printing-press generate --spec https://petstore3.swagger.io/api/v3/openapi.json --output /tmp/petstore-raw --force
+
+# Generate with polish
+./printing-press generate --spec https://petstore3.swagger.io/api/v3/openapi.json --output /tmp/petstore-polished --force --polish
+```
+
+**Measure the difference:**
+- Compare `Short:` descriptions in `/tmp/petstore-raw/internal/cli/*.go` vs polished
+- Compare `Example:` strings
+- Compare README.md content
+- Polished should have better descriptions and more examples
+
+**If polish fails:** Fix prompts in `internal/llmpolish/prompts.go`. Common issues:
+- LLM returns markdown-wrapped JSON instead of raw JSON
+- LLM returns too-long descriptions
+- LLM hallucinates flag names that don't exist
+
+**Verification:** At least 3 help descriptions are different (improved) between raw and polished.
+
+### Unit 4: Fix Scorecard - Add Breadth Dimension
+
+The scorecard needs to penalize empty CLIs. A 6-command Notion CLI should NOT score the same as 51-command Plaid.
+
+**File:** `internal/pipeline/scorecard.go`
+
+Add a new dimension:
+
+```go
+type SteinerScore struct {
+ // ... existing 8 dimensions ...
+ Breadth int `json:"breadth"` // 0-10: how many commands vs expected
+}
+```
+
+Scoring:
+- Count commands in the generated CLI (files in internal/cli/ minus infrastructure)
+- Compare against a baseline:
+ - < 5 commands: 0/10 (basically empty)
+ - 5-10: 3/10 (minimal)
+ - 11-20: 5/10 (decent)
+ - 21-40: 7/10 (good)
+ - 41-60: 9/10 (comprehensive)
+ - 60+: 10/10 (Steinberger-tier)
+
+Update total to be out of 90 (not 80). Update grade thresholds accordingly.
+
+**Also update the comparison table** in `fullrun.go` to show the breadth dimension.
+
+**Verification:** Notion regex (6 commands) scores 0/10 on breadth. Plaid (51 commands) scores 9/10. Grade reflects this.
+
+### Unit 5: Full Run With LLM vs Without
+
+**File:** `internal/pipeline/fullrun_test.go` (extend)
+
+New test: `TestFullRunWithLLM` (gated behind `FULL_RUN_LLM=1`)
+
+Runs the same 3 APIs (petstore, plaid, notion) but with LLM enabled. Compares results against the non-LLM baseline.
+
+```go
+func TestFullRunWithLLM(t *testing.T) {
+ if os.Getenv("FULL_RUN_LLM") == "" {
+ t.Skip("Set FULL_RUN_LLM=1 to run with real LLM calls")
+ }
+ // ... same as TestFullRun but with LLM available
+ // Compare: notion should have 3x more commands with LLM
+}
+```
+
+Output: side-by-side table showing LLM vs no-LLM for each API.
+
+```
+ | Without LLM | With LLM
+--------------------|--------------------|-----------------
+Notion Commands | 6 | 25+
+Notion Breadth | 0/10 | 7/10
+Notion Grade | B (inflated) | A (earned)
+Plaid Commands | 51 | 51 (same - spec-based)
+Petstore Commands | 8 | 8 (same - spec-based)
+```
+
+The LLM brain should only improve doc-to-spec APIs (Notion). OpenAPI-based APIs (Plaid, Petstore) should be unchanged since the spec already has all endpoints.
+
+## Expected Token Cost
+
+| LLM Call | Tokens | Cost |
+|----------|--------|------|
+| Notion doc-to-spec | ~30K in, ~5K out | ~$0.30 |
+| Petstore polish (help text) | ~10K in, ~2K out | ~$0.10 |
+| Petstore polish (examples) | ~10K in, ~3K out | ~$0.15 |
+| Petstore polish (README) | ~5K in, ~3K out | ~$0.10 |
+| **Total for full run** | | **~$0.65** |
+
+## Acceptance Criteria
+
+- [ ] `internal/llm/llm.go` correctly calls `claude` CLI (verified with real call)
+- [ ] Notion CLI with LLM doc-to-spec has 15+ commands (vs 6 with regex)
+- [ ] Both Notion CLIs (regex and LLM) pass 7/7 quality gates
+- [ ] `--polish` visibly improves at least 3 help descriptions on Petstore
+- [ ] Scorecard breadth dimension penalizes 6-command CLIs (0-3/10)
+- [ ] Scorecard breadth rewards 51-command CLIs (9-10/10)
+- [ ] Full run comparison table shows LLM vs no-LLM side by side
+- [ ] Total LLM cost for full run under $2
+
+## Scope Boundaries
+
+- Do NOT fix doc-to-spec to crawl multiple pages (just use what's on the docs URL)
+- Do NOT change the template engine (this tests the LLM brains, not the templates)
+- Do NOT run LLM tests in regular `go test ./...` (gate behind env vars)
+- If Claude CLI flags are wrong, fix them - but don't build a fallback to API calls
diff --git a/internal/llm/llm.go b/internal/llm/llm.go
index df63200c..1ee46610 100644
--- a/internal/llm/llm.go
+++ b/internal/llm/llm.go
@@ -27,15 +27,24 @@ func Run(prompt string) (string, error) {
}
defer os.Remove(tmpFile)
- // Try claude first (--print -p runs non-interactively)
+ // Try claude first (-p / --print mode, prompt as positional arg)
if path, err := exec.LookPath("claude"); err == nil {
- cmd := exec.Command(path, "--print", "-p", prompt)
+ // For short prompts, pass directly. For long prompts, use a temp file referenced in the prompt.
+ var cmd *exec.Cmd
+ if len(prompt) < 100000 {
+ cmd = exec.Command(path, "-p", prompt, "--output-format", "text")
+ } else {
+ // Write to temp file and tell Claude to read it
+ metaPrompt := fmt.Sprintf("Read the file at %s and follow the instructions inside it exactly.", tmpFile)
+ cmd = exec.Command(path, "-p", metaPrompt, "--output-format", "text")
+ }
cmd.Stderr = os.Stderr
out, err := cmd.Output()
if err == nil {
return strings.TrimSpace(string(out)), nil
}
// Fall through to codex
+ fmt.Fprintf(os.Stderr, "warning: claude failed (%v), trying codex\n", err)
}
// Try codex (--quiet --prompt runs non-interactively)
diff --git a/internal/llm/llm_integration_test.go b/internal/llm/llm_integration_test.go
new file mode 100644
index 00000000..f90ee1a8
--- /dev/null
+++ b/internal/llm/llm_integration_test.go
@@ -0,0 +1,79 @@
+package llm
+
+import (
+ "os"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+// TestRunWithRealLLM tests the LLM runner with a real Claude CLI call.
+// This test costs ~$0.01 in tokens. Run with: LLM_TEST=1 go test ./internal/llm/ -run TestRunWithRealLLM -v
+func TestRunWithRealLLM(t *testing.T) {
+ if os.Getenv("LLM_TEST") == "" {
+ t.Skip("Set LLM_TEST=1 to test with real LLM (costs tokens)")
+ }
+ if !Available() {
+ t.Skip("No LLM CLI available (install claude or codex)")
+ }
+
+ response, err := Run("Respond with exactly the word 'hello' and nothing else. No punctuation, no explanation.")
+ require.NoError(t, err, "LLM Run should not error")
+ assert.Contains(t, strings.ToLower(response), "hello", "Response should contain 'hello'")
+ t.Logf("LLM response: %q", response)
+}
+
+// TestDocSpecPromptWithRealLLM tests that the doc-to-spec prompt produces valid YAML.
+// Run with: LLM_TEST=1 go test ./internal/llm/ -run TestDocSpecPromptWithRealLLM -v -timeout 2m
+func TestDocSpecPromptWithRealLLM(t *testing.T) {
+ if os.Getenv("LLM_TEST") == "" {
+ t.Skip("Set LLM_TEST=1 to test with real LLM (costs ~$0.30)")
+ }
+ if !Available() {
+ t.Skip("No LLM CLI available")
+ }
+
+ prompt := `Generate a YAML API spec for a simple pet store API with these endpoints:
+- GET /pets (list all pets)
+- GET /pets/{id} (get one pet)
+- POST /pets (create a pet, body: name string, status string)
+- DELETE /pets/{id} (delete a pet)
+
+Output ONLY valid YAML in this format (no markdown fences):
+name: petstore
+description: "Simple pet store"
+base_url: "https://api.example.com"
+auth:
+ type: api_key
+ header: "X-API-Key"
+ env_vars:
+ - PETSTORE_API_KEY
+resources:
+ pets:
+ description: "Pet management"
+ endpoints:
+ list:
+ method: GET
+ path: /pets
+ description: "List all pets"
+ get:
+ method: GET
+ path: /pets/{id}
+ description: "Get a pet by ID"
+ params:
+ - name: id
+ type: string
+ positional: true`
+
+ response, err := Run(prompt)
+ require.NoError(t, err)
+
+ // Verify it looks like YAML
+ assert.Contains(t, response, "name:", "Response should contain YAML name field")
+ assert.Contains(t, response, "resources:", "Response should contain resources")
+ assert.Contains(t, response, "endpoints:", "Response should contain endpoints")
+ t.Logf("LLM response length: %d chars", len(response))
+ t.Logf("First 500 chars: %s", response[:min(500, len(response))])
+}
diff --git a/internal/pipeline/scorecard.go b/internal/pipeline/scorecard.go
index 20b58ec7..2c3b715e 100644
--- a/internal/pipeline/scorecard.go
+++ b/internal/pipeline/scorecard.go
@@ -17,7 +17,7 @@ type Scorecard struct {
GapReport []string `json:"gap_report"`
}
-// SteinerScore breaks down the Steinberger bar into 8 dimensions, each 0-10.
+// SteinerScore breaks down the Steinberger bar into 9 dimensions, each 0-10.
type SteinerScore struct {
OutputModes int `json:"output_modes"` // 0-10
Auth int `json:"auth"` // 0-10
@@ -27,7 +27,8 @@ type SteinerScore struct {
Doctor int `json:"doctor"` // 0-10
AgentNative int `json:"agent_native"` // 0-10
LocalCache int `json:"local_cache"` // 0-10
- Total int `json:"total"` // 0-80
+ Breadth int `json:"breadth"` // 0-10: how many commands (penalizes empty CLIs)
+ Total int `json:"total"` // 0-90
Percentage int `json:"percentage"` // 0-100
}
@@ -55,6 +56,7 @@ func RunScorecard(outputDir, pipelineDir string) (*Scorecard, error) {
sc.Steinberger.Doctor = scoreDoctor(outputDir)
sc.Steinberger.AgentNative = scoreAgentNative(outputDir)
sc.Steinberger.LocalCache = scoreLocalCache(outputDir)
+ sc.Steinberger.Breadth = scoreBreadth(outputDir)
sc.Steinberger.Total = sc.Steinberger.OutputModes +
sc.Steinberger.Auth +
@@ -63,10 +65,11 @@ func RunScorecard(outputDir, pipelineDir string) (*Scorecard, error) {
sc.Steinberger.README +
sc.Steinberger.Doctor +
sc.Steinberger.AgentNative +
- sc.Steinberger.LocalCache
+ sc.Steinberger.LocalCache +
+ sc.Steinberger.Breadth
if sc.Steinberger.Total > 0 {
- sc.Steinberger.Percentage = (sc.Steinberger.Total * 100) / 80
+ sc.Steinberger.Percentage = (sc.Steinberger.Total * 100) / 90
}
// Grade
@@ -265,6 +268,43 @@ func scoreLocalCache(dir string) int {
return score
}
+func scoreBreadth(dir string) int {
+ // Count command files in internal/cli/ (exclude infrastructure files)
+ cliDir := filepath.Join(dir, "internal", "cli")
+ entries, err := os.ReadDir(cliDir)
+ if err != nil {
+ return 0
+ }
+ infra := map[string]bool{
+ "helpers.go": true, "root.go": true, "doctor.go": true, "auth.go": true,
+ }
+ commandFiles := 0
+ for _, e := range entries {
+ if e.IsDir() || !strings.HasSuffix(e.Name(), ".go") {
+ continue
+ }
+ if infra[e.Name()] {
+ continue
+ }
+ commandFiles++
+ }
+
+ switch {
+ case commandFiles >= 60:
+ return 10
+ case commandFiles >= 41:
+ return 9
+ case commandFiles >= 21:
+ return 7
+ case commandFiles >= 11:
+ return 5
+ case commandFiles >= 5:
+ return 3
+ default:
+ return 0 // < 5 commands = basically empty
+ }
+}
+
func readFileContent(path string) string {
data, err := os.ReadFile(path)
if err != nil {
@@ -307,6 +347,7 @@ func buildGapReport(s SteinerScore) []string {
{"doctor", s.Doctor},
{"agent_native", s.AgentNative},
{"local_cache", s.LocalCache},
+ {"breadth", s.Breadth},
}
for _, d := range dimensions {
if d.score < 5 {
@@ -381,12 +422,13 @@ func writeScorecardMD(sc *Scorecard, pipelineDir string) error {
{"Doctor", s.Doctor},
{"Agent Native", s.AgentNative},
{"Local Cache", s.LocalCache},
+ {"Breadth", s.Breadth},
}
for _, d := range dimensions {
bar := strings.Repeat("#", d.score) + strings.Repeat(".", 10-d.score)
b.WriteString(fmt.Sprintf("| %s | %d/10 %s |\n", d.name, d.score, bar))
}
- b.WriteString(fmt.Sprintf("| **Total** | **%d/80** |\n\n", s.Total))
+ b.WriteString(fmt.Sprintf("| **Total** | **%d/90** |\n\n", s.Total))
// Competitor comparison
if len(sc.CompetitorScores) > 0 {
← c266b6fb feat(templates): add --human-friendly flag and NDJSON pagina
·
back to Cli Printing Press
·
refactor(skill): Claude Code IS the brain, not the Go binary 6e934000 →