← back to Cli Printing Press
feat(llmpolish): add LLM polish pass - the press is now smart, not just fast
76d082f5a4e560395a524ade55d3af60f15ee0c7 · 2026-03-25 09:41:01 -0700 · Matt Van Horn
Two-pass architecture: Template (10s, $0) -> LLM Polish (2-5min, ~$1)
New package internal/llmpolish/:
- polish.go: discovers claude/codex CLI, runs 3 LLM passes (help text,
examples, README), gracefully skips if no LLM available
- prompts.go: builds structured prompts from generated CLI files -
extracts commands, descriptions, flags for LLM to improve
- apply.go: writes LLM improvements back to generated Go files
(replaces Short: strings, Example: strings, README.md)
- 14 tests covering prompts, application, and skip behavior
Wiring:
- --polish flag on generate command (both --spec and --docs paths)
- MakeBestCLI runs polish as Step 2.5 between generate and dogfood
- Comparison table shows polish results (help improved, examples added)
Usage: printing-press generate --spec <url> --polish
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Files touched
A docs/plans/2026-03-25-feat-llm-polish-pass-plan.mdM internal/cli/root.goA internal/llmpolish/apply.goA internal/llmpolish/polish.goA internal/llmpolish/polish_test.goA internal/llmpolish/prompts.goM internal/pipeline/fullrun.go
Diff
commit 76d082f5a4e560395a524ade55d3af60f15ee0c7
Author: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Date: Wed Mar 25 09:41:01 2026 -0700
feat(llmpolish): add LLM polish pass - the press is now smart, not just fast
Two-pass architecture: Template (10s, $0) -> LLM Polish (2-5min, ~$1)
New package internal/llmpolish/:
- polish.go: discovers claude/codex CLI, runs 3 LLM passes (help text,
examples, README), gracefully skips if no LLM available
- prompts.go: builds structured prompts from generated CLI files -
extracts commands, descriptions, flags for LLM to improve
- apply.go: writes LLM improvements back to generated Go files
(replaces Short: strings, Example: strings, README.md)
- 14 tests covering prompts, application, and skip behavior
Wiring:
- --polish flag on generate command (both --spec and --docs paths)
- MakeBestCLI runs polish as Step 2.5 between generate and dogfood
- Comparison table shows polish results (help improved, examples added)
Usage: printing-press generate --spec <url> --polish
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
docs/plans/2026-03-25-feat-llm-polish-pass-plan.md | 343 +++++++++++++++++++++
internal/cli/root.go | 35 +++
internal/llmpolish/apply.go | 162 ++++++++++
internal/llmpolish/polish.go | 223 ++++++++++++++
internal/llmpolish/polish_test.go | 245 +++++++++++++++
internal/llmpolish/prompts.go | 170 ++++++++++
internal/pipeline/fullrun.go | 30 ++
7 files changed, 1208 insertions(+)
diff --git a/docs/plans/2026-03-25-feat-llm-polish-pass-plan.md b/docs/plans/2026-03-25-feat-llm-polish-pass-plan.md
new file mode 100644
index 00000000..ee2ce87a
--- /dev/null
+++ b/docs/plans/2026-03-25-feat-llm-polish-pass-plan.md
@@ -0,0 +1,343 @@
+---
+title: "LLM Polish Pass - Make the Press Smart, Not Just Fast"
+type: feat
+status: active
+date: 2026-03-25
+---
+
+# LLM Polish Pass
+
+## The Insight
+
+The printing press generates CLIs from templates. Templates give you the 80% - correct code structure, auth, error handling, output formats. But the last 20% requires understanding: better help text, smarter examples, missing endpoints, a README that actually sells the tool.
+
+LLMs are good at that 20%. The press should have a two-pass architecture:
+
+```
+Pass 1: Template (fast, free, deterministic)
+ - Parse spec -> generate Go code from templates
+ - Takes 10 seconds, costs $0
+ - Produces a CLI that compiles and passes quality gates
+
+Pass 2: LLM Polish (smart, costs tokens, non-deterministic)
+ - Review the generated code with an LLM
+ - Improve descriptions, examples, README
+ - Add missing endpoints from docs
+ - Takes 2-5 minutes, costs ~$0.50-2.00
+ - Produces a CLI that's actually good to use
+```
+
+## Where LLM Polish Helps Most
+
+### 1. Help Text Rewriting
+
+**Before (template - copies spec verbatim):**
+```
+$ plaid-cli transactions list --help
+Retrieve a paginated listing of transactions associated with a given access token
+```
+
+**After (LLM polish):**
+```
+$ plaid-cli transactions list --help
+List transactions for a linked bank account. Returns the most recent transactions
+first. Use --start-date and --end-date to filter by date range.
+```
+
+The LLM reads the spec description and rewrites it for a developer reading --help at 2am.
+
+### 2. Example Generation
+
+**Before (template - generic values):**
+```
+Examples:
+ plaid-cli transactions list --count 25
+```
+
+**After (LLM polish):**
+```
+Examples:
+ # Get last 30 days of transactions
+ plaid-cli transactions list --start-date 2026-02-25 --end-date 2026-03-25
+
+ # Get transactions for a specific account
+ plaid-cli transactions list --account-id acc_abc123 --count 100
+
+ # Export as JSON for processing
+ plaid-cli transactions list --start-date 2026-01-01 --json > transactions.json
+```
+
+The LLM understands what developers actually DO with the command and writes examples for real workflows.
+
+### 3. Missing Endpoint Discovery
+
+**Before (template - only what's in the spec):**
+Notion CLI from doc-to-spec: 6 commands (agents, threads)
+
+**After (LLM polish):**
+LLM reads the Notion API docs page, identifies ALL endpoints (databases, pages, blocks, search, users, comments), writes the missing endpoint definitions, regenerates.
+
+This is the doc-to-spec problem solved properly. Instead of regex, the LLM reads the docs and understands them.
+
+### 4. README That Sells
+
+**Before (template):**
+```markdown
+# plaid-cli
+
+The Plaid REST API. Please see https://plaid.com/docs/api for more details.
+
+## Quick Start
+...
+```
+
+**After (LLM polish):**
+```markdown
+# plaid-cli
+
+Connect to bank accounts, pull transactions, and verify identities - all from your terminal.
+No more spinning up a React app just to test your Plaid integration.
+
+## Why This Exists
+
+Plaid has no official CLI. The community CLI (plaid-cli, 57 stars) was abandoned in 2021.
+This CLI covers 51 endpoints and works with sandbox, development, and production environments.
+
+## Quick Start
+...
+```
+
+## Implementation Units
+
+### Unit 1: LLM Polish Interface
+
+**File:** `internal/llmpolish/polish.go` (new package)
+
+Define the interface - what goes in, what comes out:
+
+```go
+type PolishRequest struct {
+ APIName string
+ OutputDir string // path to generated CLI
+ SpecSource string // OpenAPI spec or docs URL
+ Research *pipeline.ResearchResult // competitor insights
+}
+
+type PolishResult struct {
+ HelpTextsImproved int
+ ExamplesAdded int
+ EndpointsAdded int
+ READMERewritten bool
+ TokensUsed int
+ Duration time.Duration
+}
+
+func Polish(req PolishRequest) (*PolishResult, error)
+```
+
+The `Polish` function does NOT call an LLM API directly. Instead, it writes a structured prompt to a file and shells out to the LLM tool (Claude Code, Codex, or any CLI that accepts a prompt). This keeps the press LLM-agnostic.
+
+**How it shells out:**
+
+```go
+prompt := buildPolishPrompt(req)
+promptFile := filepath.Join(os.TempDir(), "polish-prompt.md")
+os.WriteFile(promptFile, []byte(prompt), 0644)
+
+// Try Claude Code first, fall back to Codex
+cmd := exec.Command("claude", "--print", "-p", prompt)
+output, err := cmd.CombinedOutput()
+if err != nil {
+ cmd = exec.Command("codex", "--quiet", prompt)
+ output, err = cmd.CombinedOutput()
+}
+```
+
+### Unit 2: Help Text Polish Prompt
+
+**File:** `internal/llmpolish/prompts.go`
+
+Build a prompt that asks the LLM to improve help descriptions:
+
+```markdown
+You are improving the --help text for a CLI tool called {{.APIName}}-cli.
+
+For each command below, rewrite the "Short" description to be:
+- Written for a developer reading --help at 2am
+- Under 80 characters
+- Starts with a verb (List, Get, Create, Update, Delete)
+- No jargon from the API spec
+
+Current commands and descriptions:
+{{range .Commands}}
+- {{.Name}}: "{{.CurrentDescription}}"
+{{end}}
+
+Output format: JSON array of {name, description} objects.
+```
+
+The LLM returns improved descriptions. The polish function reads the generated Go files, finds the `Short:` strings, and replaces them.
+
+### Unit 3: Example Polish Prompt
+
+Build a prompt that asks the LLM to write real-world examples:
+
+```markdown
+You are writing --help examples for {{.APIName}}-cli.
+
+For each command, write 2-3 examples showing real workflows a developer would use.
+Each example should have a comment explaining what it does.
+
+Commands:
+{{range .Commands}}
+- {{.Name}} ({{.Method}} {{.Path}})
+ Flags: {{.Flags}}
+{{end}}
+
+Output: JSON array of {command, examples: [{comment, invocation}]}
+```
+
+### Unit 4: README Polish Prompt
+
+Rewrite the generated README with understanding:
+
+```markdown
+Rewrite this CLI's README to sell the tool. You are writing for developers
+who will find this on GitHub.
+
+API: {{.APIName}}
+Commands: {{.CommandCount}}
+Competitors: {{.Competitors}}
+Killer workflow: {{.KillerWorkflow}}
+
+Current README:
+{{.CurrentREADME}}
+
+Write a README with:
+1. A one-line hook that makes developers want to install it
+2. Why this exists (what gap it fills)
+3. Quick start with a real 3-command workflow
+4. The killer workflow (the one command that makes it worth installing)
+5. Full command reference
+```
+
+### Unit 5: Endpoint Discovery Prompt (for doc-to-spec)
+
+When the press generates from docs and gets few endpoints:
+
+```markdown
+Read this API documentation and list ALL endpoints.
+
+URL: {{.DocsURL}}
+Content: {{.DocsHTML}}
+
+For each endpoint, output:
+- method (GET/POST/PUT/PATCH/DELETE)
+- path (e.g. /v1/databases/{id})
+- description (one line)
+- parameters (name, type, required)
+
+Current spec has {{.CurrentEndpoints}} endpoints. Find more.
+```
+
+### Unit 6: Wire Polish into MakeBestCLI
+
+**File:** `internal/pipeline/fullrun.go` (modify)
+
+Add a polish step between Generate and Dogfood:
+
+```go
+// Step 2: Generate (template pass)
+// ... existing code ...
+
+// Step 2.5: LLM Polish (if available)
+if llmAvailable() {
+ polishResult, err := llmpolish.Polish(llmpolish.PolishRequest{
+ APIName: apiName,
+ OutputDir: outputDir,
+ SpecSource: specURL,
+ Research: result.Research,
+ })
+ if err != nil {
+ result.Errors = append(result.Errors, "LLM polish failed: "+err.Error())
+ } else {
+ result.PolishResult = polishResult
+ }
+}
+
+// Step 3: Dogfood
+// ... existing code ...
+```
+
+Polish is optional. If no LLM CLI is available, the press skips it and generates a template-only CLI (what it does today). If an LLM is available, it polishes the output.
+
+### Unit 7: Add --polish Flag to Generate Command
+
+**File:** `internal/cli/root.go`
+
+```go
+var polish bool
+cmd.Flags().BoolVar(&polish, "polish", false, "Run LLM polish pass on generated CLI (requires claude or codex CLI)")
+```
+
+When `--polish` is set, the generate command runs the polish step after template generation.
+
+### Unit 8: Polish Scorecard Dimension
+
+**File:** `internal/pipeline/scorecard.go`
+
+Add a new dimension to the scorecard that detects LLM polish:
+
+```go
+type SteinerScore struct {
+ // ... existing 8 dimensions ...
+ Polish int `json:"polish"` // 0-10: quality of descriptions, examples, README
+}
+```
+
+Polish dimension measures:
+- Help descriptions: are they under 80 chars and start with a verb? (2pts)
+- Examples: does each command have 2+ examples? (3pts)
+- README: does it have a "Why This Exists" section? (2pts)
+- README: does it have a killer workflow section? (3pts)
+
+This dimension ONLY scores well with LLM polish. Template-only CLIs score 0-2/10 here.
+
+## The Full Architecture After This Plan
+
+```
+printing-press generate --spec <url> --polish
+ |
+ v
+ [Template Pass - 10 seconds, $0]
+ Parse spec -> Go templates -> compile -> 7 quality gates
+ |
+ v
+ [LLM Polish Pass - 2-5 minutes, ~$1]
+ Improve help text -> Add examples -> Rewrite README -> Add missing endpoints
+ |
+ v
+ [Dogfood + Scorecard]
+ Run commands -> Score against Steinberger -> Grade
+```
+
+## Acceptance Criteria
+
+- [ ] `llmpolish` package with Polish() function that shells out to claude/codex
+- [ ] Help text prompt: rewrites spec jargon into developer-friendly descriptions
+- [ ] Example prompt: generates 2-3 real-world examples per command
+- [ ] README prompt: rewrites README to sell the tool
+- [ ] Endpoint discovery prompt: finds missing endpoints from docs
+- [ ] --polish flag on generate command
+- [ ] MakeBestCLI calls polish when LLM is available
+- [ ] New scorecard dimension measures polish quality
+- [ ] Petstore with --polish has better help text than without
+- [ ] Notion with --polish has more than 6 commands
+
+## Scope Boundaries
+
+- Do NOT build an LLM API client - shell out to existing CLIs (claude, codex)
+- Do NOT make polish mandatory - template-only still works
+- Do NOT call the LLM during compilation or template rendering
+- Polish edits generated FILES, not templates
+- Keep total polish cost under $2 per CLI
diff --git a/internal/cli/root.go b/internal/cli/root.go
index f4a7e8ee..0fb18199 100644
--- a/internal/cli/root.go
+++ b/internal/cli/root.go
@@ -13,6 +13,7 @@ import (
"github.com/mvanhorn/cli-printing-press/internal/docspec"
"github.com/mvanhorn/cli-printing-press/internal/generator"
+ "github.com/mvanhorn/cli-printing-press/internal/llmpolish"
"github.com/mvanhorn/cli-printing-press/internal/openapi"
"github.com/mvanhorn/cli-printing-press/internal/pipeline"
"github.com/mvanhorn/cli-printing-press/internal/spec"
@@ -48,6 +49,7 @@ func newGenerateCmd() *cobra.Command {
var force bool
var lenient bool
var docsURL string
+ var polish bool
cmd := &cobra.Command{
Use: "generate",
@@ -95,6 +97,22 @@ func newGenerateCmd() *cobra.Command {
}
}
+ if polish {
+ fmt.Fprintln(os.Stderr, "Running LLM polish pass...")
+ polishResult, polishErr := llmpolish.Polish(llmpolish.PolishRequest{
+ APIName: parsed.Name,
+ OutputDir: absOut,
+ })
+ if polishErr != nil {
+ fmt.Fprintf(os.Stderr, "warning: polish failed: %v\n", polishErr)
+ } else if polishResult.Skipped {
+ fmt.Fprintf(os.Stderr, "polish skipped: %s\n", polishResult.SkipReason)
+ } else {
+ fmt.Fprintf(os.Stderr, "Polish: %d help texts improved, %d examples added, README %v\n",
+ polishResult.HelpTextsImproved, polishResult.ExamplesAdded, polishResult.READMERewritten)
+ }
+ }
+
fmt.Fprintf(os.Stderr, "Generated %s-cli at %s (from docs)\n", parsed.Name, absOut)
return nil
}
@@ -161,6 +179,22 @@ func newGenerateCmd() *cobra.Command {
}
}
+ if polish {
+ fmt.Fprintln(os.Stderr, "Running LLM polish pass...")
+ polishResult, polishErr := llmpolish.Polish(llmpolish.PolishRequest{
+ APIName: apiSpec.Name,
+ OutputDir: absOut,
+ })
+ if polishErr != nil {
+ fmt.Fprintf(os.Stderr, "warning: polish failed: %v\n", polishErr)
+ } else if polishResult.Skipped {
+ fmt.Fprintf(os.Stderr, "polish skipped: %s\n", polishResult.SkipReason)
+ } else {
+ fmt.Fprintf(os.Stderr, "Polish: %d help texts improved, %d examples added, README %v\n",
+ polishResult.HelpTextsImproved, polishResult.ExamplesAdded, polishResult.READMERewritten)
+ }
+ }
+
fmt.Fprintf(os.Stderr, "Generated %s-cli at %s\n", apiSpec.Name, absOut)
return nil
},
@@ -174,6 +208,7 @@ func newGenerateCmd() *cobra.Command {
cmd.Flags().BoolVar(&force, "force", false, "Remove existing output directory before generating")
cmd.Flags().BoolVar(&lenient, "lenient", false, "Skip validation errors from broken $refs in OpenAPI specs")
cmd.Flags().StringVar(&docsURL, "docs", "", "API documentation URL to generate spec from")
+ cmd.Flags().BoolVar(&polish, "polish", false, "Run LLM polish pass on generated CLI (requires claude or codex CLI)")
return cmd
}
diff --git a/internal/llmpolish/apply.go b/internal/llmpolish/apply.go
new file mode 100644
index 00000000..163c4d0f
--- /dev/null
+++ b/internal/llmpolish/apply.go
@@ -0,0 +1,162 @@
+package llmpolish
+
+import (
+ "fmt"
+ "os"
+ "path/filepath"
+ "regexp"
+ "strings"
+)
+
+// HelpImprovement represents an improved command description from the LLM.
+type HelpImprovement struct {
+ Command string `json:"command"`
+ Description string `json:"description"`
+}
+
+// ExampleSet represents generated examples for a single command.
+type ExampleSet struct {
+ Command string `json:"command"`
+ Examples []string `json:"examples"`
+}
+
+// applyHelpTexts reads each command .go file, finds Short: "...", and replaces
+// it with the improved description from the LLM.
+func applyHelpTexts(outputDir string, improvements []HelpImprovement) error {
+ cliDir := filepath.Join(outputDir, "internal", "cli")
+
+ // Build a map of command name -> improved description
+ descMap := make(map[string]string)
+ for _, imp := range improvements {
+ descMap[imp.Command] = imp.Description
+ }
+
+ entries, err := os.ReadDir(cliDir)
+ if err != nil {
+ return fmt.Errorf("reading cli dir: %w", err)
+ }
+
+ shortRe := regexp.MustCompile(`(Short:\s*)"([^"]*)"`)
+ useRe := regexp.MustCompile(`Use:\s*"([^"]*)"`)
+
+ applied := 0
+ for _, e := range entries {
+ if e.IsDir() || !strings.HasSuffix(e.Name(), ".go") || strings.HasSuffix(e.Name(), "_test.go") {
+ continue
+ }
+
+ filePath := filepath.Join(cliDir, e.Name())
+ data, err := os.ReadFile(filePath)
+ if err != nil {
+ continue
+ }
+ content := string(data)
+
+ // Find the Use: value to match against the improvement map
+ cmdName := strings.TrimSuffix(e.Name(), ".go")
+ if m := useRe.FindStringSubmatch(content); len(m) > 1 {
+ cmdName = m[1]
+ }
+
+ newDesc, ok := descMap[cmdName]
+ if !ok {
+ continue
+ }
+
+ // Replace the Short: value
+ newContent := shortRe.ReplaceAllStringFunc(content, func(match string) string {
+ return shortRe.ReplaceAllString(match, fmt.Sprintf(`${1}"%s"`, escapeGoString(newDesc)))
+ })
+
+ if newContent != content {
+ if err := os.WriteFile(filePath, []byte(newContent), 0o644); err != nil {
+ return fmt.Errorf("writing %s: %w", e.Name(), err)
+ }
+ applied++
+ }
+ }
+
+ return nil
+}
+
+// applyExamples reads command files, finds Example: "...", and replaces with
+// multi-line examples from the LLM.
+func applyExamples(outputDir string, examples []ExampleSet) error {
+ cliDir := filepath.Join(outputDir, "internal", "cli")
+
+ // Build a map of command name -> examples
+ exMap := make(map[string][]string)
+ for _, ex := range examples {
+ exMap[ex.Command] = ex.Examples
+ }
+
+ entries, err := os.ReadDir(cliDir)
+ if err != nil {
+ return fmt.Errorf("reading cli dir: %w", err)
+ }
+
+ exampleRe := regexp.MustCompile("(?s)(Example:\\s*)(`[^`]*`|\"[^\"]*\")")
+ useRe := regexp.MustCompile(`Use:\s*"([^"]*)"`)
+
+ applied := 0
+ for _, e := range entries {
+ if e.IsDir() || !strings.HasSuffix(e.Name(), ".go") || strings.HasSuffix(e.Name(), "_test.go") {
+ continue
+ }
+
+ filePath := filepath.Join(cliDir, e.Name())
+ data, err := os.ReadFile(filePath)
+ if err != nil {
+ continue
+ }
+ content := string(data)
+
+ cmdName := strings.TrimSuffix(e.Name(), ".go")
+ if m := useRe.FindStringSubmatch(content); len(m) > 1 {
+ cmdName = m[1]
+ }
+
+ exStrings, ok := exMap[cmdName]
+ if !ok || len(exStrings) == 0 {
+ continue
+ }
+
+ // Build the example string with backtick quoting for multi-line
+ combined := strings.Join(exStrings, "\n\n")
+ exampleLiteral := fmt.Sprintf("`%s`", combined)
+
+ newContent := exampleRe.ReplaceAllStringFunc(content, func(match string) string {
+ m := exampleRe.FindStringSubmatch(match)
+ if len(m) < 2 {
+ return match
+ }
+ return m[1] + exampleLiteral
+ })
+
+ if newContent != content {
+ if err := os.WriteFile(filePath, []byte(newContent), 0o644); err != nil {
+ return fmt.Errorf("writing %s: %w", e.Name(), err)
+ }
+ applied++
+ }
+ }
+
+ return nil
+}
+
+// applyREADME writes the new README content to the output directory.
+func applyREADME(outputDir, newContent string) error {
+ readmePath := filepath.Join(outputDir, "README.md")
+ trimmed := strings.TrimSpace(newContent)
+ if trimmed == "" {
+ return fmt.Errorf("empty README content")
+ }
+ return os.WriteFile(readmePath, []byte(trimmed+"\n"), 0o644)
+}
+
+// escapeGoString escapes characters that would break a Go string literal.
+func escapeGoString(s string) string {
+ s = strings.ReplaceAll(s, `\`, `\\`)
+ s = strings.ReplaceAll(s, `"`, `\"`)
+ return s
+}
diff --git a/internal/llmpolish/polish.go b/internal/llmpolish/polish.go
new file mode 100644
index 00000000..54947822
--- /dev/null
+++ b/internal/llmpolish/polish.go
@@ -0,0 +1,223 @@
+package llmpolish
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "time"
+)
+
+// PolishRequest defines what the polish pass needs.
+type PolishRequest struct {
+ APIName string
+ OutputDir string
+ SpecSource string // OpenAPI spec URL or docs URL
+ Research interface{} // *pipeline.ResearchResult - use interface to avoid circular import
+}
+
+// PolishResult summarizes what the polish pass changed.
+type PolishResult struct {
+ HelpTextsImproved int
+ ExamplesAdded int
+ READMERewritten bool
+ TokensUsed int
+ Duration time.Duration
+ Skipped bool
+ SkipReason string
+}
+
+// Polish runs an LLM polish pass over a generated CLI directory.
+// It shells out to claude or codex CLI - never calls an LLM API directly.
+// If the LLM CLI is unavailable or any step fails, it returns partial results
+// rather than crashing the pipeline.
+func Polish(req PolishRequest) (*PolishResult, error) {
+ start := time.Now()
+ result := &PolishResult{}
+
+ llmCmd, llmArgs := findLLMCLI()
+ if llmCmd == "" {
+ result.Skipped = true
+ result.SkipReason = "no LLM CLI found (install claude or codex)"
+ result.Duration = time.Since(start)
+ return result, nil
+ }
+
+ // Step 1: Improve help texts
+ helpPrompt := buildHelpPrompt(req.OutputDir)
+ if helpPrompt != "" {
+ helpOutput, err := runLLM(llmCmd, llmArgs, helpPrompt)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "warning: help text polish failed: %v\n", err)
+ } else {
+ var improvements []HelpImprovement
+ if parseErr := json.Unmarshal(extractJSON(helpOutput), &improvements); parseErr == nil {
+ if applyErr := applyHelpTexts(req.OutputDir, improvements); applyErr == nil {
+ result.HelpTextsImproved = len(improvements)
+ } else {
+ fmt.Fprintf(os.Stderr, "warning: applying help texts failed: %v\n", applyErr)
+ }
+ } else {
+ fmt.Fprintf(os.Stderr, "warning: parsing help improvements failed: %v\n", parseErr)
+ }
+ }
+ }
+
+ // Step 2: Add examples
+ examplePrompt := buildExamplePrompt(req.OutputDir)
+ if examplePrompt != "" {
+ exampleOutput, err := runLLM(llmCmd, llmArgs, examplePrompt)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "warning: example polish failed: %v\n", err)
+ } else {
+ var examples []ExampleSet
+ if parseErr := json.Unmarshal(extractJSON(exampleOutput), &examples); parseErr == nil {
+ if applyErr := applyExamples(req.OutputDir, examples); applyErr == nil {
+ result.ExamplesAdded = countExamples(examples)
+ } else {
+ fmt.Fprintf(os.Stderr, "warning: applying examples failed: %v\n", applyErr)
+ }
+ } else {
+ fmt.Fprintf(os.Stderr, "warning: parsing examples failed: %v\n", parseErr)
+ }
+ }
+ }
+
+ // Step 3: Rewrite README
+ readmePrompt := buildREADMEPrompt(req.OutputDir, req.APIName)
+ if readmePrompt != "" {
+ readmeOutput, err := runLLM(llmCmd, llmArgs, readmePrompt)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "warning: README polish failed: %v\n", err)
+ } else {
+ content := extractMarkdown(readmeOutput)
+ if content != "" {
+ if applyErr := applyREADME(req.OutputDir, content); applyErr == nil {
+ result.READMERewritten = true
+ } else {
+ fmt.Fprintf(os.Stderr, "warning: applying README failed: %v\n", applyErr)
+ }
+ }
+ }
+ }
+
+ result.Duration = time.Since(start)
+ return result, nil
+}
+
+// findLLMCLI checks for available LLM CLIs in preference order.
+// Returns the command name and base args to use.
+func findLLMCLI() (string, []string) {
+ if path, err := exec.LookPath("claude"); err == nil {
+ return path, []string{"--print", "-p"}
+ }
+ if path, err := exec.LookPath("codex"); err == nil {
+ return path, []string{"--quiet", "--prompt"}
+ }
+ return "", nil
+}
+
+// runLLM shells out to the LLM CLI with the given prompt and returns stdout.
+func runLLM(llmCmd string, baseArgs []string, prompt string) (string, error) {
+ // Write prompt to temp file for debugging/logging
+ tmpFile, err := os.CreateTemp("", "polish-prompt-*.md")
+ if err != nil {
+ return "", fmt.Errorf("creating temp file: %w", err)
+ }
+ defer os.Remove(tmpFile.Name())
+
+ if _, err := tmpFile.WriteString(prompt); err != nil {
+ tmpFile.Close()
+ return "", fmt.Errorf("writing prompt: %w", err)
+ }
+ tmpFile.Close()
+
+ args := append(baseArgs, prompt)
+ cmd := exec.Command(llmCmd, args...)
+ cmd.Stderr = os.Stderr
+
+ output, err := cmd.Output()
+ if err != nil {
+ return "", fmt.Errorf("running %s: %w", filepath.Base(llmCmd), err)
+ }
+ return string(output), nil
+}
+
+// extractJSON finds and returns the first JSON array in the LLM output.
+// LLMs often wrap JSON in markdown code blocks.
+func extractJSON(output string) []byte {
+ // Try to find JSON array directly
+ start := -1
+ end := -1
+ depth := 0
+ for i, c := range output {
+ if c == '[' {
+ if depth == 0 {
+ start = i
+ }
+ depth++
+ }
+ if c == ']' {
+ depth--
+ if depth == 0 {
+ end = i + 1
+ break
+ }
+ }
+ }
+ if start >= 0 && end > start {
+ return []byte(output[start:end])
+ }
+ return []byte("[]")
+}
+
+// extractMarkdown strips markdown code fences from LLM output.
+func extractMarkdown(output string) string {
+ // Strip leading/trailing whitespace
+ s := output
+ // Remove ```markdown ... ``` wrapper if present
+ if idx := indexOf(s, "```markdown"); idx >= 0 {
+ s = s[idx+len("```markdown"):]
+ if endIdx := lastIndexOf(s, "```"); endIdx >= 0 {
+ s = s[:endIdx]
+ }
+ } else if idx := indexOf(s, "```md"); idx >= 0 {
+ s = s[idx+len("```md"):]
+ if endIdx := lastIndexOf(s, "```"); endIdx >= 0 {
+ s = s[:endIdx]
+ }
+ } else if idx := indexOf(s, "```"); idx >= 0 {
+ s = s[idx+len("```"):]
+ if endIdx := lastIndexOf(s, "```"); endIdx >= 0 {
+ s = s[:endIdx]
+ }
+ }
+ return s
+}
+
+func indexOf(s, substr string) int {
+ for i := 0; i <= len(s)-len(substr); i++ {
+ if s[i:i+len(substr)] == substr {
+ return i
+ }
+ }
+ return -1
+}
+
+func lastIndexOf(s, substr string) int {
+ for i := len(s) - len(substr); i >= 0; i-- {
+ if s[i:i+len(substr)] == substr {
+ return i
+ }
+ }
+ return -1
+}
+
+func countExamples(examples []ExampleSet) int {
+ total := 0
+ for _, e := range examples {
+ total += len(e.Examples)
+ }
+ return total
+}
diff --git a/internal/llmpolish/polish_test.go b/internal/llmpolish/polish_test.go
new file mode 100644
index 00000000..6d8127cc
--- /dev/null
+++ b/internal/llmpolish/polish_test.go
@@ -0,0 +1,245 @@
+package llmpolish
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+func TestBuildHelpPrompt(t *testing.T) {
+ dir := setupFakeCLI(t)
+
+ prompt := buildHelpPrompt(dir)
+
+ assert.NotEmpty(t, prompt)
+ assert.Contains(t, prompt, "pet")
+ assert.Contains(t, prompt, "Returns list of pets")
+ assert.Contains(t, prompt, "developer-friendly")
+}
+
+func TestBuildHelpPromptEmptyDir(t *testing.T) {
+ dir := t.TempDir()
+ prompt := buildHelpPrompt(dir)
+ assert.Empty(t, prompt)
+}
+
+func TestBuildExamplePrompt(t *testing.T) {
+ dir := setupFakeCLI(t)
+
+ prompt := buildExamplePrompt(dir)
+
+ assert.NotEmpty(t, prompt)
+ assert.Contains(t, prompt, "pet")
+ assert.Contains(t, prompt, "--status")
+ assert.Contains(t, prompt, "examples")
+}
+
+func TestBuildREADMEPrompt(t *testing.T) {
+ dir := t.TempDir()
+ readme := "# my-cli\n\nA CLI for the My API.\n\n## Quick Start\n\n```\nmy-cli pet list\n```\n"
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "README.md"), []byte(readme), 0o644))
+
+ prompt := buildREADMEPrompt(dir, "my-api")
+
+ assert.NotEmpty(t, prompt)
+ assert.Contains(t, prompt, "my-api")
+ assert.Contains(t, prompt, "Quick Start")
+ assert.Contains(t, prompt, "pet list")
+}
+
+func TestBuildREADMEPromptMissingFile(t *testing.T) {
+ dir := t.TempDir()
+ prompt := buildREADMEPrompt(dir, "nope")
+ assert.Empty(t, prompt)
+}
+
+func TestPolishSkipsWhenNoLLM(t *testing.T) {
+ // Override PATH to ensure no LLM CLI is found
+ originalPath := os.Getenv("PATH")
+ t.Setenv("PATH", t.TempDir())
+ defer os.Setenv("PATH", originalPath)
+
+ result, err := Polish(PolishRequest{
+ APIName: "test",
+ OutputDir: t.TempDir(),
+ })
+
+ require.NoError(t, err)
+ assert.True(t, result.Skipped)
+ assert.Contains(t, result.SkipReason, "no LLM CLI found")
+}
+
+func TestApplyHelpTexts(t *testing.T) {
+ dir := setupFakeCLI(t)
+
+ improvements := []HelpImprovement{
+ {Command: "pet", Description: "List all pets, filtered by status"},
+ }
+
+ err := applyHelpTexts(dir, improvements)
+ require.NoError(t, err)
+
+ data, err := os.ReadFile(filepath.Join(dir, "internal", "cli", "pet.go"))
+ require.NoError(t, err)
+ assert.Contains(t, string(data), "List all pets, filtered by status")
+ assert.NotContains(t, string(data), "Returns list of pets")
+}
+
+func TestApplyExamples(t *testing.T) {
+ dir := setupFakeCLI(t)
+
+ examples := []ExampleSet{
+ {
+ Command: "pet",
+ Examples: []string{
+ "# List available pets\npet-cli pet list --status available",
+ "# Get a specific pet\npet-cli pet get --id 123",
+ },
+ },
+ }
+
+ err := applyExamples(dir, examples)
+ require.NoError(t, err)
+
+ data, err := os.ReadFile(filepath.Join(dir, "internal", "cli", "pet.go"))
+ require.NoError(t, err)
+ assert.Contains(t, string(data), "List available pets")
+ assert.Contains(t, string(data), "pet-cli pet list --status available")
+}
+
+func TestApplyREADME(t *testing.T) {
+ dir := t.TempDir()
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "README.md"), []byte("old"), 0o644))
+
+ err := applyREADME(dir, "# New README\n\nBetter content.")
+ require.NoError(t, err)
+
+ data, err := os.ReadFile(filepath.Join(dir, "README.md"))
+ require.NoError(t, err)
+ assert.Contains(t, string(data), "Better content")
+ assert.NotContains(t, string(data), "old")
+}
+
+func TestApplyREADMEEmpty(t *testing.T) {
+ dir := t.TempDir()
+ err := applyREADME(dir, " ")
+ assert.Error(t, err)
+}
+
+func TestExtractJSON(t *testing.T) {
+ tests := []struct {
+ name string
+ input string
+ expect string
+ }{
+ {
+ name: "bare array",
+ input: `[{"command":"pet","description":"List pets"}]`,
+ expect: `[{"command":"pet","description":"List pets"}]`,
+ },
+ {
+ name: "wrapped in markdown",
+ input: "Here is the result:\n```json\n[{\"command\":\"pet\"}]\n```\n",
+ expect: `[{"command":"pet"}]`,
+ },
+ {
+ name: "no json",
+ input: "No JSON here",
+ expect: "[]",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ result := string(extractJSON(tt.input))
+ assert.Equal(t, tt.expect, result)
+ })
+ }
+}
+
+func TestExtractMarkdown(t *testing.T) {
+ tests := []struct {
+ name string
+ input string
+ expect string
+ }{
+ {
+ name: "plain text",
+ input: "# Hello\n\nWorld",
+ expect: "# Hello\n\nWorld",
+ },
+ {
+ name: "fenced markdown",
+ input: "```markdown\n# Hello\n```",
+ expect: "\n# Hello\n",
+ },
+ {
+ name: "fenced generic",
+ input: "```\n# Hello\n```",
+ expect: "\n# Hello\n",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ result := extractMarkdown(tt.input)
+ assert.Equal(t, tt.expect, result)
+ })
+ }
+}
+
+func TestCountExamples(t *testing.T) {
+ examples := []ExampleSet{
+ {Command: "a", Examples: []string{"ex1", "ex2"}},
+ {Command: "b", Examples: []string{"ex3"}},
+ }
+ assert.Equal(t, 3, countExamples(examples))
+}
+
+func TestExtractCommands(t *testing.T) {
+ dir := setupFakeCLI(t)
+ commands := extractCommands(dir)
+
+ require.Len(t, commands, 1)
+ assert.Equal(t, "pet", commands[0].Name)
+ assert.Equal(t, "Returns list of pets", commands[0].Description)
+ assert.Contains(t, commands[0].Flags, "--status")
+}
+
+// setupFakeCLI creates a minimal generated CLI directory structure for testing.
+func setupFakeCLI(t *testing.T) string {
+ t.Helper()
+ dir := t.TempDir()
+ cliDir := filepath.Join(dir, "internal", "cli")
+ require.NoError(t, os.MkdirAll(cliDir, 0o755))
+
+ // Infrastructure files (should be skipped)
+ for _, infra := range []string{"root.go", "helpers.go", "doctor.go", "auth.go"} {
+ require.NoError(t, os.WriteFile(filepath.Join(cliDir, infra), []byte("package cli\n"), 0o644))
+ }
+
+ // Command file
+ petGo := strings.Join([]string{
+ "package cli",
+ "",
+ "import \"github.com/spf13/cobra\"",
+ "",
+ "func newPetCmd() *cobra.Command {",
+ "\tvar status string",
+ "\tcmd := &cobra.Command{",
+ "\t\tUse: \"pet\",",
+ "\t\tShort: \"Returns list of pets\",",
+ "\t\tExample: \"pet-cli pet list\",",
+ "\t}",
+ "\tcmd.Flags().StringVar(&status, \"status\", \"\", \"Filter by status\")",
+ "\treturn cmd",
+ "}",
+ }, "\n")
+ require.NoError(t, os.WriteFile(filepath.Join(cliDir, "pet.go"), []byte(petGo), 0o644))
+
+ return dir
+}
diff --git a/internal/llmpolish/prompts.go b/internal/llmpolish/prompts.go
new file mode 100644
index 00000000..bb839266
--- /dev/null
+++ b/internal/llmpolish/prompts.go
@@ -0,0 +1,170 @@
+package llmpolish
+
+import (
+ "fmt"
+ "os"
+ "path/filepath"
+ "regexp"
+ "strings"
+)
+
+// commandInfo holds extracted metadata about a generated CLI command.
+type commandInfo struct {
+ Name string
+ Description string
+ Method string
+ Path string
+ Flags []string
+}
+
+// buildHelpPrompt reads command files from the generated CLI and builds a
+// prompt asking the LLM to rewrite their Short descriptions.
+func buildHelpPrompt(outputDir string) string {
+ commands := extractCommands(outputDir)
+ if len(commands) == 0 {
+ return ""
+ }
+
+ var b strings.Builder
+ b.WriteString("Rewrite these CLI command descriptions to be developer-friendly.\n")
+ b.WriteString("Rules: under 80 chars, starts with a verb, no API jargon.\n\n")
+ b.WriteString("Current:\n")
+ for _, cmd := range commands {
+ b.WriteString(fmt.Sprintf("- %s: %q\n", cmd.Name, cmd.Description))
+ }
+ b.WriteString("\nReturn ONLY a JSON array, no other text:\n")
+ b.WriteString(`[{"command": "example", "description": "Improved description here"}]`)
+ b.WriteString("\n")
+
+ return b.String()
+}
+
+// buildExamplePrompt reads commands and builds a prompt asking the LLM to
+// generate realistic examples for each command.
+func buildExamplePrompt(outputDir string) string {
+ commands := extractCommands(outputDir)
+ if len(commands) == 0 {
+ return ""
+ }
+
+ cliName := filepath.Base(outputDir)
+
+ var b strings.Builder
+ b.WriteString(fmt.Sprintf("Write 2-3 realistic examples for each command in %s.\n", cliName))
+ b.WriteString("Each example should show a real developer workflow with a comment.\n\n")
+ b.WriteString("Commands:\n")
+ for _, cmd := range commands {
+ flags := "none"
+ if len(cmd.Flags) > 0 {
+ flags = strings.Join(cmd.Flags, ", ")
+ }
+ method := cmd.Method
+ if method == "" {
+ method = "GET"
+ }
+ path := cmd.Path
+ if path == "" {
+ path = "/" + strings.ReplaceAll(cmd.Name, " ", "/")
+ }
+ b.WriteString(fmt.Sprintf("- %s (%s %s, flags: %s)\n", cmd.Name, method, path, flags))
+ }
+ b.WriteString("\nReturn ONLY a JSON array, no other text:\n")
+ b.WriteString(fmt.Sprintf(`[{"command": "example list", "examples": ["# List all examples\n%s example list --limit 10"]}]`, cliName))
+ b.WriteString("\n")
+
+ return b.String()
+}
+
+// buildREADMEPrompt reads the current README and builds a prompt asking the
+// LLM to rewrite it to sell the tool to developers.
+func buildREADMEPrompt(outputDir, apiName string) string {
+ readmePath := filepath.Join(outputDir, "README.md")
+ content, err := os.ReadFile(readmePath)
+ if err != nil {
+ return ""
+ }
+ if len(content) == 0 {
+ return ""
+ }
+
+ var b strings.Builder
+ b.WriteString(fmt.Sprintf("Rewrite this CLI README to sell the tool to developers.\n\n"))
+ b.WriteString(fmt.Sprintf("API: %s\n", apiName))
+ b.WriteString(fmt.Sprintf("Current README:\n%s\n\n", string(content)))
+ b.WriteString("Write a README with:\n")
+ b.WriteString("1. One-line hook that makes developers want to install it\n")
+ b.WriteString("2. Why this exists (what gap it fills)\n")
+ b.WriteString("3. Quick start (3 commands max)\n")
+ b.WriteString("4. Full command list from the current README\n\n")
+ b.WriteString("Keep existing section structure but improve the copy.\n")
+ b.WriteString("Return ONLY the markdown content, no code fences.\n")
+
+ return b.String()
+}
+
+// extractCommands reads .go files from the generated CLI's internal/cli/
+// directory and extracts command metadata.
+func extractCommands(outputDir string) []commandInfo {
+ cliDir := filepath.Join(outputDir, "internal", "cli")
+ entries, err := os.ReadDir(cliDir)
+ if err != nil {
+ return nil
+ }
+
+ infraFiles := map[string]bool{
+ "helpers.go": true,
+ "root.go": true,
+ "doctor.go": true,
+ "auth.go": true,
+ }
+
+ shortRe := regexp.MustCompile(`Short:\s*"([^"]*)"`)
+ useRe := regexp.MustCompile(`Use:\s*"([^"]*)"`)
+ flagRe := regexp.MustCompile(`Flags\(\)\.\w+Var\w*\([^,]+,\s*"([^"]+)"`)
+ methodRe := regexp.MustCompile(`(?:Method|method):\s*"([^"]*)"`)
+ pathRe := regexp.MustCompile(`(?:Path|path|URL|url):\s*"([^"]*)"`)
+
+ var commands []commandInfo
+ for _, e := range entries {
+ name := e.Name()
+ if e.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") {
+ continue
+ }
+ if infraFiles[name] {
+ continue
+ }
+
+ data, err := os.ReadFile(filepath.Join(cliDir, name))
+ if err != nil {
+ continue
+ }
+ content := string(data)
+
+ cmd := commandInfo{
+ Name: strings.TrimSuffix(name, ".go"),
+ }
+
+ if m := shortRe.FindStringSubmatch(content); len(m) > 1 {
+ cmd.Description = m[1]
+ }
+ if m := useRe.FindStringSubmatch(content); len(m) > 1 {
+ cmd.Name = m[1]
+ }
+ if m := methodRe.FindStringSubmatch(content); len(m) > 1 {
+ cmd.Method = m[1]
+ }
+ if m := pathRe.FindStringSubmatch(content); len(m) > 1 {
+ cmd.Path = m[1]
+ }
+
+ flags := flagRe.FindAllStringSubmatch(content, -1)
+ for _, f := range flags {
+ if len(f) > 1 {
+ cmd.Flags = append(cmd.Flags, "--"+f[1])
+ }
+ }
+
+ commands = append(commands, cmd)
+ }
+ return commands
+}
diff --git a/internal/pipeline/fullrun.go b/internal/pipeline/fullrun.go
index 8ea46877..b216a883 100644
--- a/internal/pipeline/fullrun.go
+++ b/internal/pipeline/fullrun.go
@@ -7,6 +7,8 @@ import (
"path/filepath"
"strings"
"time"
+
+ "github.com/mvanhorn/cli-printing-press/internal/llmpolish"
)
// FullRunResult holds everything the press produced for one API.
@@ -30,6 +32,9 @@ type FullRunResult struct {
SpecEndpoints int
CoveragePercent int
+ // Step 2.5: Polish
+ PolishResult *llmpolish.PolishResult
+
// Step 4: Dogfood
Dogfood *DogfoodResults
DogfoodError string
@@ -96,6 +101,17 @@ func MakeBestCLI(apiName, level, specFlag, specURL, outputDir, pressBinary strin
return result
}
+ // Step 2.5: LLM Polish
+ polishResult, polishErr := llmpolish.Polish(llmpolish.PolishRequest{
+ APIName: apiName,
+ OutputDir: outputDir,
+ })
+ if polishErr != nil {
+ result.Errors = append(result.Errors, "polish: "+polishErr.Error())
+ } else {
+ result.PolishResult = polishResult
+ }
+
// Step 3: Count commands and resources
resources := listResources(outputDir)
result.ResourceCount = len(resources)
@@ -322,6 +338,20 @@ func PrintComparisonTable(results []*FullRunResult) string {
return fmt.Sprintf("%d/%d (%d%%)", r.Dogfood.PassedCommands, r.Dogfood.TotalCommands, pct)
})
+ // LLM Polish
+ writeRow(&b, "LLM Polish", results, func(r *FullRunResult) string {
+ if r.PolishResult == nil {
+ return "n/a"
+ }
+ if r.PolishResult.Skipped {
+ return "skipped"
+ }
+ return fmt.Sprintf("%dh/%de/%v",
+ r.PolishResult.HelpTextsImproved,
+ r.PolishResult.ExamplesAdded,
+ r.PolishResult.READMERewritten)
+ })
+
// Fix plans generated
writeRow(&b, "Fix Plans", results, func(r *FullRunResult) string {
return fmt.Sprintf("%d", len(r.FixPlans))
← b862cf4c feat(templates): agent-native CLI improvements - stdin, idem
·
back to Cli Printing Press
·
feat(llm): add LLM brain before generation - the press under a1e7af4e →