← back to Cli Printing Press
fix(scorecard): replace presence checks with quality-based scoring
5c6840d59d3003296728cdeb19450883694521fc · 2026-03-25 19:16:09 -0700 · Matt Van Horn
The scorecard was grading its own homework - every generated CLI scored
90+/100 because it checked for strings the templates always produce.
Now each dimension has 3 tiers: present (0-3), functional (4-6), excellent (7-10).
Quality checks added:
- Terminal UX: samples command files for meaningful descriptions, detects
placeholder values (abc123, my-resource, your-key-here)
- README: checks for cookbook section, realistic Quick Start, human description
- Agent Native: counts --stdin examples in commands, checks for prompts
- Auth: checks secure permissions, token masking, OAuth2 refresh
- Error Handling: checks rate limit retry, idempotency, actionable suggestions
- Breadth: penalizes lazy 1-word descriptions, rewards quality descriptions
Before: fresh Petstore = 90/90 (Grade A). After: 75/100 (Grade B).
Before: polished Discord = 90/90 (Grade A). After: 86/100 (Grade A).
Grade A now requires genuine quality work, not just template rendering.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Files touched
A docs/plans/2026-03-25-fix-scorecard-too-easy-real-quality-plan.mdM internal/pipeline/scorecard.go
Diff
commit 5c6840d59d3003296728cdeb19450883694521fc
Author: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Date: Wed Mar 25 19:16:09 2026 -0700
fix(scorecard): replace presence checks with quality-based scoring
The scorecard was grading its own homework - every generated CLI scored
90+/100 because it checked for strings the templates always produce.
Now each dimension has 3 tiers: present (0-3), functional (4-6), excellent (7-10).
Quality checks added:
- Terminal UX: samples command files for meaningful descriptions, detects
placeholder values (abc123, my-resource, your-key-here)
- README: checks for cookbook section, realistic Quick Start, human description
- Agent Native: counts --stdin examples in commands, checks for prompts
- Auth: checks secure permissions, token masking, OAuth2 refresh
- Error Handling: checks rate limit retry, idempotency, actionable suggestions
- Breadth: penalizes lazy 1-word descriptions, rewards quality descriptions
Before: fresh Petstore = 90/90 (Grade A). After: 75/100 (Grade B).
Before: polished Discord = 90/90 (Grade A). After: 86/100 (Grade A).
Grade A now requires genuine quality work, not just template rendering.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
...-25-fix-scorecard-too-easy-real-quality-plan.md | 246 ++++++++++++
internal/pipeline/scorecard.go | 433 +++++++++++++++++----
2 files changed, 597 insertions(+), 82 deletions(-)
diff --git a/docs/plans/2026-03-25-fix-scorecard-too-easy-real-quality-plan.md b/docs/plans/2026-03-25-fix-scorecard-too-easy-real-quality-plan.md
new file mode 100644
index 00000000..b9dd67a0
--- /dev/null
+++ b/docs/plans/2026-03-25-fix-scorecard-too-easy-real-quality-plan.md
@@ -0,0 +1,246 @@
+---
+title: "fix: Scorecard is trivially easy - make it measure real quality"
+type: fix
+status: active
+date: 2026-03-25
+---
+
+# Scorecard is Trivially Easy - Make It Measure Real Quality
+
+## Problem
+
+Every generated CLI scores 90/100 (or 90/90 before Vision). This is absurd. The scorecard checks for the *presence* of patterns that the generator *always* includes. It's grading its own homework.
+
+### Why Every CLI Gets 10/10 on Everything
+
+| Dimension | What It Checks | Why It's Always 10/10 |
+|---|---|---|
+| Output Modes | Does root.go contain "json", "plain", "select"? | Template always includes these flags |
+| Auth | Does config.go have `os.Getenv`? | Template always uses env vars |
+| Error Handling | Does helpers.go have "hint:" and "code:" strings? | Template always includes hints + codes |
+| Terminal UX | Does helpers.go have "colorEnabled", "NO_COLOR", "isatty"? | Template always includes these |
+| README | Does README.md have "Quick Start", "Agent Usage", etc.? | Template always generates these sections |
+| Doctor | Does doctor.go have HTTP health check patterns? | Template always includes health checks |
+| Agent Native | Does root.go have "json", "select", "dry-run", "stdin", "yes"? | Template always includes all flags |
+| Local Cache | Does client.go have "cacheDir", "readCache", "no-cache"? | Template always includes caching |
+| Breadth | How many .go files in internal/cli/? | Any real API generates 60+ files |
+
+**Result:** The scorecard measures "did the template render correctly?" not "is this CLI any good?"
+
+### What a Real Scorecard Should Measure
+
+A 10/10 should be **hard to achieve**. It should require:
+- Quality, not just presence
+- Correct behavior, not just correct strings
+- Real developer experience polish, not template boilerplate
+- Comparison to the best tools in the category, not absolute presence
+
+---
+
+## Proposed Solution: Quality-Based Scoring
+
+Replace presence checks with quality checks. Each dimension gets 3 tiers:
+
+| Points | Tier | Meaning |
+|---|---|---|
+| 0-3 | **Present** | The feature exists (what current scoring gives 10 for) |
+| 4-6 | **Functional** | The feature works correctly and handles edge cases |
+| 7-10 | **Excellent** | The feature matches or exceeds best-in-class tools |
+
+### Dimension-by-Dimension Redesign
+
+#### Output Modes (0-10)
+
+**Current (always 10):** Checks if "json", "plain", "select", "table", "csv" strings exist in root.go.
+
+**Proposed:**
+| Points | Criteria | How to Check |
+|---|---|---|
+| +2 | --json flag exists | `strings.Contains(root.go, `"json"`)` |
+| +1 | --plain flag exists | `strings.Contains(root.go, "plain")` |
+| +1 | --select flag exists | `strings.Contains(root.go, "select")` |
+| +1 | --csv flag exists | `strings.Contains(root.go, "csv")` |
+| +1 | --quiet flag exists | `strings.Contains(root.go, "quiet")` |
+| +2 | Output formatting is field-aware (select actually filters fields, not just string matching) | Check helpers.go for `filterFields` function with real JSON parsing logic (json.Unmarshal + field iteration, not just string ops) |
+| +2 | Pagination outputs NDJSON progress events | Check helpers.go for "event" + "page_fetch" (progress event pattern) |
+
+**Expected baseline:** 5-6/10 (template gets presence, quality takes work)
+
+#### Auth (0-10)
+
+**Current (always 10):** Counts `os.Getenv` in config.go + checks auth.go exists.
+
+**Proposed:**
+| Points | Criteria | How to Check |
+|---|---|---|
+| +2 | At least one env var for auth | `os.Getenv` count >= 1 in config.go |
+| +1 | Auth file exists | auth.go present |
+| +2 | Token stored securely (config file has restricted permissions) | Check config.go for `0o600` or `0600` permission bits |
+| +2 | Token masking in output | Check client.go or helpers.go for token masking pattern (e.g., showing only last 4 chars) |
+| +1 | Multiple auth methods (env var + config file + flag) | Count distinct auth source patterns >= 2 |
+| +2 | OAuth2 browser flow with refresh | Check auth.go for "oauth2" + "refresh" + "browser" patterns |
+
+**Expected baseline:** 3-5/10 (simple API key gets 3, full OAuth2 gets 8+)
+
+#### Error Handling (0-10)
+
+**Current (always 10):** Checks for "hint:" and counts "code:" occurrences.
+
+**Proposed:**
+| Points | Criteria | How to Check |
+|---|---|---|
+| +2 | At least 3 distinct exit codes | Count unique `code:` values >= 3 |
+| +1 | Error messages include hints | "hint:" or "Hint:" present |
+| +2 | Rate limit handling (429 detection + retry) | Check client.go for "429" AND ("Retry-After" or "backoff") |
+| +2 | Idempotency handling (409 = success) | Check helpers.go for "409" AND "already exists" with exit 0 |
+| +1 | Not-found returns specific exit code | Check for "404" AND separate exit code |
+| +2 | Error messages include actionable suggestions (not just "failed") | Check for "Run" + "doctor" or "try" in error messages (actionable patterns, not just status codes) |
+
+**Expected baseline:** 4-6/10
+
+#### Terminal UX (0-10)
+
+**Current (always 10):** Checks for "colorEnabled", "NO_COLOR", "isatty".
+
+**Proposed:**
+| Points | Criteria | How to Check |
+|---|---|---|
+| +2 | NO_COLOR support | "NO_COLOR" in helpers.go |
+| +1 | TTY detection | "isatty" in helpers.go |
+| +1 | Color toggle flag | "no-color" in root.go |
+| +2 | Table output uses aligned columns (tabwriter or similar) | Check helpers.go for "tabwriter" |
+| +2 | Help text descriptions are meaningful (not raw spec jargon) | Sample 5 random command files, check Short descriptions are > 10 chars and don't just repeat the command name |
+| +2 | Example values are realistic (not "abc123" or "value") | Sample 5 random command files, check Example lines don't contain "abc123" or bare "value" |
+
+**Expected baseline:** 4-6/10 (template gets basics, description quality requires polish)
+
+#### README (0-10)
+
+**Current (always 10):** Checks for section header strings.
+
+**Proposed:**
+| Points | Criteria | How to Check |
+|---|---|---|
+| +1 | Has "Quick Start" section | String check |
+| +1 | Has "Agent Usage" section | String check |
+| +1 | Has "Doctor" section | String check |
+| +1 | Has "Troubleshooting" section | String check |
+| +2 | Quick Start has realistic example (not placeholder values) | Check Quick Start section doesn't contain "abc123", "your-key-here", "USER/tap" |
+| +2 | Has Cookbook or Recipes section with 3+ examples | Check for "Cookbook" or "Recipes" AND count code blocks >= 3 |
+| +2 | Commands section lists actual command count matching generated files | Compare README command count to `ls internal/cli/*.go | wc -l` - within 20% |
+
+**Expected baseline:** 4-6/10
+
+#### Doctor (0-10)
+
+**Current (always 10):** Counts HTTP patterns.
+
+**Proposed:**
+| Points | Criteria | How to Check |
+|---|---|---|
+| +2 | Doctor command exists | doctor.go present |
+| +2 | Checks auth validity | "auth" or "token" in doctor.go check logic |
+| +2 | Checks API connectivity | HTTP request pattern in doctor.go |
+| +2 | Checks config file health | "config" check in doctor.go |
+| +2 | Checks version compatibility | "version" in doctor.go check logic OR HEAD request to API endpoint |
+
+**Expected baseline:** 4-6/10
+
+#### Agent Native (0-10)
+
+**Current (always 10):** Checks for flag strings.
+
+**Proposed:**
+| Points | Criteria | How to Check |
+|---|---|---|
+| +1 | --json flag | Present |
+| +1 | --select flag | Present |
+| +1 | --dry-run flag | Present |
+| +1 | --stdin flag | Present |
+| +1 | --yes flag | Present |
+| +1 | Non-interactive (no prompts in command logic) | Absence of "Scan" or "ReadLine" or "Prompt" in command files |
+| +2 | Typed exit codes (at least 5 distinct) | Count unique exit codes >= 5 |
+| +2 | --stdin examples in at least 3 commands | Count command files containing "--stdin" in Example >= 3 |
+
+**Expected baseline:** 5-7/10 (flags are easy, stdin examples require work)
+
+#### Local Cache (0-10)
+
+**Current (always 10 for templates, 7 without sqlite):** Checks for cache patterns.
+
+**Proposed:**
+| Points | Criteria | How to Check |
+|---|---|---|
+| +2 | GET response caching | "readCache" or "writeCache" in client.go |
+| +1 | --no-cache bypass flag | "no-cache" or "NoCache" |
+| +2 | Cache has TTL (not infinite) | Time duration or expiry check near cache logic |
+| +2 | Cache directory is configurable or uses XDG | Check for ".cache" or "XDG_CACHE_HOME" |
+| +3 | SQLite or embedded DB backend (not just file cache) | "sqlite" or "bolt" or "badger" in any Go file |
+
+**Expected baseline:** 5-7/10 (file cache = 5, sqlite = 10)
+
+#### Breadth (0-10)
+
+**Current scoring is reasonable.** Keep it but add a quality gate:
+
+**Proposed addition:**
+| Points | Criteria | How to Check |
+|---|---|---|
+| (existing) | Command file count tiers | Same as current |
+| -2 penalty | More than 50% of commands have identical 1-word Short descriptions (e.g., "Get", "Create", "Delete") | Sample Short: fields, count ones that are <= 1 word |
+
+**Expected baseline:** 7-9/10 (still easy for large APIs, penalty catches lazy descriptions)
+
+#### Vision (0-10)
+
+**Already redesigned in previous commit.** Keep as-is - it's the only dimension that was already hard (pure API wrappers correctly score 0).
+
+---
+
+## Implementation
+
+### File: `internal/pipeline/scorecard.go`
+
+Rewrite each `score*` function with the new criteria. Keep the same function signatures so no callers change.
+
+### Key implementation details:
+
+1. **Sampling for quality checks:** For dimensions that check "quality" across many files (Terminal UX description check, Agent Native stdin examples), sample 5 random command files rather than checking all. Use deterministic seed (hash of API name) for reproducibility.
+
+2. **The "abc123" detector:** Create a helper `hasPlaceholderValues(content string) bool` that checks for common placeholder patterns: "abc123", "my-resource", "your-key-here", "USER/tap", bare "value" as positional arg.
+
+3. **Description quality check:** `isQualityDescription(desc string) bool` returns true if description is > 10 chars, doesn't just repeat the command verb, and contains at least one space (multi-word).
+
+### Expected Score Distribution After Fix
+
+| CLI Quality | Old Score | New Score |
+|---|---|---|
+| Fresh from generator (no polish) | 90-100/100 | 45-55/100 (Grade C) |
+| Generator + LLM polish pass | 90-100/100 | 60-70/100 (Grade B) |
+| Generator + polish + manual GOAT fixes | 90-100/100 | 75-85/100 (Grade A) |
+| Handcrafted gogcli-level quality | N/A | 90-100/100 |
+| discrawl-level (visionary + quality) | N/A | 85-95/100 |
+
+**Key insight:** Grade A should require WORK. Getting there should mean the CLI is genuinely good, not that the template rendered.
+
+---
+
+## Acceptance Criteria
+
+- [ ] Fresh-from-generator Petstore CLI scores 40-55/100 (was 90+)
+- [ ] Discord CLI (with our GOAT fixes) scores 55-70/100 (was 90+)
+- [ ] No dimension auto-scores 10/10 from template output alone
+- [ ] At least 3 dimensions require quality checks that sampling validates
+- [ ] Placeholder value detection catches "abc123", "my-resource", "your-key-here"
+- [ ] Description quality check distinguishes "Get" from "Get a channel by ID"
+- [ ] `go build ./...` and `go test ./...` pass
+- [ ] Scorecard output format unchanged (same table, same grades)
+
+---
+
+## Sources
+
+- Current scorecard: `internal/pipeline/scorecard.go`
+- Current scorecard CLI: `internal/cli/scorecard.go`
+- Generator templates: `internal/generator/templates/*.tmpl`
+- Previous Vision plan: `docs/plans/2026-03-25-feat-visionary-research-phase-plan.md`
diff --git a/internal/pipeline/scorecard.go b/internal/pipeline/scorecard.go
index 54fc58c7..c2955ef4 100644
--- a/internal/pipeline/scorecard.go
+++ b/internal/pipeline/scorecard.go
@@ -93,21 +93,35 @@ func RunScorecard(outputDir, pipelineDir string) (*Scorecard, error) {
}
func scoreOutputModes(dir string) int {
- content := readFileContent(filepath.Join(dir, "internal", "cli", "root.go"))
+ rootContent := readFileContent(filepath.Join(dir, "internal", "cli", "root.go"))
+ helpersContent := readFileContent(filepath.Join(dir, "internal", "cli", "helpers.go"))
score := 0
- if strings.Contains(content, "json") {
- score += 2
+ // Presence tier (max 5)
+ if strings.Contains(rootContent, `"json"`) {
+ score += 1
}
- if strings.Contains(content, "plain") {
- score += 2
+ if strings.Contains(rootContent, `"plain"`) {
+ score += 1
}
- if strings.Contains(content, "select") {
- score += 2
+ if strings.Contains(rootContent, `"select"`) {
+ score += 1
+ }
+ if strings.Contains(rootContent, `"csv"`) {
+ score += 1
+ }
+ if strings.Contains(rootContent, `"quiet"`) {
+ score += 1
}
- if strings.Contains(content, "table") {
+ // Quality tier: field-aware select (real JSON parsing, not string ops)
+ if strings.Contains(helpersContent, "filterFields") && strings.Contains(helpersContent, "json.Unmarshal") {
score += 2
}
- if strings.Contains(content, "csv") {
+ // Quality tier: pagination progress events
+ if strings.Contains(helpersContent, "page_fetch") || strings.Contains(helpersContent, "ndjson") {
+ score += 1
+ }
+ // Quality tier: tabwriter for aligned output
+ if strings.Contains(helpersContent, "tabwriter") {
score += 2
}
if score > 10 {
@@ -117,17 +131,43 @@ func scoreOutputModes(dir string) int {
}
func scoreAuth(dir string) int {
- score := 0
configContent := readFileContent(filepath.Join(dir, "internal", "config", "config.go"))
- envCount := strings.Count(configContent, "os.Getenv")
- if envCount >= 2 {
- score += 8
- } else if envCount >= 1 {
- score += 5
+ authContent := readFileContent(filepath.Join(dir, "internal", "cli", "auth.go"))
+ clientContent := readFileContent(filepath.Join(dir, "internal", "client", "client.go"))
+ score := 0
+ // Presence: at least one env var
+ if strings.Count(configContent, "os.Getenv") >= 1 {
+ score += 2
}
- if fileExists(filepath.Join(dir, "internal", "cli", "auth.go")) {
+ // Presence: auth file exists
+ if authContent != "" {
+ score += 1
+ }
+ // Quality: secure config file permissions (0o600 or 0600)
+ if strings.Contains(configContent, "0o600") || strings.Contains(configContent, "0600") || strings.Contains(configContent, "0o700") || strings.Contains(configContent, "0700") {
score += 2
}
+ // Quality: token masking in output (showing partial token)
+ if strings.Contains(clientContent, "mask") || strings.Contains(clientContent, "***") || strings.Contains(clientContent, "last 4") || (strings.Contains(clientContent, "Authorization") && strings.Contains(clientContent, "[:")) {
+ score += 2
+ }
+ // Quality: multiple auth methods (env var + config + flag)
+ authSources := 0
+ if strings.Contains(configContent, "os.Getenv") {
+ authSources++
+ }
+ if strings.Contains(configContent, "ReadFile") || strings.Contains(configContent, "Load") {
+ authSources++
+ }
+ if authSources >= 2 {
+ score += 1
+ }
+ // Excellence: OAuth2 browser flow with refresh
+ if strings.Contains(authContent, "oauth2") || strings.Contains(authContent, "OAuth2") {
+ if strings.Contains(authContent, "refresh") || strings.Contains(authContent, "Refresh") {
+ score += 2
+ }
+ }
if score > 10 {
score = 10
}
@@ -135,17 +175,36 @@ func scoreAuth(dir string) int {
}
func scoreErrorHandling(dir string) int {
- content := readFileContent(filepath.Join(dir, "internal", "cli", "helpers.go"))
+ helpersContent := readFileContent(filepath.Join(dir, "internal", "cli", "helpers.go"))
+ clientContent := readFileContent(filepath.Join(dir, "internal", "client", "client.go"))
score := 0
- if strings.Contains(content, "hint:") || strings.Contains(content, "Hint:") {
- score += 5
+ // Presence: error hints
+ if strings.Contains(helpersContent, "hint:") || strings.Contains(helpersContent, "Hint:") {
+ score += 1
+ }
+ // Presence: at least 3 distinct exit codes
+ exitCount := strings.Count(helpersContent, "code:")
+ if exitCount >= 3 {
+ score += 2
+ } else if exitCount >= 1 {
+ score += 1
+ }
+ // Quality: rate limit handling (429 + retry)
+ if strings.Contains(clientContent, "429") && (strings.Contains(clientContent, "Retry-After") || strings.Contains(clientContent, "backoff") || strings.Contains(clientContent, "retry")) {
+ score += 2
}
- // Count typed exit codes (cliError with code:)
- exitCount := strings.Count(content, "code:")
- if exitCount > 5 {
- exitCount = 5
+ // Quality: idempotency (409 = already exists = success)
+ if strings.Contains(helpersContent, "409") && strings.Contains(helpersContent, "already exists") {
+ score += 2
+ }
+ // Quality: 404 with specific exit code
+ if strings.Contains(helpersContent, "404") {
+ score += 1
+ }
+ // Excellence: actionable suggestions in errors (not just codes)
+ if (strings.Contains(helpersContent, "Run") || strings.Contains(helpersContent, "try")) && strings.Contains(helpersContent, "doctor") {
+ score += 2
}
- score += exitCount
if score > 10 {
score = 10
}
@@ -153,17 +212,50 @@ func scoreErrorHandling(dir string) int {
}
func scoreTerminalUX(dir string) int {
- content := readFileContent(filepath.Join(dir, "internal", "cli", "helpers.go"))
+ helpersContent := readFileContent(filepath.Join(dir, "internal", "cli", "helpers.go"))
+ rootContent := readFileContent(filepath.Join(dir, "internal", "cli", "root.go"))
score := 0
- if strings.Contains(content, "colorEnabled") {
- score += 5
+ // Presence: NO_COLOR support
+ if strings.Contains(helpersContent, "NO_COLOR") {
+ score += 1
}
- if strings.Contains(content, "NO_COLOR") {
- score += 3
+ // Presence: TTY detection
+ if strings.Contains(helpersContent, "isatty") {
+ score += 1
}
- if strings.Contains(content, "isatty") {
+ // Presence: no-color flag
+ if strings.Contains(rootContent, "no-color") {
+ score += 1
+ }
+ // Quality: tabwriter for aligned columns
+ if strings.Contains(helpersContent, "tabwriter") {
score += 2
}
+ // Quality: help text descriptions are meaningful (not just verb names)
+ cmdFiles := sampleCommandFiles(dir, 5)
+ goodDescs := 0
+ for _, content := range cmdFiles {
+ if hasQualityDescription(content) {
+ goodDescs++
+ }
+ }
+ if goodDescs >= 4 {
+ score += 2
+ } else if goodDescs >= 2 {
+ score += 1
+ }
+ // Quality: example values are realistic (not abc123 or bare "value")
+ goodExamples := 0
+ for _, content := range cmdFiles {
+ if !hasPlaceholderValues(content) {
+ goodExamples++
+ }
+ }
+ if goodExamples >= 4 {
+ score += 3
+ } else if goodExamples >= 2 {
+ score += 1
+ }
if score > 10 {
score = 10
}
@@ -173,16 +265,35 @@ func scoreTerminalUX(dir string) int {
func scoreREADME(dir string) int {
content := readFileContent(filepath.Join(dir, "README.md"))
score := 0
- sections := map[string]int{
- "Quick Start": 2,
- "Output Formats": 2,
- "Agent Usage": 2,
- "Troubleshooting": 2,
- "Doctor": 2,
- }
- for section, pts := range sections {
+ // Presence: key sections exist (1pt each, max 4)
+ for _, section := range []string{"Quick Start", "Agent Usage", "Doctor", "Troubleshooting"} {
if strings.Contains(content, section) {
- score += pts
+ score++
+ }
+ }
+ // Quality: Quick Start has no placeholder values
+ qsIdx := strings.Index(content, "Quick Start")
+ if qsIdx >= 0 {
+ qsSection := content[qsIdx:min(qsIdx+500, len(content))]
+ if !strings.Contains(qsSection, "your-key-here") && !strings.Contains(qsSection, "USER/tap") && !strings.Contains(qsSection, "abc123") {
+ score += 2
+ }
+ }
+ // Quality: has Cookbook or Recipes with 3+ code blocks
+ if strings.Contains(content, "Cookbook") || strings.Contains(content, "Recipes") {
+ codeBlocks := strings.Count(content, "```")
+ if codeBlocks >= 6 { // 3+ examples = 6+ backtick pairs
+ score += 2
+ } else {
+ score += 1
+ }
+ }
+ // Quality: README describes the API in human terms (not raw spec text)
+ lines := strings.SplitN(content, "\n", 5)
+ if len(lines) >= 3 {
+ header := strings.Join(lines[:3], " ")
+ if !strings.Contains(header, "Preview of") && !strings.Contains(header, "specification") && len(header) > 20 {
+ score += 2
}
}
if score > 10 {
@@ -193,15 +304,30 @@ func scoreREADME(dir string) int {
func scoreDoctor(dir string) int {
content := readFileContent(filepath.Join(dir, "internal", "cli", "doctor.go"))
- // Count health check patterns (both stdlib and variable-based calls)
- healthChecks := strings.Count(content, "http.Get") +
- strings.Count(content, "http.Head") +
- strings.Count(content, "http.NewRequest") +
- strings.Count(content, "http.Client") +
- strings.Count(content, "httpClient.Get") +
- strings.Count(content, "httpClient.Head") +
- strings.Count(content, "httpClient.Do")
- score := healthChecks * 2
+ if content == "" {
+ return 0
+ }
+ score := 0
+ // Presence: doctor command exists
+ score += 2
+ // Quality: checks auth/token validity
+ if strings.Contains(content, "auth") || strings.Contains(content, "token") || strings.Contains(content, "Token") {
+ score += 2
+ }
+ // Quality: checks API connectivity (makes an HTTP request)
+ hasHTTP := strings.Contains(content, "http.Get") || strings.Contains(content, "http.Head") ||
+ strings.Contains(content, "http.NewRequest") || strings.Contains(content, "httpClient")
+ if hasHTTP {
+ score += 2
+ }
+ // Quality: checks config file
+ if strings.Contains(content, "config") || strings.Contains(content, "Config") {
+ score += 2
+ }
+ // Excellence: checks version or API compatibility
+ if strings.Contains(content, "version") || strings.Contains(content, "Version") {
+ score += 2
+ }
if score > 10 {
score = 10
}
@@ -211,36 +337,60 @@ func scoreDoctor(dir string) int {
func scoreAgentNative(dir string) int {
rootContent := readFileContent(filepath.Join(dir, "internal", "cli", "root.go"))
helpersContent := readFileContent(filepath.Join(dir, "internal", "cli", "helpers.go"))
- combined := rootContent + helpersContent
-
score := 0
- if strings.Contains(combined, "json") {
- score += 2
+ // Presence: core agent flags (1pt each, max 5)
+ if strings.Contains(rootContent, `"json"`) {
+ score++
}
- if strings.Contains(combined, "select") {
- score += 2
+ if strings.Contains(rootContent, `"select"`) {
+ score++
}
- if strings.Contains(combined, "dry-run") || strings.Contains(combined, "dryRun") || strings.Contains(combined, "dry_run") {
- score += 2
+ if strings.Contains(rootContent, "dry-run") {
+ score++
}
- if strings.Contains(combined, "non-interactive") || strings.Contains(combined, "nonInteractive") {
- score += 1
+ if strings.Contains(rootContent, "stdin") {
+ score++
}
- // Check for --stdin support
- if strings.Contains(combined, "stdin") {
- score += 1
+ if strings.Contains(rootContent, `"yes"`) {
+ score++
}
- // Check for --yes flag
- if strings.Contains(combined, `"yes"`) {
- score += 1
+ // Quality: non-interactive (no prompts in command files)
+ cmdFiles := sampleCommandFiles(dir, 5)
+ hasPrompts := false
+ for _, content := range cmdFiles {
+ if strings.Contains(content, "bufio.NewScanner(os.Stdin)") || strings.Contains(content, "Prompt") || strings.Contains(content, "ReadLine") {
+ hasPrompts = true
+ break
+ }
}
- // Check for idempotency handling (409 or "already exists")
- if strings.Contains(helpersContent, "409") || strings.Contains(helpersContent, "already exists") {
- score += 1
+ if !hasPrompts && len(cmdFiles) > 0 {
+ score++
}
- // Check for --human-friendly flag (agent-safe-by-default coloring)
- if strings.Contains(combined, "human-friendly") || strings.Contains(combined, "humanFriendly") {
- score += 1
+ // Quality: typed exit codes (5+ distinct)
+ exitCount := strings.Count(helpersContent, "code:")
+ if exitCount >= 5 {
+ score += 2
+ } else if exitCount >= 3 {
+ score++
+ }
+ // Excellence: --stdin examples in command files (at least 3 commands show stdin usage)
+ stdinExamples := 0
+ for _, content := range cmdFiles {
+ if strings.Contains(content, "--stdin") && strings.Contains(content, "Example") {
+ stdinExamples++
+ }
+ }
+ // Also check all command files for stdin examples, not just sample
+ allCmdFiles := sampleCommandFiles(dir, 0) // 0 = all
+ for _, content := range allCmdFiles {
+ if strings.Contains(content, "--stdin") && strings.Contains(content, "echo") {
+ stdinExamples++
+ }
+ }
+ if stdinExamples >= 3 {
+ score += 2
+ } else if stdinExamples >= 1 {
+ score++
}
if score > 10 {
score = 10
@@ -249,20 +399,30 @@ func scoreAgentNative(dir string) int {
}
func scoreLocalCache(dir string) int {
- score := 0
- // Check client for cache-related code
clientContent := readFileContent(filepath.Join(dir, "internal", "client", "client.go"))
- if strings.Contains(clientContent, "cacheDir") || strings.Contains(clientContent, "readCache") || strings.Contains(clientContent, "writeCache") {
- score += 5
+ score := 0
+ // Presence: GET response caching
+ if strings.Contains(clientContent, "readCache") || strings.Contains(clientContent, "writeCache") || strings.Contains(clientContent, "cacheDir") {
+ score += 2
}
+ // Presence: --no-cache bypass
if strings.Contains(clientContent, "no-cache") || strings.Contains(clientContent, "NoCache") {
+ score += 1
+ }
+ // Quality: cache has TTL (time-based expiry)
+ if strings.Contains(clientContent, "time.Duration") || strings.Contains(clientContent, "ModTime") || strings.Contains(clientContent, "TTL") || strings.Contains(clientContent, "ttl") {
score += 2
}
- // Check for sqlite or advanced caching
+ // Quality: XDG or standard cache directory
+ if strings.Contains(clientContent, ".cache") || strings.Contains(clientContent, "XDG_CACHE_HOME") || strings.Contains(clientContent, "UserCacheDir") {
+ score += 2
+ }
+ // Excellence: SQLite or embedded DB
for _, name := range []string{"internal/cache/cache.go", "internal/store/store.go"} {
content := readFileContent(filepath.Join(dir, name))
if strings.Contains(content, "sqlite") || strings.Contains(content, "bolt") || strings.Contains(content, "badger") {
score += 3
+ break
}
}
if score > 10 {
@@ -272,7 +432,6 @@ func scoreLocalCache(dir string) int {
}
func scoreBreadth(dir string) int {
- // Count command files in internal/cli/ (exclude infrastructure files)
cliDir := filepath.Join(dir, "internal", "cli")
entries, err := os.ReadDir(cliDir)
if err != nil {
@@ -280,8 +439,11 @@ func scoreBreadth(dir string) int {
}
infra := map[string]bool{
"helpers.go": true, "root.go": true, "doctor.go": true, "auth.go": true,
+ "export.go": true, "import.go": true, "search.go": true, "sync.go": true,
+ "tail.go": true, "analytics.go": true,
}
commandFiles := 0
+ lazyDescs := 0
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".go") {
continue
@@ -290,22 +452,43 @@ func scoreBreadth(dir string) int {
continue
}
commandFiles++
+ // Check for lazy 1-word descriptions
+ content := readFileContent(filepath.Join(cliDir, e.Name()))
+ if hasLazyDescription(content) {
+ lazyDescs++
+ }
}
+ var score int
switch {
case commandFiles >= 60:
- return 10
+ score = 8
case commandFiles >= 41:
- return 9
+ score = 7
case commandFiles >= 21:
- return 7
+ score = 5
case commandFiles >= 11:
- return 5
+ score = 4
case commandFiles >= 5:
- return 3
+ score = 2
default:
- return 0 // < 5 commands = basically empty
+ return 0
+ }
+ // Penalty: if more than 50% of commands have lazy 1-word descriptions
+ if commandFiles > 0 && lazyDescs*2 > commandFiles {
+ score -= 2
+ }
+ // Bonus: if descriptions are mostly quality (< 20% lazy)
+ if commandFiles > 0 && lazyDescs*5 < commandFiles {
+ score += 2
+ }
+ if score > 10 {
+ score = 10
+ }
+ if score < 0 {
+ score = 0
}
+ return score
}
func scoreVision(dir string) int {
@@ -348,6 +531,92 @@ func scoreVision(dir string) int {
return score
}
+// sampleCommandFiles reads up to n command files from internal/cli/.
+// If n <= 0, reads all command files.
+func sampleCommandFiles(dir string, n int) []string {
+ cliDir := filepath.Join(dir, "internal", "cli")
+ entries, err := os.ReadDir(cliDir)
+ if err != nil {
+ return nil
+ }
+ infra := map[string]bool{
+ "helpers.go": true, "root.go": true, "doctor.go": true, "auth.go": true,
+ "export.go": true, "import.go": true, "search.go": true, "sync.go": true,
+ "tail.go": true, "analytics.go": true,
+ }
+ var files []string
+ for _, e := range entries {
+ if e.IsDir() || !strings.HasSuffix(e.Name(), ".go") {
+ continue
+ }
+ if infra[e.Name()] {
+ continue
+ }
+ content := readFileContent(filepath.Join(cliDir, e.Name()))
+ if content != "" {
+ files = append(files, content)
+ }
+ if n > 0 && len(files) >= n {
+ break
+ }
+ }
+ return files
+}
+
+// hasPlaceholderValues checks if file content contains common placeholder values
+// that indicate unpolished examples.
+func hasPlaceholderValues(content string) bool {
+ placeholders := []string{"abc123", `"value"`, "my-resource", "your-key-here", "USER/tap"}
+ for _, p := range placeholders {
+ if strings.Contains(content, p) {
+ return true
+ }
+ }
+ return false
+}
+
+// hasQualityDescription checks if a command file has a meaningful Short description.
+// Returns true if the description is multi-word and doesn't just repeat the verb.
+func hasQualityDescription(content string) bool {
+ idx := strings.Index(content, "Short:")
+ if idx < 0 {
+ return false
+ }
+ // Extract the Short value (between quotes)
+ rest := content[idx:]
+ q1 := strings.Index(rest, `"`)
+ if q1 < 0 {
+ return false
+ }
+ q2 := strings.Index(rest[q1+1:], `"`)
+ if q2 < 0 {
+ return false
+ }
+ desc := rest[q1+1 : q1+1+q2]
+ // Quality: must be > 10 chars and contain a space (multi-word)
+ return len(desc) > 10 && strings.Contains(desc, " ")
+}
+
+// hasLazyDescription checks if a command has a 1-word or very short description.
+func hasLazyDescription(content string) bool {
+ idx := strings.Index(content, "Short:")
+ if idx < 0 {
+ return false
+ }
+ rest := content[idx:]
+ q1 := strings.Index(rest, `"`)
+ if q1 < 0 {
+ return false
+ }
+ q2 := strings.Index(rest[q1+1:], `"`)
+ if q2 < 0 {
+ return false
+ }
+ desc := rest[q1+1 : q1+1+q2]
+ words := strings.Fields(desc)
+ return len(words) <= 2
+}
+
func readFileContent(path string) string {
data, err := os.ReadFile(path)
if err != nil {
← e25a1b45 feat(vision): add Phase 0 Visionary Research - the press thi
·
back to Cli Printing Press
·
feat(profiler): add API Shape Intelligence Engine f499c4b5 →