← back to Cli Printing Press
fix(cli): crowd-sniff and generator improvements from Steam retro (#82)
8a8916fdb901df04c192d530d6a6d0fc70297f79 · 2026-03-30 19:41:22 -0700 · Trevin Chow
* fix(cli): use first path segment as crowd-sniff resource key, not joined path
deriveResourceKey() was joining multiple path segments with '/' (e.g.,
'users/posts'), which crashed the generator when it tried to create
files at paths like internal/cli/users/posts.go. Now uses only the
first significant segment as the resource key, grouping all methods
under it. Same fix applied to websniff specgen.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): detect auth patterns from npm SDK source in crowd-sniff
Scans npm SDK source code for API key (query/header), bearer token,
and X-Api-Key patterns. Populates the spec's auth section automatically
so generated CLIs wire auth correctly without manual spec edits.
Derives env var names from API name (e.g., STEAM_API_KEY from 'steam').
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): conditionally emit helpers and remove dead code from generator
helpers.go.tmpl now gates classifyDeleteError on HasDelete flag (spec
has DELETE endpoints). Permanently removes firstNonEmpty,
printOutputFiltered, and selectFieldsGlobal which were dead across
all tested APIs (Postman Explore, Steam).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): add top-level command promotion to generator
The generator now emits user-friendly promoted commands alongside
nested API commands. ISteamUser → steam-user, SteamUserStats →
steam-user-stats. Skips promotion for names colliding with built-in
commands (doctor, auth, sync, etc.). Confirmed needed across two APIs:
Postman Explore (7 manual commands) and Steam (10 manual commands).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): add YAML format safety net with JSON round-trip fallback
Spec parser now tolerates YAML style variations from non-Go tools
(Python yaml.dump, etc.) by attempting a yaml→map→json→struct
round-trip when direct unmarshaling produces empty resources. Added
json struct tags to all spec types. Safety net for crowd-sniff specs
that might be post-processed by external tools.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(cli): add Steam retro and comprehensive improvement plan
Steam retro identified 7 findings across crowd-sniff and generator.
Plan covers 5 implementation units: resource naming, auth detection,
command promotion, dead code emission, and YAML safety.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(cli): mark crowd-sniff retro plan as completed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(cli): fix gofmt, staticcheck raw string, and unused var in crowd-sniff
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Files touched
A docs/plans/2026-03-30-004-fix-crowd-sniff-and-generator-retro-improvements-plan.mdA docs/retros/2026-03-30-steam-retro.mdM internal/cli/crowd_sniff.goM internal/crowdsniff/aggregate.goM internal/crowdsniff/npm.goM internal/crowdsniff/npm_test.goM internal/crowdsniff/specgen.goM internal/crowdsniff/specgen_test.goM internal/crowdsniff/types.goM internal/generator/generator.goM internal/generator/generator_test.goA internal/generator/templates/command_promoted.go.tmplM internal/generator/templates/helpers.go.tmplM internal/generator/templates/root.go.tmplM internal/spec/spec.goM internal/spec/spec_test.goM internal/websniff/specgen.go
Diff
commit 8a8916fdb901df04c192d530d6a6d0fc70297f79
Author: Trevin Chow <trevin@trevinchow.com>
Date: Mon Mar 30 19:41:22 2026 -0700
fix(cli): crowd-sniff and generator improvements from Steam retro (#82)
* fix(cli): use first path segment as crowd-sniff resource key, not joined path
deriveResourceKey() was joining multiple path segments with '/' (e.g.,
'users/posts'), which crashed the generator when it tried to create
files at paths like internal/cli/users/posts.go. Now uses only the
first significant segment as the resource key, grouping all methods
under it. Same fix applied to websniff specgen.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): detect auth patterns from npm SDK source in crowd-sniff
Scans npm SDK source code for API key (query/header), bearer token,
and X-Api-Key patterns. Populates the spec's auth section automatically
so generated CLIs wire auth correctly without manual spec edits.
Derives env var names from API name (e.g., STEAM_API_KEY from 'steam').
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): conditionally emit helpers and remove dead code from generator
helpers.go.tmpl now gates classifyDeleteError on HasDelete flag (spec
has DELETE endpoints). Permanently removes firstNonEmpty,
printOutputFiltered, and selectFieldsGlobal which were dead across
all tested APIs (Postman Explore, Steam).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(cli): add top-level command promotion to generator
The generator now emits user-friendly promoted commands alongside
nested API commands. ISteamUser → steam-user, SteamUserStats →
steam-user-stats. Skips promotion for names colliding with built-in
commands (doctor, auth, sync, etc.). Confirmed needed across two APIs:
Postman Explore (7 manual commands) and Steam (10 manual commands).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(cli): add YAML format safety net with JSON round-trip fallback
Spec parser now tolerates YAML style variations from non-Go tools
(Python yaml.dump, etc.) by attempting a yaml→map→json→struct
round-trip when direct unmarshaling produces empty resources. Added
json struct tags to all spec types. Safety net for crowd-sniff specs
that might be post-processed by external tools.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(cli): add Steam retro and comprehensive improvement plan
Steam retro identified 7 findings across crowd-sniff and generator.
Plan covers 5 implementation units: resource naming, auth detection,
command promotion, dead code emission, and YAML safety.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs(cli): mark crowd-sniff retro plan as completed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(cli): fix gofmt, staticcheck raw string, and unused var in crowd-sniff
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
...-sniff-and-generator-retro-improvements-plan.md | 284 +++++++++++++
docs/retros/2026-03-30-steam-retro.md | 175 ++++++++
internal/cli/crowd_sniff.go | 3 +-
internal/crowdsniff/aggregate.go | 32 ++
internal/crowdsniff/npm.go | 157 ++++++-
internal/crowdsniff/npm_test.go | 459 ++++++++++++++++++++-
internal/crowdsniff/specgen.go | 67 ++-
internal/crowdsniff/specgen_test.go | 97 ++++-
internal/crowdsniff/types.go | 11 +
internal/generator/generator.go | 165 ++++++++
internal/generator/generator_test.go | 433 ++++++++++++++++++-
.../generator/templates/command_promoted.go.tmpl | 105 +++++
internal/generator/templates/helpers.go.tmpl | 22 +-
internal/generator/templates/root.go.tmpl | 3 +
internal/spec/spec.go | 134 +++---
internal/spec/spec_test.go | 114 +++++
internal/websniff/specgen.go | 9 +-
17 files changed, 2147 insertions(+), 123 deletions(-)
diff --git a/docs/plans/2026-03-30-004-fix-crowd-sniff-and-generator-retro-improvements-plan.md b/docs/plans/2026-03-30-004-fix-crowd-sniff-and-generator-retro-improvements-plan.md
new file mode 100644
index 00000000..855f8c2d
--- /dev/null
+++ b/docs/plans/2026-03-30-004-fix-crowd-sniff-and-generator-retro-improvements-plan.md
@@ -0,0 +1,284 @@
+---
+title: "fix: Crowd-sniff and generator improvements from Steam retro"
+type: fix
+status: completed
+date: 2026-03-30
+origin: docs/retros/2026-03-30-steam-retro.md
+---
+
+# fix: Crowd-sniff and generator improvements from Steam retro
+
+## Overview
+
+The Steam CLI generation surfaced 5 actionable systemic issues (plus 1 skip). This plan addresses all worthwhile findings: crowd-sniff resource naming, auth detection, top-level command promotion, dead code emission, and YAML format safety. These are validated across two APIs (Postman Explore and Steam).
+
+## Problem Frame
+
+Crowd-sniff produces specs that crash the generator (slashes in resource names), miss auth config (every API call returns 401), and require Python post-processing that breaks Go tools. The generator still emits dead helpers and only produces nested API-path commands, requiring 7-10 manual command files per CLI. (see origin: docs/retros/2026-03-30-steam-retro.md)
+
+## Requirements Trace
+
+- R1. Crowd-sniff resource names never contain `/` — methods grouped under their interface/resource prefix
+- R2. Crowd-sniff detects API key and bearer token auth patterns from npm SDK source code
+- R3. Generator emits top-level user-friendly command aliases alongside nested API commands
+- R4. Generator conditionally emits helpers based on which are actually referenced by the spec's endpoints
+- R5. No regressions for standard REST APIs or existing catalog entries
+
+## Scope Boundaries
+
+- No changes to the profiler, OpenAPI parser, or spec parser (except YAML leniency if needed)
+- No OAuth flow detection (complex multi-step auth) — only API key, bearer token, basic auth
+- IPlayerService `input_json` wrapping is a Steam-only quirk, skipped
+- Docs fallback for missing params (`--docs-fallback`) is deferred — significant new capability, needs separate design
+
+## Context & Research
+
+### Relevant Code and Patterns
+
+- **`deriveResourceKey()`** (`internal/crowdsniff/specgen.go:114`): Returns `strings.Join(segments, "/")` — this is the slash bug. Same bug exists in `internal/websniff/specgen.go:392`.
+- **`sanitizeResourceName()`** (`internal/openapi/parser.go:1760`): Existing precedent for stripping slashes from resource names. Strips `.`, `/`, `\` and trims underscores.
+- **`significantSegments()`** (`internal/crowdsniff/specgen.go:125`): Filters path segments, removing params, version segments, and "api". The filtered segments become the resource key.
+- **Generator resource name usage** (`internal/generator/generator.go:181`): `filepath.Join("internal", "cli", name+".go")` — a slash in `name` creates subdirectories that don't exist.
+- **`helpers.go.tmpl`**: 25+ functions emitted unconditionally. `classifyDeleteError`, `firstNonEmpty`, `printOutputFiltered` are consistently dead across multiple APIs.
+
+### Institutional Learnings
+
+- **filepath.Join traversal** (`docs/solutions/security-issues/filepath-join-traversal-with-user-input-2026-03-29.md`): Resource names used in `filepath.Join` need traversal protection. Belt-and-suspenders: reject `/`, `\`, `..` at input AND verify resolved path.
+- **Multi-source discovery** (`docs/solutions/best-practices/multi-source-api-discovery-design-2026-03-30.md`): Two-step path normalization for crowd-sniff. Param discovery infrastructure (brace scanner) already exists and can be extended for auth detection.
+
+## Key Technical Decisions
+
+- **Resource grouping by first significant segment**: For `/ISteamUser/GetPlayerSummaries/v2`, the resource is `ISteamUser` and the endpoint is `get_player_summaries`. For `/v1/users/{id}/posts`, the resource is `users` and the endpoint is `create_posts`. Only the first significant segment becomes the resource key — joining multiple segments with `/` is the bug we're fixing.
+- **Auth detection is additive, not override**: Crowd-sniff populates auth only when patterns are detected AND the spec doesn't already have auth configured. The `--auth` flag or user-provided spec auth takes precedence.
+- **Command promotion uses resource names from spec**: The generator already has the resource names (e.g., `ISteamUser`, `collections`). Promotion creates top-level commands that delegate to the nested ones. The `api` group stays as an escape hatch.
+- **Dead code: conditional emission over post-processing**: Checking which helpers the spec's endpoints actually need at generation time is cheaper than a separate `polish` command. The spec tells us: no DELETE endpoints → no `classifyDeleteError`.
+- **Fix websniff specgen.go too**: The same `deriveResourceKey` slash bug exists in `internal/websniff/specgen.go`. Fix both to prevent the same issue for sniffed specs.
+
+## Open Questions
+
+### Resolved During Planning
+
+- **Q: Should resource key use only the first segment or allow multi-segment?** First segment only. Multi-segment keys (`users/posts`) create filesystem problems and confusing CLI structure. Sub-resources should be nested endpoints within a resource, not separate resources.
+- **Q: How does the generator determine which helpers are needed?** From the spec: if there are DELETE endpoints → emit `classifyDeleteError`. If there are endpoints with `response_path` → emit `extractItemsAtPath`. The spec's endpoint methods and features determine the helper set.
+
+### Deferred to Implementation
+
+- **Q: Exact helper-to-spec-feature mapping** — Need to trace each helper to find which spec features trigger its use. Build this mapping during implementation.
+- **Q: How to handle resource name collisions after flattening** — If two paths produce the same first-segment resource name (e.g., `/v1/users` and `/v2/users`), need a collision strategy. Likely: append version as suffix (`users_v2`).
+
+## Implementation Units
+
+- [ ] **Unit 1: Fix crowd-sniff resource naming in specgen.go**
+
+ **Goal:** `deriveResourceKey()` returns a single-segment resource name (no slashes) by using only the first significant path segment as the resource key and grouping all methods under it.
+
+ **Requirements:** R1, R5
+
+ **Dependencies:** None
+
+ **Files:**
+ - Modify: `internal/crowdsniff/specgen.go`
+ - Modify: `internal/websniff/specgen.go` (same bug)
+ - Test: `internal/crowdsniff/specgen_test.go`
+
+ **Approach:**
+ - Change `deriveResourceKey()` to return `segments[0]` as the resource key (not `strings.Join(segments, "/")`)
+ - The endpoint name derivation (`deriveEndpointName`) already uses the last segment, so endpoints within a resource will have distinct names
+ - Handle the collision case: if two endpoints produce the same resource key + endpoint name, use `uniqueEndpointName()` (already exists)
+ - Apply the same fix to `internal/websniff/specgen.go:392`
+
+ **Patterns to follow:**
+ - `sanitizeResourceName()` in `internal/openapi/parser.go:1760` — existing precedent for cleaning resource names
+
+ **Test scenarios:**
+ - Happy path: Steam-like path `/ISteamUser/GetPlayerSummaries/v2` → resource `ISteamUser`, endpoint `get_player_summaries`
+ - Happy path: REST path `/v1/users/{id}/posts` → resource `users`, endpoint `create_posts` (not `users/posts`)
+ - Happy path: Simple path `/v1/users` → resource `users`, endpoint `list_users` (unchanged)
+ - Edge case: Path with only params and version `/v1/{id}` → resource `default` (fallback)
+ - Edge case: Two paths same first segment `/v1/users` and `/v1/users/{id}/posts` → both in `users` resource
+ - Negative test: No resource key ever contains `/` — assert across all test cases
+
+ **Verification:**
+ - `go build ./...` and `go test ./internal/crowdsniff/...` pass
+ - Run `printing-press crowd-sniff --api steam` → spec resource names have no slashes
+ - Run `printing-press generate --spec <crowd-sniff-spec>` → no directory creation errors
+
+- [ ] **Unit 2: Add auth pattern detection to crowd-sniff**
+
+ **Goal:** Crowd-sniff detects API key and bearer token auth patterns from npm SDK source code and populates the spec's auth section.
+
+ **Requirements:** R2, R5
+
+ **Dependencies:** None (independent of Unit 1)
+
+ **Files:**
+ - Modify: `internal/crowdsniff/npm.go`
+ - Modify: `internal/crowdsniff/types.go`
+ - Modify: `internal/crowdsniff/aggregate.go`
+ - Modify: `internal/crowdsniff/specgen.go`
+ - Test: `internal/crowdsniff/npm_test.go`
+ - Test: `internal/crowdsniff/specgen_test.go`
+
+ **Approach:**
+ - Add `DiscoveredAuth` type to `types.go`: `{Type, Header, In, Format, EnvVarHint string}`
+ - In `npm.go`, after extracting endpoints and params from SDK source, scan for auth patterns:
+ - Constructor: `new Client(apiKey)`, `new SteamAPI(key)` → detect key param name
+ - Header: `headers['Authorization'] = 'Bearer ' + token` → `type: bearer_token`
+ - Header: `headers['X-Api-Key'] = key` → `type: api_key, header: X-Api-Key, in: header`
+ - Query: `params.key = this.apiKey` → `type: api_key, header: key, in: query`
+ - Add `Auth []DiscoveredAuth` to `SourceResult` and merge in `aggregate.go` (highest-tier source wins)
+ - In `specgen.go`, populate `spec.Auth` from aggregated auth when no auth is already configured
+
+ **Patterns to follow:**
+ - Param discovery in `npm.go` / `params.go` — same pattern of scanning SDK source with regex + brace scanner
+ - `DiscoveredParam` type structure — follow same pattern for `DiscoveredAuth`
+
+ **Test scenarios:**
+ - Happy path: SDK with `params.key = this.apiKey` → auth detected as `api_key, in: query, header: key`
+ - Happy path: SDK with `Authorization: Bearer ${token}` → auth detected as `bearer_token`
+ - Happy path: SDK with `headers['X-Api-Key'] = apiKey` → auth detected as `api_key, in: header, header: X-Api-Key`
+ - Edge case: SDK with no auth patterns → `auth.type: none` (unchanged)
+ - Edge case: Multiple SDKs with conflicting auth → highest-tier source wins
+ - Negative test: Existing auth config in spec → not overridden by crowd-sniff detection
+
+ **Verification:**
+ - `go test ./internal/crowdsniff/...` passes
+ - Run `printing-press crowd-sniff --api steam` → spec has `auth.type: api_key`, `auth.in: query`
+
+- [ ] **Unit 3: Add top-level command promotion to generator**
+
+ **Goal:** Generator emits user-friendly top-level commands (e.g., `player`, `games`, `search`) alongside the nested API commands (e.g., `ISteamUser get_player_summaries`). The nested commands stay as an escape hatch.
+
+ **Requirements:** R3, R5
+
+ **Dependencies:** None (independent)
+
+ **Files:**
+ - Modify: `internal/generator/generator.go`
+ - Create: `internal/generator/templates/command_promoted.go.tmpl`
+ - Modify: `internal/generator/templates/root.go.tmpl`
+ - Test: `internal/generator/generator_test.go`
+
+ **Approach:**
+ - After generating the nested API commands, add a promotion pass
+ - For each spec resource with a "list" endpoint (GET without path params), emit a top-level command named after the resource (pluralized or as-is depending on conventions)
+ - The promoted command delegates to the nested command's handler — it's a thin wrapper with better UX (vanity name, clearer help text)
+ - Resources with entity-type enum params (like Postman's `entityType=collection`) get one promoted command per enum value
+ - The `command_promoted.go.tmpl` template receives the resource name, the target endpoint, and the promoted command name
+ - Root template registers both the nested group and the promoted commands
+
+ **Patterns to follow:**
+ - Existing command generation in `generator.go` lines 166-245 — same data structs, same template rendering
+ - How the postman-explore CLI's manual `cmd_browse.go` works — that's what the promoted command should look like
+
+ **Test scenarios:**
+ - Happy path: Spec with `users` resource having list/get/create endpoints → top-level `users` command emitted alongside nested `users list-users`
+ - Happy path: Spec with `ISteamUser` resource → top-level `ISteamUser` command (or promoted as `steam-user`)
+ - Edge case: Resource with no list endpoint → no promoted command
+ - Edge case: Promoted command name collides with a built-in command (e.g., `version`, `help`) → skip promotion for that resource
+ - Negative test: Existing commands (doctor, auth, sync, search) not affected by promotion
+
+ **Verification:**
+ - `go build ./...` and `go test ./internal/generator/...` pass
+ - Generated CLI has both `ISteamUser get_player_summaries` AND a top-level shortcut
+
+- [ ] **Unit 4: Conditional helper emission in generator**
+
+ **Goal:** `helpers.go.tmpl` only emits helper functions that the spec's endpoints actually need, eliminating dead code at generation time.
+
+ **Requirements:** R4, R5
+
+ **Dependencies:** None (independent)
+
+ **Files:**
+ - Modify: `internal/generator/templates/helpers.go.tmpl`
+ - Modify: `internal/generator/generator.go` (pass helper-selection flags to template)
+ - Test: `internal/generator/generator_test.go`
+
+ **Approach:**
+ - Analyze the spec at generation time to determine which helpers are needed:
+ - Has DELETE endpoints → emit `classifyDeleteError`
+ - Has endpoints (always) → emit `classifyAPIError`, `usageErr`, `notFoundErr`, `authErr`, `apiErr`, `rateLimitErr`
+ - Has list endpoints → emit `paginatedGet`
+ - Has any output → emit `printOutput`, `printOutputWithFlags`, `filterFields`, `compactFields`, `printCSV`, `wantsHumanTable`, `printAutoTable`
+ - Has text fields → emit `truncate`
+ - Has pagination → emit `formatCompact` (used in progress reporting)
+ - Functions that are always needed (error types, output formatting, tab writer, color utilities) stay unconditional
+ - Functions gated by spec features: `classifyDeleteError` (DELETE), `firstNonEmpty` (never used — remove entirely), `printOutputFiltered` (never used — remove entirely)
+ - Pass a `HelperFlags` struct to the template: `HasDelete bool`, etc.
+
+ **Patterns to follow:**
+ - VisionSet conditional logic in `generator.go` — same pattern of checking spec features to decide what to emit
+ - `{{if .VisionSet.Store}}` in root.go.tmpl — same conditional template pattern
+
+ **Test scenarios:**
+ - Happy path: Spec with no DELETE endpoints → generated helpers.go does not contain `classifyDeleteError`
+ - Happy path: Spec with DELETE endpoints → generated helpers.go contains `classifyDeleteError`
+ - Happy path: Any spec → `firstNonEmpty` and `printOutputFiltered` never emitted (dead code across all tested APIs)
+ - Edge case: Minimal spec (1 GET endpoint) → only essential helpers emitted
+ - Negative test: Spec with full feature set → all helpers present (no regression)
+
+ **Verification:**
+ - `go build ./...` and `go test ./internal/generator/...` pass
+ - Generate a CLI with no DELETE endpoints → `grep classifyDeleteError` finds nothing
+ - Scorecard dead_code improves from 0/5
+
+- [ ] **Unit 5: YAML format safety net for spec parser**
+
+ **Goal:** Spec parser tolerates common YAML style variations so specs from non-Go tools (Python yaml.dump) don't break dogfood/verify.
+
+ **Requirements:** R5
+
+ **Dependencies:** Unit 1 (if resource naming is fixed, Python restructuring is no longer needed — but this is a safety net)
+
+ **Files:**
+ - Modify: `internal/spec/reader.go`
+ - Test: `internal/spec/spec_test.go`
+
+ **Approach:**
+ - The spec parser currently validates strictly. Python's `yaml.dump` produces valid YAML that Go's strict parser rejects (possibly due to indentation style, string quoting, or flow vs block style differences)
+ - Add a pre-parse normalization step or relax validation to accept common YAML variations
+ - This is a safety net — with Unit 1 fixed, Python restructuring should be unnecessary. But specs from other tools (JS, Python, manual editing) should work too.
+
+ **Patterns to follow:**
+ - Current validation in `reader.go` — understand what specifically fails before deciding the fix
+
+ **Execution note:** Start by reproducing the failure with a Python-generated YAML to identify what specifically the parser rejects.
+
+ **Test scenarios:**
+ - Happy path: Python yaml.dump output with 2-space indent → parses correctly
+ - Happy path: Go-generated YAML → still parses correctly (no regression)
+ - Edge case: YAML with quoted string values where Go would use unquoted → parses
+ - Edge case: YAML with flow-style arrays `[a, b]` → parses
+
+ **Verification:**
+ - `go test ./internal/spec/...` passes
+ - `printing-press dogfood --spec <python-generated-yaml>` no longer fails with "at least one resource is required"
+
+## System-Wide Impact
+
+- **Crowd-sniff changes (Units 1-2)** affect every crowd-sniff-generated spec. The resource naming fix is universal; auth detection is additive.
+- **Generator changes (Units 3-4)** affect every generated CLI. Command promotion is additive (doesn't remove existing commands). Helper conditional emission reduces generated code size.
+- **Spec parser changes (Unit 5)** affect all spec consumers (generator, dogfood, verify, scorecard). Must not break any existing specs.
+- **websniff parity**: Unit 1 also fixes `internal/websniff/specgen.go` which has the same slash bug.
+- **Unchanged invariants**: The internal YAML spec format, the generator's template rendering pipeline, the profiler, and the OpenAPI parser are unchanged.
+
+## Risks & Dependencies
+
+| Risk | Mitigation |
+|------|------------|
+| Resource name flattening changes existing crowd-sniff specs | Specs are regenerated each run — no backward compatibility concern |
+| Auth detection produces false positives | Only populate auth when confidence is high (pattern seen in 2+ sources). Never override existing auth. |
+| Command promotion names collide with built-in commands | Skip promotion for resource names matching built-in commands (doctor, auth, sync, search, version, help, etc.) |
+| Helper conditional emission accidentally removes a needed helper | Test with full-featured specs (Stripe, Stytch) to verify no regressions |
+| YAML leniency introduces security risk (traversal in parsed values) | The leniency is about formatting, not content. Resource names still go through `sanitizeResourceName`. |
+
+## Sources & References
+
+- **Origin document:** [docs/retros/2026-03-30-steam-retro.md](docs/retros/2026-03-30-steam-retro.md)
+- **Prior retro:** [docs/retros/2026-03-30-postman-explore-retro.md](docs/retros/2026-03-30-postman-explore-retro.md) (findings #1, #4, #7 confirmed on second API)
+- Crowd-sniff specgen: `internal/crowdsniff/specgen.go`
+- Generator resource usage: `internal/generator/generator.go:166-245`
+- sanitizeResourceName precedent: `internal/openapi/parser.go:1760`
+- filepath-join traversal learning: `docs/solutions/security-issues/filepath-join-traversal-with-user-input-2026-03-29.md`
+- Param discovery gap doc: `manuscripts/postman-explore/20260330-105847/proofs/2026-03-30-crowd-sniff-param-discovery-gap.md`
diff --git a/docs/retros/2026-03-30-steam-retro.md b/docs/retros/2026-03-30-steam-retro.md
new file mode 100644
index 00000000..344da71a
--- /dev/null
+++ b/docs/retros/2026-03-30-steam-retro.md
@@ -0,0 +1,175 @@
+# Printing Press Retro: Steam API
+
+## Session Stats
+- API: Steam Web API (player profiles, game libraries, achievements, friends, news)
+- Spec source: Crowd-sniff (v2, with auto-discovered params) — zero OpenAPI spec
+- Scorecard: 73 (v1, manual params) → 70 (v2, auto params + generator improvements)
+- Verify pass rate: 31% (8/26 pass; 9 parent commands fail, 9 positional-arg commands fail in mock)
+- Fix loops: 1
+- Manual code edits: 4 (auth config, resource restructuring, root description, input_json wrapping)
+- Features built from scratch: 13 (10 top-level commands + 3 sync subcommands)
+- Time to ship: ~20 min (v2 run, reusing v1 research)
+
+## Findings
+
+### 1. Crowd-sniff produces flat Interface/Method resource names with slashes (bug)
+- **What happened:** Crowd-sniff output had resource names like `ISteamUser/GetPlayerSummaries` which contain `/`. The generator tried to create directories matching these names, causing `open /path/ISteamUser/GetPlayerSummaries.go: no such file or directory`.
+- **Root cause:** `internal/crowdsniff/specgen.go` — the spec builder uses the raw endpoint path components as resource names without grouping methods under their interface.
+- **Cross-API check:** Standard REST (Stripe): endpoints are `/v1/charges`, `/v1/customers` — crowd-sniff would produce `v1/charges` with slashes. Sniffed API: same pattern if paths have segments. This hits ANY API where crowd-sniff discovers paths with more than one segment.
+- **Frequency:** Most APIs. Any API with path-structured endpoints (which is almost all of them).
+- **Fallback if machine doesn't fix it:** Claude runs a Python script to restructure the spec (~3 min). But if forgotten, the generator crashes — **critical fallback**.
+- **Tradeoff:** freq(3) × fallback(4) / effort(1) + risk(0) = 12.0. High priority.
+- **Inherent or fixable:** Fixable. The spec builder should group endpoints by their first path segment (the interface/resource name) and use the method as the endpoint key within that resource.
+- **Durable fix:** In `specgen.go`, when building the resource map, split the endpoint path and use the first significant segment as the resource name. Group all methods sharing that prefix under one resource. For Steam: `/ISteamUser/GetPlayerSummaries/v2` → resource `ISteamUser`, endpoint `get_player_summaries`.
+ - Condition: Always active — resource names should never contain `/`
+ - Guard: None needed — this is a universal fix
+- **Test:** Generate a crowd-sniff spec from Steam → resource names have no slashes. Generate from a standard REST API → resources are clean path segments.
+- **Evidence:** Generator crash on first attempt; Python restructuring script needed.
+
+### 2. Crowd-sniff doesn't detect auth patterns from SDK code (missing scaffolding)
+- **What happened:** Crowd-sniff set `auth.type: none` for Steam even though every npm SDK passes `key=XXX` as a query parameter. The auth config had to be manually edited to `type: api_key`, `in: query`, `header: key`.
+- **Root cause:** `internal/crowdsniff/npm.go` — the npm analyzer extracts endpoint paths and (now) params, but doesn't look for authentication patterns in the SDK code.
+- **Cross-API check:** Standard REST (Stripe): SDKs use `Authorization: Bearer sk_...` — detectable. Sniffed API: auth varies. Most npm SDKs set auth in their constructor or per-request headers. This applies to most APIs that need auth.
+- **Frequency:** Most APIs. The crowd-sniff param discovery doc (`manuscripts/postman-explore/20260330-105847/proofs/2026-03-30-crowd-sniff-param-discovery-gap.md`) already identified auth pattern detection as a gap.
+- **Fallback if machine doesn't fix it:** Claude edits 4 lines in the spec YAML (~1 min). Low cost per edit, but if missed the CLI generates with no auth wiring and every API call returns 401 — **critical if missed**.
+- **Tradeoff:** freq(3) × fallback(3) / effort(2) + risk(0) = 4.5. Medium priority.
+- **Inherent or fixable:** Fixable. SDK code reveals auth patterns: look for constructor params like `new SteamAPI(key)`, header setting like `headers: { 'X-Api-Key': key }`, or query param patterns like `params.key = this.key`. The param discovery infrastructure (multi-line brace scanner) already exists to extract these.
+- **Durable fix:** In `npm.go`, after extracting endpoints and params, scan the SDK source for auth patterns:
+ 1. Constructor: look for `constructor(apiKey)` or `new Client(key)` → detect key param name
+ 2. Header setting: look for `headers[...] = ` patterns → detect header name and format
+ 3. Query param: look for `params.key = ` or `{ key: this.apiKey }` → detect `in: query`
+ 4. Write detected auth into the spec's `auth` section
+ - Condition: Only when auth patterns are detected in SDK source
+ - Guard: Don't override if the user passed `--auth` explicitly
+ - Frequency estimate: ~80% of APIs need auth; ~60% of npm SDKs have extractable auth patterns
+- **Test:** Run crowd-sniff for Steam → spec has `auth.type: api_key`, `auth.in: query`. Run for a bearer-token API → spec has `auth.type: bearer_token`.
+- **Evidence:** Manual edit to spec; every npm SDK for Steam passes `key` as query param.
+
+### 3. IPlayerService "Service interface" needs input_json wrapping (assumption mismatch)
+- **What happened:** Steam's "Service" interfaces (IPlayerService, IGameServersService) require params wrapped as `input_json={"steamid":"xxx"}` URL-encoded, not as regular query params. Every top-level command calling these interfaces needed manual `input_json` wrapping code.
+- **Root cause:** The spec has no way to express "this endpoint requires input_json wrapping" — it's a Steam-specific convention for Valve's "Service" interfaces.
+- **Cross-API check:** Standard REST (Stripe): no. Sniffed APIs: unlikely. This is specific to Steam's Service interface pattern. Potentially other Valve APIs (Dota 2, CS2) use the same pattern.
+- **Frequency:** This API only (Steam/Valve family). The `input_json` convention is Valve-specific.
+- **Fallback if machine doesn't fix it:** Claude writes wrapper code for each command (~5 min total, medium cost). Not critical — the commands work, just with a different param format.
+- **Tradeoff:** freq(1) × fallback(2) / effort(2) + risk(1) = 0.67. Low priority — too narrow.
+- **Inherent or fixable:** Inherent to the Steam API's design. Not worth a machine fix for a single API family. The skill instruction to handle Service interfaces during Phase 3 is the right level.
+- **Durable fix:** Skip machine fix. Document in the skill as a known Steam API quirk that Claude handles during Phase 3. If Valve APIs become a significant fraction of the catalog, revisit.
+- **Test:** N/A — this is a skip recommendation.
+- **Evidence:** All IPlayerService commands needed manual input_json wrapping.
+
+### 4. dogfood/verify reject Python-restructured YAML specs (tool limitation)
+- **What happened:** After restructuring the spec with Python's `yaml.dump`, dogfood and verify returned "at least one resource is required" — they couldn't parse the resources. The generator and scorecard (via JSON conversion) accepted the same spec.
+- **Root cause:** Python's `yaml.dump` produces slightly different YAML formatting than Go's YAML libraries expect. The spec parser in `internal/spec/reader.go` is stricter about some YAML conventions (possibly indentation or string quoting) than the generator's spec loading path.
+- **Cross-API check:** This affects any spec modified by non-Go tools (Python, JS). Since crowd-sniff produces specs via Go, the issue only arises when Python post-processes the spec (as we did for resource restructuring). If finding #1 is fixed (no restructuring needed), this issue goes away.
+- **Frequency:** API subclass: crowd-sniff specs that need post-processing. Currently 100% of crowd-sniff specs need restructuring, so effectively "most APIs using crowd-sniff."
+- **Fallback if machine doesn't fix it:** Convert to JSON for dogfood/verify (~30 sec). Low cost.
+- **Tradeoff:** freq(2) × fallback(1) / effort(1) + risk(0) = 2.0. Low priority — especially if finding #1 eliminates the need for Python restructuring.
+- **Inherent or fixable:** Mostly eliminated by fixing #1. If specs never need Python restructuring, the Go-generated YAML always works. Residual fix: make the spec parser more lenient about YAML style variations.
+- **Durable fix:** Primary: fix #1 (resource naming in specgen.go). Residual: add a YAML normalization pass in `reader.go` that canonicalizes the YAML before parsing.
+- **Test:** Run dogfood with a crowd-sniff-generated spec → passes without JSON conversion.
+- **Evidence:** dogfood/verify rejected the restructured YAML; scorecard accepted JSON version.
+
+### 5. Crowd-sniff misses params for endpoints not in popular SDKs (missing scaffolding)
+- **What happened:** GetNewsForApp, GetOwnedGames, and several other endpoints had `params: []` even after v2 param discovery. The `steamapi` npm package doesn't expose all Steam endpoints — less popular endpoints aren't covered.
+- **Root cause:** `internal/crowdsniff/npm.go` — param discovery only works for endpoints that appear in the npm packages' source code. Endpoints not covered by any SDK get no params.
+- **Cross-API check:** Standard REST (Stripe): Stripe's npm SDK is comprehensive, covering ~95% of endpoints. Less popular APIs with smaller SDKs will have more gaps. This affects APIs where community coverage is patchy.
+- **Frequency:** Most APIs using crowd-sniff. The coverage depends on SDK quality.
+- **Fallback if machine doesn't fix it:** Claude adds missing params manually from docs (~5 min). Medium cost.
+- **Tradeoff:** freq(3) × fallback(2) / effort(3) + risk(1) = 1.5. Low priority — the fallback is manageable and the fix is complex (would need a secondary source like Steamworks docs).
+- **Inherent or fixable:** Partially inherent — crowd-sniff's value is community-driven discovery, and coverage gaps are expected. The fix is to add a secondary param source: scrape the official docs page to fill gaps. This is a significant new capability.
+- **Durable fix:** Add a `--docs-fallback <url>` flag to crowd-sniff that scrapes official API docs to fill param gaps for endpoints where npm/GitHub produced no params. Lower priority than the resource naming fix.
+- **Test:** Run crowd-sniff for Steam with docs fallback → GetNewsForApp has `appid` param.
+- **Evidence:** GetNewsForApp `--appid 570` failed because the flag didn't exist.
+
+### 6. Dead code scores 0/5 on every generation (recurring friction)
+- **What happened:** Scorecard dead_code: 0/5. The generator emits helpers like `classifyDeleteError`, `firstNonEmpty`, `printOutputFiltered` that no command calls.
+- **Root cause:** `internal/generator/templates/helpers.go.tmpl` emits all utility functions unconditionally.
+- **Cross-API check:** Every API. This is the same finding as the postman-explore retro (#7).
+- **Frequency:** Every API. 0/5 dead code score on both postman-explore and steam.
+- **Fallback if machine doesn't fix it:** Claude deletes 3-5 functions (<2 min). Low cost.
+- **Tradeoff:** freq(4) × fallback(1) / effort(2) + risk(1) = 1.3. Same as before.
+- **Inherent or fixable:** Same analysis as postman-explore retro: partially inherent, mitigated by `printing-press polish --dead-code` or conditional emission.
+- **Durable fix:** Already planned in the postman-explore retro (Tier 3, backlog). No new action needed.
+- **Evidence:** 0/5 dead code on both postman-explore (85/100) and steam (70/100).
+
+### 7. Top-level commands always built from scratch (template gap)
+- **What happened:** 10 top-level commands (player, games, friends, achievements, news, players, resolve, search, compare, completionist) were written from scratch. The generator only produces `ISteamUser get_player_summaries` nested commands.
+- **Root cause:** Same as postman-explore retro finding #1 — the generator derives command structure from spec paths, not user-facing UX patterns.
+- **Cross-API check:** Every API. Same finding, same analysis as postman-explore retro.
+- **Frequency:** Every API. Confirmed across two different APIs now (Postman Explore and Steam).
+- **Fallback if machine doesn't fix it:** Claude writes 5-10 command files (~15-20 min). High cost.
+- **Tradeoff:** freq(4) × fallback(3) / effort(3) + risk(2) = 2.4. Same score as before, but now confirmed on a second API.
+- **Inherent or fixable:** Same analysis. The "command promotion" approach from the postman-explore retro plan is the right fix. Now validated on a second API.
+- **Durable fix:** Already planned as Tier 2 in the postman-explore retro.
+- **Evidence:** 10 commands written from scratch for Steam, 7 for Postman Explore — same pattern.
+
+## Prioritized Improvements
+
+### Tier 1: Do Now
+| # | Fix | Component | Frequency | Fallback Cost | Effort | Guards |
+|---|-----|-----------|-----------|--------------|--------|--------|
+| 1 | Fix crowd-sniff resource naming (no slashes) | `internal/crowdsniff/specgen.go` | most | critical (generator crash) | 4 hrs | none — universal |
+
+### Tier 2: Plan
+| # | Fix | Component | Frequency | Fallback Cost | Effort | Guards |
+|---|-----|-----------|-----------|--------------|--------|--------|
+| 2 | Auth pattern detection in crowd-sniff | `internal/crowdsniff/npm.go` | most | critical if missed, medium per edit | 2 days | only when auth patterns detected |
+| 7 | Top-level command promotion | `internal/generator/` | every | high (10+ files from scratch) | 3 days | keep `api` group as escape hatch |
+
+### Tier 3: Backlog
+| # | Fix | Component | Frequency | Fallback Cost | Effort | Guards |
+|---|-----|-----------|-----------|--------------|--------|--------|
+| 5 | Docs fallback for missing params | `internal/crowdsniff/` | most (crowd-sniff) | medium (manual param adds) | 1 week | `--docs-fallback` flag |
+| 6 | Dead code emission | `internal/generator/templates/helpers.go.tmpl` | every | low (delete 3 funcs) | 1 day | conditional emission |
+| 4 | YAML format normalization | `internal/spec/reader.go` | subclass: crowd-sniff | low (JSON convert) | 4 hrs | none |
+
+### Skip
+| # | Fix | Reason |
+|---|-----|--------|
+| 3 | IPlayerService input_json | Steam/Valve-only quirk. Not worth machine complexity for one API family |
+
+## Work Units
+
+### WU-1: Fix crowd-sniff resource naming (finding #1)
+- **Goal:** Crowd-sniff outputs specs with clean resource names (no slashes) by grouping methods under their interface/resource prefix.
+- **Target files:**
+ - `internal/crowdsniff/specgen.go` — resource grouping logic
+ - `internal/crowdsniff/specgen_test.go` — tests
+- **Acceptance criteria:**
+ - Run crowd-sniff for Steam → resource names are `ISteamUser`, `IPlayerService` (no slashes)
+ - Each resource has multiple endpoints grouped under it
+ - Generator accepts the spec without Python restructuring
+ - dogfood/verify accept the spec directly (no JSON conversion)
+ - Run crowd-sniff for a standard REST API → resources are clean path segments (e.g., `users`, `charges`)
+- **Scope boundary:** Does NOT include auth detection or param gap filling
+- **Effort:** 4 hours
+
+### WU-2: Auth pattern detection in crowd-sniff (finding #2)
+- **Goal:** Crowd-sniff detects authentication patterns from npm SDK source code and populates the spec's auth section.
+- **Target files:**
+ - `internal/crowdsniff/npm.go` — auth extraction alongside endpoint/param extraction
+ - `internal/crowdsniff/types.go` — add auth pattern type
+ - `internal/crowdsniff/aggregate.go` — merge auth patterns across sources
+ - `internal/crowdsniff/specgen.go` — write auth into spec
+ - `internal/crowdsniff/npm_test.go` — tests
+- **Acceptance criteria:**
+ - Run crowd-sniff for Steam → spec has `auth.type: api_key`, `auth.in: query`, `auth.header: key`, `auth.env_vars: [STEAM_API_KEY]`
+ - Run crowd-sniff for a bearer-token API → spec has `auth.type: bearer_token`
+ - Run crowd-sniff for a no-auth API → spec has `auth.type: none` (unchanged)
+- **Scope boundary:** Does NOT include OAuth flow detection (complex multi-step). Focuses on simple patterns: API key (query/header), bearer token, basic auth.
+- **Dependencies:** None
+- **Effort:** 2 days
+
+## Anti-patterns
+
+- **"Python yaml.dump is interchangeable with Go yaml"** — It's not. Go's strict YAML parser rejects some valid YAML that Python produces. Avoid round-tripping specs through Python when Go tools need to consume them. Fix the Go code to produce correct specs from the start.
+- **"Crowd-sniff is done when it finds endpoints"** — Endpoints without params, auth, and clean resource names aren't usable. Crowd-sniff needs to be a complete spec builder, not just an endpoint discoverer.
+- **"The generator improvements from the postman-explore retro are validated"** — They are! Pagination-aware sync, batch upsert, DB path consolidation, and query param auth all worked correctly on the Steam run. But top-level command promotion and dead code fixes are still unresolved — confirmed on a second API.
+
+## What the Machine Got Right
+
+- **Crowd-sniff v2 param discovery** — Discovered 25 params across 17 endpoints from npm SDK source code. Eliminated 100% of manual param editing (vs 24 manual params in v1). The multi-line brace scanner and function signature cross-referencing worked as designed.
+- **Generator improvements from retro** — All five fixes from the postman-explore retro worked on first try: pagination-aware sync generated correctly, batch upsert methods generated, single defaultDBPath(), query param auth (`in: query`) wired correctly, proxy route propagation verified.
+- **Prior research reuse** — Phase 0 found manuscripts from the v1 run and reused the brief, absorb manifest, and crowd-sniff spec. Research phase took ~0 minutes instead of ~10.
+- **Manuscript archiving (Phase 5.5)** — The fix we made earlier (archive unconditionally after shipcheck) worked — v1 manuscripts were available for v2 to reuse.
+- **Live API verification** — 6/6 live tests passed. The auth, params, and endpoint routing all worked against the real Steam Web API with Gabe Newell's profile as test data.
diff --git a/internal/cli/crowd_sniff.go b/internal/cli/crowd_sniff.go
index 2e2a3786..80e54146 100644
--- a/internal/cli/crowd_sniff.go
+++ b/internal/cli/crowd_sniff.go
@@ -102,6 +102,7 @@ func runCrowdSniff(ctx context.Context, apiName, baseURL, outputPath string, asJ
}
aggregated, baseURLCandidates := crowdsniff.Aggregate(results)
+ mergedAuth := crowdsniff.AggregateAuth(results)
if len(aggregated) == 0 {
return fmt.Errorf("no endpoints discovered for %q", apiName)
@@ -116,7 +117,7 @@ func runCrowdSniff(ctx context.Context, apiName, baseURL, outputPath string, asJ
return fmt.Errorf("base URL must use HTTPS: %s", resolvedBaseURL)
}
- apiSpec, err := crowdsniff.BuildSpec(apiName, resolvedBaseURL, aggregated)
+ apiSpec, err := crowdsniff.BuildSpec(apiName, resolvedBaseURL, aggregated, mergedAuth)
if err != nil {
return fmt.Errorf("building spec: %w", err)
}
diff --git a/internal/crowdsniff/aggregate.go b/internal/crowdsniff/aggregate.go
index 1c4c6317..8d90acf1 100644
--- a/internal/crowdsniff/aggregate.go
+++ b/internal/crowdsniff/aggregate.go
@@ -189,6 +189,38 @@ func paramFieldCount(p DiscoveredParam) int {
return count
}
+// AggregateAuth merges auth patterns from multiple source results and returns
+// the single best auth pattern. Auth from higher-tier sources takes precedence.
+// Returns nil if no auth patterns were detected.
+func AggregateAuth(results []SourceResult) *DiscoveredAuth {
+ var best *DiscoveredAuth
+ bestRank := -1
+
+ for _, result := range results {
+ for i := range result.Auth {
+ auth := &result.Auth[i]
+ rank := tierRank(auth.SourceTier)
+ if rank > bestRank {
+ bestRank = rank
+ best = auth
+ } else if rank == bestRank && best != nil {
+ // Same tier: prefer auth with more metadata (env var hint).
+ if auth.EnvVarHint != "" && best.EnvVarHint == "" {
+ best = auth
+ }
+ }
+ }
+ }
+
+ if best == nil {
+ return nil
+ }
+
+ // Return a copy so callers don't modify the original.
+ cp := *best
+ return &cp
+}
+
func deduplicateStrings(items []string) []string {
seen := make(map[string]struct{}, len(items))
result := make([]string, 0, len(items))
diff --git a/internal/crowdsniff/npm.go b/internal/crowdsniff/npm.go
index c1cb7441..ad6540cc 100644
--- a/internal/crowdsniff/npm.go
+++ b/internal/crowdsniff/npm.go
@@ -11,6 +11,7 @@ import (
"net/url"
"os"
"path/filepath"
+ "regexp"
"strings"
"time"
)
@@ -157,7 +158,7 @@ func (s *NPMSource) Discover(ctx context.Context, apiName string) (SourceResult,
}
// Download and extract tarball.
- endpoints, baseURLs, processErr := s.processPackageTarball(ctx, tarballURL, pkg.Name, tier, downloads[pkg.Name])
+ endpoints, baseURLs, authPatterns, processErr := s.processPackageTarball(ctx, tarballURL, pkg.Name, tier, downloads[pkg.Name])
if processErr != nil {
fmt.Fprintf(os.Stderr, "crowd-sniff: failed to process %s: %v\n", pkg.Name, processErr)
continue
@@ -165,6 +166,7 @@ func (s *NPMSource) Discover(ctx context.Context, apiName string) (SourceResult,
result.Endpoints = append(result.Endpoints, endpoints...)
result.BaseURLCandidates = append(result.BaseURLCandidates, baseURLs...)
+ result.Auth = append(result.Auth, authPatterns...)
}
return result, nil
@@ -285,48 +287,49 @@ func (s *NPMSource) fetchTarballURL(ctx context.Context, name, version string) (
return version_.Dist.Tarball, nil
}
-// processPackageTarball downloads a tarball, extracts it, and greps for endpoints.
-func (s *NPMSource) processPackageTarball(ctx context.Context, tarballURL, pkgName, tier string, weeklyDownloads int) ([]DiscoveredEndpoint, []string, error) {
+// processPackageTarball downloads a tarball, extracts it, and greps for endpoints, auth patterns, and base URLs.
+func (s *NPMSource) processPackageTarball(ctx context.Context, tarballURL, pkgName, tier string, weeklyDownloads int) ([]DiscoveredEndpoint, []string, []DiscoveredAuth, error) {
// Security: validate tarball URL is HTTPS.
parsed, err := url.Parse(tarballURL)
if err != nil {
- return nil, nil, fmt.Errorf("invalid tarball URL: %w", err)
+ return nil, nil, nil, fmt.Errorf("invalid tarball URL: %w", err)
}
if parsed.Scheme != "https" {
- return nil, nil, fmt.Errorf("tarball URL must be HTTPS, got %s", parsed.Scheme)
+ return nil, nil, nil, fmt.Errorf("tarball URL must be HTTPS, got %s", parsed.Scheme)
}
// Download tarball.
req, err := http.NewRequestWithContext(ctx, http.MethodGet, tarballURL, nil)
if err != nil {
- return nil, nil, fmt.Errorf("creating tarball request: %w", err)
+ return nil, nil, nil, fmt.Errorf("creating tarball request: %w", err)
}
resp, err := s.httpClient.Do(req)
if err != nil {
- return nil, nil, fmt.Errorf("downloading tarball: %w", err)
+ return nil, nil, nil, fmt.Errorf("downloading tarball: %w", err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
- return nil, nil, fmt.Errorf("tarball download returned status %d", resp.StatusCode)
+ return nil, nil, nil, fmt.Errorf("tarball download returned status %d", resp.StatusCode)
}
// Create temp directory.
tmpDir, err := os.MkdirTemp("", "crowd-sniff-npm-*")
if err != nil {
- return nil, nil, fmt.Errorf("creating temp dir: %w", err)
+ return nil, nil, nil, fmt.Errorf("creating temp dir: %w", err)
}
defer func() { _ = os.RemoveAll(tmpDir) }()
// Extract tarball with size limit.
if err := extractTarball(resp.Body, tmpDir); err != nil {
- return nil, nil, fmt.Errorf("extracting tarball: %w", err)
+ return nil, nil, nil, fmt.Errorf("extracting tarball: %w", err)
}
- // Grep extracted files for endpoint patterns.
+ // Grep extracted files for endpoint patterns and auth patterns.
var allEndpoints []DiscoveredEndpoint
var allBaseURLs []string
+ var allContent strings.Builder
_ = filepath.Walk(tmpDir, func(path string, info os.FileInfo, err error) error {
if err != nil {
@@ -353,17 +356,23 @@ func (s *NPMSource) processPackageTarball(ctx context.Context, tarballURL, pkgNa
return nil // skip unreadable files
}
- endpoints, baseURLs := GrepEndpoints(string(content), pkgName, tier)
- endpoints = EnrichWithParams(string(content), endpoints)
+ contentStr := string(content)
+ endpoints, baseURLs := GrepEndpoints(contentStr, pkgName, tier)
+ endpoints = EnrichWithParams(contentStr, endpoints)
allEndpoints = append(allEndpoints, endpoints...)
allBaseURLs = append(allBaseURLs, baseURLs...)
+ allContent.WriteString(contentStr)
+ allContent.WriteByte('\n')
return nil
})
// Adjust tier for low-download packages (still community, but we note it).
_ = weeklyDownloads // Used for future priority sorting; tier stays the same.
- return allEndpoints, allBaseURLs, nil
+ // Extract auth patterns from the combined source content.
+ authPatterns := GrepAuth(allContent.String(), tier)
+
+ return allEndpoints, allBaseURLs, authPatterns, nil
}
// extractTarball extracts a gzipped tar archive to the destination directory.
@@ -462,3 +471,123 @@ func classifyPackage(pkg npmPackageInfo, apiNameLower string) string {
return TierCommunitySDK
}
+
+// --- Auth pattern detection ---
+
+var (
+ // bearerHeaderPattern matches Bearer token auth in headers.
+ // Examples:
+ // headers['Authorization'] = 'Bearer ' + token
+ // Authorization: `Bearer ${token}`
+ // headers.Authorization = `Bearer ${this.token}`
+ // 'Authorization': 'Bearer ' + apiKey
+ bearerHeaderPattern = regexp.MustCompile(`(?i)(?:headers\s*[\[.]\s*['"]?Authorization['"]?\s*[\])]?\s*=|['"]Authorization['"]\s*:)\s*['` + "`" + `]?\s*Bearer\b`)
+
+ // bearerTemplatePattern matches template literal Bearer patterns.
+ // Examples:
+ // `Bearer ${token}`
+ // `Bearer ${this.apiKey}`
+ bearerTemplatePattern = regexp.MustCompile(`(?i)Bearer\s+\$\{`)
+
+ // xApiKeyHeaderPattern matches X-Api-Key style headers.
+ // Examples:
+ // headers['X-Api-Key'] = apiKey
+ // 'X-Api-Key': this.key
+ // headers.set('x-api-key', key)
+ xApiKeyHeaderPattern = regexp.MustCompile(`(?i)(?:headers\s*[\[.]\s*)?['"]X-Api-Key['"]\s*[\])]?\s*[:=]`)
+
+ // queryKeyAuthPattern matches API key passed as a query parameter named "key".
+ // Examples:
+ // params.key = this.apiKey
+ // query.key = apiKey
+ // { key: this.apiKey }
+ // 'key': this.apiKey
+ // ?key=${apiKey}
+ // ?key=' + this.key
+ queryKeyAuthPattern = regexp.MustCompile(`(?i)(?:params|query)\s*\.\s*key\s*=|(?:\bkey\b\s*:\s*(?:this\s*\.\s*)?(?:api_?[Kk]ey|key)\b)|(?:['"]key['"]\s*:\s*(?:this\s*\.\s*)?(?:api_?[Kk]ey|key))|[?&]key\s*=\s*['` + "`" + `]\s*\+?\s*\$?\{?`)
+
+ // envVarHintPattern matches environment variable references that suggest auth.
+ // Examples:
+ // process.env.STEAM_API_KEY
+ // process.env.API_KEY
+ // process.env['NOTION_API_KEY']
+ envVarHintPattern = regexp.MustCompile(`process\.env\s*(?:\.\s*([A-Z][A-Z0-9_]*(?:API|KEY|TOKEN|SECRET)[A-Z0-9_]*)|\[\s*['"]([A-Z][A-Z0-9_]*(?:API|KEY|TOKEN|SECRET)[A-Z0-9_]*)['"]\s*\])`)
+)
+
+// GrepAuth scans SDK source code for authentication patterns and returns
+// any detected auth configurations. Designed for high precision — it only
+// reports patterns that are clearly auth-related. False negatives are
+// acceptable; false positives are not.
+func GrepAuth(content string, sourceTier string) []DiscoveredAuth {
+ var auths []DiscoveredAuth
+ seen := make(map[string]bool) // dedup by Type+In+Header
+
+ // Check for Bearer token auth.
+ if bearerHeaderPattern.MatchString(content) || bearerTemplatePattern.MatchString(content) {
+ key := "bearer_token:header:Authorization"
+ if !seen[key] {
+ seen[key] = true
+ auths = append(auths, DiscoveredAuth{
+ Type: "bearer_token",
+ Header: "Authorization",
+ In: "header",
+ Format: "Bearer {token}",
+ SourceTier: sourceTier,
+ })
+ }
+ }
+
+ // Check for X-Api-Key header auth.
+ if xApiKeyHeaderPattern.MatchString(content) {
+ key := "api_key:header:X-Api-Key"
+ if !seen[key] {
+ seen[key] = true
+ auths = append(auths, DiscoveredAuth{
+ Type: "api_key",
+ Header: "X-Api-Key",
+ In: "header",
+ SourceTier: sourceTier,
+ })
+ }
+ }
+
+ // Check for query param "key" auth.
+ if queryKeyAuthPattern.MatchString(content) {
+ key := "api_key:query:key"
+ if !seen[key] {
+ seen[key] = true
+ auths = append(auths, DiscoveredAuth{
+ Type: "api_key",
+ Header: "key",
+ In: "query",
+ SourceTier: sourceTier,
+ })
+ }
+ }
+
+ // Look for env var hints.
+ envVarHint := extractEnvVarHint(content)
+
+ // Apply env var hint to the first detected auth.
+ if envVarHint != "" && len(auths) > 0 {
+ auths[0].EnvVarHint = envVarHint
+ }
+
+ return auths
+}
+
+// extractEnvVarHint scans content for process.env references that look
+// auth-related and returns the first match.
+func extractEnvVarHint(content string) string {
+ matches := envVarHintPattern.FindAllStringSubmatch(content, -1)
+ for _, m := range matches {
+ // Group 1 is process.env.VAR, group 2 is process.env['VAR'].
+ if m[1] != "" {
+ return m[1]
+ }
+ if m[2] != "" {
+ return m[2]
+ }
+ }
+ return ""
+}
diff --git a/internal/crowdsniff/npm_test.go b/internal/crowdsniff/npm_test.go
index 92b4580f..a054e610 100644
--- a/internal/crowdsniff/npm_test.go
+++ b/internal/crowdsniff/npm_test.go
@@ -651,7 +651,7 @@ class API {
HTTPClient: tarballServer.Client(),
})
- endpoints, baseURLs, err := src.processPackageTarball(
+ endpoints, baseURLs, _, err := src.processPackageTarball(
context.Background(),
tarballServer.URL+"/tarball.tgz",
"test-sdk",
@@ -679,7 +679,7 @@ class API {
t.Parallel()
src := NewNPMSource(NPMOptions{})
- _, _, err := src.processPackageTarball(
+ _, _, _, err := src.processPackageTarball(
context.Background(),
"http://evil.com/tarball.tgz",
"evil-sdk",
@@ -709,7 +709,7 @@ class API {
HTTPClient: tarballServer.Client(),
})
- endpoints, _, err := src.processPackageTarball(
+ endpoints, _, _, err := src.processPackageTarball(
context.Background(),
tarballServer.URL+"/tarball.tgz",
"test-sdk",
@@ -743,7 +743,7 @@ class API {
HTTPClient: tarballServer.Client(),
})
- endpoints, _, err := src.processPackageTarball(
+ endpoints, _, _, err := src.processPackageTarball(
context.Background(),
tarballServer.URL+"/tarball.tgz",
"symlink-sdk",
@@ -875,7 +875,7 @@ class SteamAPI {
HTTPClient: tarballServer.Client(),
})
- endpoints, baseURLs, err := src.processPackageTarball(
+ endpoints, baseURLs, _, err := src.processPackageTarball(
context.Background(),
tarballServer.URL+"/tarball.tgz",
"steam-sdk",
@@ -949,3 +949,452 @@ func TestNPMSource_MaxPackagesLimit(t *testing.T) {
// Only first 10 should have version lookups attempted.
assert.LessOrEqual(t, versionCallCount, maxPackagesToProcess)
}
+
+func TestGrepAuth(t *testing.T) {
+ t.Parallel()
+
+ t.Run("detects query param key auth", func(t *testing.T) {
+ t.Parallel()
+
+ content := `
+class SteamAPI {
+ getOwnedGames(steamid) {
+ return this.get("/IPlayerService/GetOwnedGames/v1", {
+ key: this.apiKey,
+ steamid: steamid
+ });
+ }
+}
+`
+ auths := GrepAuth(content, TierCommunitySDK)
+
+ require.Len(t, auths, 1)
+ assert.Equal(t, "api_key", auths[0].Type)
+ assert.Equal(t, "key", auths[0].Header)
+ assert.Equal(t, "query", auths[0].In)
+ assert.Equal(t, TierCommunitySDK, auths[0].SourceTier)
+ })
+
+ t.Run("detects bearer token auth from header assignment", func(t *testing.T) {
+ t.Parallel()
+
+ content := `
+class Client {
+ constructor(token) {
+ this.token = token;
+ }
+ request(path) {
+ headers['Authorization'] = 'Bearer ' + this.token;
+ return fetch(this.baseUrl + path, { headers });
+ }
+}
+`
+ auths := GrepAuth(content, TierOfficialSDK)
+
+ require.Len(t, auths, 1)
+ assert.Equal(t, "bearer_token", auths[0].Type)
+ assert.Equal(t, "Authorization", auths[0].Header)
+ assert.Equal(t, "header", auths[0].In)
+ assert.Equal(t, "Bearer {token}", auths[0].Format)
+ assert.Equal(t, TierOfficialSDK, auths[0].SourceTier)
+ })
+
+ t.Run("detects bearer token from template literal", func(t *testing.T) {
+ t.Parallel()
+
+ content := "const headers = { 'Authorization': `Bearer ${this.apiKey}` };"
+
+ auths := GrepAuth(content, TierCommunitySDK)
+
+ require.Len(t, auths, 1)
+ assert.Equal(t, "bearer_token", auths[0].Type)
+ assert.Equal(t, "Authorization", auths[0].Header)
+ assert.Equal(t, "header", auths[0].In)
+ })
+
+ t.Run("detects X-Api-Key header auth", func(t *testing.T) {
+ t.Parallel()
+
+ content := `
+function makeRequest(url, apiKey) {
+ headers['X-Api-Key'] = apiKey;
+ return fetch(url, { headers });
+}
+`
+ auths := GrepAuth(content, TierCommunitySDK)
+
+ require.Len(t, auths, 1)
+ assert.Equal(t, "api_key", auths[0].Type)
+ assert.Equal(t, "X-Api-Key", auths[0].Header)
+ assert.Equal(t, "header", auths[0].In)
+ })
+
+ t.Run("no auth patterns returns empty", func(t *testing.T) {
+ t.Parallel()
+
+ content := `
+class Client {
+ listUsers() { return this.get("/v1/users"); }
+ createUser(data) { return this.post("/v1/users", data); }
+}
+`
+ auths := GrepAuth(content, TierCommunitySDK)
+ assert.Empty(t, auths)
+ })
+
+ t.Run("detects env var hint and attaches to first auth", func(t *testing.T) {
+ t.Parallel()
+
+ content := `
+const apiKey = process.env.STEAM_API_KEY;
+class SteamAPI {
+ request(url) {
+ return fetch(url + '?key=' + apiKey);
+ }
+}
+`
+ auths := GrepAuth(content, TierCommunitySDK)
+
+ require.NotEmpty(t, auths)
+ assert.Equal(t, "STEAM_API_KEY", auths[0].EnvVarHint)
+ })
+
+ t.Run("detects env var from bracket notation", func(t *testing.T) {
+ t.Parallel()
+
+ content := `const key = process.env['NOTION_API_KEY'];`
+
+ auths := GrepAuth(content, TierOfficialSDK)
+ // No specific auth pattern found for the key usage, but env var hint
+ // won't be applied without an auth match. This verifies extractEnvVarHint works.
+ hint := extractEnvVarHint(content)
+ assert.Equal(t, "NOTION_API_KEY", hint)
+
+ // Even without a matching auth pattern, we get no false positives.
+ _ = auths
+ })
+
+ t.Run("multiple auth patterns detected", func(t *testing.T) {
+ t.Parallel()
+
+ content := `
+class DualAuthClient {
+ request(url) {
+ headers['X-Api-Key'] = this.apiKey;
+ headers['Authorization'] = 'Bearer ' + this.token;
+ return fetch(url, { headers });
+ }
+}
+`
+ auths := GrepAuth(content, TierCommunitySDK)
+
+ // Should detect both bearer and X-Api-Key.
+ assert.GreaterOrEqual(t, len(auths), 2)
+
+ types := make(map[string]bool)
+ for _, a := range auths {
+ types[a.Type+":"+a.Header] = true
+ }
+ assert.True(t, types["bearer_token:Authorization"])
+ assert.True(t, types["api_key:X-Api-Key"])
+ })
+
+ t.Run("params.key assignment detected", func(t *testing.T) {
+ t.Parallel()
+
+ content := `
+function getUser(userId) {
+ params.key = this.apiKey;
+ return this.get("/users/" + userId, params);
+}
+`
+ auths := GrepAuth(content, TierCommunitySDK)
+
+ require.Len(t, auths, 1)
+ assert.Equal(t, "api_key", auths[0].Type)
+ assert.Equal(t, "query", auths[0].In)
+ assert.Equal(t, "key", auths[0].Header)
+ })
+}
+
+func TestAggregateAuth(t *testing.T) {
+ t.Parallel()
+
+ t.Run("highest tier source wins", func(t *testing.T) {
+ t.Parallel()
+
+ results := []SourceResult{
+ {
+ Auth: []DiscoveredAuth{
+ {Type: "api_key", Header: "key", In: "query", SourceTier: TierCommunitySDK},
+ },
+ },
+ {
+ Auth: []DiscoveredAuth{
+ {Type: "bearer_token", Header: "Authorization", In: "header", SourceTier: TierOfficialSDK},
+ },
+ },
+ }
+
+ merged := AggregateAuth(results)
+ require.NotNil(t, merged)
+ assert.Equal(t, "bearer_token", merged.Type)
+ assert.Equal(t, "Authorization", merged.Header)
+ assert.Equal(t, TierOfficialSDK, merged.SourceTier)
+ })
+
+ t.Run("same tier prefers env var hint", func(t *testing.T) {
+ t.Parallel()
+
+ results := []SourceResult{
+ {
+ Auth: []DiscoveredAuth{
+ {Type: "api_key", Header: "key", In: "query", SourceTier: TierCommunitySDK},
+ },
+ },
+ {
+ Auth: []DiscoveredAuth{
+ {Type: "api_key", Header: "key", In: "query", SourceTier: TierCommunitySDK, EnvVarHint: "STEAM_API_KEY"},
+ },
+ },
+ }
+
+ merged := AggregateAuth(results)
+ require.NotNil(t, merged)
+ assert.Equal(t, "STEAM_API_KEY", merged.EnvVarHint)
+ })
+
+ t.Run("no auth returns nil", func(t *testing.T) {
+ t.Parallel()
+
+ results := []SourceResult{
+ {Endpoints: []DiscoveredEndpoint{{Method: "GET", Path: "/users"}}},
+ }
+
+ merged := AggregateAuth(results)
+ assert.Nil(t, merged)
+ })
+
+ t.Run("empty results returns nil", func(t *testing.T) {
+ t.Parallel()
+
+ merged := AggregateAuth(nil)
+ assert.Nil(t, merged)
+ })
+}
+
+func TestBuildSpec_WithAuth(t *testing.T) {
+ t.Parallel()
+
+ t.Run("auth applied to spec when detected", func(t *testing.T) {
+ t.Parallel()
+
+ endpoints := []AggregatedEndpoint{
+ {Method: "GET", Path: "/v1/users", SourceTier: TierOfficialSDK, SourceCount: 1},
+ }
+ auth := &DiscoveredAuth{
+ Type: "bearer_token",
+ Header: "Authorization",
+ In: "header",
+ Format: "Bearer {token}",
+ EnvVarHint: "NOTION_TOKEN",
+ SourceTier: TierOfficialSDK,
+ }
+
+ apiSpec, err := BuildSpec("notion", "https://api.notion.com", endpoints, auth)
+ require.NoError(t, err)
+
+ assert.Equal(t, "bearer_token", apiSpec.Auth.Type)
+ assert.Equal(t, "Authorization", apiSpec.Auth.Header)
+ assert.Equal(t, "header", apiSpec.Auth.In)
+ assert.Equal(t, "Bearer {token}", apiSpec.Auth.Format)
+ assert.Equal(t, []string{"NOTION_TOKEN"}, apiSpec.Auth.EnvVars)
+ })
+
+ t.Run("auth defaults to none when nil", func(t *testing.T) {
+ t.Parallel()
+
+ endpoints := []AggregatedEndpoint{
+ {Method: "GET", Path: "/v1/users", SourceTier: TierCodeSearch, SourceCount: 1},
+ }
+
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints, nil)
+ require.NoError(t, err)
+
+ assert.Equal(t, "none", apiSpec.Auth.Type)
+ })
+
+ t.Run("env var derived from API name when no hint", func(t *testing.T) {
+ t.Parallel()
+
+ endpoints := []AggregatedEndpoint{
+ {Method: "GET", Path: "/v1/games", SourceTier: TierCommunitySDK, SourceCount: 1},
+ }
+ auth := &DiscoveredAuth{
+ Type: "api_key",
+ Header: "key",
+ In: "query",
+ SourceTier: TierCommunitySDK,
+ }
+
+ apiSpec, err := BuildSpec("steam", "https://api.steampowered.com", endpoints, auth)
+ require.NoError(t, err)
+
+ assert.Equal(t, "api_key", apiSpec.Auth.Type)
+ assert.Equal(t, "key", apiSpec.Auth.Header)
+ assert.Equal(t, "query", apiSpec.Auth.In)
+ assert.Equal(t, []string{"STEAM_API_KEY"}, apiSpec.Auth.EnvVars)
+ })
+
+ t.Run("bearer token derives TOKEN env var", func(t *testing.T) {
+ t.Parallel()
+
+ endpoints := []AggregatedEndpoint{
+ {Method: "GET", Path: "/v1/users", SourceTier: TierOfficialSDK, SourceCount: 1},
+ }
+ auth := &DiscoveredAuth{
+ Type: "bearer_token",
+ Header: "Authorization",
+ In: "header",
+ Format: "Bearer {token}",
+ SourceTier: TierOfficialSDK,
+ }
+
+ apiSpec, err := BuildSpec("notion", "https://api.notion.com", endpoints, auth)
+ require.NoError(t, err)
+
+ assert.Equal(t, []string{"NOTION_TOKEN"}, apiSpec.Auth.EnvVars)
+ })
+
+ t.Run("API name with special chars normalized in env var", func(t *testing.T) {
+ t.Parallel()
+
+ endpoints := []AggregatedEndpoint{
+ {Method: "GET", Path: "/v1/events", SourceTier: TierOfficialSDK, SourceCount: 1},
+ }
+ auth := &DiscoveredAuth{
+ Type: "api_key",
+ Header: "key",
+ In: "query",
+ SourceTier: TierOfficialSDK,
+ }
+
+ apiSpec, err := BuildSpec("cal.com", "https://api.cal.com", endpoints, auth)
+ require.NoError(t, err)
+
+ assert.Equal(t, []string{"CAL_COM_API_KEY"}, apiSpec.Auth.EnvVars)
+ })
+}
+
+func TestDeriveEnvVar(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ apiName string
+ authType string
+ want string
+ }{
+ {name: "api_key from steam", apiName: "steam", authType: "api_key", want: "STEAM_API_KEY"},
+ {name: "bearer_token from notion", apiName: "notion", authType: "bearer_token", want: "NOTION_TOKEN"},
+ {name: "api name with dot", apiName: "cal.com", authType: "api_key", want: "CAL_COM_API_KEY"},
+ {name: "api name with dash", apiName: "dub-co", authType: "bearer_token", want: "DUB_CO_TOKEN"},
+ {name: "basic auth", apiName: "myapi", authType: "basic", want: "MYAPI_API_KEY"},
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+ assert.Equal(t, tt.want, deriveEnvVar(tt.apiName, tt.authType))
+ })
+ }
+}
+
+func TestProcessPackageTarball_AuthDetection(t *testing.T) {
+ t.Parallel()
+
+ t.Run("detects auth from SDK tarball", func(t *testing.T) {
+ t.Parallel()
+
+ sdkContent := `
+const baseUrl = "https://api.steampowered.com";
+class SteamAPI {
+ constructor(apiKey) {
+ this.apiKey = apiKey;
+ }
+ getOwnedGames(steamid) {
+ return this.get("/IPlayerService/GetOwnedGames/v1", {
+ key: this.apiKey,
+ steamid: steamid
+ });
+ }
+}
+`
+ tarball := buildTarball(t, map[string]string{
+ "package/src/client.js": sdkContent,
+ })
+
+ tarballServer := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Write(tarball)
+ }))
+ defer tarballServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ HTTPClient: tarballServer.Client(),
+ })
+
+ _, _, authPatterns, err := src.processPackageTarball(
+ context.Background(),
+ tarballServer.URL+"/tarball.tgz",
+ "steam-sdk",
+ TierCommunitySDK,
+ 500,
+ )
+
+ require.NoError(t, err)
+ require.NotEmpty(t, authPatterns, "expected auth patterns to be detected")
+
+ // Should detect query key auth.
+ var foundQueryKey bool
+ for _, auth := range authPatterns {
+ if auth.Type == "api_key" && auth.In == "query" && auth.Header == "key" {
+ foundQueryKey = true
+ }
+ }
+ assert.True(t, foundQueryKey, "expected api_key query auth with header=key")
+ })
+
+ t.Run("no auth from tarball without auth patterns", func(t *testing.T) {
+ t.Parallel()
+
+ sdkContent := `
+const baseUrl = "https://api.test.com";
+class API {
+ listUsers() { return this.get("/v1/users"); }
+}
+`
+ tarball := buildTarball(t, map[string]string{
+ "package/src/client.js": sdkContent,
+ })
+
+ tarballServer := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Write(tarball)
+ }))
+ defer tarballServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ HTTPClient: tarballServer.Client(),
+ })
+
+ _, _, authPatterns, err := src.processPackageTarball(
+ context.Background(),
+ tarballServer.URL+"/tarball.tgz",
+ "test-sdk",
+ TierCommunitySDK,
+ 0,
+ )
+
+ require.NoError(t, err)
+ assert.Empty(t, authPatterns)
+ })
+}
diff --git a/internal/crowdsniff/specgen.go b/internal/crowdsniff/specgen.go
index aa2d2f9f..6ef4ee9a 100644
--- a/internal/crowdsniff/specgen.go
+++ b/internal/crowdsniff/specgen.go
@@ -9,7 +9,9 @@ import (
)
// BuildSpec assembles a valid spec.APISpec from aggregated endpoints.
-func BuildSpec(name, baseURL string, endpoints []AggregatedEndpoint) (*spec.APISpec, error) {
+// If auth is non-nil and the spec would otherwise default to "none", the
+// detected auth is applied.
+func BuildSpec(name, baseURL string, endpoints []AggregatedEndpoint, auth *DiscoveredAuth) (*spec.APISpec, error) {
if len(endpoints) == 0 {
return nil, fmt.Errorf("no endpoints to build spec from")
}
@@ -74,12 +76,17 @@ func BuildSpec(name, baseURL string, endpoints []AggregatedEndpoint) (*spec.APIS
resources[resourceKey] = resource
}
+ authConfig := spec.AuthConfig{Type: "none"}
+ if auth != nil {
+ authConfig = buildAuthConfig(name, auth)
+ }
+
apiSpec := &spec.APISpec{
Name: name,
Description: fmt.Sprintf("Discovered API spec for %s (crowd-sniff)", name),
Version: "0.1.0",
BaseURL: baseURL,
- Auth: spec.AuthConfig{Type: "none"},
+ Auth: authConfig,
Config: spec.ConfigSpec{
Format: "toml",
Path: fmt.Sprintf("~/.config/%s-pp-cli/config.toml", name),
@@ -116,10 +123,10 @@ func deriveResourceKey(path string) (string, string) {
if len(segments) == 0 {
return "", ""
}
- if len(segments) > 3 {
- segments = segments[:3]
- }
- return strings.Join(segments, "/"), segments[len(segments)-1]
+ // Use only the first significant segment as the resource key.
+ // This prevents slashes in resource names which break the generator's
+ // filepath.Join and Cobra Use field.
+ return segments[0], segments[len(segments)-1]
}
func significantSegments(path string) []string {
@@ -185,3 +192,51 @@ func uniqueEndpointName(endpoints map[string]spec.Endpoint, base string) string
}
}
}
+
+// buildAuthConfig converts a DiscoveredAuth into a spec.AuthConfig.
+// It also derives an env var name from the API name if no hint was detected.
+func buildAuthConfig(apiName string, auth *DiscoveredAuth) spec.AuthConfig {
+ cfg := spec.AuthConfig{
+ Type: auth.Type,
+ Header: auth.Header,
+ In: auth.In,
+ Format: auth.Format,
+ }
+
+ envVar := auth.EnvVarHint
+ if envVar == "" {
+ envVar = deriveEnvVar(apiName, auth.Type)
+ }
+ if envVar != "" {
+ cfg.EnvVars = []string{envVar}
+ }
+
+ return cfg
+}
+
+// deriveEnvVar generates an environment variable name from the API name and auth type.
+// Example: apiName="steam", authType="api_key" → "STEAM_API_KEY"
+func deriveEnvVar(apiName, authType string) string {
+ // Normalize: replace non-alphanumeric with underscore, uppercase.
+ var b strings.Builder
+ for _, r := range apiName {
+ if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') {
+ b.WriteRune(r)
+ } else {
+ b.WriteByte('_')
+ }
+ }
+ prefix := strings.ToUpper(b.String())
+ prefix = strings.Trim(prefix, "_")
+
+ switch authType {
+ case "bearer_token":
+ return prefix + "_TOKEN"
+ case "api_key":
+ return prefix + "_API_KEY"
+ case "basic":
+ return prefix + "_API_KEY"
+ default:
+ return prefix + "_API_KEY"
+ }
+}
diff --git a/internal/crowdsniff/specgen_test.go b/internal/crowdsniff/specgen_test.go
index 05887f37..4d63678a 100644
--- a/internal/crowdsniff/specgen_test.go
+++ b/internal/crowdsniff/specgen_test.go
@@ -19,7 +19,7 @@ func TestBuildSpec(t *testing.T) {
{Method: "POST", Path: "/v1/users", SourceTier: TierCommunitySDK, SourceCount: 1},
}
- apiSpec, err := BuildSpec("notion", "https://api.notion.com", endpoints)
+ apiSpec, err := BuildSpec("notion", "https://api.notion.com", endpoints, nil)
require.NoError(t, err)
assert.Equal(t, "notion", apiSpec.Name)
@@ -38,7 +38,7 @@ func TestBuildSpec(t *testing.T) {
{Method: "GET", Path: "/v1/users", SourceTier: TierOfficialSDK, SourceCount: 3},
}
- apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints, nil)
require.NoError(t, err)
for _, resource := range apiSpec.Resources {
@@ -56,7 +56,7 @@ func TestBuildSpec(t *testing.T) {
{Method: "GET", Path: "/v1/projects", SourceTier: TierCodeSearch, SourceCount: 1},
}
- apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints, nil)
require.NoError(t, err)
_, hasUsers := apiSpec.Resources["users"]
@@ -67,7 +67,7 @@ func TestBuildSpec(t *testing.T) {
t.Run("empty endpoints returns error", func(t *testing.T) {
t.Parallel()
- _, err := BuildSpec("test", "https://api.example.com", nil)
+ _, err := BuildSpec("test", "https://api.example.com", nil, nil)
assert.Error(t, err)
assert.Contains(t, err.Error(), "no endpoints")
})
@@ -76,7 +76,7 @@ func TestBuildSpec(t *testing.T) {
t.Parallel()
_, err := BuildSpec("", "https://api.example.com", []AggregatedEndpoint{
{Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceCount: 1},
- })
+ }, nil)
assert.Error(t, err)
assert.Contains(t, err.Error(), "name is required")
})
@@ -85,7 +85,7 @@ func TestBuildSpec(t *testing.T) {
t.Parallel()
_, err := BuildSpec("test", "", []AggregatedEndpoint{
{Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceCount: 1},
- })
+ }, nil)
assert.Error(t, err)
assert.Contains(t, err.Error(), "base_url is required")
})
@@ -96,7 +96,7 @@ func TestBuildSpec(t *testing.T) {
{Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceCount: 1},
}
- apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints, nil)
require.NoError(t, err)
assert.NoError(t, apiSpec.Validate())
})
@@ -145,6 +145,81 @@ func TestResolveBaseURL(t *testing.T) {
}
}
+func TestDeriveResourceKey(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ path string
+ wantResource string
+ wantName string
+ }{
+ {
+ name: "Steam-like path uses first segment as resource",
+ path: "/ISteamUser/GetPlayerSummaries/v2",
+ wantResource: "ISteamUser",
+ wantName: "GetPlayerSummaries",
+ },
+ {
+ name: "REST nested path uses first segment only",
+ path: "/v1/users/{id}/posts",
+ wantResource: "users",
+ wantName: "posts",
+ },
+ {
+ name: "simple path unchanged",
+ path: "/v1/users",
+ wantResource: "users",
+ wantName: "users",
+ },
+ {
+ name: "only params returns empty",
+ path: "/v1/{id}",
+ wantResource: "",
+ wantName: "",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+ resource, name := deriveResourceKey(tt.path)
+ assert.Equal(t, tt.wantResource, resource)
+ assert.Equal(t, tt.wantName, name)
+ })
+ }
+}
+
+func TestDeriveResourceKey_NoSlashes(t *testing.T) {
+ t.Parallel()
+
+ // Resource keys must never contain slashes — they are used as filenames
+ // and Cobra command Use fields.
+ paths := []string{
+ "/v1/users",
+ "/v1/users/{id}/posts",
+ "/v1/users/{id}/posts/{postId}/comments",
+ "/api/v2/organizations/{orgId}/members",
+ "/ISteamUser/GetPlayerSummaries/v2",
+ }
+
+ for _, p := range paths {
+ resource, _ := deriveResourceKey(p)
+ assert.NotContains(t, resource, "/", "resource key for %q must not contain slashes", p)
+ }
+}
+
+func TestDeriveResourceKey_SharedFirstSegment(t *testing.T) {
+ t.Parallel()
+
+ // Two paths sharing the same first significant segment must map to the
+ // same resource key so they end up in one file.
+ res1, _ := deriveResourceKey("/v1/users")
+ res2, _ := deriveResourceKey("/v1/users/{id}/posts")
+ assert.Equal(t, res1, res2, "paths sharing first segment should have same resource key")
+ assert.Equal(t, "users", res1)
+}
+
func TestBuildSpec_ParamMapping(t *testing.T) {
t.Parallel()
@@ -164,7 +239,7 @@ func TestBuildSpec_ParamMapping(t *testing.T) {
},
}
- apiSpec, err := BuildSpec("steam", "https://api.steampowered.com", endpoints)
+ apiSpec, err := BuildSpec("steam", "https://api.steampowered.com", endpoints, nil)
require.NoError(t, err)
// Find the endpoint in the spec.
@@ -210,7 +285,7 @@ func TestBuildSpec_ParamMapping(t *testing.T) {
},
}
- apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints, nil)
require.NoError(t, err)
var found *spec.Endpoint
@@ -251,7 +326,7 @@ func TestBuildSpec_ParamMapping(t *testing.T) {
},
}
- apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints, nil)
require.NoError(t, err)
for _, resource := range apiSpec.Resources {
@@ -285,7 +360,7 @@ func TestBuildSpec_ParamMapping(t *testing.T) {
},
}
- apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints, nil)
require.NoError(t, err)
for _, resource := range apiSpec.Resources {
diff --git a/internal/crowdsniff/types.go b/internal/crowdsniff/types.go
index faac188e..40fa95a2 100644
--- a/internal/crowdsniff/types.go
+++ b/internal/crowdsniff/types.go
@@ -25,10 +25,21 @@ type DiscoveredEndpoint struct {
SourceName string // e.g., "@notionhq/client", "github-code-search"
}
+// DiscoveredAuth represents an authentication pattern detected in SDK source code.
+type DiscoveredAuth struct {
+ Type string // "api_key", "bearer_token", "basic"
+ Header string // header name or query param name (e.g., "key", "X-Api-Key", "Authorization")
+ In string // "header" or "query"
+ Format string // e.g., "Bearer {token}", "{api_key}"
+ EnvVarHint string // detected env var name if visible (e.g., "STEAM_API_KEY")
+ SourceTier string // tier of the source that found this auth pattern
+}
+
// SourceResult is returned by each discovery source.
type SourceResult struct {
Endpoints []DiscoveredEndpoint
BaseURLCandidates []string // e.g., "https://api.notion.com"
+ Auth []DiscoveredAuth
}
// AggregatedEndpoint is a deduplicated endpoint with cross-source metadata.
diff --git a/internal/generator/generator.go b/internal/generator/generator.go
index 94a18b56..90775f14 100644
--- a/internal/generator/generator.go
+++ b/internal/generator/generator.go
@@ -95,10 +95,42 @@ func New(s *spec.APISpec, outputDir string) *Generator {
"exampleLine": g.exampleLine,
"currentYear": func() string { return strconv.Itoa(time.Now().Year()) },
"modulePath": func() string { return naming.CLI(s.Name) },
+ "kebab": toKebab,
}
return g
}
+// HelperFlags controls which helper functions are emitted in helpers.go.
+type HelperFlags struct {
+ HasDelete bool // spec has DELETE endpoints → emit classifyDeleteError
+}
+
+// computeHelperFlags scans the spec's resources to determine which helpers are needed.
+func computeHelperFlags(s *spec.APISpec) HelperFlags {
+ var flags HelperFlags
+ for _, r := range s.Resources {
+ for _, e := range r.Endpoints {
+ if e.Method == "DELETE" {
+ flags.HasDelete = true
+ }
+ }
+ for _, sub := range r.SubResources {
+ for _, e := range sub.Endpoints {
+ if e.Method == "DELETE" {
+ flags.HasDelete = true
+ }
+ }
+ }
+ }
+ return flags
+}
+
+// helpersTemplateData wraps APISpec with flags controlling conditional helper emission.
+type helpersTemplateData struct {
+ *spec.APISpec
+ HelperFlags
+}
+
// readmeTemplateData wraps APISpec with additional fields for README rendering.
type readmeTemplateData struct {
*spec.APISpec
@@ -149,6 +181,11 @@ func (g *Generator) Generate() error {
data := any(g.Spec)
if tmplName == "readme.md.tmpl" {
data = g.readmeData()
+ } else if tmplName == "helpers.go.tmpl" {
+ data = &helpersTemplateData{
+ APISpec: g.Spec,
+ HelperFlags: computeHelperFlags(g.Spec),
+ }
}
if err := g.renderTemplate(tmplName, outPath, data); err != nil {
return fmt.Errorf("rendering %s: %w", tmplName, err)
@@ -413,16 +450,40 @@ func (g *Generator) Generate() error {
}
}
+ // Generate promoted top-level commands (user-friendly aliases for nested API commands)
+ promotedCommands := buildPromotedCommands(g.Spec)
+ for _, pc := range promotedCommands {
+ promotedData := struct {
+ PromotedName string
+ ResourceName string
+ EndpointName string
+ Endpoint spec.Endpoint
+ *spec.APISpec
+ }{
+ PromotedName: pc.PromotedName,
+ ResourceName: pc.ResourceName,
+ EndpointName: pc.EndpointName,
+ Endpoint: pc.Endpoint,
+ APISpec: g.Spec,
+ }
+ promotedPath := filepath.Join("internal", "cli", "promoted_"+pc.PromotedName+".go")
+ if err := g.renderTemplate("command_promoted.go.tmpl", promotedPath, promotedData); err != nil {
+ return fmt.Errorf("rendering promoted command %s: %w", pc.PromotedName, err)
+ }
+ }
+
rootData := struct {
*spec.APISpec
VisionSet VisionTemplateSet
WorkflowConstructors []string
InsightConstructors []string
+ PromotedCommands []PromotedCommand
}{
APISpec: g.Spec,
VisionSet: g.VisionSet,
WorkflowConstructors: renderedWorkflowConstructors,
InsightConstructors: renderedInsightConstructors,
+ PromotedCommands: promotedCommands,
}
if err := g.renderTemplate("root.go.tmpl", filepath.Join("internal", "cli", "root.go"), rootData); err != nil {
return fmt.Errorf("rendering root: %w", err)
@@ -847,6 +908,110 @@ func safeTypeName(name string) string {
return result
}
+// toKebab converts PascalCase, camelCase, or mixed names to kebab-case.
+// It also strips a leading "I" if it looks like an interface prefix (e.g., ISteamUser → steam-user).
+func toKebab(s string) string {
+ // Strip leading "I" when followed by an uppercase letter (interface prefix convention)
+ if len(s) > 1 && s[0] == 'I' && unicode.IsUpper(rune(s[1])) {
+ s = s[1:]
+ }
+ var result strings.Builder
+ for i, r := range s {
+ if unicode.IsUpper(r) && i > 0 {
+ prev := rune(s[i-1])
+ // Insert hyphen before uppercase letter if preceded by lowercase,
+ // or if preceding char is uppercase AND next char is lowercase (e.g., "APIKey" → "api-key")
+ if unicode.IsLower(prev) || (unicode.IsUpper(prev) && i+1 < len(s) && unicode.IsLower(rune(s[i+1]))) {
+ result.WriteByte('-')
+ }
+ }
+ result.WriteRune(unicode.ToLower(r))
+ }
+ return result.String()
+}
+
+// PromotedCommand represents a top-level user-friendly command that wraps a nested API endpoint.
+type PromotedCommand struct {
+ PromotedName string
+ ResourceName string
+ Endpoint spec.Endpoint
+ EndpointName string
+}
+
+// builtinCommands lists command names that must not be used for promoted commands
+// because they collide with the CLI's own built-in commands.
+var builtinCommands = map[string]bool{
+ "version": true,
+ "help": true,
+ "doctor": true,
+ "auth": true,
+ "sync": true,
+ "search": true,
+ "export": true,
+ "import": true,
+ "completion": true,
+ "workflow": true,
+ "tail": true,
+ "analytics": true,
+}
+
+// buildPromotedCommands scans spec resources and returns promotable top-level commands.
+// For each resource, it finds the "primary" GET endpoint (no path params, or first GET)
+// and creates a promoted command with a cleaner name.
+func buildPromotedCommands(s *spec.APISpec) []PromotedCommand {
+ var promoted []PromotedCommand
+ usedNames := make(map[string]bool)
+
+ for name, resource := range s.Resources {
+ // Find the primary GET endpoint: prefer GET without positional params, else first GET
+ var bestName string
+ var bestEndpoint spec.Endpoint
+ found := false
+
+ for eName, ep := range resource.Endpoints {
+ if ep.Method != "GET" {
+ continue
+ }
+ hasPositional := false
+ for _, p := range ep.Params {
+ if p.Positional {
+ hasPositional = true
+ break
+ }
+ }
+ if !found || !hasPositional {
+ bestName = eName
+ bestEndpoint = ep
+ found = true
+ if !hasPositional {
+ break // Ideal: GET without path params (list endpoint)
+ }
+ }
+ }
+
+ if !found {
+ continue
+ }
+
+ promotedName := toKebab(name)
+ if builtinCommands[promotedName] {
+ continue
+ }
+ if usedNames[promotedName] {
+ continue
+ }
+ usedNames[promotedName] = true
+
+ promoted = append(promoted, PromotedCommand{
+ PromotedName: promotedName,
+ ResourceName: name,
+ Endpoint: bestEndpoint,
+ EndpointName: bestName,
+ })
+ }
+ return promoted
+}
+
func envVarPlaceholder(envVar string) string {
// STYTCH_PROJECT_ID -> project_id (the placeholder in the format string)
parts := strings.Split(envVar, "_")
diff --git a/internal/generator/generator_test.go b/internal/generator/generator_test.go
index 2689bb74..8015cf00 100644
--- a/internal/generator/generator_test.go
+++ b/internal/generator/generator_test.go
@@ -22,9 +22,9 @@ func TestGenerateProjectsCompile(t *testing.T) {
specPath string
expectedFiles int
}{
- {name: "stytch", specPath: filepath.Join("..", "..", "testdata", "stytch.yaml"), expectedFiles: 30},
- {name: "clerk", specPath: filepath.Join("..", "..", "testdata", "clerk.yaml"), expectedFiles: 35},
- {name: "loops", specPath: filepath.Join("..", "..", "testdata", "loops.yaml"), expectedFiles: 33},
+ {name: "stytch", specPath: filepath.Join("..", "..", "testdata", "stytch.yaml"), expectedFiles: 32},
+ {name: "clerk", specPath: filepath.Join("..", "..", "testdata", "clerk.yaml"), expectedFiles: 38},
+ {name: "loops", specPath: filepath.Join("..", "..", "testdata", "loops.yaml"), expectedFiles: 34},
}
for _, tt := range tests {
@@ -457,3 +457,430 @@ func TestGeneratedOutput_GetCommandsLackEnvelope(t *testing.T) {
assert.NotContains(t, content, "envelope")
assert.NotContains(t, content, "statusCode")
}
+
+// --- Unit 4: Conditional Helper Emission Tests ---
+
+func TestComputeHelperFlags(t *testing.T) {
+ t.Parallel()
+
+ t.Run("spec with DELETE endpoints sets HasDelete", func(t *testing.T) {
+ s := &spec.APISpec{
+ Name: "test",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Resources: map[string]spec.Resource{
+ "items": {
+ Endpoints: map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/items"},
+ "delete": {Method: "DELETE", Path: "/items/{id}"},
+ },
+ },
+ },
+ }
+ flags := computeHelperFlags(s)
+ assert.True(t, flags.HasDelete)
+ })
+
+ t.Run("spec without DELETE endpoints clears HasDelete", func(t *testing.T) {
+ s := &spec.APISpec{
+ Name: "test",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Resources: map[string]spec.Resource{
+ "items": {
+ Endpoints: map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/items"},
+ "create": {Method: "POST", Path: "/items"},
+ },
+ },
+ },
+ }
+ flags := computeHelperFlags(s)
+ assert.False(t, flags.HasDelete)
+ })
+
+ t.Run("DELETE in sub-resource sets HasDelete", func(t *testing.T) {
+ s := &spec.APISpec{
+ Name: "test",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Resources: map[string]spec.Resource{
+ "projects": {
+ Endpoints: map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/projects"},
+ },
+ SubResources: map[string]spec.Resource{
+ "tasks": {
+ Endpoints: map[string]spec.Endpoint{
+ "delete": {Method: "DELETE", Path: "/projects/{id}/tasks/{task_id}"},
+ },
+ },
+ },
+ },
+ },
+ }
+ flags := computeHelperFlags(s)
+ assert.True(t, flags.HasDelete)
+ })
+}
+
+func TestGeneratedHelpers_ConditionalClassifyDeleteError(t *testing.T) {
+ t.Parallel()
+
+ baseSpec := func(endpoints map[string]spec.Endpoint) *spec.APISpec {
+ return &spec.APISpec{
+ Name: "testhelpers",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Auth: spec.AuthConfig{Type: "api_key", Header: "X-Api-Key", EnvVars: []string{"TEST_API_KEY"}},
+ Config: spec.ConfigSpec{Format: "toml", Path: "~/.config/testhelpers-pp-cli/config.toml"},
+ Resources: map[string]spec.Resource{
+ "items": {
+ Description: "Manage items",
+ Endpoints: endpoints,
+ },
+ },
+ }
+ }
+
+ t.Run("no DELETE endpoints omits classifyDeleteError", func(t *testing.T) {
+ apiSpec := baseSpec(map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/items", Description: "List items"},
+ })
+
+ outputDir := filepath.Join(t.TempDir(), "testhelpers-pp-cli")
+ gen := New(apiSpec, outputDir)
+ require.NoError(t, gen.Generate())
+
+ helpersGo, err := os.ReadFile(filepath.Join(outputDir, "internal", "cli", "helpers.go"))
+ require.NoError(t, err)
+ content := string(helpersGo)
+ assert.NotContains(t, content, "classifyDeleteError")
+ // classifyAPIError should always be present
+ assert.Contains(t, content, "classifyAPIError")
+ })
+
+ t.Run("with DELETE endpoints includes classifyDeleteError", func(t *testing.T) {
+ apiSpec := baseSpec(map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/items", Description: "List items"},
+ "delete": {Method: "DELETE", Path: "/items/{id}", Description: "Delete item"},
+ })
+
+ outputDir := filepath.Join(t.TempDir(), "testhelpers-pp-cli")
+ gen := New(apiSpec, outputDir)
+ require.NoError(t, gen.Generate())
+
+ helpersGo, err := os.ReadFile(filepath.Join(outputDir, "internal", "cli", "helpers.go"))
+ require.NoError(t, err)
+ content := string(helpersGo)
+ assert.Contains(t, content, "classifyDeleteError")
+ assert.Contains(t, content, "classifyAPIError")
+ })
+}
+
+// --- Unit 3: Top-Level Command Promotion Tests ---
+
+func TestToKebab(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ input string
+ expected string
+ }{
+ {"ISteamUser", "steam-user"},
+ {"SteamUser", "steam-user"},
+ {"users", "users"},
+ {"IPlayerService", "player-service"},
+ {"camelCase", "camel-case"},
+ {"PascalCase", "pascal-case"},
+ {"APIKey", "api-key"},
+ {"simpleresource", "simpleresource"},
+ {"ABC", "abc"},
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.input, func(t *testing.T) {
+ t.Parallel()
+ assert.Equal(t, tt.expected, toKebab(tt.input))
+ })
+ }
+}
+
+func TestBuildPromotedCommands(t *testing.T) {
+ t.Parallel()
+
+ t.Run("resource with list endpoint gets promoted", func(t *testing.T) {
+ t.Parallel()
+ s := &spec.APISpec{
+ Name: "test",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Resources: map[string]spec.Resource{
+ "users": {
+ Endpoints: map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/users", Description: "List users"},
+ },
+ },
+ },
+ }
+ promoted := buildPromotedCommands(s)
+ require.Len(t, promoted, 1)
+ assert.Equal(t, "users", promoted[0].PromotedName)
+ assert.Equal(t, "users", promoted[0].ResourceName)
+ assert.Equal(t, "list", promoted[0].EndpointName)
+ })
+
+ t.Run("ISteamUser resource becomes steam-user", func(t *testing.T) {
+ t.Parallel()
+ s := &spec.APISpec{
+ Name: "test",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Resources: map[string]spec.Resource{
+ "ISteamUser": {
+ Endpoints: map[string]spec.Endpoint{
+ "get_player_summaries": {Method: "GET", Path: "/ISteamUser/GetPlayerSummaries/v2", Description: "Get player summaries"},
+ },
+ },
+ },
+ }
+ promoted := buildPromotedCommands(s)
+ require.Len(t, promoted, 1)
+ assert.Equal(t, "steam-user", promoted[0].PromotedName)
+ assert.Equal(t, "ISteamUser", promoted[0].ResourceName)
+ })
+
+ t.Run("resource named version is skipped (collides with built-in)", func(t *testing.T) {
+ t.Parallel()
+ s := &spec.APISpec{
+ Name: "test",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Resources: map[string]spec.Resource{
+ "version": {
+ Endpoints: map[string]spec.Endpoint{
+ "get": {Method: "GET", Path: "/version", Description: "Get version"},
+ },
+ },
+ },
+ }
+ promoted := buildPromotedCommands(s)
+ assert.Empty(t, promoted)
+ })
+
+ t.Run("resource with no GET endpoints is skipped", func(t *testing.T) {
+ t.Parallel()
+ s := &spec.APISpec{
+ Name: "test",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Resources: map[string]spec.Resource{
+ "items": {
+ Endpoints: map[string]spec.Endpoint{
+ "create": {Method: "POST", Path: "/items", Description: "Create item"},
+ "delete": {Method: "DELETE", Path: "/items/{id}", Description: "Delete item"},
+ },
+ },
+ },
+ }
+ promoted := buildPromotedCommands(s)
+ assert.Empty(t, promoted)
+ })
+
+ t.Run("prefers GET without positional params", func(t *testing.T) {
+ t.Parallel()
+ s := &spec.APISpec{
+ Name: "test",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Resources: map[string]spec.Resource{
+ "items": {
+ Endpoints: map[string]spec.Endpoint{
+ "get": {Method: "GET", Path: "/items/{id}", Description: "Get item",
+ Params: []spec.Param{{Name: "id", Type: "string", Positional: true}}},
+ "list": {Method: "GET", Path: "/items", Description: "List items"},
+ },
+ },
+ },
+ }
+ promoted := buildPromotedCommands(s)
+ require.Len(t, promoted, 1)
+ assert.Equal(t, "list", promoted[0].EndpointName)
+ assert.Equal(t, "/items", promoted[0].Endpoint.Path)
+ })
+
+ t.Run("all built-in names are skipped", func(t *testing.T) {
+ t.Parallel()
+ resources := map[string]spec.Resource{}
+ for name := range builtinCommands {
+ resources[name] = spec.Resource{
+ Endpoints: map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/" + name, Description: "List " + name},
+ },
+ }
+ }
+ s := &spec.APISpec{
+ Name: "test",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Resources: resources,
+ }
+ promoted := buildPromotedCommands(s)
+ assert.Empty(t, promoted)
+ })
+}
+
+func TestGeneratedOutput_PromotedCommandExists(t *testing.T) {
+ t.Parallel()
+
+ apiSpec := &spec.APISpec{
+ Name: "promtest",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Auth: spec.AuthConfig{Type: "api_key", Header: "X-Api-Key", EnvVars: []string{"PROM_API_KEY"}},
+ Config: spec.ConfigSpec{Format: "toml", Path: "~/.config/promtest-pp-cli/config.toml"},
+ Resources: map[string]spec.Resource{
+ "users": {
+ Description: "Manage users",
+ Endpoints: map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/users", Description: "List all users"},
+ },
+ },
+ },
+ }
+
+ outputDir := filepath.Join(t.TempDir(), "promtest-pp-cli")
+ gen := New(apiSpec, outputDir)
+ require.NoError(t, gen.Generate())
+
+ // Promoted command file should exist
+ promotedFile := filepath.Join(outputDir, "internal", "cli", "promoted_users.go")
+ assert.FileExists(t, promotedFile)
+
+ // Read it and verify key content
+ content, err := os.ReadFile(promotedFile)
+ require.NoError(t, err)
+ contentStr := string(content)
+ assert.Contains(t, contentStr, "newUsersPromotedCmd")
+ assert.Contains(t, contentStr, `Use: "users"`)
+ assert.Contains(t, contentStr, `/users`)
+
+ // Root.go should register the promoted command
+ rootContent, err := os.ReadFile(filepath.Join(outputDir, "internal", "cli", "root.go"))
+ require.NoError(t, err)
+ assert.Contains(t, string(rootContent), "newUsersPromotedCmd")
+}
+
+func TestGeneratedOutput_PromotedCommandCompiles(t *testing.T) {
+ t.Parallel()
+
+ apiSpec := &spec.APISpec{
+ Name: "compiletest",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Auth: spec.AuthConfig{Type: "api_key", Header: "X-Api-Key", EnvVars: []string{"CT_API_KEY"}},
+ Config: spec.ConfigSpec{Format: "toml", Path: "~/.config/compiletest-pp-cli/config.toml"},
+ Resources: map[string]spec.Resource{
+ "ISteamUser": {
+ Description: "Steam user interface",
+ Endpoints: map[string]spec.Endpoint{
+ "get_player_summaries": {Method: "GET", Path: "/ISteamUser/GetPlayerSummaries/v2", Description: "Get player summaries",
+ Params: []spec.Param{{Name: "steamids", Type: "string", Description: "Comma-separated Steam IDs"}}},
+ },
+ },
+ "items": {
+ Description: "Manage items",
+ Endpoints: map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/items", Description: "List items"},
+ "create": {Method: "POST", Path: "/items", Description: "Create item"},
+ },
+ },
+ },
+ }
+
+ outputDir := filepath.Join(t.TempDir(), "compiletest-pp-cli")
+ gen := New(apiSpec, outputDir)
+ require.NoError(t, gen.Generate())
+
+ // Both promoted files should exist
+ assert.FileExists(t, filepath.Join(outputDir, "internal", "cli", "promoted_steam-user.go"))
+ assert.FileExists(t, filepath.Join(outputDir, "internal", "cli", "promoted_items.go"))
+
+ // Must compile
+ runGoCommand(t, outputDir, "mod", "tidy")
+ runGoCommand(t, outputDir, "build", "./...")
+}
+
+func TestGeneratedOutput_PromotedCommandNotForBuiltins(t *testing.T) {
+ t.Parallel()
+
+ apiSpec := &spec.APISpec{
+ Name: "builtintest",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Auth: spec.AuthConfig{Type: "api_key", Header: "X-Api-Key", EnvVars: []string{"BT_API_KEY"}},
+ Config: spec.ConfigSpec{Format: "toml", Path: "~/.config/builtintest-pp-cli/config.toml"},
+ Resources: map[string]spec.Resource{
+ "version": {
+ Description: "Version info",
+ Endpoints: map[string]spec.Endpoint{
+ "get": {Method: "GET", Path: "/version", Description: "Get version"},
+ },
+ },
+ "users": {
+ Description: "Manage users",
+ Endpoints: map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/users", Description: "List users"},
+ },
+ },
+ },
+ }
+
+ outputDir := filepath.Join(t.TempDir(), "builtintest-pp-cli")
+ gen := New(apiSpec, outputDir)
+ require.NoError(t, gen.Generate())
+
+ // "version" should NOT have a promoted command (collides with built-in)
+ assert.NoFileExists(t, filepath.Join(outputDir, "internal", "cli", "promoted_version.go"))
+ // "users" should have a promoted command
+ assert.FileExists(t, filepath.Join(outputDir, "internal", "cli", "promoted_users.go"))
+}
+
+func TestGeneratedHelpers_DeadCodeRemoved(t *testing.T) {
+ t.Parallel()
+
+ // Dead code should never appear regardless of spec contents
+ apiSpec := &spec.APISpec{
+ Name: "deadcode",
+ Version: "0.1.0",
+ BaseURL: "https://api.example.com",
+ Auth: spec.AuthConfig{Type: "api_key", Header: "X-Api-Key", EnvVars: []string{"DEAD_API_KEY"}},
+ Config: spec.ConfigSpec{Format: "toml", Path: "~/.config/deadcode-pp-cli/config.toml"},
+ Resources: map[string]spec.Resource{
+ "items": {
+ Description: "Manage items",
+ Endpoints: map[string]spec.Endpoint{
+ "list": {Method: "GET", Path: "/items", Description: "List items"},
+ "delete": {Method: "DELETE", Path: "/items/{id}", Description: "Delete item"},
+ },
+ },
+ },
+ }
+
+ outputDir := filepath.Join(t.TempDir(), "deadcode-pp-cli")
+ gen := New(apiSpec, outputDir)
+ require.NoError(t, gen.Generate())
+
+ helpersGo, err := os.ReadFile(filepath.Join(outputDir, "internal", "cli", "helpers.go"))
+ require.NoError(t, err)
+ content := string(helpersGo)
+
+ assert.NotContains(t, content, "firstNonEmpty", "firstNonEmpty is dead code and should not be emitted")
+ assert.NotContains(t, content, "printOutputFiltered", "printOutputFiltered is dead code and should not be emitted")
+ assert.NotContains(t, content, "selectFieldsGlobal", "selectFieldsGlobal is dead code and should not be emitted")
+
+ // Verify useful functions are still present
+ assert.Contains(t, content, "printOutputWithFlags")
+ assert.Contains(t, content, "filterFields")
+ assert.Contains(t, content, "classifyAPIError")
+}
diff --git a/internal/generator/templates/command_promoted.go.tmpl b/internal/generator/templates/command_promoted.go.tmpl
new file mode 100644
index 00000000..d92a0473
--- /dev/null
+++ b/internal/generator/templates/command_promoted.go.tmpl
@@ -0,0 +1,105 @@
+// Copyright {{currentYear}} {{.Owner}}. Licensed under Apache-2.0. See LICENSE.
+// Generated by CLI Printing Press (https://github.com/mvanhorn/cli-printing-press). DO NOT EDIT.
+
+package cli
+
+import (
+ "encoding/json"
+ "fmt"
+ "io"
+ "os"
+ "strings"
+
+ "github.com/spf13/cobra"
+)
+
+var _ = strings.ReplaceAll // ensure import
+var _ = fmt.Sprintf // ensure import
+var _ = io.ReadAll // ensure import
+var _ = os.Stdin // ensure import
+var _ json.RawMessage // ensure import
+
+func new{{camel .PromotedName}}PromotedCmd(flags *rootFlags) *cobra.Command {
+{{- range .Endpoint.Params}}
+{{- if not .Positional}}
+ var flag{{camel .Name}} {{goType .Type}}
+{{- end}}
+{{- end}}
+{{- if .Endpoint.Pagination}}
+ var flagAll bool
+{{- end}}
+
+ cmd := &cobra.Command{
+ Use: "{{.PromotedName}}{{positionalArgs .Endpoint}}",
+ Short: "{{oneline .Endpoint.Description}}",
+ Long: "Shortcut for '{{.ResourceName}} {{.EndpointName}}'. {{oneline .Endpoint.Description}}",
+ Example: " {{modulePath}} {{.PromotedName}}",
+ RunE: func(cmd *cobra.Command, args []string) error {
+ c, err := flags.newClient()
+ if err != nil {
+ return err
+ }
+
+ path := "{{.Endpoint.Path}}"
+{{- range $i, $p := .Endpoint.Params}}
+{{- if .Positional}}
+ if len(args) < {{add $i 1}} {
+ return usageErr(fmt.Errorf("{{.Name}} is required\nUsage: %s %s <%s>", cmd.Root().Name(), cmd.CommandPath(), "{{.Name}}"))
+ }
+ path = replacePathParam(path, "{{.Name}}", args[{{$i}}])
+{{- end}}
+{{- end}}
+
+{{- if .Endpoint.Pagination}}
+ data, err := paginatedGet(c, path, map[string]string{
+{{- range .Endpoint.Params}}
+{{- if not .Positional}}
+ "{{.Name}}": fmt.Sprintf("%v", flag{{camel .Name}}),
+{{- end}}
+{{- end}}
+ }, flagAll, "{{.Endpoint.Pagination.CursorParam}}", "{{.Endpoint.Pagination.NextCursorPath}}", "{{.Endpoint.Pagination.HasMoreField}}")
+{{- else}}
+ params := map[string]string{}
+{{- range .Endpoint.Params}}
+{{- if not .Positional}}
+ if flag{{camel .Name}} != {{zeroVal .Type}} {
+ params["{{.Name}}"] = fmt.Sprintf("%v", flag{{camel .Name}})
+ }
+{{- end}}
+{{- end}}
+ data, err := c.Get(path, params)
+{{- end}}
+ if err != nil {
+ return classifyAPIError(err)
+ }
+
+ if wantsHumanTable(cmd.OutOrStdout(), flags) {
+ var items []map[string]any
+ if json.Unmarshal(data, &items) == nil && len(items) > 0 {
+ if err := printAutoTable(cmd.OutOrStdout(), items); err != nil {
+ return err
+ }
+ if len(items) >= 25 {
+ fmt.Fprintf(os.Stderr, "\nShowing %d results. To narrow: add --limit, --json --select, or filter flags.\n", len(items))
+ }
+ return nil
+ }
+ }
+ return printOutputWithFlags(cmd.OutOrStdout(), data, flags)
+ },
+ }
+
+{{- range .Endpoint.Params}}
+{{- if not .Positional}}
+ cmd.Flags().{{cobraFlagFunc .Type}}(&flag{{camel .Name}}, "{{flagName .Name}}", {{defaultVal .}}, "{{oneline .Description}}")
+{{- if .Required}}
+ _ = cmd.MarkFlagRequired("{{flagName .Name}}")
+{{- end}}
+{{- end}}
+{{- end}}
+{{- if .Endpoint.Pagination}}
+ cmd.Flags().BoolVar(&flagAll, "all", false, "Fetch all pages")
+{{- end}}
+
+ return cmd
+}
diff --git a/internal/generator/templates/helpers.go.tmpl b/internal/generator/templates/helpers.go.tmpl
index c10e3a43..4962ef6c 100644
--- a/internal/generator/templates/helpers.go.tmpl
+++ b/internal/generator/templates/helpers.go.tmpl
@@ -137,6 +137,7 @@ func classifyAPIError(err error) error {
}
}
+{{- if .HasDelete}}
// classifyDeleteError treats 404 as success for DELETE (already deleted = idempotent no-op).
func classifyDeleteError(err error) error {
msg := err.Error()
@@ -146,6 +147,7 @@ func classifyDeleteError(err error) error {
}
return classifyAPIError(err)
}
+{{- end}}
func truncate(s string, max int) string {
if len(s) <= max {
@@ -157,15 +159,6 @@ func truncate(s string, max int) string {
return s[:max-3] + "..."
}
-func firstNonEmpty(items ...string) string {
- for _, s := range items {
- if s != "" {
- return s
- }
- }
- return ""
-}
-
func newTabWriter(w io.Writer) *tabwriter.Writer {
return tabwriter.NewWriter(w, 2, 4, 2, ' ', 0)
}
@@ -263,17 +256,6 @@ func paginatedGet(c interface {
return json.RawMessage(result), nil
}
-// selectFields is set from the --select flag; used by printOutputFiltered.
-var selectFieldsGlobal string
-
-// printOutputFiltered applies --select field filtering before rendering.
-func printOutputFiltered(w io.Writer, data json.RawMessage, asJSON bool, selectExpr string) error {
- if selectExpr != "" {
- data = filterFields(data, selectExpr)
- }
- return printOutput(w, data, asJSON)
-}
-
// filterFields keeps only the specified comma-separated fields from JSON objects/arrays.
func filterFields(data json.RawMessage, fields string) json.RawMessage {
wanted := map[string]bool{}
diff --git a/internal/generator/templates/root.go.tmpl b/internal/generator/templates/root.go.tmpl
index 29dfbd67..03d5794b 100644
--- a/internal/generator/templates/root.go.tmpl
+++ b/internal/generator/templates/root.go.tmpl
@@ -120,6 +120,9 @@ func Execute() error {
{{- end}}
{{- range .InsightConstructors}}
rootCmd.AddCommand(new{{.}}Cmd(&flags))
+{{- end}}
+{{- range .PromotedCommands}}
+ rootCmd.AddCommand(new{{camel .PromotedName}}PromotedCmd(&flags))
{{- end}}
rootCmd.AddCommand(newVersionCliCmd())
diff --git a/internal/spec/spec.go b/internal/spec/spec.go
index 104d3366..8877f0a9 100644
--- a/internal/spec/spec.go
+++ b/internal/spec/spec.go
@@ -1,6 +1,7 @@
package spec
import (
+ "encoding/json"
"fmt"
"os"
@@ -8,89 +9,89 @@ import (
)
type APISpec struct {
- Name string `yaml:"name"`
- Description string `yaml:"description"`
- Version string `yaml:"version"`
- BaseURL string `yaml:"base_url"`
- BasePath string `yaml:"base_path,omitempty"`
- Owner string `yaml:"owner,omitempty"` // GitHub owner for import paths and Homebrew tap
- SpecSource string `yaml:"spec_source,omitempty"` // official, community, sniffed, docs — affects generated client defaults
- ClientPattern string `yaml:"client_pattern,omitempty"` // rest (default), proxy-envelope — affects generated HTTP client
- ProxyRoutes map[string]string `yaml:"proxy_routes,omitempty"` // path prefix → service name for proxy-envelope routing
- Auth AuthConfig `yaml:"auth"`
- Config ConfigSpec `yaml:"config"`
- Resources map[string]Resource `yaml:"resources"`
- Types map[string]TypeDef `yaml:"types"`
+ Name string `yaml:"name" json:"name"`
+ Description string `yaml:"description" json:"description"`
+ Version string `yaml:"version" json:"version"`
+ BaseURL string `yaml:"base_url" json:"base_url"`
+ BasePath string `yaml:"base_path,omitempty" json:"base_path,omitempty"`
+ Owner string `yaml:"owner,omitempty" json:"owner,omitempty"` // GitHub owner for import paths and Homebrew tap
+ SpecSource string `yaml:"spec_source,omitempty" json:"spec_source,omitempty"` // official, community, sniffed, docs — affects generated client defaults
+ ClientPattern string `yaml:"client_pattern,omitempty" json:"client_pattern,omitempty"` // rest (default), proxy-envelope — affects generated HTTP client
+ ProxyRoutes map[string]string `yaml:"proxy_routes,omitempty" json:"proxy_routes,omitempty"` // path prefix → service name for proxy-envelope routing
+ Auth AuthConfig `yaml:"auth" json:"auth"`
+ Config ConfigSpec `yaml:"config" json:"config"`
+ Resources map[string]Resource `yaml:"resources" json:"resources"`
+ Types map[string]TypeDef `yaml:"types" json:"types"`
}
type AuthConfig struct {
- Type string `yaml:"type"` // api_key, oauth2, bearer_token, cookie, none
- Header string `yaml:"header"`
- Format string `yaml:"format"`
- EnvVars []string `yaml:"env_vars"`
- Scheme string `yaml:"scheme,omitempty"` // OpenAPI security scheme name
- In string `yaml:"in,omitempty"` // header, query, cookie
- AuthorizationURL string `yaml:"authorization_url,omitempty"`
- TokenURL string `yaml:"token_url,omitempty"`
- Scopes []string `yaml:"scopes,omitempty"`
+ Type string `yaml:"type" json:"type"` // api_key, oauth2, bearer_token, cookie, none
+ Header string `yaml:"header" json:"header"`
+ Format string `yaml:"format" json:"format"`
+ EnvVars []string `yaml:"env_vars" json:"env_vars"`
+ Scheme string `yaml:"scheme,omitempty" json:"scheme,omitempty"` // OpenAPI security scheme name
+ In string `yaml:"in,omitempty" json:"in,omitempty"` // header, query, cookie
+ AuthorizationURL string `yaml:"authorization_url,omitempty" json:"authorization_url,omitempty"`
+ TokenURL string `yaml:"token_url,omitempty" json:"token_url,omitempty"`
+ Scopes []string `yaml:"scopes,omitempty" json:"scopes,omitempty"`
}
type ConfigSpec struct {
- Format string `yaml:"format"` // toml, yaml
- Path string `yaml:"path"`
+ Format string `yaml:"format" json:"format"` // toml, yaml
+ Path string `yaml:"path" json:"path"`
}
type Resource struct {
- Description string `yaml:"description"`
- Endpoints map[string]Endpoint `yaml:"endpoints"`
- SubResources map[string]Resource `yaml:"sub_resources,omitempty"`
+ Description string `yaml:"description" json:"description"`
+ Endpoints map[string]Endpoint `yaml:"endpoints" json:"endpoints"`
+ SubResources map[string]Resource `yaml:"sub_resources,omitempty" json:"sub_resources,omitempty"`
}
type Endpoint struct {
- Method string `yaml:"method"`
- Path string `yaml:"path"`
- Description string `yaml:"description"`
- Params []Param `yaml:"params"`
- Body []Param `yaml:"body"`
- Response ResponseDef `yaml:"response"`
- Pagination *Pagination `yaml:"pagination"`
- ResponsePath string `yaml:"response_path,omitempty"` // path to extract data array from response (e.g., "data", "results.items")
- Meta map[string]string `yaml:"meta,omitempty"` // per-endpoint metadata (e.g., source_tier, source_count from crowd-sniff)
- Alias string `yaml:"-"` // computed, not from YAML
+ Method string `yaml:"method" json:"method"`
+ Path string `yaml:"path" json:"path"`
+ Description string `yaml:"description" json:"description"`
+ Params []Param `yaml:"params" json:"params"`
+ Body []Param `yaml:"body" json:"body"`
+ Response ResponseDef `yaml:"response" json:"response"`
+ Pagination *Pagination `yaml:"pagination" json:"pagination"`
+ ResponsePath string `yaml:"response_path,omitempty" json:"response_path,omitempty"` // path to extract data array from response (e.g., "data", "results.items")
+ Meta map[string]string `yaml:"meta,omitempty" json:"meta,omitempty"` // per-endpoint metadata (e.g., source_tier, source_count from crowd-sniff)
+ Alias string `yaml:"-" json:"-"` // computed, not from YAML
}
type Param struct {
- Name string `yaml:"name"`
- Type string `yaml:"type"`
- Required bool `yaml:"required"`
- Positional bool `yaml:"positional"`
- Default any `yaml:"default"`
- Description string `yaml:"description"`
- Fields []Param `yaml:"fields"` // for nested objects
- Enum []string `yaml:"enum,omitempty"` // enum constraints for the parameter
- Format string `yaml:"format,omitempty"` // OpenAPI format hints (date-time, email, uri, etc.)
+ Name string `yaml:"name" json:"name"`
+ Type string `yaml:"type" json:"type"`
+ Required bool `yaml:"required" json:"required"`
+ Positional bool `yaml:"positional" json:"positional"`
+ Default any `yaml:"default" json:"default"`
+ Description string `yaml:"description" json:"description"`
+ Fields []Param `yaml:"fields" json:"fields"` // for nested objects
+ Enum []string `yaml:"enum,omitempty" json:"enum,omitempty"` // enum constraints for the parameter
+ Format string `yaml:"format,omitempty" json:"format,omitempty"` // OpenAPI format hints (date-time, email, uri, etc.)
}
type ResponseDef struct {
- Type string `yaml:"type"` // object, array
- Item string `yaml:"item"` // type name
+ Type string `yaml:"type" json:"type"` // object, array
+ Item string `yaml:"item" json:"item"` // type name
}
type Pagination struct {
- Type string `yaml:"type"` // cursor, offset, page_token
- LimitParam string `yaml:"limit_param"` // query param name for page size (limit, maxResults, pageSize)
- CursorParam string `yaml:"cursor_param"` // query param name for cursor (after, pageToken, offset)
- NextCursorPath string `yaml:"next_cursor_path"` // response field with next cursor (nextPageToken, cursor)
- HasMoreField string `yaml:"has_more_field"` // response field indicating more pages (has_more)
+ Type string `yaml:"type" json:"type"` // cursor, offset, page_token
+ LimitParam string `yaml:"limit_param" json:"limit_param"` // query param name for page size (limit, maxResults, pageSize)
+ CursorParam string `yaml:"cursor_param" json:"cursor_param"` // query param name for cursor (after, pageToken, offset)
+ NextCursorPath string `yaml:"next_cursor_path" json:"next_cursor_path"` // response field with next cursor (nextPageToken, cursor)
+ HasMoreField string `yaml:"has_more_field" json:"has_more_field"` // response field indicating more pages (has_more)
}
type TypeDef struct {
- Fields []TypeField `yaml:"fields"`
+ Fields []TypeField `yaml:"fields" json:"fields"`
}
type TypeField struct {
- Name string `yaml:"name"`
- Type string `yaml:"type"`
+ Name string `yaml:"name" json:"name"`
+ Type string `yaml:"type" json:"type"`
}
func Parse(path string) (*APISpec, error) {
@@ -104,8 +105,25 @@ func Parse(path string) (*APISpec, error) {
func ParseBytes(data []byte) (*APISpec, error) {
var s APISpec
- if err := yaml.Unmarshal(data, &s); err != nil {
- return nil, fmt.Errorf("parsing yaml: %w", err)
+ yamlErr := yaml.Unmarshal(data, &s)
+ if yamlErr != nil || len(s.Resources) == 0 {
+ // Try JSON round-trip: yaml → map → json → struct.
+ // This handles YAML style variations (flow arrays, Python-style
+ // quoting, non-standard indentation) that can cause struct mapping
+ // to silently produce empty fields even though the YAML is valid.
+ var raw map[string]any
+ if yaml.Unmarshal(data, &raw) == nil && len(raw) > 0 {
+ if jsonBytes, err := json.Marshal(raw); err == nil {
+ var fallback APISpec
+ if json.Unmarshal(jsonBytes, &fallback) == nil && len(fallback.Resources) > 0 {
+ s = fallback
+ yamlErr = nil
+ }
+ }
+ }
+ }
+ if yamlErr != nil {
+ return nil, fmt.Errorf("parsing yaml: %w", yamlErr)
}
if err := s.Validate(); err != nil {
return nil, fmt.Errorf("validation: %w", err)
diff --git a/internal/spec/spec_test.go b/internal/spec/spec_test.go
index a5d5f38b..bfe226db 100644
--- a/internal/spec/spec_test.go
+++ b/internal/spec/spec_test.go
@@ -198,3 +198,117 @@ resources:
assert.NotContains(t, string(data), "meta")
})
}
+
+// --- Unit 5: YAML Format Safety Net Tests ---
+
+func TestParseBytesYAMLVariations(t *testing.T) {
+ t.Parallel()
+
+ t.Run("Python-style YAML with 2-space indent and flow arrays", func(t *testing.T) {
+ t.Parallel()
+ // Python's yaml.dump produces this style with flow sequences
+ input := `name: steamapi
+description: "Steam Web API"
+version: '0.1.0'
+base_url: "https://api.steampowered.com"
+auth:
+ type: api_key
+ header: key
+ in: query
+ env_vars: [STEAM_API_KEY]
+config:
+ format: toml
+ path: "~/.config/steamapi-pp-cli/config.toml"
+resources:
+ users:
+ description: "Manage users"
+ endpoints:
+ list:
+ method: GET
+ path: /users
+ description: "List users"
+`
+ s, err := ParseBytes([]byte(input))
+ require.NoError(t, err)
+ assert.Equal(t, "steamapi", s.Name)
+ assert.Equal(t, "Steam Web API", s.Description)
+ assert.Len(t, s.Resources, 1)
+ assert.Contains(t, s.Resources, "users")
+ assert.Equal(t, "api_key", s.Auth.Type)
+ assert.Equal(t, []string{"STEAM_API_KEY"}, s.Auth.EnvVars)
+ })
+
+ t.Run("Go-style YAML still works (no regression)", func(t *testing.T) {
+ t.Parallel()
+ input := `name: testapi
+base_url: https://api.example.com
+auth:
+ type: api_key
+ header: X-Api-Key
+ env_vars:
+ - TEST_API_KEY
+config:
+ format: toml
+ path: ~/.config/testapi-pp-cli/config.toml
+resources:
+ items:
+ description: Manage items
+ endpoints:
+ list:
+ method: GET
+ path: /items
+ description: List items
+`
+ s, err := ParseBytes([]byte(input))
+ require.NoError(t, err)
+ assert.Equal(t, "testapi", s.Name)
+ assert.Len(t, s.Resources, 1)
+ assert.Equal(t, "api_key", s.Auth.Type)
+ })
+
+ t.Run("YAML with quoted string values", func(t *testing.T) {
+ t.Parallel()
+ input := `"name": "quotedapi"
+"base_url": "https://api.example.com"
+"auth":
+ "type": "bearer_token"
+ "header": "Authorization"
+ "format": "Bearer {token}"
+ "env_vars":
+ - "QUOTED_TOKEN"
+"config":
+ "format": "toml"
+ "path": "~/.config/quotedapi-pp-cli/config.toml"
+"resources":
+ "things":
+ "description": "Manage things"
+ "endpoints":
+ "list":
+ "method": "GET"
+ "path": "/things"
+ "description": "List things"
+`
+ s, err := ParseBytes([]byte(input))
+ require.NoError(t, err)
+ assert.Equal(t, "quotedapi", s.Name)
+ assert.Len(t, s.Resources, 1)
+ assert.Equal(t, "bearer_token", s.Auth.Type)
+ })
+
+ t.Run("invalid YAML still returns error", func(t *testing.T) {
+ t.Parallel()
+ input := `{{{not valid yaml at all`
+ _, err := ParseBytes([]byte(input))
+ require.Error(t, err)
+ })
+
+ t.Run("valid YAML but missing required fields still fails validation", func(t *testing.T) {
+ t.Parallel()
+ input := `name: incomplete
+description: Missing base_url and resources
+`
+ _, err := ParseBytes([]byte(input))
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "base_url is required")
+ })
+}
diff --git a/internal/websniff/specgen.go b/internal/websniff/specgen.go
index 441cbb84..7bd44291 100644
--- a/internal/websniff/specgen.go
+++ b/internal/websniff/specgen.go
@@ -395,11 +395,10 @@ func deriveResourceKey(path string) (string, string) {
return "", ""
}
- if len(segments) > 3 {
- segments = segments[:3]
- }
-
- return strings.Join(segments, "/"), segments[len(segments)-1]
+ // Use only the first significant segment as the resource key.
+ // This prevents slashes in resource names which break the generator's
+ // filepath.Join and Cobra Use field.
+ return segments[0], segments[len(segments)-1]
}
func significantSegments(path string) []string {
← 64ded0ae fix(skills): improve retro skill prioritization and skip/do
·
back to Cli Printing Press
·
fix(skills): merge codex detection into setup contract 43d21f22 →