← back to Cli Printing Press
feat(cli): add crowd-sniff command for community-based API discovery (#67)
4a9843dff245d8427c4a28c5da38db5ea80bfc9d · 2026-03-30 18:03:05 -0700 · Trevin Chow
Files touched
A .claude/skills/printing-press-retro/SKILL.mdM .gitignoreA docs/brainstorms/2026-03-29-crowd-sniff-requirements.mdA docs/plans/2026-03-29-003-feat-crowd-sniff-plan.mdA docs/plans/2026-03-30-003-feat-crowd-sniff-param-discovery-plan.mdA docs/retros/2026-03-30-postman-explore-retro.mdA docs/solutions/best-practices/multi-source-api-discovery-design-2026-03-30.mdA docs/solutions/best-practices/sniff-and-crowd-sniff-complementary-discovery-2026-03-30.mdM go.modM go.sumA internal/cli/crowd_sniff.goA internal/cli/crowd_sniff_test.goM internal/cli/root.goA internal/crowdsniff/aggregate.goA internal/crowdsniff/aggregate_test.goA internal/crowdsniff/github.goA internal/crowdsniff/github_test.goA internal/crowdsniff/npm.goA internal/crowdsniff/npm_test.goA internal/crowdsniff/params.goA internal/crowdsniff/params_test.goA internal/crowdsniff/patterns.goA internal/crowdsniff/patterns_test.goA internal/crowdsniff/specgen.goA internal/crowdsniff/specgen_test.goA internal/crowdsniff/types.goM internal/spec/spec.goM internal/spec/spec_test.goM lefthook.ymlM skills/printing-press/SKILL.mdA testdata/crowdsniff/github-code-search-response.jsonA testdata/crowdsniff/github-repo-response.jsonA testdata/crowdsniff/sample-sdk.js
Diff
commit 4a9843dff245d8427c4a28c5da38db5ea80bfc9d
Author: Trevin Chow <trevin@trevinchow.com>
Date: Mon Mar 30 18:03:05 2026 -0700
feat(cli): add crowd-sniff command for community-based API discovery (#67)
---
.claude/skills/printing-press-retro/SKILL.md | 487 +++++++++++
.gitignore | 1 +
.../2026-03-29-crowd-sniff-requirements.md | 127 +++
docs/plans/2026-03-29-003-feat-crowd-sniff-plan.md | 456 ++++++++++
...30-003-feat-crowd-sniff-param-discovery-plan.md | 352 ++++++++
docs/retros/2026-03-30-postman-explore-retro.md | 202 +++++
...multi-source-api-discovery-design-2026-03-30.md | 167 ++++
...owd-sniff-complementary-discovery-2026-03-30.md | 132 +++
go.mod | 1 +
go.sum | 2 +
internal/cli/crowd_sniff.go | 204 +++++
internal/cli/crowd_sniff_test.go | 439 ++++++++++
internal/cli/root.go | 1 +
internal/crowdsniff/aggregate.go | 203 +++++
internal/crowdsniff/aggregate_test.go | 323 +++++++
internal/crowdsniff/github.go | 411 +++++++++
internal/crowdsniff/github_test.go | 407 +++++++++
internal/crowdsniff/npm.go | 464 ++++++++++
internal/crowdsniff/npm_test.go | 951 +++++++++++++++++++++
internal/crowdsniff/params.go | 658 ++++++++++++++
internal/crowdsniff/params_test.go | 493 +++++++++++
internal/crowdsniff/patterns.go | 229 +++++
internal/crowdsniff/patterns_test.go | 264 ++++++
internal/crowdsniff/specgen.go | 187 ++++
internal/crowdsniff/specgen_test.go | 302 +++++++
internal/crowdsniff/types.go | 41 +
internal/spec/spec.go | 19 +-
internal/spec/spec_test.go | 72 ++
lefthook.yml | 4 +
skills/printing-press/SKILL.md | 120 +++
.../crowdsniff/github-code-search-response.json | 74 ++
testdata/crowdsniff/github-repo-response.json | 4 +
testdata/crowdsniff/sample-sdk.js | 41 +
33 files changed, 7829 insertions(+), 9 deletions(-)
diff --git a/.claude/skills/printing-press-retro/SKILL.md b/.claude/skills/printing-press-retro/SKILL.md
new file mode 100644
index 00000000..344f0825
--- /dev/null
+++ b/.claude/skills/printing-press-retro/SKILL.md
@@ -0,0 +1,487 @@
+---
+name: printing-press-retro
+description: >
+ Run a retrospective after generating a CLI with the printing press. Identifies
+ systemic improvements to the machine — generator templates, Go binary, skill
+ instructions, catalog — not patches to the specific CLI. Covers bugs, but also
+ recurring friction (like dead code), features that had to be built manually,
+ and optimizations discovered during the session. Use after any /printing-press
+ run. Trigger phrases: "retro", "retrospective", "what went wrong", "improve
+ the press", "post-mortem", "lessons learned", "what can we improve".
+allowed-tools:
+ - Bash
+ - Read
+ - Glob
+ - Grep
+ - Write
+ - Agent
+ - AskUserQuestion
+---
+
+# /printing-press-retro
+
+Analyze a printing press session to find ways to make the *machine* better. Not
+fixes to the CLI that was just printed — improvements to the generator, binary,
+skill, and catalog so the *next* CLI comes out stronger with less manual effort.
+
+This goes beyond bugs. The most valuable findings are often the work that *succeeded
+but shouldn't have been necessary* — features you built by hand that the generator
+should have emitted, friction that recurs on every generation, and optimizations you
+discovered that should become defaults.
+
+## When to run
+
+Run in the same conversation where the CLI was generated (post-shipcheck). The retro
+needs the full conversation history — every error, retry, manual edit, and discovery.
+
+If running in a fresh conversation, point it at the manuscripts directory, but know
+you'll miss the in-conversation debugging context.
+
+## Setup
+
+```bash
+PRESS_HOME="$HOME/printing-press"
+PRESS_MANUSCRIPTS="$PRESS_HOME/manuscripts"
+
+# Find the most recent run across all APIs
+LATEST_RUN=$(find "$PRESS_MANUSCRIPTS" -name "*shipcheck*" -type f -exec stat -f "%m %N" {} \; | sort -rn | head -1 | awk '{print $2}')
+
+if [ -z "$LATEST_RUN" ]; then
+ echo "No shipcheck proofs found. Run /printing-press first."
+ exit 1
+fi
+
+API_NAME=$(echo "$LATEST_RUN" | sed "s|$PRESS_MANUSCRIPTS/||" | cut -d'/' -f1)
+RUN_ID=$(echo "$LATEST_RUN" | sed "s|$PRESS_MANUSCRIPTS/||" | cut -d'/' -f2)
+RUN_DIR="$PRESS_MANUSCRIPTS/$API_NAME/$RUN_ID"
+
+echo "Retro for: $API_NAME (run $RUN_ID)"
+echo "Manuscripts: $RUN_DIR"
+```
+
+If the user passed an API name as an argument, use that instead of auto-detecting.
+
+## Phase 1: Gather evidence
+
+Read all artifacts from the run:
+
+1. **Research brief** — `$RUN_DIR/research/*brief*`
+2. **Absorb manifest** — `$RUN_DIR/research/*absorb*`
+3. **Shipcheck proof** — `$RUN_DIR/proofs/*shipcheck*`
+4. **Build log** — `$RUN_DIR/proofs/*build-log*` (if exists)
+5. **Live smoke log** — `$RUN_DIR/proofs/*live-smoke*` (if exists)
+6. **The generated CLI** — `$PRESS_HOME/library/<api>-pp-cli/`
+
+Also gather the scorecard, verify pass rate, and dogfood report (from the shipcheck
+proof or by re-running the tools).
+
+## Phase 2: Mine the session
+
+Scan the full conversation history for five categories of signal. Every finding
+becomes a row in Phase 3 — don't filter yet, just collect.
+
+### 2a. Errors and retries
+
+Any time a command failed and was re-run, a build broke, or the generator produced
+code that didn't compile. What broke, what fixed it, and how long did it take?
+
+### 2b. Manual code edits
+
+Every hand-edit to generated code is a signal. Each one means the generator *should
+have* gotten it right but didn't. These are the highest-value findings because they
+point directly at template gaps.
+
+Examples from real sessions:
+- Rewriting the root command `Short:` description from API-speak to user-speak
+- Adding top-level commands to wrap deeply-nested generated commands
+- Fixing `serviceForPath` routing for proxy-envelope APIs
+- Rewriting the sync command for offset-based pagination
+- Adding entity-specific store tables the generator didn't create
+
+### 2c. Features built from scratch
+
+Features in the absorb manifest or transcendence list that had to be written entirely
+by hand during Phase 3. The generator produced no scaffolding for them. Ask: is this
+a feature class the generator could reasonably emit, or is it genuinely custom?
+
+For example: if every CLI needs a `trending` command that queries local SQLite, maybe
+the generator should emit a trending template when it detects time-series metrics in
+the spec.
+
+### 2d. Recurring friction
+
+Work that happens on *every* generation, not just this one. The key question for each:
+**is this inherent to the approach, or can the machine eliminate it?**
+
+Examples:
+- **Dead code** — The generator emits generic helpers (CSV, delete-classify, etc.) and
+ some are never called. Is the fix to stop emitting them (risk: some CLIs need them)?
+ Or to add a post-generation dead-code sweep? Or to make the generator smarter about
+ which helpers each API actually needs?
+- **Default resource mismatch** — `defaultSyncResources()` always returns a placeholder.
+ Could the generator derive the right resources from the spec's entity types?
+- **DB path inconsistency** — Different generated commands use different default paths.
+ Could the generator emit a single `defaultDBPath()` and reference it everywhere?
+
+For each piece of friction, propose at least two possible fixes at different levels
+(generator, binary post-processing, skill instruction) and assess which is most durable.
+
+### 2e. Discovered optimizations
+
+Improvements noticed during the session that weren't fixing a problem — they were
+making something better. These might be UX ideas, performance improvements, new
+command patterns, or output format improvements that emerged from actually using the
+CLI.
+
+Ask: could this optimization be detected automatically and applied by the generator?
+
+## Phase 3: Classify findings
+
+For each finding from Phase 2, answer these questions. Skip findings that only
+affect this specific API and wouldn't recur.
+
+### The Six Questions
+
+**1. What happened?**
+One sentence. Describe the symptom or the work that was done, not the fix.
+
+**2. What category is this?**
+
+| Category | Description | Example |
+|----------|-------------|---------|
+| **Bug** | Generated code is wrong | serviceForPath returns wrong service |
+| **Template gap** | Generator has no template for a common pattern | No top-level command aliases |
+| **Assumption mismatch** | Generator assumes X but API uses Y | Cursor pagination vs offset |
+| **Recurring friction** | Happens every generation, might be inherent | Dead code cleanup |
+| **Missing scaffolding** | Feature class the generator could emit but doesn't | Entity-specific store tables |
+| **Default gap** | Generator emits a wrong or placeholder default | Sync resources list, DB path |
+| **Discovered optimization** | Improvement found during use | Compact number formatting |
+| **Skill instruction gap** | Skill told Claude wrong thing or missed a step | Phase ordering issue |
+| **Tool limitation** | Verify/dogfood/scorecard missed or mis-reported | False positive dead code |
+
+**3. Where in the machine does this originate?**
+
+| Component | Path | Controls |
+|-----------|------|----------|
+| Generator templates | `internal/generator/` | Go code emitted for commands, store, client |
+| Spec parser | `internal/spec/` | Internal YAML spec parsing |
+| OpenAPI parser | `internal/openapi/` | OpenAPI 3.0+ parsing |
+| Catalog | `catalog/` | API entries and metadata |
+| Main skill | `skills/printing-press/SKILL.md` | Orchestration instructions |
+| Verify/dogfood/scorecard | CLI commands | Quality checking tools |
+
+**4. Blast radius and fallback cost — should the machine handle this?**
+
+This is the most important question and the easiest to get wrong. The retro runs
+right after a session with one API, and pattern-matching from a single example is
+unreliable. A finding that felt universal during the Postman Explore session might
+be specific to proxy-envelope APIs, or to sniffed specs, or to APIs with entity-type
+enum params.
+
+**Step A: Cross-API stress test.** Mentally test each finding against at least three
+different API shapes:
+
+- A standard REST API with clean OpenAPI spec (e.g., Stripe, GitHub)
+- A minimal/undocumented API discovered via sniff or HAR
+- An API with a different auth model or response format
+
+For each, ask: "Would this exact problem occur? Would the proposed fix help, be
+irrelevant, or actively hurt?"
+
+**Step B: Estimate frequency.** Based on the stress test, assign a blast radius:
+
+- **Every API** — occurs regardless of API shape. Be skeptical of this label — it
+ must fail the stress test for all three shapes.
+- **Most APIs** — affects common patterns. Name the triggering condition.
+- **API subclass** — affects a specific pattern. Name the subclass precisely:
+ proxy-envelope, GraphQL, sniffed-only, offset-paginated, etc.
+- **This API only** — isolated quirk.
+
+**Step C: Assess fallback cost.** This is what happens if the machine does NOT have
+the fix. For each finding, the fallback is one of:
+
+| Fallback | Cost | Example |
+|----------|------|---------|
+| **Claude rewrites from scratch** | High — 10+ min, error-prone, may forget | Rewriting the entire sync command for offset pagination |
+| **Claude makes targeted edits** | Medium — 2-5 min, usually succeeds | Fixing serviceForPath routing, changing a default path |
+| **Claude deletes/tweaks one thing** | Low — <1 min, mechanical | Removing 3 dead functions, rewriting a Short description |
+| **CLI ships broken** | Critical — user hits the bug at runtime | Infinite sync loop, wrong API responses, empty search |
+
+**Step D: Make the tradeoff.** The decision to add conditional logic to the machine
+is NOT just "is it general enough?" — it's a cost-benefit:
+
+```
+machine fix justified when:
+ (frequency × fallback cost) > (implementation effort + regression risk)
+```
+
+This means:
+- A finding affecting only 20% of APIs (API subclass) can still justify a machine fix
+ if the fallback is "Claude completely rewrites the sync command" (high cost) or
+ "CLI ships with an infinite loop" (critical).
+- A finding affecting 100% of APIs might NOT justify a machine fix if the fallback is
+ "Claude changes one line" (low cost) and the fix is complex to implement.
+- Narrow-scope fixes should include conditional logic (activate when X, skip otherwise)
+ so they don't regress the simple case. But the mere fact that a condition is required
+ is not a reason to skip the fix — it's a reason to scope it carefully.
+
+**The counterpoint to being conservative:** if the machine doesn't cover a wide enough
+breadth of cases, the printed CLIs suffer. Best case is Claude dynamically fixes the
+problem during generation — which is inefficient, error-prone, and might not happen.
+Worst case is the CLI ships with the defect. Every finding left out of the machine is
+a bet that Claude will catch it every time, and Claude won't.
+
+When the finding applies to an API subclass, the recommendation must include:
+- **Condition:** When to activate (e.g., "spec has `x-proxy-routes`")
+- **Guard:** When to skip (e.g., "standard REST APIs without proxy pattern")
+- **Frequency estimate:** How common is this subclass? If it's >20% of APIs the
+ printing press targets, the conditional logic is likely worth the complexity.
+
+**5. Is this inherent or fixable?**
+This question matters most for recurring friction. Some friction is structural — code
+generation will always produce some unused code because templates are generic. But
+"inherent" shouldn't be the default answer. Push hard on whether a smarter generator,
+a post-processing step, or better spec analysis could eliminate the friction.
+
+If inherent: propose the cheapest mitigation (e.g., "dogfood auto-deletes dead helpers
+as a post-generation step").
+
+If fixable: propose the fix at the right level.
+
+**6. What is the durable fix?**
+A concrete change to the machine. Prefer this hierarchy:
+
+1. **Generator template fix** — Code is emitted correctly from the start. Zero manual work.
+2. **Binary post-processing** — A printing-press command that auto-fixes after generation
+ (like a `printing-press polish` that removes dead code and aligns paths).
+3. **Skill instruction** — Tell Claude to do it during generation. Last resort because
+ Claude might forget or get it wrong. Every instruction is a tax on every future run.
+
+Describe what test would verify the fix: "Generate a CLI for an API with offset
+pagination and verify sync terminates after fetching all pages."
+
+## Phase 4: Prioritize
+
+Score each finding using the tradeoff from Question 4:
+
+```
+priority = (frequency × fallback cost) / (implementation effort + regression risk)
+```
+
+Where:
+- **Frequency**: every=4, most=3, subclass=2, this-API=1
+- **Fallback cost**: critical=4, high(rewrite)=3, medium(targeted edit)=2, low(one-liner)=1
+- **Implementation effort**: 1(hours) to 4(weeks)
+- **Regression risk**: 0(conditional/guarded) to 3(blanket change touching all APIs)
+
+Present as a ranked table. Group into tiers:
+- **Tier 1: Do now** — high frequency×fallback, low effort, guarded implementation
+- **Tier 2: Plan** — high frequency×fallback but needs design work or careful guards
+- **Tier 3: Backlog** — low fallback cost (Claude handles it fine dynamically) or
+ inherent friction with cheap mitigations
+- **Skip** — this-API-only findings or cases where the dynamic fix is genuinely easier
+ than the machine fix
+
+## Phase 5: Write the retro
+
+```markdown
+# Printing Press Retro: <API name>
+
+## Session Stats
+- API: <name>
+- Spec source: <catalog/sniffed/docs/HAR>
+- Scorecard: <before> -> <after> (if applicable)
+- Verify pass rate: <X>%
+- Fix loops: <N>
+- Manual code edits: <N>
+- Features built from scratch: <N>
+- Time to ship: ~<X>m
+
+## Findings
+
+### 1. <Title> (<category>)
+- **What happened:** ...
+- **Root cause:** Component + what's specifically wrong
+- **Cross-API check:** Would this occur for [standard REST]? [sniffed API]? [different auth]?
+- **Frequency:** every API / most / subclass:<name> / this API only
+- **Fallback if machine doesn't fix it:** What Claude has to do dynamically (rewrite,
+ edit, one-liner) or what ships broken
+- **Tradeoff:** Is the machine fix worth it given frequency × fallback cost vs
+ implementation effort + regression risk?
+- **Inherent or fixable:** ...
+- **Durable fix:** Concrete machine change. If subclass-scoped, include:
+ - Condition: when to activate
+ - Guard: when to skip
+ - Frequency estimate: how common is this subclass?
+- **Test:** How to verify, including a negative test for APIs that should NOT be affected
+- **Evidence:** Session moment that surfaced this
+
+### 2. ...
+
+## Prioritized Improvements
+
+### Tier 1: Do Now
+| # | Fix | Component | Frequency | Fallback Cost | Effort | Guards |
+|---|-----|-----------|-----------|--------------|--------|--------|
+
+### Tier 2: Plan
+| # | Fix | Component | Frequency | Fallback Cost | Effort | Guards |
+|---|-----|-----------|-----------|--------------|--------|--------|
+
+### Tier 3: Backlog
+| # | Fix | Component | Frequency | Fallback Cost | Effort | Guards |
+|---|-----|-----------|-----------|--------------|--------|--------|
+
+## Work Units
+
+### WU-1: <Title> (findings #N, #M)
+- **Goal:** ...
+- **Target files:** actual paths from Glob/Grep
+- **Acceptance criteria:**
+ - positive test: ...
+ - negative test: ...
+- **Scope boundary:** ...
+- **Effort:** ...
+
+### WU-2: ...
+
+## Anti-patterns
+
+Patterns that looked right but led to problems. These should become warnings in
+AGENTS.md or the relevant skill:
+- ...
+
+## What the Machine Got Right
+
+Patterns to preserve and extend — things that worked well and should not be
+accidentally degraded by future changes:
+- ...
+```
+
+## Phase 5.5: Plannable work units
+
+The retro's findings are analytical. To bridge to implementation planning (e.g.,
+via `/compound-engineering:ce-plan`), group related findings into coherent work units that a planner
+could pick up directly.
+
+For each tier 1 or tier 2 group, produce a work unit block:
+
+```markdown
+## Work Units
+
+### WU-1: <Title> (from findings #N, #M, ...)
+- **Goal:** One sentence describing the outcome
+- **Target files:** Specific file paths in the printing-press repo to modify
+ (use Glob/Grep to resolve component names to actual files)
+- **Acceptance criteria:** 2-3 concrete, testable scenarios:
+ - "Generate from postman-explore spec → sync terminates without manual fix"
+ - "Generate from Stripe spec → sync still uses cursor-based pagination (negative test)"
+- **Scope boundary:** What this does NOT include
+- **Dependencies:** Other work units that must complete first (if any)
+- **Estimated effort:** hours/days
+```
+
+To resolve target files, actually look at the printing-press repo:
+
+```bash
+# Find generator template files
+find <repo>/internal/generator -name "*.go" -o -name "*.tmpl" | head -20
+
+# Find where sync code is generated
+grep -rl "syncResource\|defaultSyncResources\|determinePaginationDefaults" <repo>/internal/
+```
+
+Group related findings into work units when they touch the same files or when
+one fix enables another. For example:
+- Response envelope unwrapping + pagination detection + sync resource derivation
+ → "WU: Data layer generation pipeline"
+- Entity-specific store tables + FTS index generation + typed columns
+ → "WU: Schema-driven store generation"
+
+A good work unit is something one person could implement in 1-3 days with a clear
+definition of done. If a work unit is bigger than that, split it.
+
+## Phase 6: Save and present
+
+### Save locations
+
+Save the retro to two places:
+
+1. **Manuscripts** (ephemeral, tied to the run):
+ ```
+ $PRESS_MANUSCRIPTS/<api>/<run-id>/proofs/<stamp>-retro-<api>-pp-cli.md
+ ```
+
+2. **Repo** (durable, checkable-into-git, readable by future sessions):
+ ```bash
+ REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || echo "$PWD")"
+ RETRO_DIR="$REPO_ROOT/docs/retros"
+ mkdir -p "$RETRO_DIR"
+
+ # Filename: YYYY-MM-DD-<api>-retro.md
+ RETRO_FILE="$RETRO_DIR/$(date +%Y-%m-%d)-<api>-retro.md"
+ ```
+
+ The repo copy is the canonical one. It accumulates over time so future retros
+ can reference past findings ("this was also flagged in the notion retro on 2026-03-15").
+
+### Present summary
+
+Show the user: top 3 findings, the tier 1 table, and the work units.
+
+### Offer to plan
+
+Check whether `/compound-engineering:ce-plan` is available (the compound-engineering plugin is in
+`.claude/settings.json` as a dependency, so it should be — but might not be for
+standalone installs). Use `AskUserQuestion` to offer next steps:
+
+> "Retro saved to `docs/retros/<date>-<api>-retro.md`. Found <N> findings across
+> <M> work units. Want to plan implementation?"
+>
+> 1. **Plan Tier 1 work units** — invoke `/compound-engineering:ce-plan` with the retro's Tier 1 work
+> units as input
+> 2. **Plan a specific work unit** — pick one WU to plan
+> 3. **Done for now** — retro is saved, plan later
+
+If the user picks option 1 or 2, invoke the `compound-engineering:ce:plan` skill
+(if that name doesn't resolve, try `compound-engineering:ce-plan` as a fallback)
+with a prompt like:
+
+```
+Create a plan to improve the printing-press CLI generation system in this repo.
+We just generated a CLI for <API> and encountered systemic problems and
+opportunities documented in the retro. The retro includes prioritized work units
+with target files, acceptance criteria, and scope boundaries:
+docs/retros/<date>-<api>-retro.md
+[If option 2: Focus on work unit WU-<N>: <title>.]
+```
+
+If neither skill name resolves, fall back to:
+- Tell the user the retro is saved and they can invoke it manually
+- Print the prompt they'd use:
+ `/compound-engineering:ce-plan Create a plan to improve the printing-press system given the retro at docs/retros/<file>`
+
+## Rules
+
+- The retro is about the machine, not the CLI. Do not propose fixes to the generated
+ CLI.
+- Do not add more phases, documents, or gates to the main skill. It's already long.
+ Propose making existing phases smarter or the generator emit better defaults.
+- Prefer automatic fixes (generator, binary) over instructional fixes (skill).
+- For recurring friction, always answer "inherent or fixable?" honestly. Don't
+ dismiss friction as inherent without considering alternatives.
+- Be honest about what went well. Protecting good patterns is as important as
+ fixing bad ones.
+- **Resist over-generalization.** You just spent an entire session with one API. Your
+ intuition about "every API needs this" is based on a sample size of one. Stress-test
+ every finding against other API shapes before claiming broad blast radius. A fix
+ that helps proxy-envelope APIs but adds unnecessary complexity to a clean REST API
+ is a regression, not an improvement. When in doubt, scope the fix narrowly with
+ conditional logic and let future retros widen it if the pattern recurs.
+- When a fix only applies to a subclass of APIs, the recommendation must include the
+ condition (when to activate) AND the guard (when to skip). A generator change
+ without a guard is a blanket change, and blanket changes break simple cases.
+- Be thorough. The retro document is a reference for future planning — include
+ enough detail that someone reading it months later can understand the finding,
+ the tradeoff reasoning, and the proposed fix without needing the original
+ conversation.
diff --git a/.gitignore b/.gitignore
index 188490ef..1a07ac56 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,6 @@
.DS_Store
.cache/
+.claude/worktrees/
/printing-press
library/
dist/
diff --git a/docs/brainstorms/2026-03-29-crowd-sniff-requirements.md b/docs/brainstorms/2026-03-29-crowd-sniff-requirements.md
new file mode 100644
index 00000000..cbd021fd
--- /dev/null
+++ b/docs/brainstorms/2026-03-29-crowd-sniff-requirements.md
@@ -0,0 +1,127 @@
+---
+date: 2026-03-29
+topic: crowd-sniff
+---
+
+# Crowd Sniff: API Discovery from Community Signals
+
+## Problem Frame
+
+Printing-press generates CLIs from API specs, but many public APIs lack published specs. The existing `sniff` command solves this by observing live web traffic -- but it only captures what one browsing session happens to hit. It can't tell you which endpoints developers actually use, which params matter, or how popular each operation is.
+
+Meanwhile, thousands of developers have already mapped these APIs in npm packages, GitHub code, and Postman collections. This crowd knowledge is structured, tested, and popularity-weighted -- but nobody systematically extracts it for CLI generation.
+
+Crowd sniff mines these community signals to either discover API endpoints (when no spec exists) or enrich existing specs with real-world usage data (which endpoints matter most, which params people actually pass, which auth patterns work).
+
+## How It Fits
+
+```
+Phase 1 Research
+ |
+ v
+Phase 1.7: Sniff Gate (existing)
+ |
+ v
+Phase 1.8: Crowd Sniff Gate (NEW)
+ |
+ v
+ +----+----+
+ | |
+ Has spec No spec
+ | |
+ v v
+Enrichment Primary Discovery
+mode mode
+ | |
+ v v
+crowd-sniff-spec.yaml
+ |
+ v
+Phase 2: Generate
+(--spec original --spec crowd)
+```
+
+Crowd sniff is a **new CLI command** (`printing-press crowd-sniff`) that outputs spec YAML -- same integration surface as `sniff`. The skill invokes it during a new Phase 1.8 gate when appropriate.
+
+## Requirements
+
+**Sources & Search**
+
+- R1. Search npm registry for SDK packages matching the target API (by name, keywords, scope). For each candidate package (capped at 10), download the tarball from the registry, decompress to a temp directory, grep source for endpoint patterns (URL paths, HTTP method calls, TypeScript interfaces), extract findings, and clean up. Skip tarballs over 10MB. No AST parsing in v1.
+- R2. Search GitHub code for usage patterns (`fetch("https://api.example.com/`, `requests.get(`, client library method calls). Aggregate across repos to extract endpoint URLs, common query parameters, and per-endpoint frequency counts (number of distinct repos using that endpoint). Use GitHub's code search API. GitHub code search requires authentication (`GITHUB_TOKEN` or `gh` CLI) and has a rate limit of 10 requests/minute -- use minimum 6-second intervals between requests.
+- R3. Search Postman's public API network for collections matching the target API. Parse collection JSON to extract endpoints, request/response examples, auth configurations, and environment variables. Programmatic access to Postman Explore is an open question -- a printing-press CLI for the Postman Explore site is in progress, which will clarify the API surface.
+- R4. Each source is optional and independently useful. When a source is unavailable (no auth token, API unreachable, zero results), it silently returns zero endpoints and the other sources still run.
+
+**Recency**
+
+- R5. Strict 6-month recency cutoff on all sources. npm packages must have been published/updated within 6 months. GitHub code must come from repos pushed within 6 months. Postman collections must have been updated within 6 months. Sources outside this window are excluded entirely.
+- R6. Recency is a binary filter only (in or out). It does not affect confidence scoring within the window.
+
+**Confidence**
+
+- R7. Each discovered endpoint carries two metadata fields:
+ - `source_tier`: one of `official-sdk` (published by the API vendor), `community-sdk` (third-party npm package), `code-search` (GitHub code), or `postman`.
+ - `source_count`: how many independent sources found this endpoint.
+- R8. Source tier reflects authority: official SDK > community SDK > code search > Postman. Cross-source agreement (higher `source_count`) is an additional quality signal. The command outputs these fields; the skill/generator decides how to use them.
+- R9. Confidence metadata is stored in a `Meta map[string]string` field on `spec.Endpoint` (with `yaml:"meta,omitempty"`). Crowd sniff sets `source_tier` and `source_count` as map entries. Existing specs, the generator, and `mergeSpecs` are unaffected because the field is omitempty and templates don't reference it.
+
+**Input & Output**
+
+- R10. Output is a valid printing-press spec YAML file, same format as `sniff` output. Can be passed to `printing-press generate --spec <path>` or merged with other specs via multi-`--spec`.
+- R11. The command is `printing-press crowd-sniff --api <name-or-url>`. Accepts an API name ("notion", "stripe") or a base URL. Outputs spec YAML to a default cache path or `--output <path>`.
+- R12. Report summary on completion: how many endpoints discovered, from which sources, source tier distribution. Similar to sniff's "N endpoints across M resources" output.
+- R13. When `--api` is ambiguous (e.g., "cal" could match cal.com or Google Calendar), display candidates with context (npm download counts, GitHub stars) and prompt the user to select. Do not silently pick one.
+- R14. Base URL resolution: prefer base URL from official SDK configuration/constants, fall back to most frequently observed domain prefix from GitHub code search results, fail with error asking user to provide `--base-url` if no URL can be inferred.
+
+**Skill Integration**
+
+- R15. The printing-press skill adds a Phase 1.8 "Crowd Sniff Gate" after the existing Phase 1.7 Sniff Gate. Decision matrix mirrors sniff's: offer crowd sniff when spec has gaps or no spec exists. Skip when spec appears complete.
+- R16. Crowd sniff can run independently of sniff. They are complementary -- sniff discovers from live traffic, crowd sniff discovers from community usage. Both output spec YAML that merges via `--spec`.
+- R17. For npm SDK discovery, prefer official SDKs (scope matches API vendor, e.g., `@notionhq/client`) over community packages. Use npm registry search API with keyword and scope filters.
+
+## Success Criteria
+
+- The source tier correctly ranks official SDK endpoints above code-search-only endpoints in the output metadata.
+- End-to-end: `printing-press crowd-sniff --api notion` produces a spec YAML that `printing-press generate` can consume without errors.
+- Recency filtering successfully excludes deprecated endpoints from abandoned packages/collections.
+- Manual validation against 3 popular APIs (e.g., Notion, Discord, Stripe) shows crowd sniff discovers a meaningful subset of known endpoints with high accuracy (discovered endpoints actually exist in the current API).
+
+## Scope Boundaries
+
+- **No AST parsing in v1.** Heuristic grep patterns only. AST parsing for JS/TS is a potential v2 enhancement.
+- **No PyPI/RubyGems in v1.** npm only for SDK analysis. Other registries are a natural extension but not initial scope.
+- **No automatic threshold decisions.** The command outputs all discovered endpoints with confidence metadata. The skill decides the threshold, not the command.
+- **No live API probing.** Crowd sniff is passive -- it reads what others have published. Active endpoint probing (hitting the API to check 200 vs 404) is a separate concern.
+- **GraphQL introspection is out of scope.** Valuable but mechanically different from crowd signal mining. Could be a separate command.
+- **Postman access is an open question.** A printing-press CLI for the Postman Explore site is in progress separately. Its outcome will determine how crowd sniff integrates with Postman. If no programmatic access materializes, Postman is dropped from v1.
+
+## Key Decisions
+
+- **Standalone CLI command** over skill-only integration: Enables manual invocation, testability, and clean separation from orchestration logic. Same pattern as `sniff`.
+- **Strict 6-month cutoff** over adaptive scoring: Simpler, avoids false confidence from stale sources. We'd rather miss a deprecated endpoint than include one.
+- **source_tier + source_count** over weighted numerical scoring: Achieves ranking without specifying a formula nobody consumes in v1. The skill/generator can interpret tiers however it wants. Weighted scoring is a v2 upgrade if needed.
+- **Meta map on Endpoint** over sidecar file: `Meta map[string]string` with `yaml:"meta,omitempty"` on `spec.Endpoint`. Zero impact on existing specs (omitempty), generator (templates don't reference it), and mergeSpecs (copies the struct wholesale). Same pattern as existing optional fields like `ResponsePath`.
+- **Heuristic grep** over AST parsing: Language-agnostic, fast, proven by OSC's convention analysis patterns. Good enough for v1; AST parsing is a v2 upgrade path.
+- **Phase 1.8 (after sniff)** rather than replacing sniff: They discover different things. Sniff finds what the web app does; crowd sniff finds what developers need. Complementary, not competing.
+
+## Dependencies / Assumptions
+
+- GitHub code search API requires authentication (`GITHUB_TOKEN` env var or `gh` CLI auth). Rate limit: 10 requests/minute. When no token is available, the GitHub source silently returns zero results (per R4).
+- npm registry search API is public and unauthenticated.
+- The existing `mergeSpecs()` in `root.go` handles combining crowd-sniff output with other spec sources. Note: `mergeSpecs` does not currently preserve per-endpoint metadata -- this may need updating if confidence fields are embedded in the spec struct.
+
+## Outstanding Questions
+
+### Deferred to Planning
+
+- [Affects R2][Needs research] GitHub code search API has known limitations (only indexed repos, max 1000 results per query). What's the practical endpoint discovery ceiling for a mid-popularity API?
+- [Affects R3][Blocked on other work] Postman Explore API access depends on the in-progress printing-press CLI for Postman Explore. That work will reveal the API surface. Crowd sniff's Postman source can be built once that CLI ships.
+- [Affects R1][Technical] What grep patterns reliably extract endpoint URLs from SDK source across different coding styles? Prototype against 3-5 real SDKs (Notion, Linear, Discord, Stripe, Twilio).
+- [Affects R1][Technical] npm tarball workflow: search registry, download tarballs, decompress, grep, clean up. Prototype the end-to-end path in Go including temp directory management.
+- [Affects R14][Technical] Base URL extraction from SDK source -- what patterns identify the base URL constant in common SDK styles?
+- [Affects R7][Needs research] What npm download count threshold reliably separates maintained community SDKs from abandoned ones? Proposed: 100/week. Validate against real data for mid-tier APIs.
+- [Affects performance] Expected wall-clock time given GitHub rate limits (~6s between requests). Should sources be queried in parallel (npm + GitHub concurrently)?
+
+## Next Steps
+
+-> `/ce:plan` for structured implementation planning. No blocking questions remain.
diff --git a/docs/plans/2026-03-29-003-feat-crowd-sniff-plan.md b/docs/plans/2026-03-29-003-feat-crowd-sniff-plan.md
new file mode 100644
index 00000000..c6a33e71
--- /dev/null
+++ b/docs/plans/2026-03-29-003-feat-crowd-sniff-plan.md
@@ -0,0 +1,456 @@
+---
+title: "feat: Add crowd-sniff command for API discovery from community signals"
+type: feat
+status: active
+date: 2026-03-29
+origin: docs/brainstorms/2026-03-29-crowd-sniff-requirements.md
+deepened: 2026-03-29
+---
+
+# feat: Add crowd-sniff command for API discovery from community signals
+
+## Overview
+
+Add a `printing-press crowd-sniff --api <name>` command that discovers API endpoints from npm SDKs and GitHub code search, producing spec YAML compatible with the existing `generate` pipeline. This complements the existing `sniff` command (which discovers from live web traffic) by mining what developers have already mapped in published packages and code.
+
+## Problem Frame
+
+Many public APIs lack published specs. `sniff` captures what one browsing session hits; crowd-sniff captures what thousands of developers actually use. The two are complementary -- sniff finds what the web app does, crowd-sniff finds what developers need. (see origin: `docs/brainstorms/2026-03-29-crowd-sniff-requirements.md`)
+
+## Requirements Trace
+
+- R1. npm SDK search, tarball download, heuristic grep for endpoints
+- R2. GitHub code search for API usage patterns with frequency aggregation
+- R3. Postman source (deferred -- blocked on in-progress Postman Explore CLI)
+- R4. Graceful degradation: each source is optional and independent
+- R5. Strict 6-month recency cutoff on all sources
+- R6. Recency is binary filter only (in or out), no confidence weighting within window
+- R7/R8. `source_tier` and `source_count` metadata per endpoint
+- R9. `Meta map[string]string` field on `spec.Endpoint` with `yaml:"meta,omitempty"`
+- R10. Output is valid spec YAML consumable by `printing-press generate`
+- R11. `printing-press crowd-sniff --api <name-or-url>` with `--output` and `--base-url` flags
+- R12. Report summary on completion: endpoint count, source breakdown, tier distribution
+- R13. Disambiguation prompt when API name is ambiguous
+- R14. Base URL resolution cascade: `--base-url` flag > SDK constants > GitHub code frequency
+- R15. Phase 1.8 Crowd Sniff Gate in the skill
+- R16. Crowd sniff runs independently of sniff (complementary, not dependent)
+- R17. Prefer official SDKs (vendor-scoped npm packages) over community
+
+## Scope Boundaries
+
+- No AST parsing -- heuristic grep only (see origin)
+- No PyPI/RubyGems -- npm only for v1 (see origin)
+- No Postman in v1 -- blocked on separate work (see origin)
+- No live API probing (see origin)
+- No automatic threshold decisions -- output all endpoints with metadata (see origin)
+- No GraphQL introspection (see origin)
+
+## Context & Research
+
+### Relevant Code and Patterns
+
+| Purpose | File |
+|---------|------|
+| Command pattern to mirror | `internal/cli/sniff.go` -- `newSniffCmd()` returning `*cobra.Command` |
+| Spec struct (add Meta) | `internal/spec/spec.go:46` -- `Endpoint` struct |
+| Reusable spec writer | `internal/websniff/specgen.go` -- `WriteSpec()`, `DefaultCachePath()` |
+| HTTP client + GitHub API | `internal/pipeline/research.go` -- `newGitHubRequest()`, error handling patterns |
+| Spec caching pattern | `internal/cli/root.go` -- `fetchOrCacheSpec()` with SHA256 cache keys |
+| Temp dir pattern | `internal/cli/scorecard.go:32` -- `os.MkdirTemp("", "prefix-*")` with `defer os.RemoveAll` |
+| Command registration | `internal/cli/root.go:48` -- `rootCmd.AddCommand(...)` |
+| Spec merging | `internal/cli/root.go:314` -- `mergeSpecs()` copies Resources/Types with collision prefixing |
+| Test style | `internal/websniff/classifier_test.go` -- table-driven, `testify/assert`, `t.Parallel()` |
+| Path sanitization | `internal/openapi/parser.go` -- `sanitizeResourceName()` |
+
+### Institutional Learnings
+
+- **Path traversal**: External identifiers (npm package names, GitHub repo names) must be sanitized before use in `filepath.Join`. Use belt-and-suspenders: validate for `..`/`/`/`\` AND verify resolved path stays within expected root. (from `docs/solutions/security-issues/filepath-join-traversal-with-user-input-2026-03-29.md`)
+- **Validation must not mutate source**: Any temp artifacts from tarball extraction must use `os.MkdirTemp` with `defer` cleanup. (from `docs/solutions/best-practices/validation-must-not-mutate-source-directory-2026-03-29.md`)
+
+### External API Constraints
+
+| API | Auth | Rate Limit | Max Results | Recency Filter |
+|-----|------|------------|-------------|----------------|
+| npm search (`registry.npmjs.org/-/v1/search`) | None | Generous (no known cap) | Unlimited pagination | `package.date` in response |
+| npm downloads (`api.npmjs.org/downloads/point/last-week/`) | None | Generous | Bulk: 128 packages | N/A |
+| npm package meta (`registry.npmjs.org/<pkg>`) | None | Generous | N/A | N/A |
+| GitHub code search (`api.github.com/search/code`) | Required (`GITHUB_TOKEN`) | **10 req/min** | **1000 total** | **Not supported in query** |
+| GitHub repo info (`api.github.com/repos/{owner}/{repo}`) | Optional (recommended) | 5000 req/hr | N/A | `pushed_at` in response |
+
+## Key Technical Decisions
+
+- **New package `internal/crowdsniff/`**: Mirrors `internal/websniff/` -- separate package for discovery logic, CLI command in `internal/cli/crowd_sniff.go`. Keeps the new code isolated and testable.
+
+- **Parallel source execution with name-based GitHub fallback**: npm and GitHub run concurrently via goroutines. npm searches by keyword directly. GitHub code search uses the base URL domain when `--base-url` is provided (e.g., `"api.notion.com" language:javascript`). When no base URL is known, GitHub falls back to name-based queries (e.g., `"notion" api fetch language:javascript`) — noisier but still useful, and compatible with parallel execution. npm has no rate limits and typically completes in 10-30s. GitHub is the bottleneck (~1-4 min). Total time is ~max(npm, github).
+
+- **GitHub recency via repo info calls**: Code search can't filter by push date. After collecting unique repo names from code search results (typically 50-200 repos from 1000 max results), batch-check `pushed_at` via the standard repos API (5000 req/hr limit, separate pool from code search). Filter out repos not pushed within 6 months.
+
+- **npm tarball workflow**: Search registry -> filter by `package.date` (6-month cutoff) -> fetch package metadata for tarball URL -> download tarball to `os.MkdirTemp` -> `archive/tar` + `compress/gzip` extraction -> grep source files -> cleanup. Cap at 10 packages, skip tarballs > 10MB.
+
+- **Heuristic grep patterns over AST**: Match URL string literals (`"/v1/users"`, `"/api/projects"`), HTTP method calls (`this.get(`, `this.post(`, `fetch(`), base URL constants (`baseUrl`, `BASE_URL`, `apiBase`), and TypeScript type exports. Language-agnostic regex, no parser dependency.
+
+- **`SourceResult` wrapper instead of bare slice**: Each source returns `SourceResult{Endpoints []DiscoveredEndpoint, BaseURLCandidates []string}` rather than just endpoints. This threads base URL signals (from SDK constants, from code frequency) through the interface so the aggregation layer can resolve R14 without a side channel. Without this, the CLI command would need to reach into source internals to extract base URLs.
+
+- **Parameter syntax normalization**: SDK code uses `:id`, `{user_id}`, `<userId>`, `$id` for path parameters. GitHub code uses `{user_id}`, f-strings, etc. These must all normalize to `{id}` for deduplication to work. This is different from websniff's normalization (which replaces concrete values like `123`). Crowd-sniff needs both: syntax unification AND concrete-value replacement.
+
+- **Testable HTTP clients with separate base URLs per service**: Sources accept configurable base URLs instead of hardcoding. npm needs two: `RegistryBaseURL` (defaults to `registry.npmjs.org`) and `DownloadsBaseURL` (defaults to `api.npmjs.org`) since these are different hosts. GitHub needs one `BaseURL` (defaults to `api.github.com`). Tests use `httptest.NewServer` with injected base URLs. This avoids repeating `research.go`'s untestable pattern.
+
+- **Input sanitization at CLI boundary**: The `--api` value flows into HTTP URLs and file paths. URL-encode with `url.QueryEscape` before embedding in any API request. Validate at CLI boundary: reject newlines, null bytes, path separators, and `..`. Apply the belt-and-suspenders pattern from the learnings doc to the output cache path (validate input AND verify resolved path stays within cache root).
+
+- **Tarball security: zip-slip AND symlink protection**: Skip `tar.TypeSymlink` and `tar.TypeLink` entries during extraction (a malicious symlink to `/etc/passwd` would let grep read arbitrary files). Use `io.LimitReader` capped at 10MB+1 as the size gate (Content-Length is unreliable with chunked encoding). Validate tarball URL is HTTPS before downloading.
+
+- **Base URL candidates must be HTTPS**: All base URL candidates (from SDK constants, GitHub code frequency) must use HTTPS. Reject non-HTTPS candidates. The catalog validation already enforces HTTPS for `spec_url` — apply the same standard here.
+
+- **`errgroup` for parallel execution (no `WithContext` cancellation)**: New `go.mod` dependency (`golang.org/x/sync`). Use `errgroup.Group` (not `errgroup.WithContext`) so that one source's failure does not cancel the other — this matches R4's requirement that each source is independent. Sources must never return errors to the errgroup; instead they log warnings to stderr and return empty `SourceResult`. The errgroup is for synchronization and panic recovery only. This is the first use of `context.Context` in the codebase (for timeouts); it is deliberate and should not be back-propagated to existing commands.
+
+- **Duplicate GitHub request helper, don't import pipeline**: `newGitHubRequest()` from `research.go` is only 8 lines (set Accept header, add Bearer token from env). Duplicating it in `crowdsniff` avoids a coupling from a discovery package to the pipeline package.
+
+- **Use plain string constants for source tiers**: `const TierOfficialSDK = "official-sdk"` etc., matching how `AuthConfig.Type` uses plain strings (`"api_key"`, `"bearer_token"`). No named type.
+
+- **Postman source deferred**: Postman is not stubbed as an interface in v1 -- it's simply not implemented. When the Postman Explore CLI ships, adding a source is one new file implementing the same function signature.
+
+## Open Questions
+
+### Resolved During Planning
+
+- **GitHub recency filtering**: Use separate `/repos/{owner}/{repo}` calls after code search to check `pushed_at`. Different rate limit pool (5000/hr vs 10/min), so not a bottleneck. Unique repo count is bounded by the 1000-result cap.
+- **npm download counts**: Use the bulk downloads API (`api.npmjs.org/downloads/point/last-week/pkg1,pkg2,...`) to fetch weekly downloads for up to 128 packages in one call. Use 100/week as the threshold for "popular community SDK" vs low-confidence community package.
+- **Wall-clock time**: ~2-4 minutes typical. npm completes in 10-30s (no rate limits). GitHub takes 1-4 min (10 req/min for code search + repo info checks). Sources run in parallel.
+- **Output location**: Default to `~/.cache/printing-press/crowd-sniff/<name>-spec.yaml`, mirroring `websniff.DefaultCachePath()` pattern.
+
+- **mergeSpecs preserves Meta via shallow copy**: `mergeSpecs()` copies `Resource` structs by value. The `Meta` map header is copied, preserving access to the same underlying data. This is safe because nothing mutates Meta after construction — Meta must be treated as immutable once set. No mergeSpecs changes needed. Verified by examining `root.go:314-357`.
+
+### Deferred to Implementation
+
+- Exact grep patterns for SDK source analysis -- prototype against Notion, Stripe, Discord SDKs during implementation
+- Base URL extraction patterns from SDK constants -- validate during npm source implementation
+- Parameter syntax normalization regex coverage -- prototype against real SDK path patterns (`:id`, `{user_id}`, `<userId>`, `$id`)
+
+## High-Level Technical Design
+
+> *This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.*
+
+```
+ crowd-sniff --api "notion"
+ |
+ +--------------+
+ | Resolve API | npm search + GitHub search
+ | Identity | -> disambiguate if needed
+ +--------------+
+ |
+ +------------+------------+
+ | |
+ +------+------+ +------+------+
+ | npm Source | | GitHub Source|
+ | (goroutine) | | (goroutine) |
+ +------+------+ +------+------+
+ | 1. Search | | 1. Code |
+ | 2. Filter | | search |
+ | recency | | 2. Collect |
+ | 3. Downloads | | repos |
+ | API | | 3. Check |
+ | 4. Tarball | | pushed_at|
+ | + grep | | 4. Aggregate|
+ +------+------+ +------+------+
+ | |
+ +------------+------------+
+ |
+ +--------------+
+ | Aggregate | Deduplicate endpoints,
+ | & Rank | compute source_tier,
+ | | source_count
+ +--------------+
+ |
+ +--------------+
+ | Build spec | Resolve base URL,
+ | APISpec | group into resources,
+ | | set Meta on endpoints
+ +--------------+
+ |
+ +--------------+
+ | Write YAML | websniff.WriteSpec()
+ +--------------+
+```
+
+## Implementation Units
+
+- [ ] **Unit 1: Add Meta field to spec.Endpoint**
+
+ **Goal:** Add `Meta map[string]string` with `yaml:"meta,omitempty"` to the `Endpoint` struct so crowd-sniff (and future features) can attach per-endpoint metadata.
+
+ **Requirements:** R9
+
+ **Dependencies:** None
+
+ **Files:**
+ - Modify: `internal/spec/spec.go`
+ - Test: `internal/spec/spec_test.go`
+
+ **Approach:**
+ Add one field to the `Endpoint` struct, placed before `Alias` (keeping YAML-serialized fields grouped, non-YAML fields at bottom). The `omitempty` tag ensures zero impact on existing YAML round-trips. No changes needed to `Validate()`, generator templates, or `mergeSpecs()`.
+
+ **Patterns to follow:**
+ - Existing optional fields on `Endpoint`: `ResponsePath string yaml:"response_path,omitempty"`, `Pagination *Pagination yaml:"pagination"`
+
+ **Test scenarios:**
+ - Happy path: Parse spec YAML with `meta` field populated -> Endpoint.Meta contains expected key-value pairs
+ - Happy path: Parse spec YAML without `meta` field -> Endpoint.Meta is nil (zero value)
+ - Happy path: Marshal Endpoint with Meta set -> YAML output contains `meta:` section
+ - Happy path: Marshal Endpoint with Meta nil -> YAML output omits `meta:` entirely
+ - Integration: Meta survives a `mergeSpecs()` round-trip -- create two specs with Meta-bearing endpoints, merge them, verify Meta is preserved on all endpoints
+ - Edge case: Existing `TestVersionConsistencyAcrossFiles` and other spec tests still pass unchanged
+
+ **Verification:**
+ - `go test ./internal/spec/...` passes
+ - `go test ./...` passes (no regressions from field addition)
+
+---
+
+- [ ] **Unit 2: Create internal/crowdsniff package with types and aggregation**
+
+ **Goal:** Define core types (`DiscoveredEndpoint`, `SourceResult`, source tier constants), the aggregation engine (deduplication, source_tier/source_count), parameter syntax normalization, and the spec builder (group into resources, resolve base URL, produce `*spec.APISpec` with Meta).
+
+ **Requirements:** R4, R7, R8, R10, R14
+
+ **Dependencies:** Unit 1
+
+ **Files:**
+ - Create: `internal/crowdsniff/types.go` (DiscoveredEndpoint, SourceResult, tier constants)
+ - Create: `internal/crowdsniff/aggregate.go` (dedup, normalization, tier computation)
+ - Create: `internal/crowdsniff/specgen.go` (APISpec construction from aggregated endpoints)
+ - Test: `internal/crowdsniff/aggregate_test.go`
+ - Test: `internal/crowdsniff/specgen_test.go`
+
+ **Approach:**
+ - `SourceResult` struct: `Endpoints []DiscoveredEndpoint` + `BaseURLCandidates []string` -- each source returns this so base URL signals are threaded through the interface
+ - `DiscoveredEndpoint`: method, path, params (optional), source tier (string constant), source name (e.g., `@notionhq/client`)
+ - Source tier constants: `const TierOfficialSDK = "official-sdk"`, `TierCommunitySDK = "community-sdk"`, `TierCodeSearch = "code-search"`, `TierPostman = "postman"`
+ - `Aggregate(results []SourceResult) ([]AggregatedEndpoint, []string)`: deduplicate by normalized method+path, compute `source_tier` (highest tier) and `source_count` (distinct source count), collect base URL candidates
+ - `BuildSpec(name, baseURL string, endpoints []AggregatedEndpoint) *spec.APISpec`: group into resources (reuse `deriveResourceKey` pattern from `websniff/specgen.go`), set `Meta` on each endpoint
+ - **Path normalization (two-step)**: First, unify parameter syntax (`:id`, `{user_id}`, `<userId>`, `$id` all become `{id}`). Second, apply websniff-style normalization (replace concrete UUIDs, numeric IDs, hashes with placeholders). Copy the normalization functions from `websniff/classifier.go` rather than importing to keep packages independent.
+ - Base URL resolution: accept `--base-url` flag, then SDK candidates, then GitHub frequency candidates, pick first non-empty
+
+ **Patterns to follow:**
+ - `internal/websniff/specgen.go` -- `AnalyzeCapture()` flow: classify -> deduplicate -> build endpoints -> assemble spec
+ - `internal/websniff/classifier.go` -- `normalizeEntryPath()`, `deriveResourceKey()`, `deriveEndpointName()` (copy, don't import)
+ - File naming: responsibility-based (`types.go`, `aggregate.go`, `specgen.go`) matching websniff convention
+
+ **Test scenarios:**
+ - Happy path: Aggregate endpoints from two sources -> source_count=2, source_tier=highest tier
+ - Happy path: Aggregate with one official-sdk and one code-search source for same endpoint -> source_tier="official-sdk", source_count=2
+ - Happy path: BuildSpec produces valid APISpec that passes `spec.Validate()`
+ - Happy path: Meta on each endpoint contains `source_tier` and `source_count` as strings
+ - Happy path: Base URL candidates from multiple sources -> first non-empty selected
+ - Edge case: Single source provides all endpoints -> source_count=1 for all
+ - Edge case: Same endpoint from same source twice -> deduplicated, source_count=1
+ - Edge case: Empty endpoint list -> returns error (at least one endpoint required)
+ - Edge case: Endpoints with different path normalizations that should merge (e.g., `/users/123` and `/users/456` both become `/users/{id}`)
+ - Edge case: Parameter syntax normalization -- `/users/:id`, `/users/{user_id}`, `/users/<id>` all merge into `/users/{id}`
+ - Happy path: Resource grouping from paths (e.g., `/v1/users` and `/v1/users/{id}` -> `users` resource)
+
+ **Verification:**
+ - `go test ./internal/crowdsniff/...` passes
+ - Aggregated output matches expected source_tier/source_count for known inputs
+
+---
+
+- [ ] **Unit 3: npm source implementation**
+
+ **Goal:** Implement the npm `Source` that searches the registry, filters by recency, fetches download counts, downloads/extracts tarballs, and greps for endpoint patterns.
+
+ **Requirements:** R1, R5, R6, R17
+
+ **Dependencies:** Unit 2
+
+ **Files:**
+ - Create: `internal/crowdsniff/npm.go`
+ - Create: `internal/crowdsniff/patterns.go` (shared grep patterns, also used by Unit 4)
+ - Test: `internal/crowdsniff/npm_test.go`
+ - Test: `internal/crowdsniff/patterns_test.go`
+ - Create: `testdata/crowdsniff/npm-search-response.json`
+ - Create: `testdata/crowdsniff/npm-package-meta.json`
+ - Create: `testdata/crowdsniff/sdk-sample/` (small extracted SDK file set for pattern testing)
+
+ **Approach:**
+ - **Constructor**: `NewNPMSource(opts NPMOptions)` where `NPMOptions` includes optional `BaseURL string` (defaults to `registry.npmjs.org`) and `HTTPClient *http.Client` (defaults to `&http.Client{Timeout: 15 * time.Second}`). Configurable base URL enables testing with `httptest.NewServer`. Use 30-second timeout for tarball downloads (larger payloads than JSON responses).
+ - **Search**: `GET <baseURL>/-/v1/search?text=<api-name>&size=25` -> filter by `package.date` (6-month cutoff) -> take top 10
+ - **Classify SDK type**: official if `package.scope` matches API vendor name (e.g., `@notionhq` for notion); else community
+ - **Downloads**: Bulk call `api.npmjs.org/downloads/point/last-week/pkg1,pkg2,...` to get weekly counts. >100/week = popular community, else low-confidence community
+ - **Tarball**: For each package, `GET <baseURL>/<pkg>/<version>` for `dist.tarball` URL. Download to `os.MkdirTemp("", "crowd-sniff-npm-*")`, extract with `archive/tar` + `compress/gzip`. Skip if tarball > 10MB (check Content-Length header first).
+ - **Grep**: Apply patterns from `patterns.go` to all `.js`, `.ts`, `.mjs` files in extracted tree. Patterns match URL path literals, HTTP method calls, base URL constants. Sanitize all file paths (learnings: path traversal).
+ - **Base URL extraction**: Also grep for base URL constants (`baseUrl`, `BASE_URL`, `this.baseUrl`, constructor defaults). Return as `SourceResult.BaseURLCandidates`.
+ - **Cleanup**: `defer os.RemoveAll(tmpDir)` per package
+
+ **Patterns to follow:**
+ - `internal/cli/scorecard.go:32` -- `os.MkdirTemp` + `defer os.RemoveAll`
+ - `internal/pipeline/research.go` -- HTTP client with explicit timeout, non-fatal error handling
+
+ **Test scenarios:**
+ - Happy path: Mock npm search response with 3 packages -> returns endpoints from all 3
+ - Happy path: Official SDK (scoped `@notionhq/client`) -> source_tier="official-sdk"
+ - Happy path: Popular community SDK (>100 downloads/week) -> source_tier="community-sdk"
+ - Happy path: Grep finds `"/v1/users"` with `this.get` call -> DiscoveredEndpoint{Method:"GET", Path:"/v1/users"}
+ - Edge case: Package `date` older than 6 months -> excluded from results
+ - Edge case: Tarball > 10MB -> skipped with warning to stderr
+ - Edge case: npm search returns 0 results -> returns empty slice, no error
+ - Edge case: Download API unavailable -> still returns endpoints, just without download count classification
+ - Error path: Tarball download fails -> skip package, continue with others
+ - Error path: Tarball extraction encounters path traversal attempt -> sanitize and skip the malicious entry
+ - Error path: Tarball contains symlink entry -> skipped (tar.TypeSymlink/TypeLink rejected)
+ - Error path: Tarball URL is non-HTTPS -> skipped with warning
+ - Edge case: Tarball has no Content-Length header -> io.LimitReader caps at 10MB
+
+ **Verification:**
+ - `go test ./internal/crowdsniff/...` passes
+ - Pattern tests cover at least 3 SDK coding styles (class method, fetch wrapper, axios instance)
+
+---
+
+- [ ] **Unit 4: GitHub code search source implementation**
+
+ **Goal:** Implement the GitHub `Source` that searches for API usage patterns in code, checks repo freshness, aggregates endpoint frequency, and extracts common parameters.
+
+ **Requirements:** R2, R5, R6
+
+ **Dependencies:** Unit 2
+
+ **Files:**
+ - Create: `internal/crowdsniff/github.go`
+ - Test: `internal/crowdsniff/github_test.go`
+ - Create: `testdata/crowdsniff/github-code-search-response.json`
+ - Create: `testdata/crowdsniff/github-repo-response.json`
+
+ **Approach:**
+ - **Constructor**: `NewGitHubSource(opts GitHubOptions)` where `GitHubOptions` includes optional `BaseURL string` (defaults to `api.github.com`), `HTTPClient *http.Client`, and `Token string` (defaults to `os.Getenv("GITHUB_TOKEN")`). Configurable for testing.
+ - **Code search**: When base URL is known (from `--base-url`), use domain-based queries: `"api.notion.com" language:javascript`. When no base URL is known, fall back to name-based queries: `"notion" api fetch language:javascript` (noisier but still useful). Use `Accept: application/vnd.github.text-match+json` for matched fragments. Paginate up to 1000 results (10 pages x 100).
+ - **Rate limiting**: Use a plain `time.Ticker` at 6-second intervals for code search requests. No need for a reusable rate limiter type — only two call sites, both in this file. Testability is handled by the configurable base URL + `httptest.NewServer` approach (test server responds instantly).
+ - **Repo freshness**: Collect unique `repository.full_name` from code search results. For each, `GET /repos/{owner}/{repo}` (5000 req/hr limit, separate pool) and check `pushed_at`. Discard repos not pushed within 6 months.
+ - **Endpoint extraction**: From text matches and file URLs, extract URL path patterns. Use patterns from `patterns.go` (shared with npm source). Aggregate by normalized method+path, count distinct repos per endpoint. Return most frequent domain as `SourceResult.BaseURLCandidates`.
+ - **Auth**: If token is empty, return empty `SourceResult` immediately (per R4 graceful degradation). Duplicate the 8-line `newGitHubRequest()` helper locally (set Accept header, add Bearer token) -- do not import `pipeline` package.
+
+ **Patterns to follow:**
+ - `internal/pipeline/research.go` -- `newGitHubRequest()` logic to duplicate, error handling patterns
+ - `internal/websniff/classifier.go` -- `extractPath()`, `normalizeEntryPath()` patterns (copied to crowdsniff in Unit 2)
+
+ **Test scenarios:**
+ - Happy path: Mock code search response with 5 results across 3 repos -> returns aggregated endpoints with frequency counts
+ - Happy path: Endpoint found in 10+ repos -> DiscoveredEndpoint with high frequency signal
+ - Happy path: `text_matches` contain `/v1/users` and `/v1/projects` -> two distinct endpoints extracted
+ - Edge case: No GITHUB_TOKEN set -> returns empty slice immediately, no error
+ - Edge case: Repo pushed_at is 8 months ago -> all endpoints from that repo excluded
+ - Edge case: All 1000 results are from the same repo -> source_count still 1 for code-search source
+ - Edge case: Code search returns 0 results for an obscure API -> returns empty slice
+ - Error path: Rate limit hit (429 response) -> wait and retry once, then skip remaining queries
+ - Error path: GitHub API returns 5xx -> skip GitHub source, log warning to stderr
+ - Integration: Code search and repo freshness checks use different rate limit pools -> both complete without mutual blocking
+
+ **Verification:**
+ - `go test ./internal/crowdsniff/...` passes
+ - Rate limiter test confirms minimum 6-second spacing between code search requests
+
+---
+
+- [ ] **Unit 5: CLI command and orchestration**
+
+ **Goal:** Add the `printing-press crowd-sniff` cobra command that accepts `--api`, orchestrates sources in parallel, handles disambiguation, and writes the output spec YAML.
+
+ **Requirements:** R4, R10, R11, R12, R13, R14
+
+ **Dependencies:** Units 2, 3, 4
+
+ **Files:**
+ - Create: `internal/cli/crowd_sniff.go`
+ - Modify: `internal/cli/root.go` (register command)
+ - Test: `internal/cli/crowd_sniff_test.go`
+
+ **Approach:**
+ - `newCrowdSniffCmd()` returns `*cobra.Command` with flags: `--api` (required), `--output`, `--base-url`, `--json`
+ - No `--github-token` flag -- use `os.Getenv("GITHUB_TOKEN")` only, matching `research.go` convention
+ - **Disambiguation**: If npm search + GitHub search both return results for multiple distinct APIs (e.g., `--api cal` matches `cal.com` and Google Calendar), present candidates via `fmt.Fprintf` to stderr and prompt user. If running non-interactively (check `os.Stdin` is not a terminal), error with "ambiguous API name, use --api <specific-name> or --base-url".
+ - **Parallel execution**: Use `errgroup.Group` (not `WithContext`). Sources never return errors to the group — they log warnings to stderr and return empty `SourceResult`. This matches R4: each source is independent. Adds `golang.org/x/sync` to `go.mod`.
+ - **Input sanitization**: URL-encode `--api` with `url.QueryEscape` before any HTTP request. Validate at CLI boundary: reject newlines, null bytes, path separators, `..`. Apply belt-and-suspenders to output cache path per learnings doc.
+ - **Aggregation**: Call `crowdsniff.Aggregate(results)` which returns aggregated endpoints and merged base URL candidates. Then `crowdsniff.BuildSpec()`.
+ - **Base URL**: First check `--base-url` flag. If not set, use candidates from `Aggregate()` (SDK constants first, then GitHub frequency). Fail with error asking user to provide `--base-url` if no URL resolved.
+ - **Output**: Default cache path `~/.cache/printing-press/crowd-sniff/<name>-spec.yaml`. Call `websniff.WriteSpec()` from the CLI layer (crowdsniff package returns `*spec.APISpec` only, doesn't import websniff).
+ - **Summary**: Print to stdout: endpoint count, resource count, source breakdown (N from npm, M from GitHub), source_tier distribution. `--json` flag outputs structured JSON instead.
+
+ **Patterns to follow:**
+ - `internal/cli/sniff.go` -- command structure, flag declaration, RunE flow, success message format
+ - `internal/cli/root.go:48` -- command registration
+
+ **Test scenarios:**
+ - Happy path: `crowd-sniff --api notion` with mock sources -> produces valid spec YAML at expected output path
+ - Happy path: `--output custom/path.yaml` -> writes to specified path
+ - Happy path: `--base-url https://api.example.com` overrides auto-detected base URL
+ - Happy path: `--json` flag produces structured JSON output
+ - Happy path: Summary output includes correct endpoint and source counts
+ - Edge case: Both sources return 0 endpoints -> error message "no endpoints discovered"
+ - Edge case: Only npm source returns results (no GITHUB_TOKEN) -> still produces valid spec
+ - Edge case: GitHub source returns error, npm returns 5 endpoints -> spec produced, warning logged to stderr about GitHub failure
+ - Edge case: `--api` flag not provided -> cobra reports missing required flag
+ - Error path: Output directory doesn't exist -> created automatically (same as sniff)
+ - Error path: `--api "../../.ssh/evil"` -> rejected at CLI boundary (path traversal)
+ - Error path: `--api "notion&size=9999"` -> api name URL-encoded in HTTP requests, no query injection
+
+ **Verification:**
+ - `go build -o ./printing-press ./cmd/printing-press` succeeds
+ - `./printing-press crowd-sniff --help` shows expected flags and description
+ - `go test ./internal/cli/...` passes
+
+---
+
+- [ ] **Unit 6: Skill integration (Phase 1.8 Crowd Sniff Gate)**
+
+ **Goal:** Add Phase 1.8 to the printing-press skill that offers crowd-sniff when the research phase identifies spec gaps or no spec exists.
+
+ **Requirements:** R15, R16
+
+ **Dependencies:** Unit 5
+
+ **Files:**
+ - Modify: `skills/printing-press/SKILL.md`
+
+ **Approach:**
+ - Insert Phase 1.8 after Phase 1.7 (Sniff Gate). Same structure: decision matrix, user prompt via `AskUserQuestion`, fallback on failure.
+ - Decision matrix mirrors sniff: offer crowd-sniff when spec has gaps or no spec found. Skip when spec appears complete or user already provided `--spec`.
+ - Gate prompt: "Want me to search npm packages and GitHub code for `<api>` to discover additional endpoints? This typically takes 2-4 minutes."
+ - On approval: run `printing-press crowd-sniff --api <api> --output "$RESEARCH_DIR/<api>-crowd-spec.yaml"`
+ - On success: report endpoint count and feed into Phase 2 as additional `--spec`
+ - On failure: "Crowd sniff found no additional endpoints -- proceeding with existing spec."
+ - Time budget: 5 minutes (longer than sniff's 3 minutes due to GitHub rate limits)
+
+ **Test expectation: none** -- skill SKILL.md changes are validated by manual testing and the existing quality gate workflow, not Go tests.
+
+ **Verification:**
+ - SKILL.md contains Phase 1.8 section with decision matrix, prompt, and fallback
+ - Phase 1.8 is referenced in Phase 2 generate command construction (merge crowd-spec if it exists)
+
+## System-Wide Impact
+
+- **Spec struct change**: Adding `Meta map[string]string` to `spec.Endpoint` is the only change to shared types. With `omitempty`, it is invisible to all existing producers and consumers. No template changes, no validation changes, no mergeSpecs changes.
+- **New external API dependencies**: npm registry (unauthenticated, generous limits) and GitHub code search (authenticated, 10 req/min). Both are optional per R4 -- the command works with either or both.
+- **New go.mod dependency**: `golang.org/x/sync` for `errgroup`. Well-maintained, officially part of the Go project.
+- **Binary size**: New `archive/tar` and `compress/gzip` imports add to binary. These are stdlib packages, minimal impact.
+- **Unchanged invariants**: The `generate` command, OpenAPI/GraphQL parsers, existing `sniff` command, catalog system, and all templates are unmodified.
+
+## Risks & Dependencies
+
+| Risk | Mitigation |
+|------|------------|
+| GitHub code search 10 req/min makes the command slow | Run npm and GitHub in parallel. Set user expectation in CLI output ("Searching GitHub... this takes 1-3 minutes due to rate limits"). |
+| Heuristic grep patterns produce false positives (match non-API URLs) | Filter by known API domain. Use path normalization to deduplicate. Cross-source agreement (source_count > 1) naturally filters noise. |
+| npm tarballs contain malicious paths (zip slip) | Sanitize every extracted file path. Validate resolved path stays within temp dir root. Existing learnings doc covers this pattern. |
+| GitHub code search results are stale (old repos) | Separate repo freshness check via `/repos/` API. 6-month hard cutoff on `pushed_at`. |
+| Parameter syntax varies across SDKs (`:id`, `{id}`, `<id>`) | Two-step normalization: unify syntax first, then replace concrete values. Dedup depends on this working correctly. Prototype against 3+ real SDKs. |
+| Malicious npm tarballs (zip-slip, symlinks) | Reject symlinks and hard links during extraction. Validate all paths stay within temp dir. Require HTTPS tarball URLs. Use io.LimitReader for size gate. |
+| --api value used in HTTP URLs and file paths | URL-encode for HTTP. Validate at CLI boundary for path traversal. Belt-and-suspenders on output path. |
+| Base URL candidates from untrusted sources | Require HTTPS. Reject localhost/private IPs. Validate before writing to spec YAML. |
+| Postman source not available for v1 | Postman is simply not implemented in v1. When the Postman Explore CLI ships, adding it is one new file with the same function signature. |
+
+## Sources & References
+
+- **Origin document:** [docs/brainstorms/2026-03-29-crowd-sniff-requirements.md](docs/brainstorms/2026-03-29-crowd-sniff-requirements.md)
+- npm registry API: `registry.npmjs.org/-/v1/search`, `api.npmjs.org/downloads/point/last-week/`
+- GitHub code search API: `api.github.com/search/code`
+- Related code: `internal/cli/sniff.go`, `internal/websniff/`, `internal/pipeline/research.go`
+- Security learning: `docs/solutions/security-issues/filepath-join-traversal-with-user-input-2026-03-29.md`
diff --git a/docs/plans/2026-03-30-003-feat-crowd-sniff-param-discovery-plan.md b/docs/plans/2026-03-30-003-feat-crowd-sniff-param-discovery-plan.md
new file mode 100644
index 00000000..371e3f74
--- /dev/null
+++ b/docs/plans/2026-03-30-003-feat-crowd-sniff-param-discovery-plan.md
@@ -0,0 +1,352 @@
+---
+title: "feat: Add parameter discovery to crowd-sniff"
+type: feat
+status: completed
+date: 2026-03-30
+origin: docs/brainstorms/2026-03-29-crowd-sniff-requirements.md
+deepened: 2026-03-30
+---
+
+# feat: Add parameter discovery to crowd-sniff
+
+## Overview
+
+Extend crowd-sniff's npm source to extract query parameters from SDK source code alongside endpoints. Currently crowd-sniff discovers method + path correctly but emits `params: []` for every endpoint. The parameter data is present in the code crowd-sniff already downloads and greps — it just isn't extracted.
+
+## Problem Frame
+
+The Steam API crowd-sniff found 23 correct endpoints but zero parameters. All 24 params had to be added manually during the generate phase. The params were visible in three places the npm source already reads: function signatures (`getOwnedGames(steamid, opts = {})`), request builder calls (`this.get('/path', { steamid, include_appinfo: includeAppInfo })`), and README documentation. Eliminating this manual enrichment step makes crowd-sniff a complete discovery source rather than a skeleton that requires human intervention. (see origin: `docs/brainstorms/2026-03-29-crowd-sniff-requirements.md`, gap analysis: `~/printing-press/manuscripts/postman-explore/20260330-105847/proofs/2026-03-30-crowd-sniff-param-discovery-gap.md`)
+
+## Requirements Trace
+
+- P1. Extract query parameter names from object literals passed as the second argument to HTTP method calls (`.get(url, { key1, key2: val })`)
+- P2. Infer parameter types using heuristic rules: `.join(',')` → string, numeric defaults → integer, `true`/`false` → boolean, name matches whole-word `id` → string (use `\b` word boundary to avoid matching `userId`, `guild_id`), name is `count`/`limit`/`offset`/`page`/`maxlength` → integer, default → string
+- P3. Infer required vs optional from function signatures: positional args without defaults → required, destructured args with defaults → optional. Default to `required: false` when the function signature cannot be correlated
+- P4. Handle multi-line object literals (params object spanning 2-10 lines with one key per line)
+- P5. Carry extracted params through aggregation (merge across sources) and into `spec.Endpoint.Params`
+- P6. Preserve all existing endpoint discovery behavior — param extraction is additive only. `GrepEndpoints` returns unchanged `DiscoveredEndpoint` (Params field is nil). `EnrichWithParams` is an optional enrichment step called after `GrepEndpoints`
+- P7. GitHub source is out of scope for param extraction (text match fragments are too short to contain full function bodies). npm source only
+
+## Scope Boundaries
+
+- **npm source only** — GitHub code search returns text fragments, not full function bodies. Param extraction from GitHub is a future enhancement
+- **No AST parsing** — lightweight brace-matching scanner, not a JS/TS parser
+- **No README/JSDoc parsing** — v2 enhancement. Requires multi-line section extraction beyond what heuristic scanning handles well
+- **No body field extraction** — POST/PUT body structure is harder to infer from SDK code (often just `data` or `body`). Future enhancement
+- **No parameter description generation** — params get `Name`, `Type`, `Required`, `Default` only. Descriptions are empty
+
+## Context & Research
+
+### Relevant Code and Patterns
+
+| Purpose | File |
+|---------|------|
+| Types to extend (add Params) | `internal/crowdsniff/types.go` — `DiscoveredEndpoint`, `AggregatedEndpoint` |
+| Extraction target (multi-line scanner) | `internal/crowdsniff/patterns.go` — `GrepEndpoints()`, `extractMethodCallEndpoints()` |
+| Aggregation to extend (merge params) | `internal/crowdsniff/aggregate.go` — `Aggregate()`, accumulator struct |
+| Spec builder to wire up | `internal/crowdsniff/specgen.go` — `BuildSpec()` |
+| Reference implementation for `[]spec.Param` | `internal/websniff/specgen.go:182` — `inferURLParams()` |
+| Target spec type | `internal/spec/spec.go:62` — `Param` struct with Name, Type, Required, Default, etc. |
+| Test patterns | `internal/crowdsniff/patterns_test.go` — inline SDK content strings, `t.Parallel()`, `testify/assert` |
+| Test patterns | `internal/crowdsniff/aggregate_test.go` — table-driven, subtest groups |
+
+### Institutional Learnings
+
+- **Cartesian product risk** (from `docs/solutions/best-practices/multi-source-api-discovery-design-2026-03-30.md`): `extractMethodCallEndpoints` previously had a bug where every method was paired with every path on the same line. The fix used `FindAllStringSubmatchIndex` for positional correlation. The same cross-product risk exists when extracting params from a line with multiple function calls — use positional matching.
+- **Word-boundary regex** (from `docs/solutions/logic-errors/scorecard-accuracy-broadened-pattern-matching-2026-03-27.md`): `strings.Contains` produces false positives on substrings. When extracting param names like `id` that are substrings of `userId`, `guild_id`, etc., use `\b` word boundaries.
+- **Scope broadening breaks downstream** (same source): When broadening from single-line to multi-line extraction, audit all downstream consumers of the extracted data.
+
+## Key Technical Decisions
+
+- **Brace-matching scanner over multi-line regex**: Go's `regexp` does not support lookahead/lookbehind. A lightweight scanner that finds `{` after the URL match and counts brace depth until balanced `}` is more reliable than attempting multi-line regex on concatenated content. ~50 lines of Go, handles nested objects and trailing commas naturally.
+
+- **Content-level scanning (not line-by-line) for params**: The existing `GrepEndpoints` line loop stays for endpoint discovery (works well). A new `EnrichWithParams` function operates on full file content to handle multi-line object literals. It is called after `GrepEndpoints` and enriches the discovered endpoints with params by matching on method+path.
+
+- **Separate `DiscoveredParam` type over reusing `spec.Param`**: A lightweight `DiscoveredParam{Name, Type, Required, Default}` in types.go keeps the discovery layer decoupled from the spec layer. `specgen.go` maps `DiscoveredParam` → `spec.Param`. This avoids importing `spec` into the core discovery types.
+
+- **Union merge for params in aggregation**: When the same endpoint is discovered by multiple sources, take the union of param names. For conflicts on the same param name, prefer metadata from the higher-tier source (official-sdk > community-sdk > code-search). This mirrors how `source_tier` is already resolved.
+
+- **Default `required: false`**: When the function signature cannot be correlated with the HTTP call (common), default to `required: false`. A false-required param breaks CLI usage; a false-optional param is merely inconvenient. Safer default.
+
+- **Enrichment pass architecture**: `GrepEndpoints` returns endpoints as before. A new `EnrichWithParams(content string, endpoints []DiscoveredEndpoint) []DiscoveredEndpoint` function does a second pass over the same content, matching each discovered endpoint's path to HTTP calls, then extracting the params object. This avoids modifying the existing extraction functions and their carefully-tested positional logic. Note: `EnrichWithParams` operates on full content (not line-by-line) — a deliberate architectural asymmetry with `GrepEndpoints` that should be documented in the function's doc comment.
+
+- **EnrichWithParams must match raw URL patterns, not cleaned paths**: `GrepEndpoints` transforms paths during extraction (e.g., `${userId}` → `{id}` via `cleanPath`). The enrichment pass must search content for the raw URL string as it appears in source code, not the cleaned `DiscoveredEndpoint.Path`. Use the same `httpMethodCall` + `urlPathLiteral` regex patterns independently to locate HTTP calls, then correlate by position — do not substring-match `endpoint.Path` against content.
+
+- **String-typed defaults, no type coercion**: `DiscoveredParam.Default` is `string` while `spec.Param.Default` is `any`. Numeric defaults like `10` are stored as `"10"` and passed through as strings. Do not over-engineer type coercion — the data is heuristic. This means YAML output will emit `default: "10"` not `default: 10`, which is acceptable for crowd-sniffed params.
+
+- **Per-param tier tracking in aggregation**: The accumulator's `bestTier` is per-endpoint, but param merge needs per-param tier resolution. Chosen approach: params inherit tier from their parent endpoint's `SourceTier`. The accumulator stores `params map[string]paramEntry` where `paramEntry` is `struct{ param DiscoveredParam; tier string }`. When a param name conflicts, compare `tierRank(incoming.tier)` vs `tierRank(stored.tier)` — higher tier wins. This keeps `DiscoveredParam` itself lightweight (no tier field) while giving the accumulator the data it needs for tier-aware merge.
+
+## Open Questions
+
+### Resolved During Planning
+
+- **Where to extract params — same pass or second pass?** Second pass. The existing line-by-line extraction in `GrepEndpoints` is well-tested and uses positional index matching. Adding multi-line brace scanning into the same loop would mix two scanning strategies. A separate enrichment pass is cleaner.
+- **Should we extract from GitHub source too?** No. GitHub code search returns text match fragments (~300 chars), not full function bodies. Params require seeing both the function signature and the HTTP call, which are typically 3-10 lines apart. npm tarballs have full source files.
+- **Normalize param names across sources?** No normalization needed for v1. Param names from SDK code are already the canonical API param names (e.g., `steamid`, `include_appinfo`). Path params need normalization because different syntaxes represent the same concept; query params don't have this problem.
+
+### Deferred to Implementation
+
+- Exact regex patterns for object literal key extraction — prototype against Steam, Notion, Discord SDK source
+- Whether function-signature correlation works reliably enough to set `required: true` — if too noisy in practice, fall back to all-optional
+- Whether the brace scanner should track string literal state (skip `{`/`}` inside quotes) — adds ~10 lines but prevents misparse on params like `{ query: '{complex}' }`. Prototype against real SDKs to gauge frequency
+- Param name normalization across sources is deferred. If two sources use different names for the same parameter (e.g., `steamid` vs `steam_id`), both will appear in the union-merged param list. Acceptable for v1 but may produce duplicate params in edge cases
+
+## High-Level Technical Design
+
+> *This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.*
+
+```
+SDK source content (from npm tarball)
+ |
+ v
+ GrepEndpoints() ← existing, unchanged
+ returns []DiscoveredEndpoint (method+path only)
+ |
+ v
+ EnrichWithParams() ← NEW
+ For each endpoint found:
+ 1. Find the HTTP call in content: .get("/matching/path"
+ 2. Scan forward from the path for , { (params object start)
+ 3. Brace-match to find the closing }
+ 4. Extract keys from the captured block
+ 5. Look backward for enclosing function definition
+ 6. Correlate function args with param keys for required/optional
+ 7. Apply type heuristics to values
+ returns []DiscoveredEndpoint (now with Params populated)
+ |
+ v
+ Aggregate() ← extended with param merging
+ Union-merge params for same method+path
+ |
+ v
+ BuildSpec() ← extended to map DiscoveredParam → spec.Param
+ |
+ v
+ spec.Endpoint.Params populated
+```
+
+**Brace-matching scanner pseudocode:**
+
+```
+given: content string, position after the URL path match
+1. scan forward for comma followed by { (skipping whitespace/newlines)
+2. if not found within 500 chars, no params → return nil
+3. set depth = 1, start collecting
+4. for each char after {:
+ if { → depth++
+ if } → depth--; if depth == 0 → stop
+5. the captured block between { and } contains param entries
+6. split on commas (respecting nested braces)
+7. for each entry, extract the key (before : or the bare identifier for shorthand)
+```
+
+## Implementation Units
+
+- [x] **Unit 1: Add DiscoveredParam type and Params field to discovery types**
+
+ **Goal:** Extend the type hierarchy so params can flow from extraction through aggregation to spec generation.
+
+ **Requirements:** P5
+
+ **Dependencies:** None
+
+ **Files:**
+ - Modify: `internal/crowdsniff/types.go`
+
+ **Approach:**
+ - Add `DiscoveredParam` struct with `Name string`, `Type string`, `Required bool`, `Default string` (all-string default to keep it simple)
+ - Add `Params []DiscoveredParam` field to `DiscoveredEndpoint`
+ - Add `Params []DiscoveredParam` field to `AggregatedEndpoint`
+ - No test file for this unit — pure type definitions. Tests come with the functions that use them.
+
+ **Patterns to follow:**
+ - `DiscoveredEndpoint` and `AggregatedEndpoint` struct style in `types.go`
+
+ **Test expectation: none** — struct definitions with no behavior. Exercised extensively by Unit 2 (EnrichWithParams), Unit 3 (Aggregate), and Unit 4 (BuildSpec), which instantiate and assert on these structs.
+
+ **Verification:**
+ - `go build ./internal/crowdsniff/...` succeeds
+ - Existing tests still pass (`go test ./internal/crowdsniff/...`)
+
+---
+
+- [x] **Unit 2: Multi-line parameter extraction scanner**
+
+ **Goal:** Implement `EnrichWithParams` that does a second pass over SDK source content, finding the params object literal after each HTTP method call and extracting param names, types, and required/optional status.
+
+ **Requirements:** P1, P2, P3, P4, P6
+
+ **Dependencies:** Unit 1
+
+ **Files:**
+ - Create: `internal/crowdsniff/params.go`
+ - Test: `internal/crowdsniff/params_test.go`
+
+ **Approach:**
+ - `EnrichWithParams(content string, endpoints []DiscoveredEndpoint) []DiscoveredEndpoint` — the main entry point
+ - Independently scan content for all HTTP method calls using the same `httpMethodCall` + `urlPathLiteral` regex patterns. For each raw match, apply `cleanPath` to the extracted raw URL, then match the resulting `(method, cleanedPath)` pair against the `DiscoveredEndpoint` list by equality. This avoids substring-matching `endpoint.Path` against content (see Key Technical Decisions on raw-vs-cleaned matching). For matching endpoints, extract params from that call position.
+ - From the HTTP call position, scan forward for the params object:
+ - Skip whitespace/newlines after the path string's closing quote
+ - Look for `, {` or `,\n{` (comma then opening brace)
+ - If found, run the brace-matching scanner to capture the full object literal
+ - If not found within a reasonable distance (~200 chars), assume no params
+ - Extract keys from the captured object block:
+ - Shorthand properties: bare identifier on its own (e.g., `steamid` without `:`)
+ - Key-value pairs: `key: value` or `key: value.join(',')` — extract key
+ - Ignore nested objects (depth > 1)
+ - Handle trailing commas
+ - Type inference heuristics (from the gap artifact's table):
+ - Value contains `.join(',')` → `"string"`
+ - Value is `true` or `false` → `"boolean"`
+ - Value is a numeric literal → `"integer"`
+ - Key name matches `count`, `limit`, `offset`, `page`, `maxlength` → `"integer"`
+ - Default → `"string"`
+ - Function signature correlation (best-effort):
+ - From the HTTP call position, scan backward (max 1000 chars) for the nearest function definition pattern: `function\s+\w+\s*\(`, `async\s+function\s+\w+\s*\(`, `\w+\s*\([^)]*\)\s*\{` (class method shorthand), or `\w+\s*=\s*(?:async\s+)?(?:function)?\s*\(` (arrow/assigned). Do NOT use bare `\w+\s*\(` — it matches ordinary function calls like `console.log(`, `JSON.stringify(` and will almost always false-positive
+ - Extract the parameter list
+ - Positional args without `=` → required
+ - Destructured args `{ key = default }` → optional
+ - If no function definition is found within 1000 chars, or correlation fails, default all to `required: false`
+ - **Critical: use positional matching** — if content has multiple HTTP calls, each endpoint must only extract params from its own call, not cross-product with others
+
+ **Patterns to follow:**
+ - Inline SDK content strings in tests (see `patterns_test.go`)
+ - `t.Parallel()` on every test and subtest
+ - `testify/assert` for assertions
+ - Include negative fixtures: lines that should NOT produce params (comments, string literals in non-SDK code)
+
+ **Test scenarios:**
+ - Happy path: Single-line params `this.get('/path', { key1: val1, key2: val2 })` → extracts key1 and key2
+ - Happy path: Multi-line params spanning 4 lines → extracts all keys
+ - Happy path: Shorthand property `{ steamid }` → extracts `steamid`
+ - Happy path: Key with `.join(',')` value → type is `"string"`
+ - Happy path: Key with boolean value `true`/`false` → type is `"boolean"`
+ - Happy path: Key with numeric default → type is `"integer"`
+ - Happy path: Key name `count` or `limit` → type is `"integer"`
+ - Happy path: Function signature `fn(steamid)` with matching param → `required: true`
+ - Happy path: Destructured `fn(id, { opt1 = true } = {})` → `id` required, `opt1` optional
+ - Edge case: No params object after URL → returns endpoint with nil params (not empty slice — consistent with GrepEndpoints zero-value)
+ - Edge case: Nested object in params `{ filter: { type: "active" } }` → extracts `filter` only (depth 1), ignores nested keys
+ - Edge case: Trailing comma after last param → handles correctly
+ - Edge case: Content with multiple HTTP calls → each endpoint gets only its own params, no cross-product
+ - Edge case: Multi-line call with whitespace/newlines between URL and params `this.get('/path',\n { key: val })` → still extracts params correctly
+ - Error path: Malformed object literal (unbalanced braces) → returns endpoint without params, does not panic
+ - Integration: `GrepEndpoints` then `EnrichWithParams` on same content → endpoints have both path and params populated
+ - Negative: Existing `GrepEndpoints` behavior unchanged — calling `GrepEndpoints` alone still returns endpoints with nil params
+
+ **Verification:**
+ - `go test ./internal/crowdsniff/...` passes
+ - Tests cover the Steam SDK patterns from the gap artifact (`.get('/path', { steamid: steamids.join(',') })` and `getOwnedGames(steamid, { includeAppInfo = true } = {})`)
+
+---
+
+- [x] **Unit 3: Param merging in aggregation**
+
+ **Goal:** Extend `Aggregate()` to merge params across sources using union-merge, preferring metadata from higher-tier sources.
+
+ **Requirements:** P5
+
+ **Dependencies:** Units 1, 2
+
+ **Files:**
+ - Modify: `internal/crowdsniff/aggregate.go`
+ - Modify: `internal/crowdsniff/aggregate_test.go`
+
+ **Approach:**
+ - Extend the `accumulator` struct to hold `params map[string]paramEntry` where `paramEntry` is `struct{ param DiscoveredParam; tier string }` — keyed by param name, tier tracks the source that contributed each param
+ - During aggregation, for each endpoint's params:
+ - If param name not yet seen → add it with the endpoint's `SourceTier`
+ - If param name already exists → compare `tierRank(incoming)` vs `tierRank(stored)`: higher tier wins. If same tier, keep the one with more fields populated (non-empty type, has default, etc.)
+ - Convert the accumulated params map to a sorted `[]DiscoveredParam` on the `AggregatedEndpoint` (sort by name for deterministic output — this is a core correctness requirement for reproducible YAML, not just an edge case)
+ - Per-file dedup in `GrepEndpoints` uses `{method, path, sourceName}` as key — params ride along on the winning entry. If two files in the same SDK define the same endpoint with different param sets, only the first-walked file's params are kept. This is an accepted minor data-quality trade-off
+ - `deduplicateEndpoints` key is unchanged — params do not affect dedup
+
+ **Patterns to follow:**
+ - Existing `Aggregate()` accumulator pattern (bestTier, sources map)
+ - Table-driven tests in `aggregate_test.go`
+
+ **Test scenarios:**
+ - Happy path: Two sources discover same endpoint, source A has `{steamid: string}`, source B has `{steamid: string, count: integer}` → aggregated has both params (union)
+ - Happy path: Same param from official-sdk (type: string) and code-search (type: integer) → keeps official-sdk's version
+ - Happy path: One source has params, other has none → params from the source that has them are preserved
+ - Happy path: Params are sorted alphabetically by name in output (deterministic YAML)
+ - Edge case: Same param name from same-tier sources with different types → first-seen wins (deterministic)
+ - Edge case: Endpoints with no params from any source → AggregatedEndpoint.Params is nil
+ - Negative: Existing aggregate tests still pass unchanged (endpoints without params work exactly as before)
+
+ **Verification:**
+ - `go test ./internal/crowdsniff/...` passes
+ - Existing `TestAggregate` subtests pass without modification
+
+---
+
+- [x] **Unit 4: Wire params through spec generation and npm source**
+
+ **Goal:** Map `DiscoveredParam` → `spec.Param` in `BuildSpec()` and call `EnrichWithParams` in the npm source after `GrepEndpoints`.
+
+ **Requirements:** P5, P7
+
+ **Dependencies:** Units 1, 2, 3
+
+ **Files:**
+ - Modify: `internal/crowdsniff/specgen.go`
+ - Modify: `internal/crowdsniff/specgen_test.go`
+ - Modify: `internal/crowdsniff/npm.go`
+ - Modify: `internal/crowdsniff/npm_test.go`
+ - Modify: `internal/cli/crowd_sniff.go` (add `param_count` to JSON output)
+ - Modify: `internal/cli/crowd_sniff_test.go` (add param flow integration test)
+
+ **Approach:**
+ - **specgen.go**: In `BuildSpec()`, after constructing `spec.Endpoint`, map each `AggregatedEndpoint.Params` entry to a `spec.Param{Name, Type, Required, Default}`. Set `Positional: false` (these are query params, not path params). Set `Description: ""` (per scope boundary). Note: `DiscoveredParam.Default` is `string`, `spec.Param.Default` is `any` — assign the string directly, no type coercion.
+ - **npm.go**: In `processPackageTarball`, inside the `filepath.Walk` callback, after `GrepEndpoints(string(content), pkgName, tier)`, call `EnrichWithParams(string(content), endpoints)` while `content` is still in scope. This enriches per-file, before appending to `allEndpoints`; cross-source dedup happens later in `Aggregate()`.
+ - **crowd_sniff.go**: Add `param_count` to the `--json` output struct. Compute by summing `len(endpoint.Params)` across all spec endpoints. Low-cost additive change that gives downstream tooling (retro skill, provenance) visibility into param discovery quality.
+ - **crowd_sniff_test.go**: Add one test case with a mock source returning endpoints with params, assert the written spec YAML contains param entries. The existing mock tests pass without modification but don't verify param flow through the CLI layer.
+ - **No changes to github.go** — per P7, GitHub source is out of scope for param extraction.
+
+ **Patterns to follow:**
+ - `websniff/specgen.go:inferURLParams()` for how `spec.Param` is constructed
+ - Existing `BuildSpec` field mapping pattern (Method, Path, Description, Meta)
+
+ **Test scenarios:**
+ - Happy path: `BuildSpec` with aggregated endpoints that have params → `spec.Endpoint.Params` populated with correct Name, Type, Required values
+ - Happy path: Param type mapping: DiscoveredParam{Type: "integer"} → spec.Param{Type: "integer"}
+ - Happy path: npm source returns endpoints with params populated (end-to-end with mock tarball containing SDK code with params)
+ - Edge case: AggregatedEndpoint with nil Params → spec.Endpoint.Params is nil (not empty slice)
+ - Edge case: Mix of endpoints with and without params in same spec → endpoints without params serialize as `params: []` (existing behavior — `spec.Endpoint.Params` YAML tag lacks `omitempty`)
+ - Happy path: `--json` output includes `param_count` field with correct total
+ - Happy path: CLI integration — mock source with params → written spec YAML contains `params:` entries
+ - Negative: GitHub source endpoints still have no params (verify github_test.go still passes)
+ - Negative: Existing specgen tests pass without modification
+ - Integration: Full pipeline — mock npm tarball with Steam-like SDK code → crowd-sniff produces spec with both endpoints AND params populated
+
+ **Verification:**
+ - `go test ./internal/crowdsniff/...` passes
+ - `go test ./...` passes (no regressions anywhere)
+ - `go build -o ./printing-press ./cmd/printing-press` succeeds
+
+## System-Wide Impact
+
+- **Spec output changes**: Generated crowd-sniff spec YAML files will now include `params:` sections on endpoints where params were discovered. This is purely additive — existing consumers (generate, mergeSpecs, templates) already handle `Params` from other spec sources (OpenAPI parser, websniff).
+- **No generator template changes**: Templates already iterate `endpoint.Params` when present. crowd-sniff endpoints that previously had empty params now have populated params — the templates handle this naturally.
+- **Minor CLI output addition**: `--json` output gains a `param_count` field (additive, non-breaking). Terminal summary unchanged. Params are visible in the output spec YAML.
+- **Unchanged invariants**: `GrepEndpoints` return contract is unchanged (params field is nil when `EnrichWithParams` is not called). Existing endpoint dedup logic is unaffected. GitHub source is unmodified. All existing tests pass.
+
+## Risks & Dependencies
+
+| Risk | Mitigation |
+|------|------------|
+| Brace-matching scanner breaks on edge cases (template literals with `${}`, regex literals with `{n}`) | Cap scan distance at 500 chars from URL match. SDK HTTP calls typically have params within 200 chars. Test with diverse real-world SDK samples. |
+| Raw-vs-cleaned path mismatch — endpoint.Path contains `{id}` but source code contains `${userId}` | EnrichWithParams must independently scan content with raw regex patterns, not substring-match the cleaned endpoint.Path. See Key Technical Decisions. Most likely implementation bug if missed. |
+| Cross-product bug — params from one HTTP call attributed to another endpoint | Use positional matching (match endpoint's specific path in content to locate the right call). Include multi-call test fixtures. Known bug pattern in this codebase (cartesian product learnings doc). |
+| Function signature correlation is unreliable for complex code | Default to `required: false` when correlation fails. This is the safe default — false-optional is inconvenient, false-required breaks the CLI. |
+| Type inference heuristics are wrong for some APIs | Heuristics are conservative (default to `"string"`). Wrong type on a query param degrades UX slightly but doesn't break functionality. |
+| Multi-line scanning introduces quadratic behavior on large files | Bound the scan: max 500 chars forward from URL match, max 1000 chars backward for function signature. SDK source files are typically 5-50KB. |
+
+## Sources & References
+
+- **Origin document:** [docs/brainstorms/2026-03-29-crowd-sniff-requirements.md](docs/brainstorms/2026-03-29-crowd-sniff-requirements.md)
+- **Gap analysis:** `~/printing-press/manuscripts/postman-explore/20260330-105847/proofs/2026-03-30-crowd-sniff-param-discovery-gap.md`
+- **Original crowd-sniff plan:** [docs/plans/2026-03-29-003-feat-crowd-sniff-plan.md](docs/plans/2026-03-29-003-feat-crowd-sniff-plan.md)
+- **Cartesian product learning:** `docs/solutions/best-practices/multi-source-api-discovery-design-2026-03-30.md` (Pattern 6)
+- **Word-boundary learning:** `docs/solutions/logic-errors/scorecard-accuracy-broadened-pattern-matching-2026-03-27.md`
+- Related code: `internal/crowdsniff/`, `internal/websniff/specgen.go:inferURLParams()`, `internal/spec/spec.go:Param`
diff --git a/docs/retros/2026-03-30-postman-explore-retro.md b/docs/retros/2026-03-30-postman-explore-retro.md
new file mode 100644
index 00000000..3f4c7467
--- /dev/null
+++ b/docs/retros/2026-03-30-postman-explore-retro.md
@@ -0,0 +1,202 @@
+# Printing Press Retro: Postman Explore
+
+## Session Stats
+- API: Postman Explore (public API network directory)
+- Spec source: Catalog (sniffed via crowd-sniff, proxy-envelope client pattern)
+- Scorecard: 76 -> 85/100 Grade A
+- Verify pass rate: 62% (9/24 failures are positional-arg commands the verifier can't test)
+- Manual code edits: 7 (serviceForPath routing, sync rewrite, DB path, FTS table wiring, envelope unwrap, command restructuring, description rewrite)
+- Features built from scratch: 17 (7 top-level commands + 10 transcendence features)
+- Dead code removed: 3 functions (classifyDeleteError, firstNonEmpty, printOutputFiltered); 7 dogfood false positives
+- Time to ship: ~45m
+
+## Findings
+
+### 1. Sync Hardcodes Cursor Pagination (assumption mismatch)
+- **What happened:** Generated sync assumes cursor-based pagination (`after` param) but Postman uses offset-based (`offset` integer param). Teams endpoint has no pagination at all -- sync looped infinitely.
+- **Root cause:** `internal/generator/templates/sync.go.tmpl` -- `determinePaginationDefaults()` hardcodes `cursorParam: "after"`. The spec already declares `Pagination.Type` (cursor/offset/page_token) per endpoint but the sync template ignores it.
+- **Cross-API check:** Standard REST (Stripe): uses cursor -- current template works. Sniffed API (Postman): uses offset -- breaks. Different auth (GitHub): uses page/per_page -- also breaks. This hits any API not using cursor pagination.
+- **Frequency:** Most APIs. Offset pagination is at least as common as cursor pagination. The spec struct already has `Pagination.Type` field -- the data is there, just unused.
+- **Fallback if machine doesn't fix it:** Claude rewrites the entire sync command from scratch (high cost, ~15 min, error-prone -- the teams infinite loop proves it).
+- **Tradeoff:** freq(3) x fallback(3) / effort(2) + risk(1) = 3.0. The spec already carries pagination metadata. The sync template just needs to branch on it. Low regression risk because cursor stays the default.
+- **Inherent or fixable:** Fixable. The spec's `Pagination` struct has `Type`, `CursorParam`, `LimitParam`. The sync template should read these instead of hardcoding.
+- **Durable fix:** In `sync.go.tmpl`, change `determinePaginationDefaults()` to read from the spec's pagination config. When `Type == "offset"`, use offset+limit loop. When no pagination exists on the endpoint, fetch once (no loop). When `Type == "cursor"`, use current logic.
+- **Test:** Generate from postman-explore spec -> sync uses offset and terminates. Generate from Stripe-like spec -> sync still uses cursor. Generate for single-page endpoint -> no loop.
+- **Evidence:** Complete sync.go rewrite required; teams endpoint looped infinitely before fix.
+
+### 2. Proxy serviceForPath Ignores x-proxy-routes (bug)
+- **What happened:** `serviceForPath()` was hardcoded to return the API slug ("postman-explore") for all paths. The proxy needs different service names per path prefix (publishing, search, notebook).
+- **Root cause:** `internal/generator/templates/client.go.tmpl` -- the template already has conditional logic for `.ProxyRoutes` but the OpenAPI parser writes them to `spec.ProxyRoutes` while the generator's template context may not propagate them. The sniffed spec had routes in `x-proxy-routes` extension and the parser reads them (line 153-164 of `internal/openapi/parser.go`), but the generated output had a stub.
+- **Cross-API check:** Standard REST: not affected (no proxy pattern). Sniffed proxy API: breaks every time. Different auth: irrelevant. Affects only proxy-envelope APIs.
+- **Frequency:** API subclass: proxy-envelope. Currently only postman-explore in catalog, but the pattern exists in other sniffed APIs (e.g., any SPA with a backend proxy).
+- **Fallback if machine doesn't fix it:** Claude makes a targeted edit to client.go (~5 min, medium cost). But if missed, CLI ships broken -- every API call goes to wrong service (critical).
+- **Tradeoff:** freq(2) x fallback(4) / effort(1) + risk(0) = 8.0. The template already has the conditional -- this is a wiring fix.
+- **Inherent or fixable:** Fixable. The template has `{{- if .ProxyRoutes}}` logic. Need to verify the spec's `proxy_routes` are being passed through to the template context.
+- **Durable fix:** Verify the generator passes `spec.ProxyRoutes` to the client template context. Add a test: generate from a spec with `proxy_routes` and assert `serviceForPath` contains the route map.
+ - **Condition:** `spec.ClientPattern == "proxy-envelope"` AND `spec.ProxyRoutes` is non-empty
+ - **Guard:** Standard REST APIs skip this entirely (template already guards with `{{if eq .ClientPattern "proxy-envelope"}}`)
+ - **Frequency estimate:** ~10% of catalog APIs, but 100% of proxy-envelope APIs
+- **Test:** Generate from postman-explore spec -> `serviceForPath("/search-all")` returns "search". Generate from standard REST spec -> no serviceForPath function emitted.
+- **Evidence:** First fix applied in session; manual edit to client.go.
+
+### 3. Response Envelope Unwrapping Not Used in Sync (assumption mismatch)
+- **What happened:** Categories and teams sync failed because responses are wrapped in `{"data": [...]}`. The generated sync tried direct array unmarshal.
+- **Root cause:** `internal/generator/templates/sync.go.tmpl` -- `syncResource()` uses `extractPageItems()` which tries `"data"` as a wrapper key, but the custom per-resource sync functions in the final CLI (`syncCategories`, `syncTeams`) were written from scratch because the generic `syncResource` couldn't handle the entity-specific store methods.
+- **Cross-API check:** Standard REST (Stripe): wraps in `{"data": [...]}` -- same problem. GitHub: returns direct arrays -- works. Most modern APIs use envelopes.
+- **Frequency:** Most APIs. The `extractPageItems` helper already handles common wrapper keys. The issue is that the sync template's generic `syncResource` uses it, but the per-endpoint sync code doesn't.
+- **Fallback if machine doesn't fix it:** Claude writes an `unwrapDataArray()` helper (~5 min). Medium cost but easy to forget.
+- **Tradeoff:** freq(3) x fallback(2) / effort(1) + risk(0) = 6.0. The spec's `ResponsePath` field already exists and could be used.
+- **Inherent or fixable:** Fixable. The spec's `Endpoint.ResponsePath` field (e.g., "data") tells the generator exactly where the array lives. The sync template should use it.
+- **Durable fix:** When generating per-resource sync functions, use `Endpoint.ResponsePath` to emit unwrap code. If not set, fall through to the generic `extractPageItems` heuristic.
+- **Test:** Generate from a spec with `response_path: "data"` -> sync unwraps correctly. Generate from a spec without it -> uses heuristic.
+- **Evidence:** `unwrapDataArray()` written by hand; categories and teams failed on first sync attempt.
+
+### 4. Top-Level Commands Not Generated (template gap)
+- **What happened:** Generator produced nested commands (`api list-network-entities`, `search-all search_all`). 7 top-level user-facing commands (search, browse, categories, teams, stats, show, open) had to be written from scratch.
+- **Root cause:** `internal/generator/templates/command_endpoint.go.tmpl` -- derives command names from OpenAPI operationIds and path segments. Produces API-internal structure, not user-facing CLI structure.
+- **Cross-API check:** Standard REST (Stripe): `api list-charges` vs `charges list` -- same gap. Sniffed API: worse because operationIds are often auto-generated. Every API needs user-friendly top-level commands.
+- **Frequency:** Every API. But the fix is complex -- mapping API paths to user-friendly command names requires heuristics that could produce bad names for some APIs.
+- **Fallback if machine doesn't fix it:** Claude writes 5-10 command files from scratch (high cost, ~20 min). The absorb manifest defines the right names, but Claude has to implement each one.
+- **Tradeoff:** freq(4) x fallback(3) / effort(3) + risk(2) = 2.4. High value but high implementation complexity. The heuristic for naming commands from path segments is brittle.
+- **Inherent or fixable:** Partially fixable. The generator could emit top-level aliases for resources with a single entity type. For multi-entity endpoints (like networkentity with entityType enum), the mapping is API-specific. A reasonable middle ground: emit the `api` group as-is AND a set of resource-noun commands that delegate to the `api` commands.
+- **Durable fix:** Add a "command promotion" pass: for each spec resource, if it has a list endpoint, emit a top-level command named after the resource (pluralized). For resources with entity-type enum params, emit one command per enum value. Keep `api *` as escape hatch.
+- **Test:** Generate from postman-explore spec -> verify `browse`, `categories`, `teams` commands exist alongside `api list-*`.
+- **Evidence:** 7 command files written from scratch.
+
+### 5. Entity-Specific Store Methods Not Generated (missing scaffolding)
+- **What happened:** Needed 24 store methods (UpsertCollectionBatch, UpsertCategoryBatch, UpsertTeamBatch, SearchCollections, etc.) and 3 entity-specific FTS tables. Generator emits generic `resources` table + `UpsertBatch`.
+- **Root cause:** `internal/generator/schema_builder.go` already computes entity-specific tables with typed columns and FTS via `BuildSchema()`. `internal/generator/templates/store.go.tmpl` emits these tables in migrations. But the template only generates `Upsert<Entity>` for single-object inserts -- no batch upsert or entity-specific search methods.
+- **Cross-API check:** Every API that uses sync + offline search needs batch upsert and entity-specific FTS queries. The schema builder already detects high-gravity entities.
+- **Frequency:** Every API with store/sync. The schema is generated correctly; only the methods are missing.
+- **Fallback if machine doesn't fix it:** Claude writes batch upsert and search methods per entity (high cost, ~20 min for 3+ entities). Error-prone -- the FTS table was wired wrong on first attempt (queried `resources_fts` instead of `collections_fts`).
+- **Tradeoff:** freq(4) x fallback(3) / effort(2) + risk(1) = 4.0. The schema is already computed. Emitting methods for it is incremental.
+- **Inherent or fixable:** Fixable. The `BuildSchema()` output already knows which tables have FTS. The store template should emit `UpsertBatch<Entity>`, `Search<Entity>`, and `Get<Entity>ByID` for each high-gravity table.
+- **Durable fix:** Extend `store.go.tmpl` to iterate over `Tables` and emit: (1) `Upsert<Entity>Batch` for tables with >3 columns, (2) `Search<Entity>` for tables with FTS5, (3) `Get<Entity>ByID` for all entity tables. The template data already has `Tables` with `FTS5`, `FTS5Fields`, and `Columns`.
+- **Test:** Generate a CLI -> verify `store.go` contains `UpsertCollectionBatch` and `SearchCollections` methods. Verify FTS queries use entity-specific FTS tables.
+- **Evidence:** 24 store methods written by hand; offline search initially used wrong FTS table.
+
+### 6. DB Path Inconsistency (default gap)
+- **What happened:** `defaultDBPath()` in channel_workflow.go returned `~/.config/<cli>/store.db` but sync used `~/.local/share/<cli>/data.db`. Trending/leaderboard silently used empty database.
+- **Root cause:** Two templates (`sync.go.tmpl` and `channel_workflow.go.tmpl`) independently define DB path defaults with different values.
+- **Cross-API check:** Every API. Every CLI with store+workflow has this mismatch.
+- **Frequency:** Every API. Both templates are always emitted when store vision is enabled.
+- **Fallback if machine doesn't fix it:** Claude changes one line in one file (low cost, <1 min).
+- **Tradeoff:** freq(4) x fallback(1) / effort(1) + risk(0) = 4.0. Trivial fix, no regression risk.
+- **Inherent or fixable:** Fixable. Emit one `defaultDBPath()` in `helpers.go.tmpl` and reference it from both templates.
+- **Durable fix:** Add `defaultDBPath()` to `helpers.go.tmpl`. Remove inline path construction from `sync.go.tmpl` and `channel_workflow.go.tmpl`. Use `~/.local/share/<cli>/data.db` as canonical path.
+- **Test:** Generate a CLI -> grep for `UserHomeDir` in all files -> only one definition of DB path.
+- **Evidence:** Manual fix to channel_workflow.go; trending returned "No data" until path aligned.
+
+### 7. Dead Code from Blanket Helper Emission (recurring friction)
+- **What happened:** Dogfood flagged 10 dead functions; 3 truly dead (classifyDeleteError, firstNonEmpty, printOutputFiltered), 7 false positives from incomplete reference tracing.
+- **Root cause:** `helpers.go.tmpl` emits all utility functions unconditionally. `classifyDeleteError` emitted even though spec has no DELETE endpoints. Dogfood's reference tracing misses cross-function calls.
+- **Cross-API check:** Every API. Template emits full helper set regardless of which are needed.
+- **Frequency:** Every API. Typically 3-5 truly dead functions per generation.
+- **Fallback if machine doesn't fix it:** Claude deletes 3-5 functions (low cost, <2 min). Mechanical.
+- **Tradeoff:** freq(4) x fallback(1) / effort(2) + risk(1) = 1.3. Low priority -- the fallback is cheap.
+- **Inherent or fixable:** Partially inherent. Template-based generation will always over-emit. Two complementary fixes: (1) conditional emission for obvious cases (no DELETE -> no classifyDeleteError), (2) post-generation `polish` command that removes unreferenced functions.
+- **Durable fix:** Short-term: add conditionals in `helpers.go.tmpl` for method-specific helpers. Long-term: `printing-press polish --dead-code` post-generation step. Fix dogfood false positive rate separately.
+- **Test:** Generate from a spec with no DELETE endpoints -> `classifyDeleteError` not present. Run polish -> no dead functions remain.
+- **Evidence:** 3 manual deletions.
+
+### 8. Verify Can't Test Positional-Arg Commands (tool limitation)
+- **What happened:** 9/24 commands reported as FAIL because verify runs `<cmd> --dry-run` without positional args. All 9 pass --help and work with correct args.
+- **Root cause:** The verify tool doesn't parse cobra `Use` fields for `<arg>` patterns.
+- **Cross-API check:** Every API with positional-arg commands (most CLIs).
+- **Frequency:** Every API. Typically 30-40% of commands need positional args.
+- **Fallback if machine doesn't fix it:** Verify pass rate is misleadingly low but not actionable -- no fix needed, just noise. Low cost.
+- **Tradeoff:** freq(4) x fallback(1) / effort(2) + risk(0) = 2.0. Improves signal quality but no runtime impact.
+- **Inherent or fixable:** Fixable. Parse `Use` field for `<...>` patterns. Extract placeholder values from `Example` field.
+- **Durable fix:** In the verify tool, parse each command's `Use` field. Supply placeholder values from the first `Example` line or use "test"/"1" defaults.
+- **Test:** Run verify on postman-explore -> all 24 commands pass.
+- **Evidence:** 62% pass rate with 9 false failures.
+
+## Prioritized Improvements
+
+### Tier 1: Do Now
+| # | Fix | Component | Frequency | Fallback Cost | Effort | Guards |
+|---|-----|-----------|-----------|--------------|--------|--------|
+| 6 | Emit single `defaultDBPath()` in helpers.go.tmpl | `internal/generator/templates/helpers.go.tmpl` | every | low (1-liner) | 1 hr | none needed |
+| 2 | Read `x-proxy-routes` into `serviceForPath` | `internal/generator/templates/client.go.tmpl` | subclass:proxy-envelope | critical (broken API) | 2 hrs | `{{if .ProxyRoutes}}` already exists |
+| 1 | Branch sync on `Pagination.Type` (offset/cursor/none) | `internal/generator/templates/sync.go.tmpl` | most | high (full rewrite) | 4 hrs | cursor stays default |
+
+### Tier 2: Plan
+| # | Fix | Component | Frequency | Fallback Cost | Effort | Guards |
+|---|-----|-----------|-----------|--------------|--------|--------|
+| 5 | Generate batch upsert + FTS search methods for entity tables | `internal/generator/templates/store.go.tmpl` | every | high (24+ methods) | 2 days | only for tables with gravity >= 6 |
+| 3 | Use `ResponsePath` for envelope unwrapping in sync | `internal/generator/templates/sync.go.tmpl` | most | medium (helper) | 4 hrs | fallback to heuristic |
+| 4 | Generate top-level command aliases from resources | `internal/generator/templates/` (new) | every | high (5-10 files) | 3 days | keep `api` group as escape hatch |
+
+### Tier 3: Backlog
+| # | Fix | Component | Frequency | Fallback Cost | Effort | Guards |
+|---|-----|-----------|-----------|--------------|--------|--------|
+| 7 | Conditional helper emission + polish command | `internal/generator/templates/helpers.go.tmpl` | every | low (delete 3 funcs) | 1 day | method-specific conditionals |
+| 8 | Verify: parse Use field for positional args | verify tool | every | low (noise only) | 4 hrs | none |
+
+## Work Units
+
+### WU-1: Pagination-Aware Sync Generation (findings #1, #3)
+- **Goal:** Generated sync command correctly handles offset, cursor, and single-page pagination without manual rewrite.
+- **Target files:**
+ - `internal/generator/templates/sync.go.tmpl` -- branch `determinePaginationDefaults()` on spec pagination type
+ - `internal/generator/generator.go` -- pass per-resource pagination metadata to sync template
+ - `internal/profiler/profiler.go` -- ensure pagination type is detected for each syncable resource
+- **Acceptance criteria:**
+ - Generate from postman-explore spec -> sync uses offset+limit for collections, no loop for teams, no loop for categories
+ - Generate from a cursor-paginated API (e.g., Notion) -> sync uses cursor-based pagination (negative test)
+ - Generate for an endpoint with no pagination -> sync fetches once (no infinite loop)
+- **Scope boundary:** Does NOT include response envelope unwrapping (WU-2) or entity-specific store methods (WU-3). Pagination metadata flows from spec to sync template only.
+- **Estimated effort:** 1 day
+
+### WU-2: Response Envelope + Proxy Route Wiring (findings #2, #3)
+- **Goal:** Generated client correctly routes proxy-envelope requests AND sync correctly unwraps response envelopes using spec metadata.
+- **Target files:**
+ - `internal/generator/templates/client.go.tmpl` -- verify ProxyRoutes data flows through
+ - `internal/generator/templates/sync.go.tmpl` -- use `Endpoint.ResponsePath` for unwrapping
+ - `internal/generator/generator.go` -- pass endpoint-level ResponsePath to sync template context
+- **Acceptance criteria:**
+ - Generate from postman-explore spec with proxy-envelope -> `serviceForPath("/search-all")` returns "search"
+ - Generate from postman-explore spec -> sync unwraps `{"data": [...]}` without manual fix
+ - Generate from standard REST spec -> no serviceForPath emitted, no unwrap code (negative test)
+- **Scope boundary:** Does NOT include entity-specific sync functions -- this fixes the generic sync path only.
+- **Estimated effort:** 1 day
+
+### WU-3: Entity-Specific Store Method Generation (finding #5)
+- **Goal:** For high-gravity entities, generate batch upsert and FTS search methods so Claude doesn't write 24 methods from scratch.
+- **Target files:**
+ - `internal/generator/templates/store.go.tmpl` -- add batch upsert, search, get-by-id method generation
+ - `internal/generator/schema_builder.go` -- already computes the needed metadata; may need minor extensions
+- **Acceptance criteria:**
+ - Generate a CLI -> `store.go` contains `UpsertCollectionBatch` for high-gravity tables
+ - Generate a CLI -> `store.go` contains `SearchCollections` using `collections_fts` (not `resources_fts`)
+ - Generate from a minimal spec with low-gravity entities -> only generic `Upsert` methods (negative test)
+- **Scope boundary:** Does NOT include command generation or sync integration -- just the store layer.
+- **Dependencies:** None (store methods are independent of sync/pagination changes)
+- **Estimated effort:** 2 days
+
+### WU-4: DB Path Consolidation (finding #6)
+- **Goal:** Single `defaultDBPath()` definition used by all store-consuming templates.
+- **Target files:**
+ - `internal/generator/templates/helpers.go.tmpl` -- add `defaultDBPath()`
+ - `internal/generator/templates/sync.go.tmpl` -- remove inline path, call `defaultDBPath()`
+ - `internal/generator/templates/channel_workflow.go.tmpl` -- remove inline path, call `defaultDBPath()`
+- **Acceptance criteria:**
+ - Generate a CLI -> `grep -r 'UserHomeDir' internal/cli/` finds exactly one definition
+ - All store-consuming commands use the same path
+- **Scope boundary:** Path consolidation only. Does not change the XDG-standard path choice.
+- **Estimated effort:** 2 hours
+
+## Anti-patterns
+
+- **"The generic sync just works"** -- The generated sync hardcodes cursor pagination, doesn't unwrap response envelopes, and uses a single resource path. It needed a complete rewrite for Postman and will need significant edits for any non-cursor API. The spec already carries the pagination metadata -- the template just ignores it.
+- **"Schema builder output = store completeness"** -- The schema builder correctly generates entity-specific tables with typed columns and FTS, but the store template only emits single-object upsert methods. Having the right schema without the right methods is half the job.
+- **"Dead code is inherent to templates"** -- Some is, but emitting `classifyDeleteError` for an API with zero DELETE endpoints is not inherent -- it's a missing conditional.
+
+## What the Machine Got Right
+
+- **Proxy-envelope client pattern** -- Once serviceForPath was fixed, the entire proxy client (envelope wrapping, rate limiting, retry, dry-run, caching) worked flawlessly. The template is well-designed; the wiring just needed to propagate.
+- **Adaptive rate limiter** -- Handled 429s during teams sync burst with zero manual intervention. The ramp-up/back-off logic is production-quality.
+- **Schema builder gravity scoring** -- `BuildSchema()` correctly identified collections, categories, and teams as high-gravity entities and emitted typed columns with FTS. The data model was right; only the store methods were missing.
+- **Output format infrastructure** -- `--json`, `--compact`, `--csv`, `--select`, `--agent` all worked from generation. The smart default (terminal=table, pipe=JSON) is the correct design.
+- **Quality gate pipeline** -- 7/7 gates passed on first generation. The dogfood + verify + scorecard trio, despite individual limitations, caught every real issue.
+- **Catalog metadata propagation** -- `client_pattern`, `spec_source`, `auth_required` from the catalog entry correctly influenced generation defaults (rate limiting, proxy client). This metadata channel works well.
diff --git a/docs/solutions/best-practices/multi-source-api-discovery-design-2026-03-30.md b/docs/solutions/best-practices/multi-source-api-discovery-design-2026-03-30.md
new file mode 100644
index 00000000..0168f2b2
--- /dev/null
+++ b/docs/solutions/best-practices/multi-source-api-discovery-design-2026-03-30.md
@@ -0,0 +1,167 @@
+---
+title: Multi-Source API Discovery Design Patterns
+date: 2026-03-30
+category: best-practices
+module: crowd-sniff API discovery
+problem_type: best_practice
+component: tooling
+severity: medium
+applies_when:
+ - Aggregating API endpoints from heterogeneous sources (npm, GitHub, etc.)
+ - Implementing resilient multi-source data collection with different auth/rate limits
+ - Designing testable HTTP client patterns for external API integration
+ - Extracting structured data from downloaded archives
+tags:
+ - api-discovery
+ - multi-source-aggregation
+ - error-resilience
+ - http-client-testing
+ - security-hardening
+ - path-normalization
+---
+
+# Multi-Source API Discovery Design Patterns
+
+## Context
+
+Building the `crowd-sniff` command required aggregating API endpoint data from multiple external sources (npm registry, GitHub code search) that have fundamentally different authentication models, rate limits, data formats, and reliability characteristics. No existing codebase pattern covered multi-source external API integration with parallel execution, graceful degradation, and cross-source deduplication. The brainstorm-plan-implement-review pipeline surfaced six reusable patterns.
+
+## Guidance
+
+### 1. Testable HTTP Clients via Injectable Base URLs
+
+Sources accept a configurable `BaseURL` in their options struct. Tests use `httptest.NewServer` and inject its URL. This avoids the anti-pattern in `research.go` where `httptest.NewServer` was created but never used because URLs were hardcoded.
+
+For APIs on multiple hosts (npm uses `registry.npmjs.org` for search and `api.npmjs.org` for downloads), use separate configurable URLs per host:
+
+```go
+type NPMOptions struct {
+ RegistryBaseURL string // defaults to "https://registry.npmjs.org"
+ DownloadsBaseURL string // defaults to "https://api.npmjs.org"
+ HTTPClient *http.Client
+}
+```
+
+### 2. errgroup.Group (not WithContext) for Independent Sources
+
+When the requirement says "each source is optional and independent," `errgroup.WithContext` is wrong. A 401 from GitHub would cancel the npm source via shared context cancellation. Use plain `errgroup.Group` for synchronization only.
+
+Sources never return errors to the group. They log warnings to stderr and return empty results:
+
+```go
+g := new(errgroup.Group)
+g.Go(func() error {
+ result, err := npmSource.Discover(ctx, apiName)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "warning: npm source: %v\n", err)
+ return nil // never propagate
+ }
+ npmResult = result
+ return nil
+})
+```
+
+### 3. Two-Step Path Normalization
+
+SDK code uses inconsistent parameter syntax. Normalization requires two steps:
+
+- **Step 1 — Unify syntax**: `:id`, `{user_id}`, `<id>`, `$id` all become `{id}`
+- **Step 2 — Replace concrete values**: UUIDs, numeric IDs, hashes become `{uuid}`, `{id}`, `{hash}`
+
+websniff only does step 2 (it sees real traffic URLs with concrete values). crowd-sniff needs both because SDK source code uses various parameter syntaxes. Without step 1, `/users/:id` from npm and `/users/{user_id}` from GitHub fail to deduplicate.
+
+### 4. Tarball Extraction Security Checklist
+
+When downloading and extracting archives from external registries:
+
+1. **HTTPS-only URLs** — reject http:// and file:// schemes before downloading
+2. **io.LimitReader at 10MB** — not Content-Length (unreliable with chunked encoding)
+3. **Reject symlinks** — skip `tar.TypeSymlink` and `tar.TypeLink` entries (prevents reading `/etc/passwd` via malicious symlink)
+4. **Path containment** — validate every extracted path resolves within the temp directory root
+5. **Deferred cleanup** — `defer os.RemoveAll(tmpDir)` ensures temp dirs are always removed
+
+### 5. SourceResult Wrapper for Multi-Signal Returns
+
+When sources produce endpoints AND other signals (base URL candidates), wrap them in a result struct:
+
+```go
+type SourceResult struct {
+ Endpoints []DiscoveredEndpoint
+ BaseURLCandidates []string
+}
+```
+
+This was caught during the plan's architecture review — the original interface only returned endpoints, leaving base URL discovery as an architectural gap the CLI command would have to work around.
+
+### 6. Document Review Catches Real Bugs
+
+The `ce:review` correctness reviewer found a cartesian product bug where `extractMethodCallEndpoints` paired every HTTP method with every URL path on the same line (cross-product of N methods x M paths). The fix: use `FindAllStringSubmatchIndex` for positional matching — find the first path after each method call's position, not all paths globally.
+
+This confirms that dedicated correctness review with structured reviewer personas catches category errors that escape typical code review.
+
+## Why This Matters
+
+These patterns compound. The next external API integration (Postman source when that CLI ships, or any future multi-source feature) can follow these patterns directly. The testable HTTP client pattern alone would have saved debugging time on the existing `research.go` code if it had been established first.
+
+The security checklist for tarball extraction is especially important — npm packages are user-uploaded content. Every archive extraction is an attack surface.
+
+## When to Apply
+
+- Any feature querying multiple external APIs in parallel
+- Any code downloading and extracting archives from untrusted sources
+- Any aggregation pipeline deduplicating results from heterogeneous sources
+- Any HTTP client code that needs test coverage without hitting real APIs
+
+## Examples
+
+### Before (research.go anti-pattern)
+
+```go
+// httptest.NewServer created in test but never used because URLs are hardcoded
+func fetchReadme(apiName string) (string, error) {
+ resp, err := http.Get("https://api.github.com/repos/...")
+ // ...
+}
+```
+
+### After (crowd-sniff pattern)
+
+```go
+// BaseURL configurable; fully testable with httptest.NewServer
+source := NewNPMSource(NPMOptions{RegistryBaseURL: server.URL})
+result, err := source.Discover(ctx, "notion")
+```
+
+### Before (errgroup.WithContext — wrong for independent sources)
+
+```go
+g, ctx := errgroup.WithContext(cmd.Context())
+g.Go(func() error {
+ // GitHub 401 cancels ctx, killing npm mid-flight
+ return githubSource.Discover(ctx, name)
+})
+```
+
+### After (errgroup.Group — sources are independent)
+
+```go
+g := new(errgroup.Group)
+g.Go(func() error {
+ result, err := githubSource.Discover(ctx, name)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "warning: %v\n", err)
+ return nil // never cancel other sources
+ }
+ githubResult = result
+ return nil
+})
+```
+
+## Related
+
+- `docs/solutions/security-issues/filepath-join-traversal-with-user-input-2026-03-29.md` — path traversal protection pattern (applied in tarball extraction)
+- `docs/solutions/best-practices/validation-must-not-mutate-source-directory-2026-03-29.md` — temp dir cleanup pattern (applied in npm source)
+- `docs/solutions/best-practices/adaptive-rate-limiting-sniffed-apis.md` — rate limiting for generated CLIs (related: GitHub code search uses 6s ticker for 10 req/min)
+- `docs/solutions/best-practices/checkout-scoped-printing-press-output-layout-2026-03-28.md` — output path conventions (crowd-sniff defaults to `~/.cache/printing-press/crowd-sniff/`)
+- `docs/brainstorms/2026-03-29-crowd-sniff-requirements.md` — origin requirements document
+- `docs/plans/2026-03-29-003-feat-crowd-sniff-plan.md` — implementation plan
diff --git a/docs/solutions/best-practices/sniff-and-crowd-sniff-complementary-discovery-2026-03-30.md b/docs/solutions/best-practices/sniff-and-crowd-sniff-complementary-discovery-2026-03-30.md
new file mode 100644
index 00000000..4b52e596
--- /dev/null
+++ b/docs/solutions/best-practices/sniff-and-crowd-sniff-complementary-discovery-2026-03-30.md
@@ -0,0 +1,132 @@
+---
+title: "Sniff and Crowd-Sniff: Complementary API Discovery for CLI Generation"
+date: 2026-03-30
+category: best-practices
+module: API discovery pipeline
+problem_type: best_practice
+component: tooling
+severity: medium
+applies_when:
+ - Generating a CLI for an API that lacks a published OpenAPI spec
+ - Deciding whether to use sniff, crowd-sniff, or both during Phase 1
+ - Evaluating which discovery method produces better CLI coverage
+tags:
+ - api-discovery
+ - sniff
+ - crowd-sniff
+ - pipeline-strategy
+ - spec-generation
+---
+
+# Sniff and Crowd-Sniff: Complementary API Discovery for CLI Generation
+
+## Context
+
+Printing-press generates CLIs from API specs. Many public APIs don't publish specs. Two discovery commands fill this gap — `sniff` (Phase 1.7) and `crowd-sniff` (Phase 1.8) — but they discover fundamentally different things and work best together.
+
+Understanding when to use each, and why they're complementary rather than competing, is essential for producing the best CLI coverage.
+
+## Guidance
+
+### What each discovers
+
+**Sniff** browses a live web app headlessly, captures HTTP traffic, and reverse-engineers a spec from observed requests/responses. It sees what the web application does.
+
+**Crowd-sniff** searches npm SDKs and GitHub code to find what developers have already mapped. It sees what developers need.
+
+These are different sets:
+
+| Signal | Sniff | Crowd-sniff |
+|--------|-------|-------------|
+| Source | One browsing session | Thousands of developers |
+| What it finds | What the web app does | What developers need |
+| Auth patterns | Cookie/session (often unusable for CLI) | API keys, bearer tokens (CLI-ready) |
+| Coverage | Whatever pages you visit in 60-90s | Whatever SDKs implement |
+| Popularity signal | No | Yes (frequency across GitHub repos) |
+| Parameter types | Inferred from response bodies | Declared in SDK source code |
+| Works without browser | No | Yes |
+| Response body examples | Yes (from live traffic) | No (only endpoint paths and methods) |
+
+### When to use each
+
+**Use sniff alone** when:
+- The API has no SDKs on npm (rare for popular APIs)
+- The web app is the primary interface and you want to capture exactly what it does
+- You need response body examples for richer spec generation
+
+**Use crowd-sniff alone** when:
+- Browser automation isn't available or is unreliable
+- The API requires login (sniff skips auth-required sites)
+- Speed matters (crowd-sniff runs without a browser, typically 2-4 minutes)
+- You want popularity-weighted endpoint coverage
+
+**Use both** when:
+- Building a GOAT CLI (this is the default recommendation)
+- The API has both a web app and published SDKs
+- You want maximum coverage: web-app endpoints AND developer-used endpoints
+
+### How they combine in the pipeline
+
+```
+Phase 1.7: Sniff Gate
+ → Produces sniff-spec.yaml (endpoints from live traffic)
+
+Phase 1.8: Crowd Sniff Gate
+ → Produces crowd-spec.yaml (endpoints from npm + GitHub)
+
+Phase 2: Generate
+ → printing-press generate --spec original.yaml --spec sniff-spec.yaml --spec crowd-spec.yaml
+```
+
+The `mergeSpecs()` function flattens all endpoints from all specs into one CLI. Collision handling prefixes duplicate resource names with the source spec name. The result: a CLI that covers what the API documents, what the web app does, AND what developers use.
+
+### Why crowd-sniff's popularity signal matters for CLI quality
+
+A spec treats all endpoints equally. Crowd-sniff doesn't — an endpoint found in 200 GitHub repos is more important than one found in 3. This frequency data is carried as `source_count` in the spec's `Meta` field, making it available to the skill during Phase 3 (Build The GOAT) for prioritization decisions:
+
+- High-frequency endpoints deserve better descriptions, examples, and polish
+- Low-frequency endpoints might be omitted from the quick-start guide
+- Zero-frequency endpoints (only in the spec, never seen in real code) might be candidates for exclusion
+
+### The core insight
+
+For any API popular enough to want a CLI for, someone has already mapped it in code. An npm SDK has every endpoint the vendor tested and ships. GitHub code from hundreds of repos shows real-world usage patterns. Crowd-sniff turns this existing community knowledge into a structured spec — no browsing, no traffic capture, no manual documentation.
+
+Sniff captures the API's behavior. Crowd-sniff captures the community's intent. Together they produce the most complete picture available.
+
+## Why This Matters
+
+Without sniff or crowd-sniff, an API without a published spec is a dead end for CLI generation. With both, printing-press can generate a CLI for virtually any public REST API — the two discovery methods close the gap between "APIs that document themselves" and "APIs that don't."
+
+The complementary nature also improves quality beyond coverage:
+- Sniff provides response body examples that crowd-sniff can't (it only finds paths and methods)
+- Crowd-sniff provides auth patterns that sniff can't (web apps use cookies; SDKs use API keys)
+- Cross-source agreement (an endpoint found by both sniff and crowd-sniff) is a strong confidence signal
+
+## When to Apply
+
+- Every time you run the printing-press skill for an API without a published spec
+- When evaluating whether a generated CLI has sufficient endpoint coverage
+- When the skill asks whether to run sniff or crowd-sniff — the answer is usually "both"
+- When adding new discovery methods to the pipeline (they should be complementary, not replacing)
+
+## Examples
+
+### API with published spec + SDK (e.g., Notion)
+
+Crowd-sniff's `@notionhq/client` SDK produces endpoints that closely match the official spec. Sniff captures the web app's internal API calls (some of which use different paths or additional endpoints not in the public API). Using both reveals the gap between the public API and the internal API.
+
+### API with no spec, no SDK (e.g., obscure SaaS)
+
+Sniff is the primary discovery method — browse the web app and capture what it does. Crowd-sniff may find GitHub code snippets from users calling the API directly, but coverage will be thinner. The skill falls back to `--docs` generation if neither produces results.
+
+### API with SDKs but no web app (e.g., infrastructure APIs)
+
+Crowd-sniff is the primary method — the SDK maps the entire API surface. Sniff has nothing to browse. This is where crowd-sniff's value is most clear: it turns published SDK code into a CLI spec without any human intervention.
+
+## Related
+
+- `docs/solutions/best-practices/multi-source-api-discovery-design-2026-03-30.md` — technical design patterns used in crowd-sniff (testable HTTP clients, errgroup, path normalization, tarball security)
+- `docs/solutions/best-practices/adaptive-rate-limiting-sniffed-apis.md` — rate limiting for CLIs generated from sniffed specs
+- `docs/brainstorms/2026-03-29-crowd-sniff-requirements.md` — origin requirements document
+- `docs/plans/2026-03-29-003-feat-crowd-sniff-plan.md` — implementation plan
diff --git a/go.mod b/go.mod
index a0ddccf9..9338cf5b 100644
--- a/go.mod
+++ b/go.mod
@@ -6,6 +6,7 @@ require (
github.com/getkin/kin-openapi v0.133.0
github.com/spf13/cobra v1.10.2
github.com/stretchr/testify v1.11.1
+ golang.org/x/sync v0.20.0
golang.org/x/text v0.35.0
gopkg.in/yaml.v3 v3.0.1
)
diff --git a/go.sum b/go.sum
index 01ecb2f2..8cebd941 100644
--- a/go.sum
+++ b/go.sum
@@ -43,6 +43,8 @@ github.com/ugorji/go/codec v1.2.7/go.mod h1:WGN1fab3R1fzQlVQTkfxVtIBhWDRqOviHU95
github.com/woodsbury/decimal128 v1.3.0 h1:8pffMNWIlC0O5vbyHWFZAt5yWvWcrHA+3ovIIjVWss0=
github.com/woodsbury/decimal128 v1.3.0/go.mod h1:C5UTmyTjW3JftjUFzOVhC20BEQa2a4ZKOB5I6Zjb+ds=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
+golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
+golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
diff --git a/internal/cli/crowd_sniff.go b/internal/cli/crowd_sniff.go
new file mode 100644
index 00000000..2e2a3786
--- /dev/null
+++ b/internal/cli/crowd_sniff.go
@@ -0,0 +1,204 @@
+package cli
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/url"
+ "os"
+ "path/filepath"
+ "strings"
+
+ "github.com/mvanhorn/cli-printing-press/internal/crowdsniff"
+ "github.com/mvanhorn/cli-printing-press/internal/websniff"
+ "github.com/spf13/cobra"
+ "golang.org/x/sync/errgroup"
+)
+
+// crowdSniffSource is the interface for discovery sources, enabling test injection.
+type crowdSniffSource interface {
+ Discover(ctx context.Context, apiName string) (crowdsniff.SourceResult, error)
+}
+
+// crowdSniffOptions holds injectable dependencies for testing.
+type crowdSniffOptions struct {
+ sources []crowdSniffSource
+ stdout io.Writer
+ stderr io.Writer
+}
+
+func newCrowdSniffCmd() *cobra.Command {
+ return newCrowdSniffCmdWithOptions(crowdSniffOptions{})
+}
+
+func newCrowdSniffCmdWithOptions(opts crowdSniffOptions) *cobra.Command {
+ var apiName string
+ var outputPath string
+ var baseURL string
+ var asJSON bool
+
+ cmd := &cobra.Command{
+ Use: "crowd-sniff",
+ Short: "Discover API endpoints from npm SDKs and GitHub code search",
+ Long: `Discover API endpoints by mining community signals: npm SDK packages
+and GitHub code search. Produces a spec YAML compatible with 'printing-press generate'.
+
+Complements 'sniff' (which discovers from live web traffic) by finding
+what developers have already mapped in published packages and code.`,
+ RunE: func(cmd *cobra.Command, args []string) error {
+ return runCrowdSniff(cmd.Context(), apiName, baseURL, outputPath, asJSON, opts)
+ },
+ }
+
+ cmd.Flags().StringVar(&apiName, "api", "", "API name or domain (e.g., 'notion', 'api.stripe.com')")
+ cmd.Flags().StringVar(&outputPath, "output", "", "Output path for generated spec YAML")
+ cmd.Flags().StringVar(&baseURL, "base-url", "", "Override auto-detected base URL (must be HTTPS)")
+ cmd.Flags().BoolVar(&asJSON, "json", false, "Output as JSON")
+ _ = cmd.MarkFlagRequired("api")
+
+ return cmd
+}
+
+func runCrowdSniff(ctx context.Context, apiName, baseURL, outputPath string, asJSON bool, opts crowdSniffOptions) error {
+ stdout := opts.stdout
+ if stdout == nil {
+ stdout = os.Stdout
+ }
+ stderr := opts.stderr
+ if stderr == nil {
+ stderr = os.Stderr
+ }
+
+ if err := validateCrowdSniffAPIName(apiName); err != nil {
+ return err
+ }
+
+ sources := opts.sources
+ if len(sources) == 0 {
+ sources = []crowdSniffSource{
+ crowdsniff.NewNPMSource(crowdsniff.NPMOptions{}),
+ crowdsniff.NewGitHubSource(crowdsniff.GitHubOptions{}),
+ }
+ }
+
+ results := make([]crowdsniff.SourceResult, len(sources))
+ g := new(errgroup.Group)
+
+ for i, src := range sources {
+ g.Go(func() error {
+ result, err := src.Discover(ctx, apiName)
+ if err != nil {
+ fmt.Fprintf(stderr, "warning: source %d: %v\n", i, err)
+ return nil
+ }
+ results[i] = result
+ return nil
+ })
+ }
+
+ if err := g.Wait(); err != nil {
+ return fmt.Errorf("running sources: %w", err)
+ }
+
+ aggregated, baseURLCandidates := crowdsniff.Aggregate(results)
+
+ if len(aggregated) == 0 {
+ return fmt.Errorf("no endpoints discovered for %q", apiName)
+ }
+
+ resolvedBaseURL := crowdsniff.ResolveBaseURL(baseURL, baseURLCandidates)
+ if resolvedBaseURL == "" {
+ return fmt.Errorf("could not determine base URL for %q; use --base-url to specify", apiName)
+ }
+
+ if !isHTTPS(resolvedBaseURL) {
+ return fmt.Errorf("base URL must use HTTPS: %s", resolvedBaseURL)
+ }
+
+ apiSpec, err := crowdsniff.BuildSpec(apiName, resolvedBaseURL, aggregated)
+ if err != nil {
+ return fmt.Errorf("building spec: %w", err)
+ }
+
+ if outputPath == "" {
+ outputPath = defaultCrowdSniffCachePath(apiName)
+ }
+
+ if err := websniff.WriteSpec(apiSpec, outputPath); err != nil {
+ return fmt.Errorf("writing spec: %w", err)
+ }
+
+ endpointCount := 0
+ paramCount := 0
+ for _, resource := range apiSpec.Resources {
+ endpointCount += len(resource.Endpoints)
+ for _, ep := range resource.Endpoints {
+ paramCount += len(ep.Params)
+ }
+ }
+
+ tierCounts := make(map[string]int)
+ for _, ep := range aggregated {
+ tierCounts[ep.SourceTier]++
+ }
+
+ if asJSON {
+ return json.NewEncoder(stdout).Encode(map[string]interface{}{
+ "spec_path": outputPath,
+ "endpoints": endpointCount,
+ "resources": len(apiSpec.Resources),
+ "param_count": paramCount,
+ "tier_breakdown": tierCounts,
+ })
+ }
+
+ fmt.Fprintf(stdout, "Spec written to %s (%d endpoints across %d resources)\n", outputPath, endpointCount, len(apiSpec.Resources))
+ if len(tierCounts) > 0 {
+ parts := make([]string, 0, len(tierCounts))
+ for tier, count := range tierCounts {
+ parts = append(parts, fmt.Sprintf("%s: %d", tier, count))
+ }
+ fmt.Fprintf(stdout, "Tiers: %s\n", strings.Join(parts, ", "))
+ }
+ fmt.Fprintf(stdout, "Run 'printing-press generate --spec %s' to build the CLI\n", outputPath)
+ return nil
+}
+
+// validateCrowdSniffAPIName rejects dangerous --api values.
+func validateCrowdSniffAPIName(name string) error {
+ if strings.TrimSpace(name) == "" {
+ return fmt.Errorf("--api value is required")
+ }
+ for _, ch := range name {
+ if ch == '\n' || ch == '\r' || ch == 0 {
+ return fmt.Errorf("--api value contains invalid characters")
+ }
+ }
+ if strings.Contains(name, "..") || strings.ContainsAny(name, `/\`) {
+ // If it looks like a URL (contains ://), allow slashes in the URL path.
+ if !strings.Contains(name, "://") {
+ return fmt.Errorf("--api value contains path traversal characters")
+ }
+ }
+ return nil
+}
+
+func defaultCrowdSniffCachePath(name string) string {
+ // Sanitize name for use in file path.
+ safeName := url.PathEscape(name)
+
+ home, err := os.UserHomeDir()
+ if err != nil || home == "" {
+ return filepath.Join(".cache", "printing-press", "crowd-sniff", safeName+"-spec.yaml")
+ }
+ return filepath.Join(home, ".cache", "printing-press", "crowd-sniff", safeName+"-spec.yaml")
+}
+
+func isHTTPS(rawURL string) bool {
+ parsed, err := url.Parse(rawURL)
+ if err != nil {
+ return false
+ }
+ return strings.EqualFold(parsed.Scheme, "https")
+}
diff --git a/internal/cli/crowd_sniff_test.go b/internal/cli/crowd_sniff_test.go
new file mode 100644
index 00000000..d0571724
--- /dev/null
+++ b/internal/cli/crowd_sniff_test.go
@@ -0,0 +1,439 @@
+package cli
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/mvanhorn/cli-printing-press/internal/crowdsniff"
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+// mockSource implements crowdSniffSource for testing.
+type mockSource struct {
+ result crowdsniff.SourceResult
+ err error
+ calledWith string // captures the apiName argument
+}
+
+func (m *mockSource) Discover(_ context.Context, apiName string) (crowdsniff.SourceResult, error) {
+ m.calledWith = apiName
+ return m.result, m.err
+}
+
+func endpointsResult(baseURL string, endpoints ...crowdsniff.DiscoveredEndpoint) crowdsniff.SourceResult {
+ var candidates []string
+ if baseURL != "" {
+ candidates = []string{baseURL}
+ }
+ return crowdsniff.SourceResult{
+ Endpoints: endpoints,
+ BaseURLCandidates: candidates,
+ }
+}
+
+func TestCrowdSniffCmd_MissingAPIFlag(t *testing.T) {
+ t.Parallel()
+ cmd := newCrowdSniffCmd()
+ cmd.SetArgs([]string{})
+ err := cmd.Execute()
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "required flag")
+}
+
+func TestCrowdSniffCmd_HelpOutput(t *testing.T) {
+ t.Parallel()
+ cmd := newCrowdSniffCmd()
+ cmd.SetArgs([]string{"--help"})
+ err := cmd.Execute()
+ assert.NoError(t, err)
+}
+
+func TestRunCrowdSniff_HappyPath(t *testing.T) {
+ t.Parallel()
+
+ outputDir := t.TempDir()
+ outputPath := filepath.Join(outputDir, "test-spec.yaml")
+ var stdout bytes.Buffer
+
+ src := &mockSource{result: endpointsResult("https://api.example.com",
+ crowdsniff.DiscoveredEndpoint{Method: "GET", Path: "/v1/users", SourceTier: crowdsniff.TierOfficialSDK, SourceName: "sdk"},
+ crowdsniff.DiscoveredEndpoint{Method: "POST", Path: "/v1/users", SourceTier: crowdsniff.TierOfficialSDK, SourceName: "sdk"},
+ )}
+
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{src},
+ stdout: &stdout,
+ stderr: &bytes.Buffer{},
+ }
+
+ err := runCrowdSniff(context.Background(), "example", "https://api.example.com", outputPath, false, opts)
+ require.NoError(t, err)
+
+ assert.Contains(t, stdout.String(), "Spec written to")
+ assert.Contains(t, stdout.String(), "2 endpoints")
+
+ // Verify API name was passed through to source.
+ assert.Equal(t, "example", src.calledWith)
+
+ // Verify spec file was written with correct content.
+ data, err := os.ReadFile(outputPath)
+ require.NoError(t, err)
+ assert.Contains(t, string(data), "base_url: https://api.example.com")
+ assert.Contains(t, string(data), "name: example")
+}
+
+func TestRunCrowdSniff_JSONOutput(t *testing.T) {
+ t.Parallel()
+
+ outputDir := t.TempDir()
+ outputPath := filepath.Join(outputDir, "test-spec.yaml")
+ var stdout bytes.Buffer
+
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{
+ &mockSource{result: endpointsResult("https://api.example.com",
+ crowdsniff.DiscoveredEndpoint{Method: "GET", Path: "/users", SourceTier: crowdsniff.TierCodeSearch, SourceName: "gh"},
+ )},
+ },
+ stdout: &stdout,
+ stderr: &bytes.Buffer{},
+ }
+
+ err := runCrowdSniff(context.Background(), "test", "https://api.example.com", outputPath, true, opts)
+ require.NoError(t, err)
+
+ var jsonOut map[string]interface{}
+ require.NoError(t, json.Unmarshal(stdout.Bytes(), &jsonOut))
+ assert.Equal(t, outputPath, jsonOut["spec_path"])
+ assert.Equal(t, float64(1), jsonOut["endpoints"])
+ assert.Equal(t, float64(1), jsonOut["resources"])
+ tierBreakdown, ok := jsonOut["tier_breakdown"].(map[string]interface{})
+ require.True(t, ok, "tier_breakdown should be a map")
+ assert.Equal(t, float64(1), tierBreakdown[crowdsniff.TierCodeSearch])
+}
+
+func TestRunCrowdSniff_NoEndpointsDiscovered(t *testing.T) {
+ t.Parallel()
+
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{
+ &mockSource{result: crowdsniff.SourceResult{}},
+ },
+ stdout: &bytes.Buffer{},
+ stderr: &bytes.Buffer{},
+ }
+
+ err := runCrowdSniff(context.Background(), "obscure-api", "", "", false, opts)
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "no endpoints discovered")
+}
+
+func TestRunCrowdSniff_NoBaseURL(t *testing.T) {
+ t.Parallel()
+
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{
+ &mockSource{result: crowdsniff.SourceResult{
+ Endpoints: []crowdsniff.DiscoveredEndpoint{
+ {Method: "GET", Path: "/users", SourceTier: crowdsniff.TierCodeSearch, SourceName: "gh"},
+ },
+ // No BaseURLCandidates.
+ }},
+ },
+ stdout: &bytes.Buffer{},
+ stderr: &bytes.Buffer{},
+ }
+
+ err := runCrowdSniff(context.Background(), "test", "", "", false, opts)
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "could not determine base URL")
+ assert.Contains(t, err.Error(), "--base-url")
+}
+
+func TestRunCrowdSniff_NonHTTPSBaseURL(t *testing.T) {
+ t.Parallel()
+
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{
+ &mockSource{result: endpointsResult("http://api.example.com",
+ crowdsniff.DiscoveredEndpoint{Method: "GET", Path: "/users", SourceTier: crowdsniff.TierCodeSearch, SourceName: "gh"},
+ )},
+ },
+ stdout: &bytes.Buffer{},
+ stderr: &bytes.Buffer{},
+ }
+
+ err := runCrowdSniff(context.Background(), "test", "", "", false, opts)
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "base URL must use HTTPS")
+}
+
+func TestRunCrowdSniff_BaseURLFlagOverridesCandidate(t *testing.T) {
+ t.Parallel()
+
+ outputDir := t.TempDir()
+ outputPath := filepath.Join(outputDir, "test-spec.yaml")
+
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{
+ &mockSource{result: endpointsResult("https://wrong.example.com",
+ crowdsniff.DiscoveredEndpoint{Method: "GET", Path: "/users", SourceTier: crowdsniff.TierCodeSearch, SourceName: "gh"},
+ )},
+ },
+ stdout: &bytes.Buffer{},
+ stderr: &bytes.Buffer{},
+ }
+
+ err := runCrowdSniff(context.Background(), "test", "https://correct.example.com", outputPath, false, opts)
+ require.NoError(t, err)
+
+ data, err := os.ReadFile(outputPath)
+ require.NoError(t, err)
+ assert.Contains(t, string(data), "https://correct.example.com")
+ assert.NotContains(t, string(data), "https://wrong.example.com")
+}
+
+func TestRunCrowdSniff_SourceErrorGracefulDegradation(t *testing.T) {
+ t.Parallel()
+
+ outputDir := t.TempDir()
+ outputPath := filepath.Join(outputDir, "test-spec.yaml")
+ var stderr bytes.Buffer
+
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{
+ // First source fails.
+ &mockSource{err: fmt.Errorf("npm registry down")},
+ // Second source succeeds.
+ &mockSource{result: endpointsResult("https://api.example.com",
+ crowdsniff.DiscoveredEndpoint{Method: "GET", Path: "/users", SourceTier: crowdsniff.TierCodeSearch, SourceName: "gh"},
+ )},
+ },
+ stdout: &bytes.Buffer{},
+ stderr: &stderr,
+ }
+
+ err := runCrowdSniff(context.Background(), "test", "", outputPath, false, opts)
+ require.NoError(t, err)
+
+ // Warning logged for failed source.
+ assert.Contains(t, stderr.String(), "warning")
+ assert.Contains(t, stderr.String(), "npm registry down")
+
+ // Spec still written from the successful source with correct content.
+ data, err := os.ReadFile(outputPath)
+ require.NoError(t, err)
+ assert.Contains(t, string(data), "/users")
+ assert.Contains(t, string(data), "https://api.example.com")
+}
+
+func TestRunCrowdSniff_AllSourcesFail(t *testing.T) {
+ t.Parallel()
+
+ var stderr bytes.Buffer
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{
+ &mockSource{err: fmt.Errorf("npm down")},
+ &mockSource{err: fmt.Errorf("github down")},
+ },
+ stdout: &bytes.Buffer{},
+ stderr: &stderr,
+ }
+
+ err := runCrowdSniff(context.Background(), "test", "https://api.example.com", "", false, opts)
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "no endpoints discovered")
+
+ // Both warnings should be logged.
+ assert.Contains(t, stderr.String(), "npm down")
+ assert.Contains(t, stderr.String(), "github down")
+}
+
+func TestRunCrowdSniff_OutputDirCreated(t *testing.T) {
+ t.Parallel()
+
+ outputDir := filepath.Join(t.TempDir(), "nested", "dir")
+ outputPath := filepath.Join(outputDir, "test-spec.yaml")
+
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{
+ &mockSource{result: endpointsResult("https://api.example.com",
+ crowdsniff.DiscoveredEndpoint{Method: "GET", Path: "/users", SourceTier: crowdsniff.TierCodeSearch, SourceName: "gh"},
+ )},
+ },
+ stdout: &bytes.Buffer{},
+ stderr: &bytes.Buffer{},
+ }
+
+ err := runCrowdSniff(context.Background(), "test", "", outputPath, false, opts)
+ require.NoError(t, err)
+
+ _, err = os.Stat(outputPath)
+ assert.NoError(t, err)
+}
+
+func TestRunCrowdSniff_CmdIntegration(t *testing.T) {
+ t.Parallel()
+
+ outputDir := t.TempDir()
+ outputPath := filepath.Join(outputDir, "test-spec.yaml")
+
+ cmd := newCrowdSniffCmdWithOptions(crowdSniffOptions{
+ sources: []crowdSniffSource{
+ &mockSource{result: endpointsResult("https://api.example.com",
+ crowdsniff.DiscoveredEndpoint{Method: "GET", Path: "/v1/users", SourceTier: crowdsniff.TierOfficialSDK, SourceName: "sdk"},
+ )},
+ },
+ stdout: &bytes.Buffer{},
+ stderr: &bytes.Buffer{},
+ })
+
+ cmd.SetArgs([]string{"--api", "example", "--output", outputPath})
+ err := cmd.Execute()
+ require.NoError(t, err)
+
+ _, err = os.Stat(outputPath)
+ assert.NoError(t, err)
+}
+
+func TestValidateCrowdSniffAPIName(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ api string
+ wantErr string
+ }{
+ {name: "valid name", api: "notion", wantErr: ""},
+ {name: "valid domain", api: "api.notion.com", wantErr: ""},
+ {name: "valid URL", api: "https://api.notion.com/v1", wantErr: ""},
+ {name: "empty", api: "", wantErr: "required"},
+ {name: "whitespace only", api: " ", wantErr: "required"},
+ {name: "newline injection", api: "notion\nHost: evil.com", wantErr: "invalid characters"},
+ {name: "null byte", api: "notion\x00evil", wantErr: "invalid characters"},
+ {name: "path traversal", api: "../../.ssh/evil", wantErr: "path traversal"},
+ {name: "backslash traversal", api: `..\..\evil`, wantErr: "path traversal"},
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+ err := validateCrowdSniffAPIName(tt.api)
+ if tt.wantErr == "" {
+ assert.NoError(t, err)
+ } else {
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), tt.wantErr)
+ }
+ })
+ }
+}
+
+func TestDefaultCrowdSniffCachePath(t *testing.T) {
+ t.Parallel()
+
+ path := defaultCrowdSniffCachePath("notion")
+ assert.Contains(t, path, "crowd-sniff")
+ assert.Contains(t, path, "notion-spec.yaml")
+}
+
+func TestRunCrowdSniff_JSONOutput_IncludesParamCount(t *testing.T) {
+ t.Parallel()
+
+ outputDir := t.TempDir()
+ outputPath := filepath.Join(outputDir, "test-spec.yaml")
+ var stdout bytes.Buffer
+
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{
+ &mockSource{result: endpointsResult("https://api.example.com",
+ crowdsniff.DiscoveredEndpoint{
+ Method: "GET",
+ Path: "/v1/games",
+ SourceTier: crowdsniff.TierOfficialSDK,
+ SourceName: "sdk",
+ Params: []crowdsniff.DiscoveredParam{
+ {Name: "steamid", Type: "string", Required: true},
+ {Name: "count", Type: "integer", Required: false},
+ },
+ },
+ crowdsniff.DiscoveredEndpoint{
+ Method: "GET",
+ Path: "/v1/users",
+ SourceTier: crowdsniff.TierOfficialSDK,
+ SourceName: "sdk",
+ Params: []crowdsniff.DiscoveredParam{
+ {Name: "limit", Type: "integer", Required: false},
+ },
+ },
+ )},
+ },
+ stdout: &stdout,
+ stderr: &bytes.Buffer{},
+ }
+
+ err := runCrowdSniff(context.Background(), "test", "https://api.example.com", outputPath, true, opts)
+ require.NoError(t, err)
+
+ var jsonOut map[string]interface{}
+ require.NoError(t, json.Unmarshal(stdout.Bytes(), &jsonOut))
+ // 2 params on first endpoint + 1 param on second endpoint = 3 total
+ assert.Equal(t, float64(3), jsonOut["param_count"], "expected param_count to be 3")
+}
+
+func TestRunCrowdSniff_ParamsInWrittenSpec(t *testing.T) {
+ t.Parallel()
+
+ outputDir := t.TempDir()
+ outputPath := filepath.Join(outputDir, "test-spec.yaml")
+
+ opts := crowdSniffOptions{
+ sources: []crowdSniffSource{
+ &mockSource{result: endpointsResult("https://api.example.com",
+ crowdsniff.DiscoveredEndpoint{
+ Method: "GET",
+ Path: "/v1/games",
+ SourceTier: crowdsniff.TierOfficialSDK,
+ SourceName: "sdk",
+ Params: []crowdsniff.DiscoveredParam{
+ {Name: "steamid", Type: "string", Required: true},
+ {Name: "include_appinfo", Type: "boolean", Required: false, Default: "true"},
+ },
+ },
+ )},
+ },
+ stdout: &bytes.Buffer{},
+ stderr: &bytes.Buffer{},
+ }
+
+ err := runCrowdSniff(context.Background(), "test", "https://api.example.com", outputPath, false, opts)
+ require.NoError(t, err)
+
+ // Read the written spec YAML and verify params are present.
+ data, err := os.ReadFile(outputPath)
+ require.NoError(t, err)
+ specContent := string(data)
+
+ assert.Contains(t, specContent, "params:")
+ assert.Contains(t, specContent, "name: steamid")
+ assert.Contains(t, specContent, "name: include_appinfo")
+ assert.Contains(t, specContent, "type: boolean")
+ assert.Contains(t, specContent, "required: true")
+}
+
+func TestIsHTTPS(t *testing.T) {
+ t.Parallel()
+
+ assert.True(t, isHTTPS("https://api.example.com"))
+ assert.True(t, isHTTPS("HTTPS://API.EXAMPLE.COM"))
+ assert.False(t, isHTTPS("http://api.example.com"))
+ assert.False(t, isHTTPS("ftp://api.example.com"))
+ assert.False(t, isHTTPS(""))
+ assert.False(t, isHTTPS("not-a-url"))
+}
diff --git a/internal/cli/root.go b/internal/cli/root.go
index b0d0f2af..12821515 100644
--- a/internal/cli/root.go
+++ b/internal/cli/root.go
@@ -46,6 +46,7 @@ func Execute() error {
rootCmd.AddCommand(newVersionCmd())
rootCmd.AddCommand(newPrintCmd())
rootCmd.AddCommand(newSniffCmd())
+ rootCmd.AddCommand(newCrowdSniffCmd())
rootCmd.AddCommand(newCatalogCmd())
rootCmd.AddCommand(newLibraryCmd())
rootCmd.AddCommand(newPublishCmd())
diff --git a/internal/crowdsniff/aggregate.go b/internal/crowdsniff/aggregate.go
new file mode 100644
index 00000000..1c4c6317
--- /dev/null
+++ b/internal/crowdsniff/aggregate.go
@@ -0,0 +1,203 @@
+package crowdsniff
+
+import (
+ "regexp"
+ "sort"
+ "strings"
+)
+
+var (
+ // Concrete value patterns (from websniff/classifier.go).
+ uuidSegmentPattern = regexp.MustCompile(`(?i)^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`)
+ hashSegmentPattern = regexp.MustCompile(`(?i)^[0-9a-f]{32,}$`)
+ numericPattern = regexp.MustCompile(`^\d+$`)
+
+ // Parameter syntax patterns — matches :id, {id}, {user_id}, <id>, <userId>, $id.
+ colonParamPattern = regexp.MustCompile(`^:[a-zA-Z_][a-zA-Z0-9_]*$`)
+ bracketParamPattern = regexp.MustCompile(`^\{[a-zA-Z_][a-zA-Z0-9_]*\}$`)
+ angleParamPattern = regexp.MustCompile(`^<[a-zA-Z_][a-zA-Z0-9_]*>$`)
+ dollarParamPattern = regexp.MustCompile(`^\$[a-zA-Z_][a-zA-Z0-9_]*$`)
+)
+
+// tierRank returns a numeric rank for tier ordering (higher = more authoritative).
+func tierRank(tier string) int {
+ switch tier {
+ case TierOfficialSDK:
+ return 4
+ case TierCommunitySDK:
+ return 3
+ case TierCodeSearch:
+ return 2
+ case TierPostman:
+ return 1
+ default:
+ return 0
+ }
+}
+
+// Aggregate deduplicates endpoints from multiple sources, computing the highest
+// source tier and distinct source count for each unique method+path combination.
+// It also collects all base URL candidates from the sources.
+func Aggregate(results []SourceResult) ([]AggregatedEndpoint, []string) {
+ type endpointKey struct {
+ method string
+ path string
+ }
+
+ type accumulator struct {
+ bestTier string
+ sources map[string]struct{} // distinct source names
+ params map[string]paramEntry
+ }
+
+ index := make(map[endpointKey]*accumulator)
+ var order []endpointKey
+ var baseURLs []string
+
+ for _, result := range results {
+ baseURLs = append(baseURLs, result.BaseURLCandidates...)
+
+ for _, ep := range result.Endpoints {
+ method := strings.ToUpper(strings.TrimSpace(ep.Method))
+ path := NormalizePath(ep.Path)
+ key := endpointKey{method: method, path: path}
+
+ acc, exists := index[key]
+ if !exists {
+ acc = &accumulator{
+ sources: make(map[string]struct{}),
+ params: make(map[string]paramEntry),
+ }
+ index[key] = acc
+ order = append(order, key)
+ }
+
+ if tierRank(ep.SourceTier) > tierRank(acc.bestTier) {
+ acc.bestTier = ep.SourceTier
+ }
+ acc.sources[ep.SourceName] = struct{}{}
+
+ // Union-merge params: prefer metadata from higher-tier source.
+ for _, p := range ep.Params {
+ existing, seen := acc.params[p.Name]
+ if !seen {
+ acc.params[p.Name] = paramEntry{param: p, tier: ep.SourceTier}
+ } else if tierRank(ep.SourceTier) > tierRank(existing.tier) {
+ acc.params[p.Name] = paramEntry{param: p, tier: ep.SourceTier}
+ } else if tierRank(ep.SourceTier) == tierRank(existing.tier) && paramFieldCount(p) > paramFieldCount(existing.param) {
+ acc.params[p.Name] = paramEntry{param: p, tier: ep.SourceTier}
+ }
+ }
+ }
+ }
+
+ aggregated := make([]AggregatedEndpoint, 0, len(order))
+ for _, key := range order {
+ acc := index[key]
+ ep := AggregatedEndpoint{
+ Method: key.method,
+ Path: key.path,
+ SourceTier: acc.bestTier,
+ SourceCount: len(acc.sources),
+ }
+ if len(acc.params) > 0 {
+ ep.Params = sortedParams(acc.params)
+ }
+ aggregated = append(aggregated, ep)
+ }
+
+ return aggregated, deduplicateStrings(baseURLs)
+}
+
+// NormalizePath unifies parameter syntax and replaces concrete values with placeholders.
+// Step 1: Convert :id, {user_id}, <id>, $id → {id}
+// Step 2: Replace UUIDs, numeric IDs, long hashes → {id}/{uuid}/{hash}
+func NormalizePath(path string) string {
+ // Strip query string if present.
+ if idx := strings.Index(path, "?"); idx >= 0 {
+ path = path[:idx]
+ }
+
+ parts := strings.Split(path, "/")
+ segments := make([]string, 0, len(parts))
+ for _, segment := range parts {
+ if segment == "" {
+ continue
+ }
+
+ // Step 1: Unify parameter syntax → {id}
+ switch {
+ case colonParamPattern.MatchString(segment):
+ segment = "{id}"
+ case bracketParamPattern.MatchString(segment):
+ segment = "{id}"
+ case angleParamPattern.MatchString(segment):
+ segment = "{id}"
+ case dollarParamPattern.MatchString(segment):
+ segment = "{id}"
+ // Step 2: Replace concrete values
+ case numericPattern.MatchString(segment):
+ segment = "{id}"
+ case uuidSegmentPattern.MatchString(segment):
+ segment = "{uuid}"
+ case hashSegmentPattern.MatchString(segment):
+ segment = "{hash}"
+ }
+ segments = append(segments, segment)
+ }
+
+ normalized := "/" + strings.Join(segments, "/")
+ if normalized == "" {
+ return "/"
+ }
+ return normalized
+}
+
+// paramEntry tracks a discovered param alongside the tier of its source,
+// enabling tier-aware merge when the same param is found by multiple sources.
+type paramEntry struct {
+ param DiscoveredParam
+ tier string
+}
+
+// sortedParams converts the accumulator's param map into a slice sorted by name
+// for deterministic YAML output.
+func sortedParams(m map[string]paramEntry) []DiscoveredParam {
+ params := make([]DiscoveredParam, 0, len(m))
+ for _, entry := range m {
+ params = append(params, entry.param)
+ }
+ sort.Slice(params, func(i, j int) bool {
+ return params[i].Name < params[j].Name
+ })
+ return params
+}
+
+// paramFieldCount returns how many fields are populated on a DiscoveredParam.
+// Used as a tiebreaker when two same-tier sources provide the same param name.
+func paramFieldCount(p DiscoveredParam) int {
+ count := 0
+ if p.Type != "" {
+ count++
+ }
+ if p.Required {
+ count++
+ }
+ if p.Default != "" {
+ count++
+ }
+ return count
+}
+
+func deduplicateStrings(items []string) []string {
+ seen := make(map[string]struct{}, len(items))
+ result := make([]string, 0, len(items))
+ for _, item := range items {
+ if _, exists := seen[item]; exists {
+ continue
+ }
+ seen[item] = struct{}{}
+ result = append(result, item)
+ }
+ return result
+}
diff --git a/internal/crowdsniff/aggregate_test.go b/internal/crowdsniff/aggregate_test.go
new file mode 100644
index 00000000..dc288e13
--- /dev/null
+++ b/internal/crowdsniff/aggregate_test.go
@@ -0,0 +1,323 @@
+package crowdsniff
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+)
+
+func TestNormalizePath(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ path string
+ want string
+ }{
+ {name: "plain path", path: "/v1/users", want: "/v1/users"},
+ {name: "colon param", path: "/users/:id", want: "/users/{id}"},
+ {name: "bracket param", path: "/users/{user_id}", want: "/users/{id}"},
+ {name: "angle param", path: "/users/<userId>", want: "/users/{id}"},
+ {name: "dollar param", path: "/users/$id", want: "/users/{id}"},
+ {name: "numeric ID", path: "/users/123", want: "/users/{id}"},
+ {name: "UUID", path: "/users/550e8400-e29b-41d4-a716-446655440000", want: "/users/{uuid}"},
+ {name: "long hash", path: "/blobs/abc123def456abc123def456abc123def456", want: "/blobs/{hash}"},
+ {name: "strip query string", path: "/users?page=1&limit=10", want: "/users"},
+ {name: "mixed params", path: "/v1/users/:id/posts/{post_id}", want: "/v1/users/{id}/posts/{id}"},
+ {name: "extra slashes", path: "///v1///users///", want: "/v1/users"},
+ {name: "double slashes", path: "/v1//users", want: "/v1/users"},
+ {name: "empty path", path: "", want: "/"},
+ {name: "root", path: "/", want: "/"},
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+ assert.Equal(t, tt.want, NormalizePath(tt.path))
+ })
+ }
+}
+
+func TestAggregate(t *testing.T) {
+ t.Parallel()
+
+ t.Run("two sources same endpoint", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "GET", Path: "/v1/users", SourceTier: TierOfficialSDK, SourceName: "@notionhq/client"},
+ },
+ BaseURLCandidates: []string{"https://api.notion.com"},
+ },
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "GET", Path: "/v1/users", SourceTier: TierCodeSearch, SourceName: "github-code-search"},
+ },
+ BaseURLCandidates: []string{"https://api.notion.com"},
+ },
+ }
+
+ aggregated, baseURLs := Aggregate(results)
+
+ assert.Len(t, aggregated, 1)
+ assert.Equal(t, TierOfficialSDK, aggregated[0].SourceTier)
+ assert.Equal(t, 2, aggregated[0].SourceCount)
+ assert.Equal(t, []string{"https://api.notion.com"}, baseURLs)
+ })
+
+ t.Run("different endpoints from different sources", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "GET", Path: "/v1/users", SourceTier: TierOfficialSDK, SourceName: "sdk"},
+ },
+ },
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "POST", Path: "/v1/users", SourceTier: TierCodeSearch, SourceName: "github"},
+ },
+ },
+ }
+
+ aggregated, _ := Aggregate(results)
+ assert.Len(t, aggregated, 2)
+ })
+
+ t.Run("parameter syntax normalization deduplicates", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "GET", Path: "/users/:id", SourceTier: TierCommunitySDK, SourceName: "npm-sdk"},
+ },
+ },
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "GET", Path: "/users/{user_id}", SourceTier: TierCodeSearch, SourceName: "github"},
+ },
+ },
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "GET", Path: "/users/<id>", SourceTier: TierCodeSearch, SourceName: "github2"},
+ },
+ },
+ }
+
+ aggregated, _ := Aggregate(results)
+ assert.Len(t, aggregated, 1)
+ assert.Equal(t, "/users/{id}", aggregated[0].Path)
+ assert.Equal(t, TierCommunitySDK, aggregated[0].SourceTier)
+ assert.Equal(t, 3, aggregated[0].SourceCount)
+ })
+
+ t.Run("single source", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceName: "github"},
+ {Method: "POST", Path: "/users", SourceTier: TierCodeSearch, SourceName: "github"},
+ },
+ },
+ }
+
+ aggregated, _ := Aggregate(results)
+ assert.Len(t, aggregated, 2)
+ for _, ep := range aggregated {
+ assert.Equal(t, 1, ep.SourceCount)
+ }
+ })
+
+ t.Run("same source same endpoint deduplicates to count 1", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceName: "github"},
+ {Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceName: "github"},
+ },
+ },
+ }
+
+ aggregated, _ := Aggregate(results)
+ assert.Len(t, aggregated, 1)
+ assert.Equal(t, 1, aggregated[0].SourceCount)
+ })
+
+ t.Run("empty results", func(t *testing.T) {
+ t.Parallel()
+ aggregated, baseURLs := Aggregate(nil)
+ assert.Empty(t, aggregated)
+ assert.Empty(t, baseURLs)
+ })
+
+ t.Run("union merge params from two sources", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {
+ Method: "GET", Path: "/v1/users", SourceTier: TierOfficialSDK, SourceName: "sdk",
+ Params: []DiscoveredParam{
+ {Name: "steamid", Type: "string", Required: true},
+ },
+ },
+ },
+ },
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {
+ Method: "GET", Path: "/v1/users", SourceTier: TierCodeSearch, SourceName: "github",
+ Params: []DiscoveredParam{
+ {Name: "steamid", Type: "string"},
+ {Name: "count", Type: "integer"},
+ },
+ },
+ },
+ },
+ }
+
+ aggregated, _ := Aggregate(results)
+ assert.Len(t, aggregated, 1)
+ assert.Len(t, aggregated[0].Params, 2)
+ // Sorted alphabetically
+ assert.Equal(t, "count", aggregated[0].Params[0].Name)
+ assert.Equal(t, "steamid", aggregated[0].Params[1].Name)
+ // steamid from official-sdk (higher tier) wins
+ assert.True(t, aggregated[0].Params[1].Required)
+ })
+
+ t.Run("higher tier param wins on conflict", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {
+ Method: "GET", Path: "/users", SourceTier: TierOfficialSDK, SourceName: "sdk",
+ Params: []DiscoveredParam{
+ {Name: "limit", Type: "string"},
+ },
+ },
+ },
+ },
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {
+ Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceName: "github",
+ Params: []DiscoveredParam{
+ {Name: "limit", Type: "integer"},
+ },
+ },
+ },
+ },
+ }
+
+ aggregated, _ := Aggregate(results)
+ assert.Len(t, aggregated[0].Params, 1)
+ assert.Equal(t, "string", aggregated[0].Params[0].Type) // official-sdk wins
+ })
+
+ t.Run("one source has params other does not", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "GET", Path: "/users", SourceTier: TierOfficialSDK, SourceName: "sdk"},
+ },
+ },
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {
+ Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceName: "github",
+ Params: []DiscoveredParam{
+ {Name: "page", Type: "integer"},
+ },
+ },
+ },
+ },
+ }
+
+ aggregated, _ := Aggregate(results)
+ assert.Len(t, aggregated[0].Params, 1)
+ assert.Equal(t, "page", aggregated[0].Params[0].Name)
+ })
+
+ t.Run("params sorted alphabetically", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {
+ Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceName: "github",
+ Params: []DiscoveredParam{
+ {Name: "zebra", Type: "string"},
+ {Name: "alpha", Type: "string"},
+ {Name: "mid", Type: "string"},
+ },
+ },
+ },
+ },
+ }
+
+ aggregated, _ := Aggregate(results)
+ assert.Equal(t, "alpha", aggregated[0].Params[0].Name)
+ assert.Equal(t, "mid", aggregated[0].Params[1].Name)
+ assert.Equal(t, "zebra", aggregated[0].Params[2].Name)
+ })
+
+ t.Run("same tier same param first seen wins", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {
+ Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceName: "github1",
+ Params: []DiscoveredParam{
+ {Name: "limit", Type: "string"},
+ },
+ },
+ },
+ },
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {
+ Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceName: "github2",
+ Params: []DiscoveredParam{
+ {Name: "limit", Type: "integer"},
+ },
+ },
+ },
+ },
+ }
+
+ aggregated, _ := Aggregate(results)
+ assert.Equal(t, "string", aggregated[0].Params[0].Type) // first seen wins
+ })
+
+ t.Run("no params from any source yields nil", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {
+ Endpoints: []DiscoveredEndpoint{
+ {Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceName: "github"},
+ },
+ },
+ }
+
+ aggregated, _ := Aggregate(results)
+ assert.Nil(t, aggregated[0].Params)
+ })
+
+ t.Run("base URL candidates deduplicated", func(t *testing.T) {
+ t.Parallel()
+ results := []SourceResult{
+ {BaseURLCandidates: []string{"https://api.example.com", "https://api.example.com"}},
+ {BaseURLCandidates: []string{"https://api.example.com", "https://other.example.com"}},
+ }
+
+ _, baseURLs := Aggregate(results)
+ assert.Equal(t, []string{"https://api.example.com", "https://other.example.com"}, baseURLs)
+ })
+}
diff --git a/internal/crowdsniff/github.go b/internal/crowdsniff/github.go
new file mode 100644
index 00000000..6ab36cb3
--- /dev/null
+++ b/internal/crowdsniff/github.go
@@ -0,0 +1,411 @@
+package crowdsniff
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/url"
+ "os"
+ "regexp"
+ "strings"
+ "time"
+)
+
+// GitHubOptions configures the GitHub code search source.
+type GitHubOptions struct {
+ BaseURL string // defaults to "https://api.github.com"
+ HTTPClient *http.Client // defaults to 15s timeout client
+ Token string // defaults to os.Getenv("GITHUB_TOKEN")
+ RecencyCutoff time.Duration // defaults to 180 days
+}
+
+// GitHubSource discovers API endpoints by searching GitHub code.
+type GitHubSource struct {
+ baseURL string
+ client *http.Client
+ token string
+ recencyCutoff time.Duration
+}
+
+// NewGitHubSource creates a new GitHub code search discovery source.
+func NewGitHubSource(opts GitHubOptions) *GitHubSource {
+ baseURL := opts.BaseURL
+ if baseURL == "" {
+ baseURL = "https://api.github.com"
+ }
+ client := opts.HTTPClient
+ if client == nil {
+ client = &http.Client{Timeout: 15 * time.Second}
+ }
+ token := opts.Token
+ if token == "" {
+ token = os.Getenv("GITHUB_TOKEN")
+ }
+ cutoff := opts.RecencyCutoff
+ if cutoff == 0 {
+ cutoff = 180 * 24 * time.Hour
+ }
+ return &GitHubSource{
+ baseURL: strings.TrimRight(baseURL, "/"),
+ client: client,
+ token: token,
+ recencyCutoff: cutoff,
+ }
+}
+
+// Discover searches GitHub code for references to the given API and returns
+// discovered endpoints. If no token is configured, it returns an empty result
+// immediately (graceful degradation).
+func (g *GitHubSource) Discover(ctx context.Context, apiName string) (SourceResult, error) {
+ if g.token == "" {
+ return SourceResult{}, nil
+ }
+
+ queries := buildSearchQueries(apiName)
+
+ var allItems []codeSearchItem
+ ticker := time.NewTicker(6 * time.Second) // 10 req/min rate limit
+ defer ticker.Stop()
+ firstRequest := true
+
+ for _, query := range queries {
+ for page := 1; page <= 10; page++ {
+ // Rate limit: wait before each request (skip the first one).
+ if !firstRequest {
+ select {
+ case <-ctx.Done():
+ return buildResult(allItems, g, ctx), ctx.Err()
+ case <-ticker.C:
+ }
+ }
+ firstRequest = false
+
+ items, total, err := g.searchCode(ctx, query, page)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "warning: github code search page %d: %v\n", page, err)
+ break // move to next query
+ }
+ allItems = append(allItems, items...)
+
+ // Stop if we've seen all results for this query.
+ if page*100 >= total {
+ break
+ }
+ }
+ }
+
+ return buildResult(allItems, g, ctx), nil
+}
+
+// buildSearchQueries returns the GitHub code search queries for the given API name.
+// Domain-like names (containing ".") get exact domain queries; plain names get
+// a broader fallback.
+func buildSearchQueries(apiName string) []string {
+ if strings.Contains(apiName, ".") {
+ return []string{
+ fmt.Sprintf(`"%s" language:javascript`, apiName),
+ fmt.Sprintf(`"%s" language:python`, apiName),
+ }
+ }
+ return []string{
+ fmt.Sprintf(`"%s" api language:javascript`, apiName),
+ fmt.Sprintf(`"%s" api language:python`, apiName),
+ }
+}
+
+// searchCode performs a single code search request and returns items + total count.
+func (g *GitHubSource) searchCode(ctx context.Context, query string, page int) ([]codeSearchItem, int, error) {
+ u := fmt.Sprintf("%s/search/code?q=%s&per_page=100&page=%d",
+ g.baseURL, url.QueryEscape(query), page)
+
+ req, err := http.NewRequestWithContext(ctx, "GET", u, nil)
+ if err != nil {
+ return nil, 0, err
+ }
+ req.Header.Set("Accept", "application/vnd.github.text-match+json")
+ if g.token != "" {
+ req.Header.Set("Authorization", "Bearer "+g.token)
+ }
+
+ resp, err := g.client.Do(req)
+ if err != nil {
+ return nil, 0, err
+ }
+ defer func() { _ = resp.Body.Close() }()
+
+ if resp.StatusCode != http.StatusOK {
+ return nil, 0, fmt.Errorf("code search returned %d", resp.StatusCode)
+ }
+
+ var result codeSearchResponse
+ if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
+ return nil, 0, err
+ }
+ return result.Items, result.TotalCount, nil
+}
+
+// fetchRepoPushedAt gets the pushed_at timestamp for a repo.
+func (g *GitHubSource) fetchRepoPushedAt(ctx context.Context, fullName string) (time.Time, error) {
+ u := fmt.Sprintf("%s/repos/%s", g.baseURL, fullName)
+
+ req, err := http.NewRequestWithContext(ctx, "GET", u, nil)
+ if err != nil {
+ return time.Time{}, err
+ }
+ req.Header.Set("Accept", "application/vnd.github+json")
+ if g.token != "" {
+ req.Header.Set("Authorization", "Bearer "+g.token)
+ }
+
+ resp, err := g.client.Do(req)
+ if err != nil {
+ return time.Time{}, err
+ }
+ defer func() { _ = resp.Body.Close() }()
+
+ if resp.StatusCode != http.StatusOK {
+ return time.Time{}, fmt.Errorf("repo fetch returned %d", resp.StatusCode)
+ }
+
+ var repo repoResponse
+ if err := json.NewDecoder(resp.Body).Decode(&repo); err != nil {
+ return time.Time{}, err
+ }
+
+ t, err := time.Parse(time.RFC3339, repo.PushedAt)
+ if err != nil {
+ return time.Time{}, fmt.Errorf("parsing pushed_at: %w", err)
+ }
+ return t, nil
+}
+
+// buildResult takes raw code search items, filters by repo freshness, extracts
+// endpoints and base URL candidates.
+func buildResult(items []codeSearchItem, g *GitHubSource, ctx context.Context) SourceResult {
+ if len(items) == 0 {
+ return SourceResult{}
+ }
+
+ // Collect unique repos.
+ repos := make(map[string]struct{})
+ for _, item := range items {
+ repos[item.Repository.FullName] = struct{}{}
+ }
+
+ // Check repo freshness.
+ freshRepos := make(map[string]bool)
+ cutoff := time.Now().Add(-g.recencyCutoff)
+ for fullName := range repos {
+ pushed, err := g.fetchRepoPushedAt(ctx, fullName)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "warning: checking repo %s freshness: %v\n", fullName, err)
+ continue // skip this repo, don't include its endpoints
+ }
+ freshRepos[fullName] = pushed.After(cutoff)
+ }
+
+ // Extract endpoints from fresh repos only and track domain frequency.
+ domainCounts := make(map[string]int)
+ var endpoints []DiscoveredEndpoint
+
+ // Track which repo contributed which endpoint to count per-repo frequency.
+ type endpointKey struct {
+ method string
+ path string
+ }
+ repoEndpoints := make(map[endpointKey]map[string]struct{})
+
+ for _, item := range items {
+ repo := item.Repository.FullName
+ if !freshRepos[repo] {
+ continue
+ }
+
+ for _, tm := range item.TextMatches {
+ extracted := extractEndpointsFromTextMatches(tm.Fragment)
+ for _, ep := range extracted {
+ key := endpointKey{method: ep.method, path: ep.path}
+ if _, ok := repoEndpoints[key]; !ok {
+ repoEndpoints[key] = make(map[string]struct{})
+ }
+ repoEndpoints[key][repo] = struct{}{}
+
+ if ep.domain != "" {
+ domainCounts[ep.domain]++
+ }
+ }
+ }
+ }
+
+ // Convert to DiscoveredEndpoint, deduplicating across repos.
+ seen := make(map[endpointKey]struct{})
+ for key := range repoEndpoints {
+ if _, exists := seen[key]; exists {
+ continue
+ }
+ seen[key] = struct{}{}
+
+ method := key.method
+ if method == "" {
+ method = "GET"
+ }
+ endpoints = append(endpoints, DiscoveredEndpoint{
+ Method: method,
+ Path: key.path,
+ SourceTier: TierCodeSearch,
+ SourceName: "github-code-search",
+ })
+ }
+
+ // Build base URL candidates from most frequent domain.
+ var baseURLs []string
+ if len(domainCounts) > 0 {
+ var bestDomain string
+ var bestCount int
+ for domain, count := range domainCounts {
+ if count > bestCount {
+ bestDomain = domain
+ bestCount = count
+ }
+ }
+ if bestDomain != "" {
+ baseURLs = append(baseURLs, "https://"+bestDomain)
+ }
+ }
+
+ return SourceResult{
+ Endpoints: endpoints,
+ BaseURLCandidates: baseURLs,
+ }
+}
+
+// extractedEndpoint is a raw endpoint extracted from text match fragments.
+type extractedEndpoint struct {
+ method string
+ path string
+ domain string
+}
+
+// urlPathPattern matches URL paths like /v1/users, /api/projects, /v2/databases/{id}
+var urlPathPattern = regexp.MustCompile(`(?:https?://([a-zA-Z0-9._-]+))?(/(?:v\d+|api)/[a-zA-Z0-9/_{}:.-]+)`)
+
+// standalonePathPattern matches standalone paths like '/v1/users' that aren't part of a full URL.
+var standalonePathPattern = regexp.MustCompile(`['"](/(?:v\d+|api)/[a-zA-Z0-9/_{}:.-]+)['"]`)
+
+// httpMethodPattern matches HTTP method hints near URL patterns.
+var httpMethodPattern = regexp.MustCompile(`(?i)\b(GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS)\b`)
+
+// extractEndpointsFromTextMatches parses a text match fragment and extracts
+// URL path patterns and associated HTTP methods.
+func extractEndpointsFromTextMatches(fragment string) []extractedEndpoint {
+ var results []extractedEndpoint
+ seen := make(map[string]struct{})
+
+ // Try full URL patterns first (with domain).
+ for _, match := range urlPathPattern.FindAllStringSubmatch(fragment, -1) {
+ domain := match[1]
+ path := cleanExtractedPath(match[2])
+ if path == "" {
+ continue
+ }
+
+ key := path
+ if _, exists := seen[key]; exists {
+ continue
+ }
+ seen[key] = struct{}{}
+
+ method := inferMethod(fragment, match[0])
+ results = append(results, extractedEndpoint{
+ method: method,
+ path: path,
+ domain: domain,
+ })
+ }
+
+ // Also try standalone paths not already captured.
+ for _, match := range standalonePathPattern.FindAllStringSubmatch(fragment, -1) {
+ path := cleanExtractedPath(match[1])
+ if path == "" {
+ continue
+ }
+
+ if _, exists := seen[path]; exists {
+ continue
+ }
+ seen[path] = struct{}{}
+
+ method := inferMethod(fragment, match[0])
+ results = append(results, extractedEndpoint{
+ method: method,
+ path: path,
+ })
+ }
+
+ return results
+}
+
+// cleanExtractedPath trims trailing punctuation and normalizes the path.
+func cleanExtractedPath(path string) string {
+ // Trim common trailing characters that are regex artifacts.
+ path = strings.TrimRight(path, ".'\")")
+ // Must start with /.
+ if !strings.HasPrefix(path, "/") {
+ return ""
+ }
+ // Filter out paths that are too short to be meaningful.
+ if len(path) < 3 {
+ return ""
+ }
+ return path
+}
+
+// inferMethod tries to find an HTTP method keyword near the URL in the fragment.
+func inferMethod(fragment, urlMatch string) string {
+ // Look for a method keyword in the same fragment.
+ idx := strings.Index(fragment, urlMatch)
+ if idx < 0 {
+ return ""
+ }
+
+ // Check a window before the URL match for method keywords.
+ start := idx - 80
+ if start < 0 {
+ start = 0
+ }
+ window := fragment[start : idx+len(urlMatch)]
+
+ matches := httpMethodPattern.FindAllString(window, -1)
+ if len(matches) > 0 {
+ return strings.ToUpper(matches[len(matches)-1])
+ }
+ return ""
+}
+
+// --- Response types (unexported) ---
+
+type codeSearchResponse struct {
+ TotalCount int `json:"total_count"`
+ Items []codeSearchItem `json:"items"`
+}
+
+type codeSearchItem struct {
+ Name string `json:"name"`
+ Path string `json:"path"`
+ HTMLURL string `json:"html_url"`
+ Repository codeSearchRepo `json:"repository"`
+ TextMatches []textMatch `json:"text_matches"`
+}
+
+type codeSearchRepo struct {
+ FullName string `json:"full_name"`
+}
+
+type textMatch struct {
+ Fragment string `json:"fragment"`
+}
+
+type repoResponse struct {
+ PushedAt string `json:"pushed_at"`
+}
diff --git a/internal/crowdsniff/github_test.go b/internal/crowdsniff/github_test.go
new file mode 100644
index 00000000..756b87c0
--- /dev/null
+++ b/internal/crowdsniff/github_test.go
@@ -0,0 +1,407 @@
+package crowdsniff
+
+import (
+ "context"
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+func loadFixture(t *testing.T, name string) []byte {
+ t.Helper()
+ data, err := os.ReadFile("../../testdata/crowdsniff/" + name)
+ require.NoError(t, err)
+ return data
+}
+
+func TestGitHubSource_Discover(t *testing.T) {
+ t.Parallel()
+
+ t.Run("no token returns empty result", func(t *testing.T) {
+ t.Parallel()
+ src := NewGitHubSource(GitHubOptions{Token: "NONE"})
+ // Override to empty after construction to simulate no token.
+ src.token = ""
+
+ result, err := src.Discover(context.Background(), "notion")
+ assert.NoError(t, err)
+ assert.Empty(t, result.Endpoints)
+ assert.Empty(t, result.BaseURLCandidates)
+ })
+
+ t.Run("happy path with fresh repos", func(t *testing.T) {
+ t.Parallel()
+
+ searchResp := loadFixture(t, "github-code-search-response.json")
+ recentPush := time.Now().Add(-30 * 24 * time.Hour).Format(time.RFC3339)
+
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch {
+ case strings.HasPrefix(r.URL.Path, "/search/code"):
+ // Verify text-match accept header.
+ assert.Equal(t, "application/vnd.github.text-match+json", r.Header.Get("Accept"))
+ assert.Equal(t, "Bearer test-token", r.Header.Get("Authorization"))
+ w.Header().Set("Content-Type", "application/json")
+ _, _ = w.Write(searchResp)
+ case strings.HasPrefix(r.URL.Path, "/repos/"):
+ w.Header().Set("Content-Type", "application/json")
+ resp := repoResponse{PushedAt: recentPush}
+ _ = json.NewEncoder(w).Encode(resp)
+ default:
+ http.NotFound(w, r)
+ }
+ }))
+ defer srv.Close()
+
+ src := NewGitHubSource(GitHubOptions{
+ BaseURL: srv.URL,
+ Token: "test-token",
+ HTTPClient: srv.Client(),
+ RecencyCutoff: 180 * 24 * time.Hour,
+ })
+
+ result, err := src.Discover(context.Background(), "api.notion.com")
+ assert.NoError(t, err)
+ assert.NotEmpty(t, result.Endpoints)
+
+ // Should have found some endpoints from text matches.
+ paths := make(map[string]bool)
+ for _, ep := range result.Endpoints {
+ paths[ep.Path] = true
+ assert.Equal(t, TierCodeSearch, ep.SourceTier)
+ assert.Equal(t, "github-code-search", ep.SourceName)
+ }
+
+ // At least /v1/users and /v1/projects should be extracted.
+ assert.True(t, paths["/v1/users"], "expected /v1/users in results")
+ assert.True(t, paths["/v1/projects"], "expected /v1/projects in results")
+
+ // Base URL should include api.notion.com.
+ assert.NotEmpty(t, result.BaseURLCandidates)
+ assert.Contains(t, result.BaseURLCandidates[0], "api.notion.com")
+ })
+
+ t.Run("text matches extract endpoints correctly", func(t *testing.T) {
+ t.Parallel()
+
+ recentPush := time.Now().Add(-10 * 24 * time.Hour).Format(time.RFC3339)
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch {
+ case strings.HasPrefix(r.URL.Path, "/search/code"):
+ resp := codeSearchResponse{
+ TotalCount: 2,
+ Items: []codeSearchItem{
+ {
+ Name: "client.js",
+ Path: "src/client.js",
+ Repository: codeSearchRepo{
+ FullName: "user/repo1",
+ },
+ TextMatches: []textMatch{
+ {Fragment: "fetch('https://api.example.com/v1/users', { method: 'GET' })"},
+ {Fragment: "axios.post('https://api.example.com/v1/projects')"},
+ },
+ },
+ {
+ Name: "api.py",
+ Path: "src/api.py",
+ Repository: codeSearchRepo{
+ FullName: "user/repo2",
+ },
+ TextMatches: []textMatch{
+ {Fragment: "requests.get('https://api.example.com/v1/users')"},
+ },
+ },
+ },
+ }
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(resp)
+ case strings.HasPrefix(r.URL.Path, "/repos/"):
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(repoResponse{PushedAt: recentPush})
+ default:
+ http.NotFound(w, r)
+ }
+ }))
+ defer srv.Close()
+
+ src := NewGitHubSource(GitHubOptions{
+ BaseURL: srv.URL,
+ Token: "test-token",
+ HTTPClient: srv.Client(),
+ })
+
+ result, err := src.Discover(context.Background(), "api.example.com")
+ assert.NoError(t, err)
+
+ paths := make(map[string]bool)
+ for _, ep := range result.Endpoints {
+ paths[ep.Path] = true
+ }
+ assert.True(t, paths["/v1/users"], "expected /v1/users")
+ assert.True(t, paths["/v1/projects"], "expected /v1/projects")
+ })
+
+ t.Run("stale repos are excluded", func(t *testing.T) {
+ t.Parallel()
+
+ stalePush := time.Now().Add(-365 * 24 * time.Hour).Format(time.RFC3339) // 1 year ago
+
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch {
+ case strings.HasPrefix(r.URL.Path, "/search/code"):
+ resp := codeSearchResponse{
+ TotalCount: 1,
+ Items: []codeSearchItem{
+ {
+ Name: "old.js",
+ Path: "src/old.js",
+ Repository: codeSearchRepo{
+ FullName: "ancient/repo",
+ },
+ TextMatches: []textMatch{
+ {Fragment: "fetch('https://api.example.com/v1/users')"},
+ },
+ },
+ },
+ }
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(resp)
+ case strings.HasPrefix(r.URL.Path, "/repos/"):
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(repoResponse{PushedAt: stalePush})
+ default:
+ http.NotFound(w, r)
+ }
+ }))
+ defer srv.Close()
+
+ src := NewGitHubSource(GitHubOptions{
+ BaseURL: srv.URL,
+ Token: "test-token",
+ HTTPClient: srv.Client(),
+ RecencyCutoff: 180 * 24 * time.Hour,
+ })
+
+ result, err := src.Discover(context.Background(), "api.example.com")
+ assert.NoError(t, err)
+ assert.Empty(t, result.Endpoints, "stale repo endpoints should be excluded")
+ })
+
+ t.Run("zero search results returns empty", func(t *testing.T) {
+ t.Parallel()
+
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if strings.HasPrefix(r.URL.Path, "/search/code") {
+ resp := codeSearchResponse{TotalCount: 0, Items: nil}
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(resp)
+ return
+ }
+ http.NotFound(w, r)
+ }))
+ defer srv.Close()
+
+ src := NewGitHubSource(GitHubOptions{
+ BaseURL: srv.URL,
+ Token: "test-token",
+ HTTPClient: srv.Client(),
+ })
+
+ result, err := src.Discover(context.Background(), "nonexistent.api.com")
+ assert.NoError(t, err)
+ assert.Empty(t, result.Endpoints)
+ assert.Empty(t, result.BaseURLCandidates)
+ })
+
+ t.Run("API 5xx returns partial results", func(t *testing.T) {
+ t.Parallel()
+
+ callCount := 0
+ recentPush := time.Now().Add(-10 * 24 * time.Hour).Format(time.RFC3339)
+
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch {
+ case strings.HasPrefix(r.URL.Path, "/search/code"):
+ callCount++
+ if callCount == 1 {
+ // First query succeeds.
+ resp := codeSearchResponse{
+ TotalCount: 1,
+ Items: []codeSearchItem{
+ {
+ Name: "ok.js",
+ Path: "src/ok.js",
+ Repository: codeSearchRepo{
+ FullName: "good/repo",
+ },
+ TextMatches: []textMatch{
+ {Fragment: "fetch('https://api.example.com/v1/users')"},
+ },
+ },
+ },
+ }
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(resp)
+ } else {
+ // Subsequent queries fail.
+ http.Error(w, "Internal Server Error", http.StatusInternalServerError)
+ }
+ case strings.HasPrefix(r.URL.Path, "/repos/"):
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(repoResponse{PushedAt: recentPush})
+ default:
+ http.NotFound(w, r)
+ }
+ }))
+ defer srv.Close()
+
+ src := NewGitHubSource(GitHubOptions{
+ BaseURL: srv.URL,
+ Token: "test-token",
+ HTTPClient: srv.Client(),
+ })
+
+ result, err := src.Discover(context.Background(), "api.example.com")
+ assert.NoError(t, err)
+ // Should still have results from the first successful request.
+ assert.NotEmpty(t, result.Endpoints, "should return partial results on 5xx")
+ })
+}
+
+func TestBuildSearchQueries(t *testing.T) {
+ t.Parallel()
+
+ t.Run("domain-like name uses exact match", func(t *testing.T) {
+ t.Parallel()
+ queries := buildSearchQueries("api.notion.com")
+ assert.Len(t, queries, 2)
+ assert.Contains(t, queries[0], `"api.notion.com"`)
+ assert.Contains(t, queries[0], "language:javascript")
+ assert.Contains(t, queries[1], "language:python")
+ })
+
+ t.Run("plain name uses broader query", func(t *testing.T) {
+ t.Parallel()
+ queries := buildSearchQueries("notion")
+ assert.Len(t, queries, 2)
+ assert.Contains(t, queries[0], `"notion" api`)
+ assert.Contains(t, queries[0], "language:javascript")
+ })
+}
+
+func TestExtractEndpointsFromTextMatches(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ fragment string
+ wantPaths []string
+ }{
+ {
+ name: "full URL with path",
+ fragment: "fetch('https://api.notion.com/v1/users', { headers })",
+ wantPaths: []string{"/v1/users"},
+ },
+ {
+ name: "multiple URLs in fragment",
+ fragment: "fetch('/v1/users')\nfetch('/v1/projects')",
+ wantPaths: []string{"/v1/users", "/v1/projects"},
+ },
+ {
+ name: "POST method inferred",
+ fragment: "axios.post('https://api.example.com/v1/items')",
+ wantPaths: []string{"/v1/items"},
+ },
+ {
+ name: "GET method inferred",
+ fragment: "requests.get('https://api.example.com/v1/users')",
+ wantPaths: []string{"/v1/users"},
+ },
+ {
+ name: "no path patterns",
+ fragment: "just some random text without API paths",
+ wantPaths: nil,
+ },
+ {
+ name: "path with api prefix",
+ fragment: "client.get('/api/projects/list')",
+ wantPaths: []string{"/api/projects/list"},
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+ eps := extractEndpointsFromTextMatches(tt.fragment)
+ var paths []string
+ for _, ep := range eps {
+ paths = append(paths, ep.path)
+ }
+ if tt.wantPaths == nil {
+ assert.Empty(t, paths)
+ } else {
+ assert.Equal(t, tt.wantPaths, paths)
+ }
+ })
+ }
+}
+
+func TestExtractEndpointsMethodInference(t *testing.T) {
+ t.Parallel()
+
+ t.Run("POST method from fragment", func(t *testing.T) {
+ t.Parallel()
+ eps := extractEndpointsFromTextMatches("axios.post('https://api.example.com/v1/items')")
+ require.Len(t, eps, 1)
+ assert.Equal(t, "POST", eps[0].method)
+ })
+
+ t.Run("GET method from fragment", func(t *testing.T) {
+ t.Parallel()
+ eps := extractEndpointsFromTextMatches("http.Get(\"https://api.example.com/v1/blocks\")")
+ require.Len(t, eps, 1)
+ assert.Equal(t, "GET", eps[0].method)
+ })
+
+ t.Run("no method defaults to empty", func(t *testing.T) {
+ t.Parallel()
+ eps := extractEndpointsFromTextMatches("fetch('https://api.example.com/v1/data')")
+ require.Len(t, eps, 1)
+ assert.Equal(t, "", eps[0].method)
+ })
+}
+
+func TestNewGitHubSource_Defaults(t *testing.T) {
+ t.Parallel()
+
+ t.Run("applies defaults", func(t *testing.T) {
+ t.Parallel()
+ src := NewGitHubSource(GitHubOptions{Token: "tok"})
+ assert.Equal(t, "https://api.github.com", src.baseURL)
+ assert.NotNil(t, src.client)
+ assert.Equal(t, 180*24*time.Hour, src.recencyCutoff)
+ })
+
+ t.Run("respects overrides", func(t *testing.T) {
+ t.Parallel()
+ client := &http.Client{Timeout: 5 * time.Second}
+ src := NewGitHubSource(GitHubOptions{
+ BaseURL: "https://custom.api.com",
+ HTTPClient: client,
+ Token: "custom-token",
+ RecencyCutoff: 90 * 24 * time.Hour,
+ })
+ assert.Equal(t, "https://custom.api.com", src.baseURL)
+ assert.Equal(t, client, src.client)
+ assert.Equal(t, "custom-token", src.token)
+ assert.Equal(t, 90*24*time.Hour, src.recencyCutoff)
+ })
+}
diff --git a/internal/crowdsniff/npm.go b/internal/crowdsniff/npm.go
new file mode 100644
index 00000000..c1cb7441
--- /dev/null
+++ b/internal/crowdsniff/npm.go
@@ -0,0 +1,464 @@
+package crowdsniff
+
+import (
+ "archive/tar"
+ "compress/gzip"
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "net/url"
+ "os"
+ "path/filepath"
+ "strings"
+ "time"
+)
+
+const (
+ defaultRegistryBaseURL = "https://registry.npmjs.org"
+ defaultDownloadsBaseURL = "https://api.npmjs.org"
+ defaultRecencyCutoff = 180 * 24 * time.Hour // 6 months
+ defaultHTTPTimeout = 15 * time.Second
+ maxTarballSize = 10 * 1024 * 1024 // 10 MB
+ maxSearchResults = 25
+ maxPackagesToProcess = 10
+ maxBulkDownloadPackages = 128
+)
+
+// NPMOptions configures the NPM source.
+type NPMOptions struct {
+ RegistryBaseURL string
+ DownloadsBaseURL string
+ HTTPClient *http.Client
+ RecencyCutoff time.Duration
+}
+
+// NPMSource discovers API endpoints by searching the npm registry,
+// downloading SDK tarballs, and grepping source code for patterns.
+type NPMSource struct {
+ registryBaseURL string
+ downloadsBaseURL string
+ httpClient *http.Client
+ recencyCutoff time.Duration
+}
+
+// NewNPMSource creates an NPMSource with the given options.
+func NewNPMSource(opts NPMOptions) *NPMSource {
+ registry := opts.RegistryBaseURL
+ if registry == "" {
+ registry = defaultRegistryBaseURL
+ }
+ downloads := opts.DownloadsBaseURL
+ if downloads == "" {
+ downloads = defaultDownloadsBaseURL
+ }
+ client := opts.HTTPClient
+ if client == nil {
+ client = &http.Client{Timeout: defaultHTTPTimeout}
+ }
+ cutoff := opts.RecencyCutoff
+ if cutoff == 0 {
+ cutoff = defaultRecencyCutoff
+ }
+ return &NPMSource{
+ registryBaseURL: strings.TrimRight(registry, "/"),
+ downloadsBaseURL: strings.TrimRight(downloads, "/"),
+ httpClient: client,
+ recencyCutoff: cutoff,
+ }
+}
+
+// npmSearchResponse represents the npm registry search API response.
+type npmSearchResponse struct {
+ Objects []npmSearchObject `json:"objects"`
+}
+
+type npmSearchObject struct {
+ Package npmPackageInfo `json:"package"`
+}
+
+type npmPackageInfo struct {
+ Name string `json:"name"`
+ Scope string `json:"scope"`
+ Version string `json:"version"`
+ Date time.Time `json:"date"`
+ Links npmLinks `json:"links"`
+ Dist *npmDistInfo `json:"dist,omitempty"`
+}
+
+type npmLinks struct {
+ NPM string `json:"npm"`
+}
+
+// npmPackageVersion represents the response from GET /<pkg>/<version>.
+type npmPackageVersion struct {
+ Dist npmDistInfo `json:"dist"`
+}
+
+type npmDistInfo struct {
+ Tarball string `json:"tarball"`
+}
+
+// npmDownloadsResponse represents the npm downloads API response.
+type npmDownloadsResponse struct {
+ Downloads int `json:"downloads"`
+ Package string `json:"package"`
+}
+
+// npmBulkDownloadsResponse maps package names to download counts.
+type npmBulkDownloadsResponse map[string]*npmDownloadsResponse
+
+// Discover searches npm for packages related to the given API name,
+// downloads their source code, and greps for endpoint patterns.
+func (s *NPMSource) Discover(ctx context.Context, apiName string) (SourceResult, error) {
+ var result SourceResult
+
+ // Step 1: Search the registry.
+ packages, err := s.search(ctx, apiName)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "crowd-sniff: npm search failed: %v\n", err)
+ return result, nil
+ }
+ if len(packages) == 0 {
+ return result, nil
+ }
+
+ // Step 2: Filter by recency.
+ cutoffTime := time.Now().Add(-s.recencyCutoff)
+ var recent []npmPackageInfo
+ for _, pkg := range packages {
+ if pkg.Date.After(cutoffTime) {
+ recent = append(recent, pkg)
+ }
+ }
+ if len(recent) == 0 {
+ return result, nil
+ }
+
+ // Step 3: Take top N packages.
+ if len(recent) > maxPackagesToProcess {
+ recent = recent[:maxPackagesToProcess]
+ }
+
+ // Step 4: Fetch download counts (non-fatal).
+ downloads := s.fetchDownloads(ctx, recent)
+
+ // Step 5: Process each package.
+ apiNameLower := strings.ToLower(apiName)
+ for _, pkg := range recent {
+ tier := classifyPackage(pkg, apiNameLower)
+
+ // Fetch tarball URL from package metadata.
+ tarballURL, fetchErr := s.fetchTarballURL(ctx, pkg.Name, pkg.Version)
+ if fetchErr != nil {
+ fmt.Fprintf(os.Stderr, "crowd-sniff: failed to get tarball URL for %s: %v\n", pkg.Name, fetchErr)
+ continue
+ }
+
+ // Download and extract tarball.
+ endpoints, baseURLs, processErr := s.processPackageTarball(ctx, tarballURL, pkg.Name, tier, downloads[pkg.Name])
+ if processErr != nil {
+ fmt.Fprintf(os.Stderr, "crowd-sniff: failed to process %s: %v\n", pkg.Name, processErr)
+ continue
+ }
+
+ result.Endpoints = append(result.Endpoints, endpoints...)
+ result.BaseURLCandidates = append(result.BaseURLCandidates, baseURLs...)
+ }
+
+ return result, nil
+}
+
+// search queries the npm registry search API.
+func (s *NPMSource) search(ctx context.Context, query string) ([]npmPackageInfo, error) {
+ u := fmt.Sprintf("%s/-/v1/search?text=%s&size=%d", s.registryBaseURL, url.QueryEscape(query), maxSearchResults)
+
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
+ if err != nil {
+ return nil, fmt.Errorf("creating search request: %w", err)
+ }
+
+ resp, err := s.httpClient.Do(req)
+ if err != nil {
+ return nil, fmt.Errorf("executing search: %w", err)
+ }
+ defer func() { _ = resp.Body.Close() }()
+
+ if resp.StatusCode != http.StatusOK {
+ return nil, fmt.Errorf("search returned status %d", resp.StatusCode)
+ }
+
+ var searchResp npmSearchResponse
+ if err := json.NewDecoder(resp.Body).Decode(&searchResp); err != nil {
+ return nil, fmt.Errorf("decoding search response: %w", err)
+ }
+
+ packages := make([]npmPackageInfo, 0, len(searchResp.Objects))
+ for _, obj := range searchResp.Objects {
+ packages = append(packages, obj.Package)
+ }
+ return packages, nil
+}
+
+// fetchDownloads fetches weekly download counts for packages.
+// Returns a map of package name -> download count. Errors are non-fatal.
+func (s *NPMSource) fetchDownloads(ctx context.Context, packages []npmPackageInfo) map[string]int {
+ result := make(map[string]int)
+
+ // Build the bulk request (up to 128 packages).
+ names := make([]string, 0, len(packages))
+ for _, pkg := range packages {
+ if len(names) >= maxBulkDownloadPackages {
+ break
+ }
+ names = append(names, pkg.Name)
+ }
+
+ if len(names) == 0 {
+ return result
+ }
+
+ // npm bulk downloads API: GET /downloads/point/last-week/<pkg1>,<pkg2>,...
+ u := fmt.Sprintf("%s/downloads/point/last-week/%s", s.downloadsBaseURL, strings.Join(names, ","))
+
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "crowd-sniff: failed to create downloads request: %v\n", err)
+ return result
+ }
+
+ resp, err := s.httpClient.Do(req)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "crowd-sniff: downloads request failed: %v\n", err)
+ return result
+ }
+ defer func() { _ = resp.Body.Close() }()
+
+ if resp.StatusCode != http.StatusOK {
+ fmt.Fprintf(os.Stderr, "crowd-sniff: downloads API returned status %d\n", resp.StatusCode)
+ return result
+ }
+
+ var bulk npmBulkDownloadsResponse
+ if err := json.NewDecoder(resp.Body).Decode(&bulk); err != nil {
+ fmt.Fprintf(os.Stderr, "crowd-sniff: failed to decode downloads response: %v\n", err)
+ return result
+ }
+
+ for name, data := range bulk {
+ if data != nil {
+ result[name] = data.Downloads
+ }
+ }
+ return result
+}
+
+// fetchTarballURL gets the tarball download URL for a specific package version.
+func (s *NPMSource) fetchTarballURL(ctx context.Context, name, version string) (string, error) {
+ u := fmt.Sprintf("%s/%s/%s", s.registryBaseURL, url.PathEscape(name), url.PathEscape(version))
+
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
+ if err != nil {
+ return "", fmt.Errorf("creating version request: %w", err)
+ }
+
+ resp, err := s.httpClient.Do(req)
+ if err != nil {
+ return "", fmt.Errorf("fetching version metadata: %w", err)
+ }
+ defer func() { _ = resp.Body.Close() }()
+
+ if resp.StatusCode != http.StatusOK {
+ return "", fmt.Errorf("version metadata returned status %d", resp.StatusCode)
+ }
+
+ var version_ npmPackageVersion
+ if err := json.NewDecoder(resp.Body).Decode(&version_); err != nil {
+ return "", fmt.Errorf("decoding version metadata: %w", err)
+ }
+
+ if version_.Dist.Tarball == "" {
+ return "", fmt.Errorf("no tarball URL in version metadata")
+ }
+
+ return version_.Dist.Tarball, nil
+}
+
+// processPackageTarball downloads a tarball, extracts it, and greps for endpoints.
+func (s *NPMSource) processPackageTarball(ctx context.Context, tarballURL, pkgName, tier string, weeklyDownloads int) ([]DiscoveredEndpoint, []string, error) {
+ // Security: validate tarball URL is HTTPS.
+ parsed, err := url.Parse(tarballURL)
+ if err != nil {
+ return nil, nil, fmt.Errorf("invalid tarball URL: %w", err)
+ }
+ if parsed.Scheme != "https" {
+ return nil, nil, fmt.Errorf("tarball URL must be HTTPS, got %s", parsed.Scheme)
+ }
+
+ // Download tarball.
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, tarballURL, nil)
+ if err != nil {
+ return nil, nil, fmt.Errorf("creating tarball request: %w", err)
+ }
+
+ resp, err := s.httpClient.Do(req)
+ if err != nil {
+ return nil, nil, fmt.Errorf("downloading tarball: %w", err)
+ }
+ defer func() { _ = resp.Body.Close() }()
+
+ if resp.StatusCode != http.StatusOK {
+ return nil, nil, fmt.Errorf("tarball download returned status %d", resp.StatusCode)
+ }
+
+ // Create temp directory.
+ tmpDir, err := os.MkdirTemp("", "crowd-sniff-npm-*")
+ if err != nil {
+ return nil, nil, fmt.Errorf("creating temp dir: %w", err)
+ }
+ defer func() { _ = os.RemoveAll(tmpDir) }()
+
+ // Extract tarball with size limit.
+ if err := extractTarball(resp.Body, tmpDir); err != nil {
+ return nil, nil, fmt.Errorf("extracting tarball: %w", err)
+ }
+
+ // Grep extracted files for endpoint patterns.
+ var allEndpoints []DiscoveredEndpoint
+ var allBaseURLs []string
+
+ _ = filepath.Walk(tmpDir, func(path string, info os.FileInfo, err error) error {
+ if err != nil {
+ return nil // skip errors
+ }
+ if info.IsDir() {
+ return nil
+ }
+
+ // Only grep JS/TS files.
+ ext := strings.ToLower(filepath.Ext(path))
+ if ext != ".js" && ext != ".ts" && ext != ".mjs" {
+ return nil
+ }
+
+ // Skip declaration files and test files.
+ base := filepath.Base(path)
+ if strings.HasSuffix(base, ".d.ts") || strings.Contains(base, ".test.") || strings.Contains(base, ".spec.") {
+ return nil
+ }
+
+ content, readErr := os.ReadFile(path)
+ if readErr != nil {
+ return nil // skip unreadable files
+ }
+
+ endpoints, baseURLs := GrepEndpoints(string(content), pkgName, tier)
+ endpoints = EnrichWithParams(string(content), endpoints)
+ allEndpoints = append(allEndpoints, endpoints...)
+ allBaseURLs = append(allBaseURLs, baseURLs...)
+ return nil
+ })
+
+ // Adjust tier for low-download packages (still community, but we note it).
+ _ = weeklyDownloads // Used for future priority sorting; tier stays the same.
+
+ return allEndpoints, allBaseURLs, nil
+}
+
+// extractTarball extracts a gzipped tar archive to the destination directory.
+// Security: rejects symlinks, hard links, path traversal, and limits total size.
+func extractTarball(r io.Reader, destDir string) error {
+ // Limit total bytes read.
+ limited := io.LimitReader(r, maxTarballSize)
+
+ gz, err := gzip.NewReader(limited)
+ if err != nil {
+ return fmt.Errorf("opening gzip reader: %w", err)
+ }
+ defer func() { _ = gz.Close() }()
+
+ tr := tar.NewReader(gz)
+ absDestDir, err := filepath.Abs(destDir)
+ if err != nil {
+ return fmt.Errorf("resolving dest dir: %w", err)
+ }
+
+ for {
+ header, err := tr.Next()
+ if err == io.EOF {
+ break
+ }
+ if err != nil {
+ return fmt.Errorf("reading tar entry: %w", err)
+ }
+
+ // Security: reject symlinks and hard links.
+ if header.Typeflag == tar.TypeSymlink || header.Typeflag == tar.TypeLink {
+ continue // skip, don't error — other files may be fine
+ }
+
+ // Only process regular files and directories.
+ if header.Typeflag != tar.TypeReg && header.Typeflag != tar.TypeDir {
+ continue
+ }
+
+ // Security: sanitize path and prevent traversal.
+ target := filepath.Join(absDestDir, filepath.Clean(header.Name))
+ absTarget, err := filepath.Abs(target)
+ if err != nil {
+ continue
+ }
+ if !strings.HasPrefix(absTarget, absDestDir+string(filepath.Separator)) && absTarget != absDestDir {
+ continue // path traversal attempt
+ }
+
+ if header.Typeflag == tar.TypeDir {
+ if err := os.MkdirAll(absTarget, 0o755); err != nil {
+ return fmt.Errorf("creating directory %s: %w", header.Name, err)
+ }
+ continue
+ }
+
+ // Ensure parent directory exists.
+ parentDir := filepath.Dir(absTarget)
+ if err := os.MkdirAll(parentDir, 0o755); err != nil {
+ return fmt.Errorf("creating parent dir for %s: %w", header.Name, err)
+ }
+
+ f, err := os.Create(absTarget)
+ if err != nil {
+ return fmt.Errorf("creating file %s: %w", header.Name, err)
+ }
+
+ // Copy with size limit per file (same global limit via LimitReader on outer reader).
+ if _, err := io.Copy(f, tr); err != nil {
+ _ = f.Close()
+ return fmt.Errorf("writing file %s: %w", header.Name, err)
+ }
+ _ = f.Close()
+ }
+
+ return nil
+}
+
+// classifyPackage determines whether a package is an official or community SDK.
+// A package is considered official if its npm scope matches the API vendor name.
+func classifyPackage(pkg npmPackageInfo, apiNameLower string) string {
+ scope := strings.TrimPrefix(pkg.Scope, "@")
+ scope = strings.ToLower(scope)
+
+ if scope != "" && (scope == apiNameLower ||
+ strings.Contains(scope, apiNameLower) ||
+ strings.Contains(apiNameLower, scope)) {
+ return TierOfficialSDK
+ }
+
+ // Also check the package name itself for official-looking names.
+ nameLower := strings.ToLower(pkg.Name)
+ if strings.HasPrefix(nameLower, "@"+apiNameLower+"/") {
+ return TierOfficialSDK
+ }
+
+ return TierCommunitySDK
+}
diff --git a/internal/crowdsniff/npm_test.go b/internal/crowdsniff/npm_test.go
new file mode 100644
index 00000000..92b4580f
--- /dev/null
+++ b/internal/crowdsniff/npm_test.go
@@ -0,0 +1,951 @@
+package crowdsniff
+
+import (
+ "archive/tar"
+ "bytes"
+ "compress/gzip"
+ "context"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+// buildTarball creates a gzipped tar archive with the given files.
+func buildTarball(t *testing.T, files map[string]string) []byte {
+ t.Helper()
+
+ var buf bytes.Buffer
+ gw := gzip.NewWriter(&buf)
+ tw := tar.NewWriter(gw)
+
+ for name, content := range files {
+ hdr := &tar.Header{
+ Name: name,
+ Mode: 0o644,
+ Size: int64(len(content)),
+ Typeflag: tar.TypeReg,
+ }
+ require.NoError(t, tw.WriteHeader(hdr))
+ _, err := tw.Write([]byte(content))
+ require.NoError(t, err)
+ }
+
+ require.NoError(t, tw.Close())
+ require.NoError(t, gw.Close())
+ return buf.Bytes()
+}
+
+// buildTarballWithSymlink creates a gzipped tar archive that includes a symlink entry.
+func buildTarballWithSymlink(t *testing.T) []byte {
+ t.Helper()
+
+ var buf bytes.Buffer
+ gw := gzip.NewWriter(&buf)
+ tw := tar.NewWriter(gw)
+
+ // Add a regular file.
+ content := `this.get("/v1/safe");`
+ hdr := &tar.Header{
+ Name: "package/index.js",
+ Mode: 0o644,
+ Size: int64(len(content)),
+ Typeflag: tar.TypeReg,
+ }
+ require.NoError(t, tw.WriteHeader(hdr))
+ _, err := tw.Write([]byte(content))
+ require.NoError(t, err)
+
+ // Add a symlink entry (should be skipped).
+ symlinkHdr := &tar.Header{
+ Name: "package/evil-link",
+ Linkname: "/etc/passwd",
+ Mode: 0o777,
+ Typeflag: tar.TypeSymlink,
+ }
+ require.NoError(t, tw.WriteHeader(symlinkHdr))
+
+ require.NoError(t, tw.Close())
+ require.NoError(t, gw.Close())
+ return buf.Bytes()
+}
+
+func searchResponse(packages ...npmPackageInfo) []byte {
+ resp := npmSearchResponse{
+ Objects: make([]npmSearchObject, len(packages)),
+ }
+ for i, pkg := range packages {
+ resp.Objects[i] = npmSearchObject{Package: pkg}
+ }
+ data, _ := json.Marshal(resp)
+ return data
+}
+
+func versionResponse(tarballURL string) []byte {
+ resp := npmPackageVersion{
+ Dist: npmDistInfo{Tarball: tarballURL},
+ }
+ data, _ := json.Marshal(resp)
+ return data
+}
+
+func downloadsResponse(packages map[string]int) []byte {
+ resp := make(npmBulkDownloadsResponse)
+ for name, count := range packages {
+ resp[name] = &npmDownloadsResponse{Downloads: count, Package: name}
+ }
+ data, _ := json.Marshal(resp)
+ return data
+}
+
+func TestNPMSource_Discover(t *testing.T) {
+ t.Parallel()
+
+ t.Run("happy path with endpoints", func(t *testing.T) {
+ t.Parallel()
+
+ sdkContent := `
+const BASE_URL = "https://api.example.com";
+class Client {
+ listUsers() { return this.get("/v1/users"); }
+ createUser(data) { return this.post("/v1/users", data); }
+ getProject(id) { return this.get("/v1/projects/" + id); }
+}
+`
+ tarball := buildTarball(t, map[string]string{
+ "package/index.js": sdkContent,
+ })
+
+ // Set up tarball server (needs to be HTTPS for validation, but httptest
+ // uses HTTP. We'll use the tarball server URL directly and test the
+ // HTTPS check separately).
+ tarballServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set("Content-Type", "application/octet-stream")
+ w.Write(tarball)
+ }))
+ defer tarballServer.Close()
+
+ versionPaths := map[string]bool{
+ "/example-sdk/1.0.0": true,
+ "/example-client/2.0.0": true,
+ "/example-api/0.5.0": true,
+ }
+ registryServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch r.URL.Path {
+ case "/-/v1/search":
+ w.Write(searchResponse(
+ npmPackageInfo{
+ Name: "example-sdk",
+ Version: "1.0.0",
+ Date: time.Now().Add(-24 * time.Hour),
+ },
+ npmPackageInfo{
+ Name: "example-client",
+ Version: "2.0.0",
+ Date: time.Now().Add(-48 * time.Hour),
+ },
+ npmPackageInfo{
+ Name: "example-api",
+ Version: "0.5.0",
+ Date: time.Now().Add(-72 * time.Hour),
+ },
+ ))
+ default:
+ if versionPaths[r.URL.Path] {
+ w.Write(versionResponse(tarballServer.URL + "/tarball.tgz"))
+ } else {
+ http.NotFound(w, r)
+ }
+ }
+ }))
+ defer registryServer.Close()
+
+ downloadsServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Write(downloadsResponse(map[string]int{
+ "example-sdk": 500,
+ "example-client": 200,
+ "example-api": 50,
+ }))
+ }))
+ defer downloadsServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ RegistryBaseURL: registryServer.URL,
+ DownloadsBaseURL: downloadsServer.URL,
+ })
+ // Override HTTPS check for test: use a custom processPackageTarball that
+ // accepts HTTP. We do this by making tarballURL validation accept the
+ // test server's scheme. Instead, we test HTTPS validation separately.
+
+ result, err := src.Discover(context.Background(), "example")
+
+ assert.NoError(t, err)
+ // Tarball URLs are http:// in tests, so they'll be rejected by HTTPS check.
+ // This test validates the search + filter + download flow works, even if
+ // tarball processing is skipped due to HTTP scheme.
+ // The endpoints will be empty because httptest uses HTTP, not HTTPS.
+ // We test the full flow including extraction in separate tests.
+ _ = result
+ })
+
+ t.Run("official SDK scope detection", func(t *testing.T) {
+ t.Parallel()
+
+ pkg := npmPackageInfo{
+ Name: "@notion/client",
+ Scope: "@notion",
+ }
+ tier := classifyPackage(pkg, "notion")
+ assert.Equal(t, TierOfficialSDK, tier)
+ })
+
+ t.Run("community SDK classification", func(t *testing.T) {
+ t.Parallel()
+
+ pkg := npmPackageInfo{
+ Name: "notion-helper",
+ Scope: "",
+ }
+ tier := classifyPackage(pkg, "notion")
+ assert.Equal(t, TierCommunitySDK, tier)
+ })
+
+ t.Run("official SDK by package name prefix", func(t *testing.T) {
+ t.Parallel()
+
+ pkg := npmPackageInfo{
+ Name: "@stripe/stripe-js",
+ Scope: "@stripe",
+ }
+ tier := classifyPackage(pkg, "stripe")
+ assert.Equal(t, TierOfficialSDK, tier)
+ })
+
+ t.Run("package date older than 6 months excluded", func(t *testing.T) {
+ t.Parallel()
+
+ registryServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ // Return packages that are all older than 6 months.
+ w.Write(searchResponse(
+ npmPackageInfo{
+ Name: "old-sdk",
+ Version: "1.0.0",
+ Date: time.Now().Add(-365 * 24 * time.Hour), // 1 year ago
+ },
+ ))
+ }))
+ defer registryServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ RegistryBaseURL: registryServer.URL,
+ })
+
+ result, err := src.Discover(context.Background(), "old-api")
+
+ assert.NoError(t, err)
+ assert.Empty(t, result.Endpoints)
+ assert.Empty(t, result.BaseURLCandidates)
+ })
+
+ t.Run("npm search returns 0 results", func(t *testing.T) {
+ t.Parallel()
+
+ registryServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Write(searchResponse()) // empty results
+ }))
+ defer registryServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ RegistryBaseURL: registryServer.URL,
+ })
+
+ result, err := src.Discover(context.Background(), "nonexistent-api")
+
+ assert.NoError(t, err)
+ assert.Empty(t, result.Endpoints)
+ assert.Empty(t, result.BaseURLCandidates)
+ })
+
+ t.Run("tarball download fails gracefully", func(t *testing.T) {
+ t.Parallel()
+
+ registryServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch r.URL.Path {
+ case "/-/v1/search":
+ w.Write(searchResponse(
+ npmPackageInfo{
+ Name: "broken-sdk",
+ Version: "1.0.0",
+ Date: time.Now().Add(-24 * time.Hour),
+ },
+ ))
+ case "/broken-sdk/1.0.0":
+ // Return a tarball URL that will fail (non-HTTPS).
+ w.Write(versionResponse("http://bad-server/tarball.tgz"))
+ default:
+ http.NotFound(w, r)
+ }
+ }))
+ defer registryServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ RegistryBaseURL: registryServer.URL,
+ })
+
+ result, err := src.Discover(context.Background(), "broken")
+
+ assert.NoError(t, err)
+ // Should gracefully skip the broken package.
+ assert.Empty(t, result.Endpoints)
+ })
+
+ t.Run("search API error is non-fatal", func(t *testing.T) {
+ t.Parallel()
+
+ registryServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(http.StatusInternalServerError)
+ }))
+ defer registryServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ RegistryBaseURL: registryServer.URL,
+ })
+
+ result, err := src.Discover(context.Background(), "some-api")
+
+ assert.NoError(t, err) // non-fatal
+ assert.Empty(t, result.Endpoints)
+ })
+
+ t.Run("version metadata 404 skips package", func(t *testing.T) {
+ t.Parallel()
+
+ registryServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch r.URL.Path {
+ case "/-/v1/search":
+ w.Write(searchResponse(
+ npmPackageInfo{
+ Name: "missing-sdk",
+ Version: "1.0.0",
+ Date: time.Now().Add(-24 * time.Hour),
+ },
+ ))
+ default:
+ http.NotFound(w, r)
+ }
+ }))
+ defer registryServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ RegistryBaseURL: registryServer.URL,
+ })
+
+ result, err := src.Discover(context.Background(), "missing")
+
+ assert.NoError(t, err)
+ assert.Empty(t, result.Endpoints)
+ })
+}
+
+func TestExtractTarball(t *testing.T) {
+ t.Parallel()
+
+ t.Run("extracts regular files", func(t *testing.T) {
+ t.Parallel()
+
+ tarball := buildTarball(t, map[string]string{
+ "package/index.js": `console.log("hello");`,
+ "package/lib/utils.js": `module.exports = {};`,
+ })
+
+ tmpDir := t.TempDir()
+ err := extractTarball(bytes.NewReader(tarball), tmpDir)
+
+ require.NoError(t, err)
+ assert.FileExists(t, tmpDir+"/package/index.js")
+ assert.FileExists(t, tmpDir+"/package/lib/utils.js")
+ })
+
+ t.Run("skips symlinks", func(t *testing.T) {
+ t.Parallel()
+
+ tarball := buildTarballWithSymlink(t)
+ tmpDir := t.TempDir()
+
+ err := extractTarball(bytes.NewReader(tarball), tmpDir)
+
+ require.NoError(t, err)
+ // Regular file should exist.
+ assert.FileExists(t, tmpDir+"/package/index.js")
+ // Symlink should NOT exist.
+ assert.NoFileExists(t, tmpDir+"/package/evil-link")
+ })
+
+ t.Run("rejects path traversal", func(t *testing.T) {
+ t.Parallel()
+
+ var buf bytes.Buffer
+ gw := gzip.NewWriter(&buf)
+ tw := tar.NewWriter(gw)
+
+ // Try to write outside the dest dir.
+ content := "malicious content"
+ hdr := &tar.Header{
+ Name: "../../../etc/evil",
+ Mode: 0o644,
+ Size: int64(len(content)),
+ Typeflag: tar.TypeReg,
+ }
+ require.NoError(t, tw.WriteHeader(hdr))
+ _, err := tw.Write([]byte(content))
+ require.NoError(t, err)
+
+ // Add a normal file too.
+ normalContent := `this.get("/v1/safe");`
+ normalHdr := &tar.Header{
+ Name: "package/safe.js",
+ Mode: 0o644,
+ Size: int64(len(normalContent)),
+ Typeflag: tar.TypeReg,
+ }
+ require.NoError(t, tw.WriteHeader(normalHdr))
+ _, err = tw.Write([]byte(normalContent))
+ require.NoError(t, err)
+
+ require.NoError(t, tw.Close())
+ require.NoError(t, gw.Close())
+
+ tmpDir := t.TempDir()
+ extractErr := extractTarball(bytes.NewReader(buf.Bytes()), tmpDir)
+
+ require.NoError(t, extractErr)
+ // Normal file should exist.
+ assert.FileExists(t, tmpDir+"/package/safe.js")
+ // Traversal file should NOT exist outside tmpDir.
+ assert.NoFileExists(t, tmpDir+"/../../../etc/evil")
+ })
+}
+
+func TestClassifyPackage(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ pkg npmPackageInfo
+ apiName string
+ wantTier string
+ }{
+ {
+ name: "scoped official SDK",
+ pkg: npmPackageInfo{Name: "@notion/client", Scope: "@notion"},
+ apiName: "notion",
+ wantTier: TierOfficialSDK,
+ },
+ {
+ name: "scoped official SDK with hq suffix",
+ pkg: npmPackageInfo{Name: "@notionhq/client", Scope: "@notionhq"},
+ apiName: "notion",
+ wantTier: TierOfficialSDK,
+ },
+ {
+ name: "unscoped community SDK",
+ pkg: npmPackageInfo{Name: "notion-helper", Scope: ""},
+ apiName: "notion",
+ wantTier: TierCommunitySDK,
+ },
+ {
+ name: "different scope community SDK",
+ pkg: npmPackageInfo{Name: "@somedev/notion-utils", Scope: "@somedev"},
+ apiName: "notion",
+ wantTier: TierCommunitySDK,
+ },
+ {
+ name: "stripe official by scope",
+ pkg: npmPackageInfo{Name: "@stripe/stripe-js", Scope: "@stripe"},
+ apiName: "stripe",
+ wantTier: TierOfficialSDK,
+ },
+ {
+ name: "api name contains scope",
+ pkg: npmPackageInfo{Name: "@cal/sdk", Scope: "@cal"},
+ apiName: "cal.com",
+ wantTier: TierOfficialSDK,
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+ got := classifyPackage(tt.pkg, tt.apiName)
+ assert.Equal(t, tt.wantTier, got)
+ })
+ }
+}
+
+func TestNPMSource_Search(t *testing.T) {
+ t.Parallel()
+
+ t.Run("parses search results", func(t *testing.T) {
+ t.Parallel()
+
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ assert.Equal(t, "/-/v1/search", r.URL.Path)
+ assert.Equal(t, "notion", r.URL.Query().Get("text"))
+ assert.Equal(t, "25", r.URL.Query().Get("size"))
+
+ w.Write(searchResponse(
+ npmPackageInfo{
+ Name: "@notionhq/client",
+ Scope: "@notionhq",
+ Version: "2.2.0",
+ Date: time.Date(2024, 1, 15, 0, 0, 0, 0, time.UTC),
+ },
+ npmPackageInfo{
+ Name: "notion-client",
+ Version: "1.0.0",
+ Date: time.Date(2024, 2, 1, 0, 0, 0, 0, time.UTC),
+ },
+ ))
+ }))
+ defer server.Close()
+
+ src := NewNPMSource(NPMOptions{RegistryBaseURL: server.URL})
+ packages, err := src.search(context.Background(), "notion")
+
+ require.NoError(t, err)
+ assert.Len(t, packages, 2)
+ assert.Equal(t, "@notionhq/client", packages[0].Name)
+ assert.Equal(t, "notion-client", packages[1].Name)
+ })
+
+ t.Run("handles API error", func(t *testing.T) {
+ t.Parallel()
+
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(http.StatusServiceUnavailable)
+ }))
+ defer server.Close()
+
+ src := NewNPMSource(NPMOptions{RegistryBaseURL: server.URL})
+ _, err := src.search(context.Background(), "test")
+
+ assert.Error(t, err)
+ assert.Contains(t, err.Error(), "status 503")
+ })
+}
+
+func TestNPMSource_FetchDownloads(t *testing.T) {
+ t.Parallel()
+
+ t.Run("parses bulk downloads", func(t *testing.T) {
+ t.Parallel()
+
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ assert.Contains(t, r.URL.Path, "/downloads/point/last-week/")
+ w.Write(downloadsResponse(map[string]int{
+ "pkg-a": 1000,
+ "pkg-b": 50,
+ }))
+ }))
+ defer server.Close()
+
+ src := NewNPMSource(NPMOptions{DownloadsBaseURL: server.URL})
+ result := src.fetchDownloads(context.Background(), []npmPackageInfo{
+ {Name: "pkg-a"},
+ {Name: "pkg-b"},
+ })
+
+ assert.Equal(t, 1000, result["pkg-a"])
+ assert.Equal(t, 50, result["pkg-b"])
+ })
+
+ t.Run("handles API error gracefully", func(t *testing.T) {
+ t.Parallel()
+
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(http.StatusInternalServerError)
+ }))
+ defer server.Close()
+
+ src := NewNPMSource(NPMOptions{DownloadsBaseURL: server.URL})
+ result := src.fetchDownloads(context.Background(), []npmPackageInfo{
+ {Name: "pkg-a"},
+ })
+
+ assert.Empty(t, result)
+ })
+
+ t.Run("empty packages returns empty map", func(t *testing.T) {
+ t.Parallel()
+
+ src := NewNPMSource(NPMOptions{})
+ result := src.fetchDownloads(context.Background(), nil)
+
+ assert.Empty(t, result)
+ })
+}
+
+func TestNPMSource_FetchTarballURL(t *testing.T) {
+ t.Parallel()
+
+ t.Run("extracts tarball URL from version metadata", func(t *testing.T) {
+ t.Parallel()
+
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ assert.Equal(t, "/example-sdk/1.0.0", r.URL.Path)
+ w.Write(versionResponse("https://registry.npmjs.org/example-sdk/-/example-sdk-1.0.0.tgz"))
+ }))
+ defer server.Close()
+
+ src := NewNPMSource(NPMOptions{RegistryBaseURL: server.URL})
+ tarballURL, err := src.fetchTarballURL(context.Background(), "example-sdk", "1.0.0")
+
+ require.NoError(t, err)
+ assert.Equal(t, "https://registry.npmjs.org/example-sdk/-/example-sdk-1.0.0.tgz", tarballURL)
+ })
+
+ t.Run("handles missing tarball URL", func(t *testing.T) {
+ t.Parallel()
+
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Write([]byte(`{"dist": {}}`))
+ }))
+ defer server.Close()
+
+ src := NewNPMSource(NPMOptions{RegistryBaseURL: server.URL})
+ _, err := src.fetchTarballURL(context.Background(), "bad-pkg", "1.0.0")
+
+ assert.Error(t, err)
+ assert.Contains(t, err.Error(), "no tarball URL")
+ })
+}
+
+func TestNPMSource_ProcessPackageTarball(t *testing.T) {
+ t.Parallel()
+
+ t.Run("extracts endpoints from tarball", func(t *testing.T) {
+ t.Parallel()
+
+ sdkContent := `
+const baseUrl = "https://api.test.com";
+class API {
+ listUsers() { return this.get("/v1/users"); }
+ createItem(data) { return this.post("/v1/items", data); }
+}
+`
+ tarball := buildTarball(t, map[string]string{
+ "package/src/client.js": sdkContent,
+ })
+
+ tarballServer := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Write(tarball)
+ }))
+ defer tarballServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ HTTPClient: tarballServer.Client(),
+ })
+
+ endpoints, baseURLs, err := src.processPackageTarball(
+ context.Background(),
+ tarballServer.URL+"/tarball.tgz",
+ "test-sdk",
+ TierCommunitySDK,
+ 500,
+ )
+
+ require.NoError(t, err)
+
+ // Check endpoints were found.
+ assert.NotEmpty(t, endpoints, "expected endpoints to be extracted")
+
+ var paths []string
+ for _, ep := range endpoints {
+ paths = append(paths, ep.Path)
+ }
+ assert.Contains(t, paths, "/v1/users")
+ assert.Contains(t, paths, "/v1/items")
+
+ // Check base URLs.
+ assert.Contains(t, baseURLs, "https://api.test.com")
+ })
+
+ t.Run("rejects non-HTTPS tarball URL", func(t *testing.T) {
+ t.Parallel()
+
+ src := NewNPMSource(NPMOptions{})
+ _, _, err := src.processPackageTarball(
+ context.Background(),
+ "http://evil.com/tarball.tgz",
+ "evil-sdk",
+ TierCommunitySDK,
+ 0,
+ )
+
+ assert.Error(t, err)
+ assert.Contains(t, err.Error(), "HTTPS")
+ })
+
+ t.Run("skips non-JS/TS files", func(t *testing.T) {
+ t.Parallel()
+
+ tarball := buildTarball(t, map[string]string{
+ "package/readme.md": `this.get("/v1/docs-only");`,
+ "package/data.json": `{"url": "/v1/json-data"}`,
+ "package/src/index.ts": `this.get("/v1/real-endpoint");`,
+ })
+
+ tarballServer := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Write(tarball)
+ }))
+ defer tarballServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ HTTPClient: tarballServer.Client(),
+ })
+
+ endpoints, _, err := src.processPackageTarball(
+ context.Background(),
+ tarballServer.URL+"/tarball.tgz",
+ "test-sdk",
+ TierCommunitySDK,
+ 0,
+ )
+
+ require.NoError(t, err)
+
+ var paths []string
+ for _, ep := range endpoints {
+ paths = append(paths, ep.Path)
+ }
+ assert.Contains(t, paths, "/v1/real-endpoint")
+ // MD and JSON files should not have been grepped.
+ assert.NotContains(t, paths, "/v1/docs-only")
+ assert.NotContains(t, paths, "/v1/json-data")
+ })
+
+ t.Run("handles tarball with symlinks", func(t *testing.T) {
+ t.Parallel()
+
+ tarball := buildTarballWithSymlink(t)
+
+ tarballServer := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Write(tarball)
+ }))
+ defer tarballServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ HTTPClient: tarballServer.Client(),
+ })
+
+ endpoints, _, err := src.processPackageTarball(
+ context.Background(),
+ tarballServer.URL+"/tarball.tgz",
+ "symlink-sdk",
+ TierCommunitySDK,
+ 0,
+ )
+
+ require.NoError(t, err)
+ // Should still extract endpoints from the safe file.
+ var paths []string
+ for _, ep := range endpoints {
+ paths = append(paths, ep.Path)
+ }
+ assert.Contains(t, paths, "/v1/safe")
+ })
+}
+
+func TestNewNPMSource_Defaults(t *testing.T) {
+ t.Parallel()
+
+ src := NewNPMSource(NPMOptions{})
+
+ assert.Equal(t, defaultRegistryBaseURL, src.registryBaseURL)
+ assert.Equal(t, defaultDownloadsBaseURL, src.downloadsBaseURL)
+ assert.Equal(t, defaultRecencyCutoff, src.recencyCutoff)
+ assert.NotNil(t, src.httpClient)
+}
+
+func TestNewNPMSource_CustomOptions(t *testing.T) {
+ t.Parallel()
+
+ client := &http.Client{Timeout: 30 * time.Second}
+ src := NewNPMSource(NPMOptions{
+ RegistryBaseURL: "https://custom-registry.com",
+ DownloadsBaseURL: "https://custom-downloads.com",
+ HTTPClient: client,
+ RecencyCutoff: 90 * 24 * time.Hour,
+ })
+
+ assert.Equal(t, "https://custom-registry.com", src.registryBaseURL)
+ assert.Equal(t, "https://custom-downloads.com", src.downloadsBaseURL)
+ assert.Equal(t, 90*24*time.Hour, src.recencyCutoff)
+ assert.Same(t, client, src.httpClient)
+}
+
+func TestNPMSource_RecencyCutoffFiltering(t *testing.T) {
+ t.Parallel()
+
+ t.Run("custom cutoff of 30 days", func(t *testing.T) {
+ t.Parallel()
+
+ registryServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch r.URL.Path {
+ case "/-/v1/search":
+ w.Write(searchResponse(
+ npmPackageInfo{
+ Name: "recent-sdk",
+ Version: "1.0.0",
+ Date: time.Now().Add(-7 * 24 * time.Hour), // 7 days ago
+ },
+ npmPackageInfo{
+ Name: "old-sdk",
+ Version: "1.0.0",
+ Date: time.Now().Add(-60 * 24 * time.Hour), // 60 days ago
+ },
+ ))
+ default:
+ // Return 404 for version lookups — old-sdk should never reach this.
+ if r.URL.Path == "/old-sdk/1.0.0" {
+ t.Error("old-sdk should have been filtered out by recency cutoff")
+ }
+ // recent-sdk version lookup will fail; that's fine — we're testing filtering.
+ http.NotFound(w, r)
+ }
+ }))
+ defer registryServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ RegistryBaseURL: registryServer.URL,
+ RecencyCutoff: 30 * 24 * time.Hour,
+ })
+
+ result, err := src.Discover(context.Background(), "test")
+
+ assert.NoError(t, err)
+ // old-sdk (60 days) should be filtered, recent-sdk (7 days) proceeds
+ // but version lookup fails so no endpoints. Key check: no error.
+ _ = result
+ })
+}
+
+func TestNPMSource_ProcessPackageTarball_WithParams(t *testing.T) {
+ t.Parallel()
+
+ t.Run("extracts endpoints with params from SDK code", func(t *testing.T) {
+ t.Parallel()
+
+ // Steam-like SDK content that has both endpoint paths and params objects.
+ sdkContent := `
+const BASE_URL = "https://api.steampowered.com";
+
+class SteamAPI {
+ getOwnedGames(steamid, { includeAppInfo = true } = {}) {
+ return this.get("/IPlayerService/GetOwnedGames/v1", {
+ steamid: steamid,
+ include_appinfo: includeAppInfo,
+ include_played_free_games: true
+ });
+ }
+
+ getRecentlyPlayed(steamid, count = 10) {
+ return this.get("/IPlayerService/GetRecentlyPlayedGames/v1", {
+ steamid,
+ count
+ });
+ }
+}
+`
+ tarball := buildTarball(t, map[string]string{
+ "package/src/client.js": sdkContent,
+ })
+
+ tarballServer := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Write(tarball)
+ }))
+ defer tarballServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ HTTPClient: tarballServer.Client(),
+ })
+
+ endpoints, baseURLs, err := src.processPackageTarball(
+ context.Background(),
+ tarballServer.URL+"/tarball.tgz",
+ "steam-sdk",
+ TierCommunitySDK,
+ 500,
+ )
+
+ require.NoError(t, err)
+ assert.NotEmpty(t, endpoints, "expected endpoints to be extracted")
+ assert.Contains(t, baseURLs, "https://api.steampowered.com")
+
+ // Check that at least one endpoint has params populated.
+ var endpointWithParams *DiscoveredEndpoint
+ for i, ep := range endpoints {
+ if len(ep.Params) > 0 {
+ endpointWithParams = &endpoints[i]
+ break
+ }
+ }
+ require.NotNil(t, endpointWithParams, "expected at least one endpoint with params")
+
+ // Verify param names exist.
+ paramNames := make(map[string]bool)
+ for _, p := range endpointWithParams.Params {
+ paramNames[p.Name] = true
+ }
+ // Should have extracted some of the params from the object literal.
+ assert.True(t, len(paramNames) > 0, "expected at least one param name")
+ })
+}
+
+func TestNPMSource_MaxPackagesLimit(t *testing.T) {
+ t.Parallel()
+
+ var versionCallCount int
+
+ // Create 15 recent packages; only 10 should be processed.
+ packages := make([]npmPackageInfo, 15)
+ for i := range packages {
+ packages[i] = npmPackageInfo{
+ Name: fmt.Sprintf("sdk-%d", i),
+ Version: "1.0.0",
+ Date: time.Now().Add(-24 * time.Hour),
+ }
+ }
+
+ registryServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch r.URL.Path {
+ case "/-/v1/search":
+ w.Write(searchResponse(packages...))
+ default:
+ versionCallCount++
+ http.NotFound(w, r) // version lookups fail; just counting
+ }
+ }))
+ defer registryServer.Close()
+
+ downloadsServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Write(downloadsResponse(map[string]int{}))
+ }))
+ defer downloadsServer.Close()
+
+ src := NewNPMSource(NPMOptions{
+ RegistryBaseURL: registryServer.URL,
+ DownloadsBaseURL: downloadsServer.URL,
+ })
+
+ _, err := src.Discover(context.Background(), "test")
+
+ assert.NoError(t, err)
+ // Only first 10 should have version lookups attempted.
+ assert.LessOrEqual(t, versionCallCount, maxPackagesToProcess)
+}
diff --git a/internal/crowdsniff/params.go b/internal/crowdsniff/params.go
new file mode 100644
index 00000000..fd020339
--- /dev/null
+++ b/internal/crowdsniff/params.go
@@ -0,0 +1,658 @@
+package crowdsniff
+
+import (
+ "regexp"
+ "strings"
+ "unicode"
+)
+
+// EnrichWithParams does a second pass over SDK source content, finding the params
+// object literal after each HTTP method call and extracting param names, types,
+// and required/optional status.
+//
+// This function operates on full content (not line-by-line) — a deliberate
+// architectural asymmetry with GrepEndpoints. GrepEndpoints works line-by-line
+// because endpoint discovery only needs method+path from a single line.
+// Param extraction needs multi-line object literals and backward scanning for
+// function signatures, which requires full-content access.
+//
+// EnrichWithParams must match raw URL patterns in the content independently,
+// then correlate to DiscoveredEndpoint entries by comparing cleaned paths.
+// It does NOT substring-match endpoint.Path against content.
+func EnrichWithParams(content string, endpoints []DiscoveredEndpoint) []DiscoveredEndpoint {
+ if len(endpoints) == 0 {
+ return endpoints
+ }
+
+ // Build a lookup from (method, path) -> index into endpoints slice.
+ // We'll collect params per endpoint index.
+ type epKey struct {
+ method string
+ path string
+ }
+ epIndex := make(map[epKey]int)
+ for i, ep := range endpoints {
+ k := epKey{method: ep.Method, path: ep.Path}
+ // First occurrence wins (consistent with dedup).
+ if _, exists := epIndex[k]; !exists {
+ epIndex[k] = i
+ }
+ }
+
+ // Independently scan content for HTTP method calls using the same patterns.
+ methodIndexes := httpMethodCall.FindAllStringSubmatchIndex(content, -1)
+
+ for _, methodIdx := range methodIndexes {
+ method := strings.ToUpper(content[methodIdx[2]:methodIdx[3]])
+ if !validHTTPMethods[method] {
+ continue
+ }
+
+ // Find the first URL path after this method call's opening paren.
+ remainder := content[methodIdx[1]:]
+ pathMatchIdx := urlPathLiteral.FindStringSubmatchIndex(remainder)
+ if pathMatchIdx == nil {
+ continue
+ }
+
+ rawPath := remainder[pathMatchIdx[2]:pathMatchIdx[3]]
+ cleaned := cleanPath(rawPath)
+
+ k := epKey{method: method, path: cleaned}
+ idx, found := epIndex[k]
+ if !found {
+ continue
+ }
+
+ // Position in content where the URL path literal ends (after closing quote).
+ urlEndPos := methodIdx[1] + pathMatchIdx[1]
+
+ // Extract params object starting from after the URL.
+ params := extractParamsFromPosition(content, urlEndPos)
+ if params == nil {
+ continue
+ }
+
+ // Look backward for function signature to determine required/optional.
+ funcStart := methodIdx[0] - maxFuncSignatureLookback
+ if funcStart < 0 {
+ funcStart = 0
+ }
+ preamble := content[funcStart:methodIdx[0]]
+ requiredNames := extractRequiredFromSignature(preamble)
+
+ // Apply required/optional from function signature correlation.
+ for i := range params {
+ if _, isReq := requiredNames[params[i].Name]; isReq {
+ params[i].Required = true
+ }
+ }
+
+ // Only set params on the first match for this endpoint.
+ // Avoids cross-product if the same endpoint path appears multiple times.
+ if endpoints[idx].Params == nil {
+ endpoints[idx].Params = params
+ }
+ }
+
+ return endpoints
+}
+
+// maxParamScanDistance is the maximum distance forward from the URL match to
+// look for the opening brace of a params object.
+const maxParamScanDistance = 500
+
+// maxFuncSignatureLookback is the maximum distance backward from the HTTP call
+// to scan for a function definition when correlating required/optional params.
+// 1000 bytes covers most real-world SDK methods while avoiding full-file scans.
+const maxFuncSignatureLookback = 1000
+
+// extractParamsFromPosition scans forward from urlEndPos in content for a
+// params object literal ({ key1: val1, key2: val2 }) and extracts the keys.
+// Returns nil if no params object is found or if the object is malformed.
+func extractParamsFromPosition(content string, urlEndPos int) []DiscoveredParam {
+ // Scan forward for ", {" or ",\n {" pattern.
+ scanEnd := urlEndPos + maxParamScanDistance
+ if scanEnd > len(content) {
+ scanEnd = len(content)
+ }
+ region := content[urlEndPos:scanEnd]
+
+ braceStart := findParamsObjectStart(region)
+ if braceStart < 0 {
+ return nil
+ }
+
+ // braceStart is relative to urlEndPos, convert to absolute.
+ absStart := urlEndPos + braceStart
+
+ // Run brace-matching scanner from absStart.
+ block := extractBraceBlock(content, absStart)
+ if block == "" {
+ return nil
+ }
+
+ return extractKeysFromBlock(block)
+}
+
+// findParamsObjectStart finds the position of the opening brace for a params
+// object in the region after a URL match. Looks for a comma followed by
+// optional whitespace/newlines then an opening brace.
+// Returns the position of '{' relative to region start, or -1 if not found.
+func findParamsObjectStart(region string) int {
+ foundComma := false
+ for i, ch := range region {
+ if !foundComma {
+ switch ch {
+ case ',':
+ foundComma = true
+ case ')':
+ // Closing paren before finding comma means no params.
+ return -1
+ }
+ continue
+ }
+ // After comma, skip whitespace/newlines until we find '{' or give up.
+ switch {
+ case ch == '{':
+ return i
+ case !unicode.IsSpace(ch):
+ // Non-whitespace, non-brace after comma — not a params object.
+ // Could be a second string argument or variable.
+ return -1
+ }
+ }
+ return -1
+}
+
+// extractBraceBlock starts at content[pos] which must be '{', and uses a
+// brace-matching scanner to find the closing '}'. Returns the content between
+// the braces (exclusive), or "" if unbalanced.
+// Tracks string literal and comment state to skip braces inside quotes and comments.
+func extractBraceBlock(content string, pos int) string {
+ if pos >= len(content) || content[pos] != '{' {
+ return ""
+ }
+
+ depth := 0
+ inString := false
+ var stringChar byte
+ inLineComment := false
+ inBlockComment := false
+
+ scanEnd := pos + maxParamScanDistance
+ if scanEnd > len(content) {
+ scanEnd = len(content)
+ }
+
+ for i := pos; i < scanEnd; i++ {
+ ch := content[i]
+
+ // Line comment: skip until newline.
+ if inLineComment {
+ if ch == '\n' {
+ inLineComment = false
+ }
+ continue
+ }
+
+ // Block comment: skip until */.
+ if inBlockComment {
+ if ch == '*' && i+1 < scanEnd && content[i+1] == '/' {
+ inBlockComment = false
+ i++ // skip the '/'
+ }
+ continue
+ }
+
+ // Track string literal state.
+ if inString {
+ if ch == '\\' {
+ i++ // skip escaped character
+ continue
+ }
+ if ch == stringChar {
+ inString = false
+ }
+ continue
+ }
+
+ // Detect comment start.
+ if ch == '/' && i+1 < scanEnd {
+ next := content[i+1]
+ if next == '/' {
+ inLineComment = true
+ i++ // skip second '/'
+ continue
+ }
+ if next == '*' {
+ inBlockComment = true
+ i++ // skip '*'
+ continue
+ }
+ }
+
+ if ch == '\'' || ch == '"' || ch == '`' {
+ inString = true
+ stringChar = ch
+ continue
+ }
+
+ switch ch {
+ case '{':
+ depth++
+ case '}':
+ depth--
+ if depth == 0 {
+ // Return content between outer braces.
+ return content[pos+1 : i]
+ }
+ }
+ }
+
+ // Unbalanced — return empty.
+ return ""
+}
+
+// extractKeysFromBlock parses the content between outer braces of an object
+// literal and extracts top-level keys with type inference.
+// Handles: shorthand properties, key-value pairs, nested objects (skipped),
+// and trailing commas.
+func extractKeysFromBlock(block string) []DiscoveredParam {
+ var params []DiscoveredParam
+ seen := make(map[string]bool)
+
+ entries := splitObjectEntries(block)
+ for _, entry := range entries {
+ entry = stripComments(entry)
+ entry = strings.TrimSpace(entry)
+ if entry == "" {
+ continue
+ }
+
+ key, value := parseObjectEntry(entry)
+ if key == "" {
+ continue
+ }
+ // Skip keys that start with spread operator.
+ if strings.HasPrefix(key, "...") {
+ continue
+ }
+ if seen[key] {
+ continue
+ }
+ seen[key] = true
+
+ paramType := inferType(key, value)
+ params = append(params, DiscoveredParam{
+ Name: key,
+ Type: paramType,
+ })
+ }
+
+ if len(params) == 0 {
+ return nil
+ }
+ return params
+}
+
+// splitObjectEntries splits an object literal body into entries, respecting
+// nested braces, string literals, and comments. Only splits on commas at depth 0.
+func splitObjectEntries(block string) []string {
+ var entries []string
+ depth := 0
+ inString := false
+ var stringChar byte
+ inLineComment := false
+ inBlockComment := false
+ start := 0
+
+ for i := 0; i < len(block); i++ {
+ ch := block[i]
+
+ if inLineComment {
+ if ch == '\n' {
+ inLineComment = false
+ }
+ continue
+ }
+
+ if inBlockComment {
+ if ch == '*' && i+1 < len(block) && block[i+1] == '/' {
+ inBlockComment = false
+ i++
+ }
+ continue
+ }
+
+ if inString {
+ if ch == '\\' {
+ i++
+ continue
+ }
+ if ch == stringChar {
+ inString = false
+ }
+ continue
+ }
+
+ // Detect comment start.
+ if ch == '/' && i+1 < len(block) {
+ next := block[i+1]
+ if next == '/' {
+ inLineComment = true
+ i++
+ continue
+ }
+ if next == '*' {
+ inBlockComment = true
+ i++
+ continue
+ }
+ }
+
+ if ch == '\'' || ch == '"' || ch == '`' {
+ inString = true
+ stringChar = ch
+ continue
+ }
+
+ if ch == '{' || ch == '[' || ch == '(' {
+ depth++
+ } else if ch == '}' || ch == ']' || ch == ')' {
+ depth--
+ } else if ch == ',' && depth == 0 {
+ entries = append(entries, block[start:i])
+ start = i + 1
+ }
+ }
+ // Last entry (no trailing comma).
+ if start < len(block) {
+ entries = append(entries, block[start:])
+ }
+ return entries
+}
+
+// stripComments removes JavaScript line comments (//) and block comments (/* */)
+// from a string, preserving content outside comments. Respects string literals.
+func stripComments(s string) string {
+ var b strings.Builder
+ b.Grow(len(s))
+ inString := false
+ var stringChar byte
+
+ for i := 0; i < len(s); i++ {
+ ch := s[i]
+
+ if inString {
+ b.WriteByte(ch)
+ if ch == '\\' && i+1 < len(s) {
+ i++
+ b.WriteByte(s[i])
+ continue
+ }
+ if ch == stringChar {
+ inString = false
+ }
+ continue
+ }
+
+ if ch == '\'' || ch == '"' || ch == '`' {
+ inString = true
+ stringChar = ch
+ b.WriteByte(ch)
+ continue
+ }
+
+ if ch == '/' && i+1 < len(s) {
+ next := s[i+1]
+ if next == '/' {
+ // Line comment: skip to end of line.
+ for i+1 < len(s) && s[i+1] != '\n' {
+ i++
+ }
+ continue
+ }
+ if next == '*' {
+ // Block comment: skip to */.
+ i += 2
+ for i+1 < len(s) {
+ if s[i] == '*' && s[i+1] == '/' {
+ i++
+ break
+ }
+ i++
+ }
+ continue
+ }
+ }
+
+ b.WriteByte(ch)
+ }
+
+ return b.String()
+}
+
+// parseObjectEntry parses a single object entry and returns (key, value).
+// Handles:
+// - "key: value" → ("key", "value")
+// - "key" (shorthand) → ("key", "")
+func parseObjectEntry(entry string) (string, string) {
+ entry = strings.TrimSpace(entry)
+ if entry == "" {
+ return "", ""
+ }
+
+ // Find the first colon that's not inside a string or nested structure.
+ colonIdx := findTopLevelColon(entry)
+ if colonIdx < 0 {
+ // Shorthand property: bare identifier.
+ key := strings.TrimSpace(entry)
+ if isValidIdentifier(key) {
+ return key, ""
+ }
+ return "", ""
+ }
+
+ key := strings.TrimSpace(entry[:colonIdx])
+ value := strings.TrimSpace(entry[colonIdx+1:])
+ if isValidIdentifier(key) {
+ return key, value
+ }
+ return "", ""
+}
+
+// findTopLevelColon finds the position of the first colon at depth 0 in entry,
+// respecting string literals. Returns -1 if not found.
+func findTopLevelColon(entry string) int {
+ inString := false
+ var stringChar byte
+
+ for i := 0; i < len(entry); i++ {
+ ch := entry[i]
+
+ if inString {
+ if ch == '\\' {
+ i++
+ continue
+ }
+ if ch == stringChar {
+ inString = false
+ }
+ continue
+ }
+
+ if ch == '\'' || ch == '"' || ch == '`' {
+ inString = true
+ stringChar = ch
+ continue
+ }
+
+ if ch == ':' {
+ return i
+ }
+ }
+ return -1
+}
+
+// isValidIdentifier checks if s is a valid JS identifier (letters, digits,
+// underscores, $, starting with a non-digit).
+func isValidIdentifier(s string) bool {
+ if s == "" {
+ return false
+ }
+ for i, ch := range s {
+ if i == 0 {
+ if !unicode.IsLetter(ch) && ch != '_' && ch != '$' {
+ return false
+ }
+ } else {
+ if !unicode.IsLetter(ch) && !unicode.IsDigit(ch) && ch != '_' && ch != '$' {
+ return false
+ }
+ }
+ }
+ return true
+}
+
+// inferType applies heuristic rules to determine the parameter type from its
+// key name and value expression.
+func inferType(key, value string) string {
+ // Value-based heuristics first (more specific).
+ if value != "" {
+ if strings.Contains(value, ".join(") {
+ return "string"
+ }
+ trimVal := strings.TrimSpace(value)
+ if trimVal == "true" || trimVal == "false" {
+ return "boolean"
+ }
+ if isNumericLiteral(trimVal) {
+ return "integer"
+ }
+ }
+
+ // Key-name-based heuristics.
+ if integerKeyName.MatchString(key) {
+ return "integer"
+ }
+
+ return "string"
+}
+
+// integerKeyName matches param names that are typically integers.
+// Uses word boundaries to avoid matching substrings like "accountId".
+var integerKeyName = regexp.MustCompile(`^(count|limit|offset|page|maxlength)$`)
+
+// isNumericLiteral checks if a string looks like a numeric literal.
+func isNumericLiteral(s string) bool {
+ if s == "" {
+ return false
+ }
+ // Allow leading minus for negative numbers.
+ start := 0
+ if s[0] == '-' {
+ start = 1
+ }
+ if start >= len(s) {
+ return false
+ }
+ hasDigit := false
+ for i := start; i < len(s); i++ {
+ if s[i] >= '0' && s[i] <= '9' {
+ hasDigit = true
+ } else if s[i] == '.' {
+ // Allow decimal point.
+ continue
+ } else {
+ return false
+ }
+ }
+ return hasDigit
+}
+
+// Function signature patterns for backward scanning.
+var funcSignaturePatterns = []*regexp.Regexp{
+ // async function name(...)
+ regexp.MustCompile(`async\s+function\s+\w+\s*\(([^)]*)\)`),
+ // function name(...)
+ regexp.MustCompile(`function\s+\w+\s*\(([^)]*)\)`),
+ // name(...) { — class method shorthand
+ regexp.MustCompile(`\w+\s*\(([^)]*)\)\s*\{`),
+ // name = async (...) or name = function(...)
+ regexp.MustCompile(`\w+\s*=\s*(?:async\s+)?(?:function)?\s*\(([^)]*)\)`),
+}
+
+// extractRequiredFromSignature scans backward (the preamble before the HTTP
+// call) for the nearest function definition and extracts which parameter names
+// are required (positional args without defaults).
+// Returns a set of required parameter names, or nil if no function signature
+// is found.
+func extractRequiredFromSignature(preamble string) map[string]bool {
+ // Find the last function signature in the preamble (nearest to the HTTP call).
+ var bestMatch string
+ bestPos := -1
+
+ for _, pat := range funcSignaturePatterns {
+ allMatches := pat.FindAllStringSubmatchIndex(preamble, -1)
+ for _, idx := range allMatches {
+ if idx[0] > bestPos {
+ bestPos = idx[0]
+ bestMatch = preamble[idx[2]:idx[3]]
+ }
+ }
+ }
+
+ if bestMatch == "" {
+ return nil
+ }
+
+ return parseSignatureParams(bestMatch)
+}
+
+// parseSignatureParams parses a function parameter list string and returns
+// a set of names that are required (positional, no default value).
+//
+// Handles:
+// - Simple: "steamid" → required
+// - With default: "count = 10" → optional
+// - Destructured: "{ opt1 = true } = {}" → opt1 is optional
+// - Mixed: "id, { opt1 = true } = {}" → id is required, opt1 is optional
+func parseSignatureParams(paramList string) map[string]bool {
+ required := make(map[string]bool)
+ paramList = strings.TrimSpace(paramList)
+ if paramList == "" {
+ return required
+ }
+
+ // Split on top-level commas (respecting nested braces).
+ parts := splitObjectEntries(paramList)
+
+ for _, part := range parts {
+ part = strings.TrimSpace(part)
+ if part == "" {
+ continue
+ }
+
+ // Check if this is a destructured param: { key1 = default, key2 } = {}
+ if strings.HasPrefix(part, "{") {
+ // Extract names from the destructured object.
+ // These are all optional (they have defaults or are in a default-assigned object).
+ continue
+ }
+
+ // Check for default assignment: name = value
+ if eqIdx := strings.Index(part, "="); eqIdx >= 0 {
+ // Has a default → optional, skip.
+ continue
+ }
+
+ // Simple positional parameter.
+ name := strings.TrimSpace(part)
+ if isValidIdentifier(name) {
+ required[name] = true
+ }
+ }
+
+ return required
+}
diff --git a/internal/crowdsniff/params_test.go b/internal/crowdsniff/params_test.go
new file mode 100644
index 00000000..ae10c996
--- /dev/null
+++ b/internal/crowdsniff/params_test.go
@@ -0,0 +1,493 @@
+package crowdsniff
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+)
+
+func TestEnrichWithParams(t *testing.T) {
+ t.Parallel()
+
+ t.Run("single-line params extracts keys", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { key1: val1, key2: val2 })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result, 1)
+ assert.Len(t, result[0].Params, 2)
+ names := paramNames(result[0].Params)
+ assert.Contains(t, names, "key1")
+ assert.Contains(t, names, "key2")
+ })
+
+ t.Run("multi-line params spanning 4 lines", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/items', {
+ steamid: steamids.join(','),
+ include_appinfo: includeAppInfo,
+ count: 10,
+ format: 'json'
+})`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/items", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result, 1)
+ assert.Len(t, result[0].Params, 4)
+ names := paramNames(result[0].Params)
+ assert.Contains(t, names, "steamid")
+ assert.Contains(t, names, "include_appinfo")
+ assert.Contains(t, names, "count")
+ assert.Contains(t, names, "format")
+ })
+
+ t.Run("shorthand property extracts key", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { steamid })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result, 1)
+ assert.Len(t, result[0].Params, 1)
+ assert.Equal(t, "steamid", result[0].Params[0].Name)
+ })
+
+ t.Run("join value infers string type", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { steamid: steamids.join(',') })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 1)
+ assert.Equal(t, "string", result[0].Params[0].Type)
+ })
+
+ t.Run("boolean value infers boolean type", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { active: true, disabled: false })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 2)
+ for _, p := range result[0].Params {
+ assert.Equal(t, "boolean", p.Type, "param %s should be boolean", p.Name)
+ }
+ })
+
+ t.Run("numeric default infers integer type", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { page: 1, size: 25 })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 2)
+ for _, p := range result[0].Params {
+ assert.Equal(t, "integer", p.Type, "param %s should be integer", p.Name)
+ }
+ })
+
+ t.Run("key name count or limit infers integer type", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { count: count, limit: userLimit, offset: off })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 3)
+ for _, p := range result[0].Params {
+ assert.Equal(t, "integer", p.Type, "param %s should be integer", p.Name)
+ }
+ })
+
+ t.Run("function signature positional arg sets required", func(t *testing.T) {
+ t.Parallel()
+ content := `
+ getSteamUser(steamid) {
+ return this.get('/user', { steamid })
+ }
+`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/user", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 1)
+ assert.Equal(t, "steamid", result[0].Params[0].Name)
+ assert.True(t, result[0].Params[0].Required, "steamid should be required")
+ })
+
+ t.Run("destructured args with defaults are optional", func(t *testing.T) {
+ t.Parallel()
+ content := `
+ getOwnedGames(id, { opt1 = true } = {}) {
+ return this.get('/games', { id, opt1 })
+ }
+`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/games", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 2)
+ paramMap := make(map[string]DiscoveredParam)
+ for _, p := range result[0].Params {
+ paramMap[p.Name] = p
+ }
+ assert.True(t, paramMap["id"].Required, "id should be required")
+ assert.False(t, paramMap["opt1"].Required, "opt1 should be optional")
+ })
+
+ t.Run("no params object after URL returns nil params", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path')`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result, 1)
+ assert.Nil(t, result[0].Params, "params should be nil, not empty slice")
+ })
+
+ t.Run("nested object extracts only top-level key", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { filter: { type: "active" }, name: val })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ names := paramNames(result[0].Params)
+ assert.Contains(t, names, "filter")
+ assert.Contains(t, names, "name")
+ assert.NotContains(t, names, "type", "nested key should not be extracted")
+ })
+
+ t.Run("trailing comma after last param", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { key1: val1, key2: val2, })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 2)
+ names := paramNames(result[0].Params)
+ assert.Contains(t, names, "key1")
+ assert.Contains(t, names, "key2")
+ })
+
+ t.Run("multiple HTTP calls get only own params", func(t *testing.T) {
+ t.Parallel()
+ content := `
+class API {
+ getUsers() {
+ return this.get('/users', { role: 'admin' })
+ }
+ getProjects() {
+ return this.get('/projects', { status: 'active' })
+ }
+}
+`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/users", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ {Method: "GET", Path: "/projects", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result, 2)
+
+ epMap := make(map[string][]DiscoveredParam)
+ for _, ep := range result {
+ epMap[ep.Path] = ep.Params
+ }
+
+ usersNames := paramNames(epMap["/users"])
+ assert.Contains(t, usersNames, "role")
+ assert.NotContains(t, usersNames, "status", "cross-product: /users should not have status")
+
+ projectsNames := paramNames(epMap["/projects"])
+ assert.Contains(t, projectsNames, "status")
+ assert.NotContains(t, projectsNames, "role", "cross-product: /projects should not have role")
+ })
+
+ t.Run("multi-line whitespace between URL and params", func(t *testing.T) {
+ t.Parallel()
+ content := "this.get('/path',\n { key: val })"
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 1)
+ assert.Equal(t, "key", result[0].Params[0].Name)
+ })
+
+ t.Run("malformed object unbalanced braces does not panic", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { key1: val1, key2: { broken`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ assert.NotPanics(t, func() {
+ result := EnrichWithParams(content, endpoints)
+ // Should return endpoint without params rather than crash
+ assert.Len(t, result, 1)
+ assert.Nil(t, result[0].Params)
+ })
+ })
+
+ t.Run("integration: GrepEndpoints then EnrichWithParams", func(t *testing.T) {
+ t.Parallel()
+ content := `
+class SteamAPI {
+ getOwnedGames(steamid, { includeAppInfo = true } = {}) {
+ return this.get('/IPlayerService/GetOwnedGames/v1', {
+ steamid: steamid,
+ include_appinfo: includeAppInfo,
+ count: 10
+ })
+ }
+}
+`
+ endpoints, _ := GrepEndpoints(content, "steamapi", TierCommunitySDK)
+ assert.NotEmpty(t, endpoints, "GrepEndpoints should find at least one endpoint")
+
+ result := EnrichWithParams(content, endpoints)
+
+ // Find the endpoint with our path
+ var found *DiscoveredEndpoint
+ for i := range result {
+ if result[i].Path == "/IPlayerService/GetOwnedGames/v1" {
+ found = &result[i]
+ break
+ }
+ }
+ assert.NotNil(t, found, "should find /IPlayerService/GetOwnedGames/v1")
+ assert.NotNil(t, found.Params, "params should be populated")
+
+ names := paramNames(found.Params)
+ assert.Contains(t, names, "steamid")
+ assert.Contains(t, names, "include_appinfo")
+ assert.Contains(t, names, "count")
+ })
+
+ t.Run("negative: GrepEndpoints alone returns nil params", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/v1/users');`
+
+ endpoints, _ := GrepEndpoints(content, "sdk", TierCommunitySDK)
+
+ assert.NotEmpty(t, endpoints)
+ for _, ep := range endpoints {
+ assert.Nil(t, ep.Params, "GrepEndpoints should not populate params")
+ }
+ })
+
+ t.Run("steam SDK pattern: steamids.join", func(t *testing.T) {
+ t.Parallel()
+ content := `
+ getPlayerSummaries(steamid) {
+ return this.get('/ISteamUser/GetPlayerSummaries/v2', {
+ steamids: steamids.join(',')
+ })
+ }
+`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/ISteamUser/GetPlayerSummaries/v2", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 1)
+ assert.Equal(t, "steamids", result[0].Params[0].Name)
+ assert.Equal(t, "string", result[0].Params[0].Type)
+ })
+
+ t.Run("template literal path matches cleaned endpoint", func(t *testing.T) {
+ t.Parallel()
+ content := "this.get(`/users/${userId}`, { expand: true })"
+ // GrepEndpoints would produce this cleaned path
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/users/{id}", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result, 1)
+ assert.Len(t, result[0].Params, 1)
+ assert.Equal(t, "expand", result[0].Params[0].Name)
+ assert.Equal(t, "boolean", result[0].Params[0].Type)
+ })
+
+ t.Run("page and maxlength key names infer integer", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { page: p, maxlength: ml })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 2)
+ for _, p := range result[0].Params {
+ assert.Equal(t, "integer", p.Type, "param %s should be integer", p.Name)
+ }
+ })
+
+ t.Run("async function signature correlation", func(t *testing.T) {
+ t.Parallel()
+ content := `
+ async function fetchUser(userId) {
+ return this.get('/user', { userId })
+ }
+`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/user", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 1)
+ assert.Equal(t, "userId", result[0].Params[0].Name)
+ assert.True(t, result[0].Params[0].Required, "userId should be required from function sig")
+ })
+
+ t.Run("string value inside quotes infers string type", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { format: 'json' })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ assert.Len(t, result[0].Params, 1)
+ assert.Equal(t, "string", result[0].Params[0].Type)
+ })
+
+ t.Run("braces inside line comments are ignored", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', {
+ key: val, // } closing brace in comment
+ more: val2
+ })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ names := paramNames(result[0].Params)
+ assert.Contains(t, names, "key")
+ assert.Contains(t, names, "more")
+ })
+
+ t.Run("braces inside block comments are ignored", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', {
+ key: val, /* { nested } brace */
+ other: val2
+ })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ names := paramNames(result[0].Params)
+ assert.Contains(t, names, "key")
+ assert.Contains(t, names, "other")
+ })
+
+ t.Run("braces inside string literals are not mismatched", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get('/path', { query: '{complex}', name: val })`
+ endpoints := []DiscoveredEndpoint{
+ {Method: "GET", Path: "/path", SourceTier: TierCommunitySDK, SourceName: "sdk"},
+ }
+
+ result := EnrichWithParams(content, endpoints)
+
+ names := paramNames(result[0].Params)
+ assert.Contains(t, names, "query")
+ assert.Contains(t, names, "name")
+ })
+}
+
+func TestParseSignatureParams(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ input string
+ required []string // expected required param names
+ }{
+ {name: "empty string", input: "", required: nil},
+ {name: "single positional", input: "steamid", required: []string{"steamid"}},
+ {name: "multiple positional", input: "steamid, appid", required: []string{"steamid", "appid"}},
+ {name: "positional with default is optional", input: "count = 10", required: nil},
+ {name: "mixed required and optional", input: "id, count = 10", required: []string{"id"}},
+ {name: "destructured object is optional", input: "{ opt1 = true, opt2 } = {}", required: nil},
+ {name: "positional then destructured", input: "steamid, { includeAppInfo = true } = {}", required: []string{"steamid"}},
+ {name: "multiple positional then destructured", input: "id, name, { page = 1 } = {}", required: []string{"id", "name"}},
+ {name: "only destructured no positional", input: "{ a, b, c } = {}", required: nil},
+ {name: "trailing comma", input: "steamid, ", required: []string{"steamid"}},
+ {name: "whitespace around names", input: " id , name ", required: []string{"id", "name"}},
+ {name: "rest parameter skipped", input: "id, ...args", required: []string{"id"}},
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+ result := parseSignatureParams(tt.input)
+
+ if tt.required == nil {
+ assert.Empty(t, result)
+ } else {
+ for _, name := range tt.required {
+ assert.True(t, result[name], "expected %q to be required", name)
+ }
+ assert.Len(t, result, len(tt.required))
+ }
+ })
+ }
+}
+
+// paramNames extracts just the names from a slice of DiscoveredParam.
+func paramNames(params []DiscoveredParam) []string {
+ names := make([]string, len(params))
+ for i, p := range params {
+ names[i] = p.Name
+ }
+ return names
+}
diff --git a/internal/crowdsniff/patterns.go b/internal/crowdsniff/patterns.go
new file mode 100644
index 00000000..ebc1e019
--- /dev/null
+++ b/internal/crowdsniff/patterns.go
@@ -0,0 +1,229 @@
+package crowdsniff
+
+import (
+ "regexp"
+ "strings"
+)
+
+// Shared grep patterns for extracting endpoints from SDK source code.
+// Used by both the NPM source (Unit 3) and GitHub source (Unit 4).
+
+var (
+ // urlPathLiteral matches URL path literals in quoted strings:
+ // "/v1/users", '/api/projects', `/users/${id}`
+ urlPathLiteral = regexp.MustCompile(`["'` + "`]" + `(/[a-zA-Z0-9_\-/{}$:.]+)["'` + "`]")
+
+ // httpMethodCall matches common SDK patterns for HTTP method calls:
+ // this.get("/path"), this.post("/path"), client.get("/path"),
+ // fetch("/path"), axios.get("/path"), http.get("/path")
+ httpMethodCall = regexp.MustCompile(`(?i)\b(?:this|self|client|api|http|axios|request)\s*\.\s*(get|post|put|patch|delete|head|options)\s*\(`)
+
+ // fetchCall matches fetch("url") or fetch('url') or fetch(`url`)
+ fetchCall = regexp.MustCompile(`(?i)\bfetch\s*\(\s*["'` + "`]" + `([^"'` + "`" + `]+)["'` + "`]")
+
+ // requestMethodLiteral matches .request({method: "GET", ...}) patterns
+ requestMethodLiteral = regexp.MustCompile(`(?i)method\s*:\s*["']?(GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS)["']?`)
+
+ // baseURLPattern matches base URL constant assignments:
+ // baseUrl = "https://...", BASE_URL = "https://...",
+ // this.baseUrl = "https://...", apiBase = "https://..."
+ baseURLPattern = regexp.MustCompile(`(?i)(?:base_?url|api_?base|base_?uri|api_?url|host|endpoint)\s*[:=]\s*["'` + "`]" + `(https?://[^"'` + "`" + `\s]+)["'` + "`]")
+
+ // validHTTPMethods is the set of methods we recognize.
+ validHTTPMethods = map[string]bool{
+ "GET": true, "POST": true, "PUT": true, "PATCH": true,
+ "DELETE": true, "HEAD": true, "OPTIONS": true,
+ }
+)
+
+// GrepEndpoints scans source code content for API endpoint patterns.
+// It returns discovered endpoints and base URL candidates.
+func GrepEndpoints(content, sourceName, sourceTier string) ([]DiscoveredEndpoint, []string) {
+ var endpoints []DiscoveredEndpoint
+ var baseURLs []string
+
+ lines := strings.Split(content, "\n")
+
+ for _, line := range lines {
+ trimmed := strings.TrimSpace(line)
+ if trimmed == "" || strings.HasPrefix(trimmed, "//") || strings.HasPrefix(trimmed, "*") {
+ continue
+ }
+
+ // Extract base URL candidates.
+ if matches := baseURLPattern.FindStringSubmatch(line); len(matches) > 1 {
+ baseURLs = append(baseURLs, matches[1])
+ }
+
+ // Try to extract method + path from method call patterns.
+ eps := extractMethodCallEndpoints(line, sourceName, sourceTier)
+ endpoints = append(endpoints, eps...)
+
+ // Try to extract from fetch() calls.
+ eps = extractFetchEndpoints(line, sourceName, sourceTier)
+ endpoints = append(endpoints, eps...)
+
+ // Try to extract paths from URL path literals when near an HTTP method.
+ eps = extractContextualPathEndpoints(line, sourceName, sourceTier)
+ endpoints = append(endpoints, eps...)
+ }
+
+ return deduplicateEndpoints(endpoints), deduplicateStrings(baseURLs)
+}
+
+// extractMethodCallEndpoints handles patterns like:
+//
+// this.get("/v1/users"), client.post("/users")
+func extractMethodCallEndpoints(line, sourceName, sourceTier string) []DiscoveredEndpoint {
+ methodIndexes := httpMethodCall.FindAllStringSubmatchIndex(line, -1)
+ if len(methodIndexes) == 0 {
+ return nil
+ }
+
+ var endpoints []DiscoveredEndpoint
+ for _, methodIdx := range methodIndexes {
+ // methodIdx[2]:methodIdx[3] is the capture group (method name).
+ method := strings.ToUpper(line[methodIdx[2]:methodIdx[3]])
+ if !validHTTPMethods[method] {
+ continue
+ }
+
+ // Find the first URL path after this method call's opening paren.
+ remainder := line[methodIdx[1]:]
+ pathMatch := urlPathLiteral.FindStringSubmatch(remainder)
+ if pathMatch == nil {
+ continue
+ }
+ path := cleanPath(pathMatch[1])
+ if isValidAPIPath(path) {
+ endpoints = append(endpoints, DiscoveredEndpoint{
+ Method: method,
+ Path: path,
+ SourceTier: sourceTier,
+ SourceName: sourceName,
+ })
+ }
+ }
+ return endpoints
+}
+
+// extractFetchEndpoints handles fetch("/path") patterns.
+func extractFetchEndpoints(line, sourceName, sourceTier string) []DiscoveredEndpoint {
+ matches := fetchCall.FindAllStringSubmatch(line, -1)
+ if len(matches) == 0 {
+ return nil
+ }
+
+ // Check if there's an explicit method in the same line or nearby.
+ method := "GET" // default for fetch
+ if methodMatch := requestMethodLiteral.FindStringSubmatch(line); len(methodMatch) > 1 {
+ method = strings.ToUpper(methodMatch[1])
+ }
+
+ var endpoints []DiscoveredEndpoint
+ for _, match := range matches {
+ path := cleanPath(match[1])
+ if isValidAPIPath(path) {
+ endpoints = append(endpoints, DiscoveredEndpoint{
+ Method: method,
+ Path: path,
+ SourceTier: sourceTier,
+ SourceName: sourceName,
+ })
+ }
+ }
+ return endpoints
+}
+
+// extractContextualPathEndpoints handles lines that have both an HTTP method
+// reference and a path literal, but not in a standard call pattern.
+// Example: .request({method: "POST", url: "/v1/users"})
+func extractContextualPathEndpoints(line, sourceName, sourceTier string) []DiscoveredEndpoint {
+ methodMatch := requestMethodLiteral.FindStringSubmatch(line)
+ if len(methodMatch) < 2 {
+ return nil
+ }
+ method := strings.ToUpper(methodMatch[1])
+
+ paths := urlPathLiteral.FindAllStringSubmatch(line, -1)
+ if len(paths) == 0 {
+ return nil
+ }
+
+ var endpoints []DiscoveredEndpoint
+ for _, pathMatch := range paths {
+ path := cleanPath(pathMatch[1])
+ if isValidAPIPath(path) {
+ endpoints = append(endpoints, DiscoveredEndpoint{
+ Method: method,
+ Path: path,
+ SourceTier: sourceTier,
+ SourceName: sourceName,
+ })
+ }
+ }
+ return endpoints
+}
+
+// cleanPath normalizes a path extracted from source code.
+// It removes template literal syntax like ${...} -> {id} and trims trailing slashes.
+func cleanPath(path string) string {
+ // Replace template literal interpolation ${varName} with {id}.
+ templateVar := regexp.MustCompile(`\$\{[^}]+\}`)
+ path = templateVar.ReplaceAllString(path, "{id}")
+
+ // Trim trailing slash (but keep leading).
+ path = strings.TrimRight(path, "/")
+ if path == "" {
+ return "/"
+ }
+ return path
+}
+
+// isValidAPIPath checks if an extracted path looks like an API path
+// rather than a file path, import path, or other false positive.
+func isValidAPIPath(path string) bool {
+ if !strings.HasPrefix(path, "/") {
+ return false
+ }
+ // Reject file extensions that are clearly not API paths.
+ lower := strings.ToLower(path)
+ for _, ext := range []string{".js", ".ts", ".mjs", ".cjs", ".json", ".css", ".html", ".md", ".yaml", ".yml", ".png", ".jpg", ".svg", ".ico"} {
+ if strings.HasSuffix(lower, ext) {
+ return false
+ }
+ }
+ // Reject paths that look like node_modules imports.
+ if strings.Contains(path, "node_modules") {
+ return false
+ }
+ // Must have at least one meaningful segment.
+ segments := strings.Split(path, "/")
+ meaningfulCount := 0
+ for _, s := range segments {
+ if s != "" {
+ meaningfulCount++
+ }
+ }
+ return meaningfulCount >= 1
+}
+
+// deduplicateEndpoints removes duplicate endpoints (same method+path+source).
+func deduplicateEndpoints(endpoints []DiscoveredEndpoint) []DiscoveredEndpoint {
+ type key struct {
+ method string
+ path string
+ sourceName string
+ }
+ seen := make(map[key]struct{})
+ var result []DiscoveredEndpoint
+ for _, ep := range endpoints {
+ k := key{method: ep.Method, path: ep.Path, sourceName: ep.SourceName}
+ if _, exists := seen[k]; exists {
+ continue
+ }
+ seen[k] = struct{}{}
+ result = append(result, ep)
+ }
+ return result
+}
diff --git a/internal/crowdsniff/patterns_test.go b/internal/crowdsniff/patterns_test.go
new file mode 100644
index 00000000..56e6a516
--- /dev/null
+++ b/internal/crowdsniff/patterns_test.go
@@ -0,0 +1,264 @@
+package crowdsniff
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+)
+
+func TestGrepEndpoints(t *testing.T) {
+ t.Parallel()
+
+ t.Run("class method style", func(t *testing.T) {
+ t.Parallel()
+ content := `
+class NotionClient {
+ async listUsers() {
+ return this.get("/v1/users");
+ }
+ async createPage(data) {
+ return this.post("/v1/pages", data);
+ }
+ async updateBlock(blockId) {
+ return this.patch("/v1/blocks/" + blockId);
+ }
+}
+`
+ endpoints, _ := GrepEndpoints(content, "test-sdk", TierCommunitySDK)
+
+ methods := make(map[string][]string)
+ for _, ep := range endpoints {
+ methods[ep.Method] = append(methods[ep.Method], ep.Path)
+ }
+
+ assert.Contains(t, methods["GET"], "/v1/users")
+ assert.Contains(t, methods["POST"], "/v1/pages")
+ assert.Contains(t, methods["PATCH"], "/v1/blocks")
+ })
+
+ t.Run("fetch wrapper", func(t *testing.T) {
+ t.Parallel()
+ content := `
+async function getUsers() {
+ return fetch("/api/users");
+}
+async function deleteUser(id) {
+ return fetch("/api/users/" + id, {method: "DELETE"});
+}
+`
+ endpoints, _ := GrepEndpoints(content, "fetch-sdk", TierCommunitySDK)
+
+ var paths []string
+ for _, ep := range endpoints {
+ paths = append(paths, ep.Path)
+ }
+
+ assert.Contains(t, paths, "/api/users")
+ })
+
+ t.Run("axios instance", func(t *testing.T) {
+ t.Parallel()
+ content := `
+const client = axios.create({ baseURL: "https://api.stripe.com" });
+const users = axios.get("/v1/customers");
+const charge = axios.post("/v1/charges");
+`
+ endpoints, baseURLs := GrepEndpoints(content, "axios-sdk", TierCommunitySDK)
+
+ methods := make(map[string][]string)
+ for _, ep := range endpoints {
+ methods[ep.Method] = append(methods[ep.Method], ep.Path)
+ }
+
+ assert.Contains(t, methods["GET"], "/v1/customers")
+ assert.Contains(t, methods["POST"], "/v1/charges")
+ assert.Contains(t, baseURLs, "https://api.stripe.com")
+ })
+
+ t.Run("base URL extraction", func(t *testing.T) {
+ t.Parallel()
+ content := `
+const BASE_URL = "https://api.notion.com";
+this.baseUrl = "https://api.example.com/v2";
+const apiBase = "https://api.github.com";
+`
+ _, baseURLs := GrepEndpoints(content, "sdk", TierCommunitySDK)
+
+ assert.Contains(t, baseURLs, "https://api.notion.com")
+ assert.Contains(t, baseURLs, "https://api.example.com/v2")
+ assert.Contains(t, baseURLs, "https://api.github.com")
+ })
+
+ t.Run("template literal paths", func(t *testing.T) {
+ t.Parallel()
+ content := "return this.get(`/v1/users/${userId}`);\n"
+
+ endpoints, _ := GrepEndpoints(content, "sdk", TierCommunitySDK)
+
+ var paths []string
+ for _, ep := range endpoints {
+ paths = append(paths, ep.Path)
+ }
+
+ assert.Contains(t, paths, "/v1/users/{id}")
+ })
+
+ t.Run("request with method literal", func(t *testing.T) {
+ t.Parallel()
+ content := `api.request({method: "PUT", url: "/v1/settings"});`
+
+ endpoints, _ := GrepEndpoints(content, "sdk", TierCommunitySDK)
+
+ found := false
+ for _, ep := range endpoints {
+ if ep.Method == "PUT" && ep.Path == "/v1/settings" {
+ found = true
+ }
+ }
+ assert.True(t, found, "expected PUT /v1/settings")
+ })
+
+ t.Run("skips comments", func(t *testing.T) {
+ t.Parallel()
+ content := `
+// this.get("/v1/should-skip")
+* this.get("/v1/also-skip")
+this.get("/v1/real-endpoint");
+`
+ endpoints, _ := GrepEndpoints(content, "sdk", TierCommunitySDK)
+
+ var paths []string
+ for _, ep := range endpoints {
+ paths = append(paths, ep.Path)
+ }
+
+ assert.Contains(t, paths, "/v1/real-endpoint")
+ assert.NotContains(t, paths, "/v1/should-skip")
+ assert.NotContains(t, paths, "/v1/also-skip")
+ })
+
+ t.Run("rejects file paths", func(t *testing.T) {
+ t.Parallel()
+ content := `
+this.get("/assets/logo.png");
+this.get("/styles/main.css");
+this.get("/v1/users");
+`
+ endpoints, _ := GrepEndpoints(content, "sdk", TierCommunitySDK)
+
+ var paths []string
+ for _, ep := range endpoints {
+ paths = append(paths, ep.Path)
+ }
+
+ assert.Contains(t, paths, "/v1/users")
+ assert.NotContains(t, paths, "/assets/logo.png")
+ assert.NotContains(t, paths, "/styles/main.css")
+ })
+
+ t.Run("empty content returns empty", func(t *testing.T) {
+ t.Parallel()
+ endpoints, baseURLs := GrepEndpoints("", "sdk", TierCommunitySDK)
+ assert.Empty(t, endpoints)
+ assert.Empty(t, baseURLs)
+ })
+
+ t.Run("source metadata preserved", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get("/v1/users");`
+
+ endpoints, _ := GrepEndpoints(content, "my-sdk", TierOfficialSDK)
+
+ assert.NotEmpty(t, endpoints)
+ assert.Equal(t, "my-sdk", endpoints[0].SourceName)
+ assert.Equal(t, TierOfficialSDK, endpoints[0].SourceTier)
+ })
+
+ t.Run("deduplicates same endpoint", func(t *testing.T) {
+ t.Parallel()
+ content := `
+this.get("/v1/users");
+this.get("/v1/users");
+`
+ endpoints, _ := GrepEndpoints(content, "sdk", TierCommunitySDK)
+
+ count := 0
+ for _, ep := range endpoints {
+ if ep.Method == "GET" && ep.Path == "/v1/users" {
+ count++
+ }
+ }
+ assert.Equal(t, 1, count)
+ })
+
+ t.Run("multi-statement line avoids cartesian product", func(t *testing.T) {
+ t.Parallel()
+ content := `this.get("/v1/users"); this.post("/v2/projects");`
+
+ endpoints, _ := GrepEndpoints(content, "sdk", TierCommunitySDK)
+
+ // Should produce exactly 2 endpoints: GET /v1/users and POST /v2/projects.
+ // Must NOT produce cross-products like GET /v2/projects or POST /v1/users.
+ type ep struct{ method, path string }
+ found := make(map[ep]bool)
+ for _, e := range endpoints {
+ found[ep{e.Method, e.Path}] = true
+ }
+
+ assert.True(t, found[ep{"GET", "/v1/users"}], "expected GET /v1/users")
+ assert.True(t, found[ep{"POST", "/v2/projects"}], "expected POST /v2/projects")
+ assert.False(t, found[ep{"GET", "/v2/projects"}], "should NOT have GET /v2/projects (cartesian)")
+ assert.False(t, found[ep{"POST", "/v1/users"}], "should NOT have POST /v1/users (cartesian)")
+ assert.Len(t, endpoints, 2)
+ })
+}
+
+func TestIsValidAPIPath(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ path string
+ valid bool
+ }{
+ {name: "valid api path", path: "/v1/users", valid: true},
+ {name: "valid nested path", path: "/api/v2/projects/tasks", valid: true},
+ {name: "no leading slash", path: "v1/users", valid: false},
+ {name: "js file", path: "/scripts/app.js", valid: false},
+ {name: "ts file", path: "/src/index.ts", valid: false},
+ {name: "json file", path: "/config/settings.json", valid: false},
+ {name: "css file", path: "/styles/main.css", valid: false},
+ {name: "node_modules", path: "/node_modules/express", valid: false},
+ {name: "root path", path: "/api", valid: true},
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+ assert.Equal(t, tt.valid, isValidAPIPath(tt.path))
+ })
+ }
+}
+
+func TestCleanPath(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ path string
+ want string
+ }{
+ {name: "plain path", path: "/v1/users", want: "/v1/users"},
+ {name: "template literal", path: "/v1/users/${userId}", want: "/v1/users/{id}"},
+ {name: "trailing slash", path: "/v1/users/", want: "/v1/users"},
+ {name: "multiple template vars", path: "/v1/orgs/${orgId}/teams/${teamId}", want: "/v1/orgs/{id}/teams/{id}"},
+ {name: "empty becomes root", path: "", want: "/"},
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+ assert.Equal(t, tt.want, cleanPath(tt.path))
+ })
+ }
+}
diff --git a/internal/crowdsniff/specgen.go b/internal/crowdsniff/specgen.go
new file mode 100644
index 00000000..aa2d2f9f
--- /dev/null
+++ b/internal/crowdsniff/specgen.go
@@ -0,0 +1,187 @@
+package crowdsniff
+
+import (
+ "fmt"
+ "strconv"
+ "strings"
+
+ "github.com/mvanhorn/cli-printing-press/internal/spec"
+)
+
+// BuildSpec assembles a valid spec.APISpec from aggregated endpoints.
+func BuildSpec(name, baseURL string, endpoints []AggregatedEndpoint) (*spec.APISpec, error) {
+ if len(endpoints) == 0 {
+ return nil, fmt.Errorf("no endpoints to build spec from")
+ }
+ if name == "" {
+ return nil, fmt.Errorf("name is required")
+ }
+ if baseURL == "" {
+ return nil, fmt.Errorf("base_url is required")
+ }
+
+ resources := make(map[string]spec.Resource)
+
+ for _, ep := range endpoints {
+ endpoint := spec.Endpoint{
+ Method: ep.Method,
+ Path: ep.Path,
+ Description: fmt.Sprintf("%s %s", ep.Method, ep.Path),
+ Meta: map[string]string{
+ "source_tier": ep.SourceTier,
+ "source_count": strconv.Itoa(ep.SourceCount),
+ },
+ }
+
+ // Map DiscoveredParam → spec.Param for each aggregated endpoint's params.
+ if ep.Params != nil {
+ specParams := make([]spec.Param, len(ep.Params))
+ for i, p := range ep.Params {
+ var defaultVal any
+ if p.Default != "" {
+ defaultVal = p.Default
+ }
+ specParams[i] = spec.Param{
+ Name: p.Name,
+ Type: p.Type,
+ Required: p.Required,
+ Positional: false,
+ Default: defaultVal,
+ }
+ }
+ endpoint.Params = specParams
+ }
+
+ resourceKey, resourceName := deriveResourceKey(ep.Path)
+ if resourceKey == "" {
+ resourceKey = "default"
+ resourceName = "default"
+ }
+
+ resource := resources[resourceKey]
+ if resource.Description == "" {
+ resource.Description = fmt.Sprintf("Operations on %s", resourceName)
+ }
+ if resource.Endpoints == nil {
+ resource.Endpoints = make(map[string]spec.Endpoint)
+ }
+
+ endpointName := deriveEndpointName(ep.Method, ep.Path)
+ if _, exists := resource.Endpoints[endpointName]; exists {
+ endpointName = uniqueEndpointName(resource.Endpoints, endpointName)
+ }
+ resource.Endpoints[endpointName] = endpoint
+ resources[resourceKey] = resource
+ }
+
+ apiSpec := &spec.APISpec{
+ Name: name,
+ Description: fmt.Sprintf("Discovered API spec for %s (crowd-sniff)", name),
+ Version: "0.1.0",
+ BaseURL: baseURL,
+ Auth: spec.AuthConfig{Type: "none"},
+ Config: spec.ConfigSpec{
+ Format: "toml",
+ Path: fmt.Sprintf("~/.config/%s-pp-cli/config.toml", name),
+ },
+ Resources: resources,
+ Types: map[string]spec.TypeDef{},
+ }
+
+ if err := apiSpec.Validate(); err != nil {
+ return nil, fmt.Errorf("validating generated spec: %w", err)
+ }
+
+ return apiSpec, nil
+}
+
+// ResolveBaseURL picks the first non-empty URL from the cascade:
+// explicit flag > source candidates (in order).
+func ResolveBaseURL(explicit string, candidates []string) string {
+ if explicit != "" {
+ return explicit
+ }
+ for _, candidate := range candidates {
+ if strings.TrimSpace(candidate) != "" {
+ return candidate
+ }
+ }
+ return ""
+}
+
+// --- Helpers adapted from websniff/specgen.go ---
+
+func deriveResourceKey(path string) (string, string) {
+ segments := significantSegments(path)
+ if len(segments) == 0 {
+ return "", ""
+ }
+ if len(segments) > 3 {
+ segments = segments[:3]
+ }
+ return strings.Join(segments, "/"), segments[len(segments)-1]
+}
+
+func significantSegments(path string) []string {
+ parts := strings.Split(path, "/")
+ segments := make([]string, 0, len(parts))
+ for _, segment := range parts {
+ segment = strings.TrimSpace(segment)
+ if segment == "" {
+ continue
+ }
+ if strings.HasPrefix(segment, "{") && strings.HasSuffix(segment, "}") {
+ continue
+ }
+ if segment == "api" || isVersionSegment(segment) {
+ continue
+ }
+ segments = append(segments, segment)
+ }
+ return segments
+}
+
+func isVersionSegment(segment string) bool {
+ if len(segment) < 2 || segment[0] != 'v' {
+ return false
+ }
+ for _, r := range segment[1:] {
+ if r < '0' || r > '9' {
+ return false
+ }
+ }
+ return true
+}
+
+func deriveEndpointName(method string, normalizedPath string) string {
+ resource := "endpoint"
+ segments := significantSegments(normalizedPath)
+ if len(segments) > 0 {
+ resource = strings.ReplaceAll(segments[len(segments)-1], "-", "_")
+ }
+
+ switch strings.ToUpper(method) {
+ case "GET":
+ if strings.Contains(normalizedPath, "{") {
+ return "get_" + resource
+ }
+ return "list_" + resource
+ case "POST":
+ return "create_" + resource
+ case "PUT", "PATCH":
+ return "update_" + resource
+ case "DELETE":
+ return "delete_" + resource
+ default:
+ return strings.ToLower(method) + "_" + resource
+ }
+}
+
+func uniqueEndpointName(endpoints map[string]spec.Endpoint, base string) string {
+ for i := 2; ; i++ {
+ name := fmt.Sprintf("%s_%d", base, i)
+ if _, exists := endpoints[name]; !exists {
+ return name
+ }
+ }
+}
diff --git a/internal/crowdsniff/specgen_test.go b/internal/crowdsniff/specgen_test.go
new file mode 100644
index 00000000..05887f37
--- /dev/null
+++ b/internal/crowdsniff/specgen_test.go
@@ -0,0 +1,302 @@
+package crowdsniff
+
+import (
+ "testing"
+
+ "github.com/mvanhorn/cli-printing-press/internal/spec"
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+func TestBuildSpec(t *testing.T) {
+ t.Parallel()
+
+ t.Run("valid spec from aggregated endpoints", func(t *testing.T) {
+ t.Parallel()
+ endpoints := []AggregatedEndpoint{
+ {Method: "GET", Path: "/v1/users", SourceTier: TierOfficialSDK, SourceCount: 2},
+ {Method: "GET", Path: "/v1/users/{id}", SourceTier: TierOfficialSDK, SourceCount: 1},
+ {Method: "POST", Path: "/v1/users", SourceTier: TierCommunitySDK, SourceCount: 1},
+ }
+
+ apiSpec, err := BuildSpec("notion", "https://api.notion.com", endpoints)
+ require.NoError(t, err)
+
+ assert.Equal(t, "notion", apiSpec.Name)
+ assert.Equal(t, "https://api.notion.com", apiSpec.BaseURL)
+ assert.NotEmpty(t, apiSpec.Resources)
+
+ // Check that the "users" resource was created.
+ users, ok := apiSpec.Resources["users"]
+ require.True(t, ok)
+ assert.Len(t, users.Endpoints, 3)
+ })
+
+ t.Run("meta contains source_tier and source_count", func(t *testing.T) {
+ t.Parallel()
+ endpoints := []AggregatedEndpoint{
+ {Method: "GET", Path: "/v1/users", SourceTier: TierOfficialSDK, SourceCount: 3},
+ }
+
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ require.NoError(t, err)
+
+ for _, resource := range apiSpec.Resources {
+ for _, ep := range resource.Endpoints {
+ assert.Equal(t, "official-sdk", ep.Meta["source_tier"])
+ assert.Equal(t, "3", ep.Meta["source_count"])
+ }
+ }
+ })
+
+ t.Run("resource grouping from paths", func(t *testing.T) {
+ t.Parallel()
+ endpoints := []AggregatedEndpoint{
+ {Method: "GET", Path: "/v1/users", SourceTier: TierCodeSearch, SourceCount: 1},
+ {Method: "GET", Path: "/v1/projects", SourceTier: TierCodeSearch, SourceCount: 1},
+ }
+
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ require.NoError(t, err)
+
+ _, hasUsers := apiSpec.Resources["users"]
+ _, hasProjects := apiSpec.Resources["projects"]
+ assert.True(t, hasUsers, "should have users resource")
+ assert.True(t, hasProjects, "should have projects resource")
+ })
+
+ t.Run("empty endpoints returns error", func(t *testing.T) {
+ t.Parallel()
+ _, err := BuildSpec("test", "https://api.example.com", nil)
+ assert.Error(t, err)
+ assert.Contains(t, err.Error(), "no endpoints")
+ })
+
+ t.Run("empty name returns error", func(t *testing.T) {
+ t.Parallel()
+ _, err := BuildSpec("", "https://api.example.com", []AggregatedEndpoint{
+ {Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceCount: 1},
+ })
+ assert.Error(t, err)
+ assert.Contains(t, err.Error(), "name is required")
+ })
+
+ t.Run("empty base_url returns error", func(t *testing.T) {
+ t.Parallel()
+ _, err := BuildSpec("test", "", []AggregatedEndpoint{
+ {Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceCount: 1},
+ })
+ assert.Error(t, err)
+ assert.Contains(t, err.Error(), "base_url is required")
+ })
+
+ t.Run("spec passes validation", func(t *testing.T) {
+ t.Parallel()
+ endpoints := []AggregatedEndpoint{
+ {Method: "GET", Path: "/users", SourceTier: TierCodeSearch, SourceCount: 1},
+ }
+
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ require.NoError(t, err)
+ assert.NoError(t, apiSpec.Validate())
+ })
+}
+
+func TestResolveBaseURL(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ explicit string
+ candidates []string
+ want string
+ }{
+ {
+ name: "explicit wins",
+ explicit: "https://explicit.com",
+ candidates: []string{"https://candidate.com"},
+ want: "https://explicit.com",
+ },
+ {
+ name: "first candidate when no explicit",
+ explicit: "",
+ candidates: []string{"https://first.com", "https://second.com"},
+ want: "https://first.com",
+ },
+ {
+ name: "skip empty candidates",
+ explicit: "",
+ candidates: []string{"", " ", "https://valid.com"},
+ want: "https://valid.com",
+ },
+ {
+ name: "empty when nothing available",
+ explicit: "",
+ candidates: nil,
+ want: "",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+ assert.Equal(t, tt.want, ResolveBaseURL(tt.explicit, tt.candidates))
+ })
+ }
+}
+
+func TestBuildSpec_ParamMapping(t *testing.T) {
+ t.Parallel()
+
+ t.Run("aggregated endpoints with params produce spec.Endpoint.Params", func(t *testing.T) {
+ t.Parallel()
+
+ endpoints := []AggregatedEndpoint{
+ {
+ Method: "GET",
+ Path: "/v1/games",
+ SourceTier: TierOfficialSDK,
+ SourceCount: 1,
+ Params: []DiscoveredParam{
+ {Name: "steamid", Type: "string", Required: true},
+ {Name: "include_appinfo", Type: "boolean", Required: false, Default: "true"},
+ },
+ },
+ }
+
+ apiSpec, err := BuildSpec("steam", "https://api.steampowered.com", endpoints)
+ require.NoError(t, err)
+
+ // Find the endpoint in the spec.
+ var found *spec.Endpoint
+ for _, resource := range apiSpec.Resources {
+ for _, ep := range resource.Endpoints {
+ if ep.Path == "/v1/games" {
+ e := ep
+ found = &e
+ break
+ }
+ }
+ }
+ require.NotNil(t, found, "expected to find endpoint with path /v1/games")
+ require.Len(t, found.Params, 2)
+
+ assert.Equal(t, "steamid", found.Params[0].Name)
+ assert.Equal(t, "string", found.Params[0].Type)
+ assert.True(t, found.Params[0].Required)
+ assert.False(t, found.Params[0].Positional)
+ assert.Equal(t, "", found.Params[0].Description)
+
+ assert.Equal(t, "include_appinfo", found.Params[1].Name)
+ assert.Equal(t, "boolean", found.Params[1].Type)
+ assert.False(t, found.Params[1].Required)
+ assert.Equal(t, "true", found.Params[1].Default)
+ })
+
+ t.Run("param type mapping preserves discovered type", func(t *testing.T) {
+ t.Parallel()
+
+ endpoints := []AggregatedEndpoint{
+ {
+ Method: "GET",
+ Path: "/v1/items",
+ SourceTier: TierCommunitySDK,
+ SourceCount: 1,
+ Params: []DiscoveredParam{
+ {Name: "count", Type: "integer", Required: false, Default: "10"},
+ {Name: "active", Type: "boolean", Required: false},
+ {Name: "query", Type: "string", Required: false},
+ },
+ },
+ }
+
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ require.NoError(t, err)
+
+ var found *spec.Endpoint
+ for _, resource := range apiSpec.Resources {
+ for _, ep := range resource.Endpoints {
+ if ep.Path == "/v1/items" {
+ e := ep
+ found = &e
+ break
+ }
+ }
+ }
+ require.NotNil(t, found)
+ require.Len(t, found.Params, 3)
+
+ // Verify types are preserved from DiscoveredParam.
+ paramsByName := make(map[string]spec.Param)
+ for _, p := range found.Params {
+ paramsByName[p.Name] = p
+ }
+
+ assert.Equal(t, "integer", paramsByName["count"].Type)
+ assert.Equal(t, "10", paramsByName["count"].Default)
+ assert.Equal(t, "boolean", paramsByName["active"].Type)
+ assert.Equal(t, "string", paramsByName["query"].Type)
+ })
+
+ t.Run("nil params on aggregated endpoint produces nil spec params", func(t *testing.T) {
+ t.Parallel()
+
+ endpoints := []AggregatedEndpoint{
+ {
+ Method: "GET",
+ Path: "/v1/users",
+ SourceTier: TierCodeSearch,
+ SourceCount: 1,
+ Params: nil,
+ },
+ }
+
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ require.NoError(t, err)
+
+ for _, resource := range apiSpec.Resources {
+ for _, ep := range resource.Endpoints {
+ if ep.Path == "/v1/users" {
+ assert.Nil(t, ep.Params, "expected nil params when AggregatedEndpoint has nil params")
+ }
+ }
+ }
+ })
+
+ t.Run("mix of endpoints with and without params", func(t *testing.T) {
+ t.Parallel()
+
+ endpoints := []AggregatedEndpoint{
+ {
+ Method: "GET",
+ Path: "/v1/games",
+ SourceTier: TierOfficialSDK,
+ SourceCount: 1,
+ Params: []DiscoveredParam{
+ {Name: "steamid", Type: "string", Required: true},
+ },
+ },
+ {
+ Method: "GET",
+ Path: "/v1/users",
+ SourceTier: TierOfficialSDK,
+ SourceCount: 1,
+ Params: nil, // no params
+ },
+ }
+
+ apiSpec, err := BuildSpec("test", "https://api.example.com", endpoints)
+ require.NoError(t, err)
+
+ for _, resource := range apiSpec.Resources {
+ for _, ep := range resource.Endpoints {
+ if ep.Path == "/v1/games" {
+ require.Len(t, ep.Params, 1)
+ assert.Equal(t, "steamid", ep.Params[0].Name)
+ }
+ // Endpoint without params will have nil Params, which
+ // serializes as `params: []` due to YAML tag lacking omitempty.
+ }
+ }
+ })
+}
diff --git a/internal/crowdsniff/types.go b/internal/crowdsniff/types.go
new file mode 100644
index 00000000..faac188e
--- /dev/null
+++ b/internal/crowdsniff/types.go
@@ -0,0 +1,41 @@
+package crowdsniff
+
+// Source tier constants for endpoint provenance.
+const (
+ TierOfficialSDK = "official-sdk"
+ TierCommunitySDK = "community-sdk"
+ TierCodeSearch = "code-search"
+ TierPostman = "postman"
+)
+
+// DiscoveredParam represents a query parameter extracted from SDK source code.
+type DiscoveredParam struct {
+ Name string // parameter name (e.g., "steamid", "include_appinfo")
+ Type string // inferred type: "string", "integer", "boolean"
+ Required bool // true if positional arg without default in function signature
+ Default string // default value as string, empty if none
+}
+
+// DiscoveredEndpoint represents a single API endpoint found by a source.
+type DiscoveredEndpoint struct {
+ Method string // HTTP method (GET, POST, etc.)
+ Path string // URL path (e.g., "/v1/users", "/users/:id")
+ Params []DiscoveredParam // query parameters extracted from SDK code
+ SourceTier string // one of the Tier* constants
+ SourceName string // e.g., "@notionhq/client", "github-code-search"
+}
+
+// SourceResult is returned by each discovery source.
+type SourceResult struct {
+ Endpoints []DiscoveredEndpoint
+ BaseURLCandidates []string // e.g., "https://api.notion.com"
+}
+
+// AggregatedEndpoint is a deduplicated endpoint with cross-source metadata.
+type AggregatedEndpoint struct {
+ Method string
+ Path string // normalized
+ Params []DiscoveredParam // union-merged params across sources, sorted by name
+ SourceTier string // highest tier across sources
+ SourceCount int // number of distinct sources
+}
diff --git a/internal/spec/spec.go b/internal/spec/spec.go
index bda4a3b1..104d3366 100644
--- a/internal/spec/spec.go
+++ b/internal/spec/spec.go
@@ -47,15 +47,16 @@ type Resource struct {
}
type Endpoint struct {
- Method string `yaml:"method"`
- Path string `yaml:"path"`
- Description string `yaml:"description"`
- Params []Param `yaml:"params"`
- Body []Param `yaml:"body"`
- Response ResponseDef `yaml:"response"`
- Pagination *Pagination `yaml:"pagination"`
- ResponsePath string `yaml:"response_path,omitempty"` // path to extract data array from response (e.g., "data", "results.items")
- Alias string `yaml:"-"` // computed, not from YAML
+ Method string `yaml:"method"`
+ Path string `yaml:"path"`
+ Description string `yaml:"description"`
+ Params []Param `yaml:"params"`
+ Body []Param `yaml:"body"`
+ Response ResponseDef `yaml:"response"`
+ Pagination *Pagination `yaml:"pagination"`
+ ResponsePath string `yaml:"response_path,omitempty"` // path to extract data array from response (e.g., "data", "results.items")
+ Meta map[string]string `yaml:"meta,omitempty"` // per-endpoint metadata (e.g., source_tier, source_count from crowd-sniff)
+ Alias string `yaml:"-"` // computed, not from YAML
}
type Param struct {
diff --git a/internal/spec/spec_test.go b/internal/spec/spec_test.go
index 97c1bfec..a5d5f38b 100644
--- a/internal/spec/spec_test.go
+++ b/internal/spec/spec_test.go
@@ -5,6 +5,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
+ "gopkg.in/yaml.v3"
)
func TestParseStytch(t *testing.T) {
@@ -126,3 +127,74 @@ func TestNewFields(t *testing.T) {
assert.Equal(t, "ApiKeyAuth", s.Auth.Scheme)
assert.Equal(t, "header", s.Auth.In)
}
+
+func TestEndpointMeta(t *testing.T) {
+ t.Parallel()
+
+ t.Run("parse YAML with meta populated", func(t *testing.T) {
+ t.Parallel()
+ input := `
+name: test
+base_url: http://x
+resources:
+ users:
+ description: Users
+ endpoints:
+ list:
+ method: GET
+ path: /users
+ meta:
+ source_tier: official-sdk
+ source_count: "2"
+`
+ var s APISpec
+ require.NoError(t, yaml.Unmarshal([]byte(input), &s))
+ require.NoError(t, s.Validate())
+ ep := s.Resources["users"].Endpoints["list"]
+ assert.Equal(t, "official-sdk", ep.Meta["source_tier"])
+ assert.Equal(t, "2", ep.Meta["source_count"])
+ })
+
+ t.Run("parse YAML without meta field", func(t *testing.T) {
+ t.Parallel()
+ input := `
+name: test
+base_url: http://x
+resources:
+ users:
+ description: Users
+ endpoints:
+ list:
+ method: GET
+ path: /users
+`
+ var s APISpec
+ require.NoError(t, yaml.Unmarshal([]byte(input), &s))
+ ep := s.Resources["users"].Endpoints["list"]
+ assert.Nil(t, ep.Meta)
+ })
+
+ t.Run("marshal with meta set includes meta section", func(t *testing.T) {
+ t.Parallel()
+ ep := Endpoint{
+ Method: "GET",
+ Path: "/users",
+ Meta: map[string]string{"source_tier": "code-search"},
+ }
+ data, err := yaml.Marshal(ep)
+ require.NoError(t, err)
+ assert.Contains(t, string(data), "meta:")
+ assert.Contains(t, string(data), "source_tier: code-search")
+ })
+
+ t.Run("marshal with nil meta omits meta section", func(t *testing.T) {
+ t.Parallel()
+ ep := Endpoint{
+ Method: "GET",
+ Path: "/users",
+ }
+ data, err := yaml.Marshal(ep)
+ require.NoError(t, err)
+ assert.NotContains(t, string(data), "meta")
+ })
+}
diff --git a/lefthook.yml b/lefthook.yml
index 1c98da7a..5cdc4818 100644
--- a/lefthook.yml
+++ b/lefthook.yml
@@ -1,5 +1,9 @@
pre-push:
commands:
+ fmt:
+ glob: "*.go"
+ run: test -z "$(gofmt -l .)" || { gofmt -l .; exit 1; }
+ fail_text: "gofmt failed. Run 'gofmt -w .' to fix."
lint:
glob: "*.go"
run: golangci-lint run --new-from-rev=origin/main ./...
diff --git a/skills/printing-press/SKILL.md b/skills/printing-press/SKILL.md
index c10e6969..750fa0ae 100644
--- a/skills/printing-press/SKILL.md
+++ b/skills/printing-press/SKILL.md
@@ -781,6 +781,94 @@ Proceed with whatever spec source exists. If no spec was found, fall back to `--
---
+## Phase 1.8: Crowd Sniff Gate
+
+After Phase 1.7 (Sniff Gate), evaluate whether mining community signals (npm SDKs and GitHub code search) would improve the spec. Skip this gate entirely if the user already passed `--spec` (spec source is already resolved and appears complete).
+
+**Time budget:** The crowd sniff gate should complete within 5 minutes. If `printing-press crowd-sniff` fails or times out, fall back immediately:
+- If a spec already exists: "Crowd sniff failed — proceeding with existing spec."
+- If no spec exists: "Crowd sniff failed — falling back to --docs generation."
+
+### When to offer crowd sniff
+
+| Spec found? | Research shows gaps? | Action |
+|-------------|---------------------|--------|
+| Yes | Yes — competitors or community projects reference more endpoints | **Offer crowd sniff as enrichment** |
+| Yes | No — spec appears complete | Skip silently |
+| No | Community SDKs exist on npm | **Offer crowd sniff as primary discovery** |
+| No | No SDKs or code found | Skip — fall back to `--docs` |
+
+### Crowd sniff as enrichment (spec exists but has gaps)
+
+Present to the user via `AskUserQuestion`:
+
+> "Found a spec with **N endpoints**, but research shows the live API likely has more. Want me to search npm packages and GitHub code for `<api>` to discover additional endpoints? This typically takes 2-4 minutes."
+>
+> Options:
+> 1. **Yes — crowd sniff and merge** (search npm SDKs and GitHub code, merge discovered endpoints with the existing spec)
+> 2. **No — use existing spec** (proceed with what we have)
+
+### Crowd sniff as primary (no spec found)
+
+Present to the user via `AskUserQuestion`:
+
+> "No OpenAPI spec found for `<API>`. Want me to search npm packages and GitHub code to discover the API from community usage? This typically takes 2-4 minutes."
+>
+> Options:
+> 1. **Yes — crowd sniff the community** (search npm SDKs and GitHub code, generate a spec from discovered endpoints)
+> 2. **No — use docs instead** (attempt `--docs` generation from documentation pages)
+> 3. **No — I'll provide a spec or HAR** (user will supply input manually)
+
+### If user approves crowd sniff
+
+Ensure the discovery directory exists:
+
+```bash
+mkdir -p "$DISCOVERY_DIR"
+```
+
+Run the crowd-sniff command and capture both the spec and JSON provenance:
+
+```bash
+printing-press crowd-sniff --api <api> --output "$RESEARCH_DIR/<api>-crowd-spec.yaml" --json > "$DISCOVERY_DIR/crowd-sniff-provenance.json"
+```
+
+If the API has a known base URL from Phase 1 research, pass it:
+
+```bash
+printing-press crowd-sniff --api <api> --base-url <known-base-url> --output "$RESEARCH_DIR/<api>-crowd-spec.yaml" --json > "$DISCOVERY_DIR/crowd-sniff-provenance.json"
+```
+
+Report the results: "Crowd sniff discovered **N endpoints** across **M resources** (X from npm, Y from GitHub)."
+
+**Feed into Phase 2:**
+- **Enrichment mode**: Phase 2 will use `--spec <original> --spec <crowd-spec> --name <api>` to merge both
+- **Primary mode**: Phase 2 will use `--spec <crowd-spec>` directly
+
+#### Write crowd-sniff discovery report
+
+Write a structured crowd-sniff provenance report to `$DISCOVERY_DIR/crowd-sniff-report.md`. This report preserves the discovery evidence so a future maintainer can see what community sources informed the spec.
+
+The report must contain these sections:
+
+1. **npm Packages Analyzed** — List each SDK package examined: name, version, download count, recency. Note which packages yielded endpoints and which were empty/irrelevant.
+
+2. **GitHub Repos Searched** — The search queries used, repos matched, and freshness of each repo. Note the GitHub token status (authenticated with broader results, or unauthenticated with rate-limited results).
+
+3. **Endpoints Discovered** — A markdown table with columns: Method, Path, Source Tier (official-sdk / community-sdk / code-search), Source Count (seen in N independent sources). Sorted by source tier then frequency.
+
+4. **Base URL Resolution** — Candidates discovered and which was selected, with rationale (e.g., "Found in 3 npm packages: https://api.notion.com").
+
+5. **Auth Patterns Detected** — Authentication patterns found in SDK code (API key headers, bearer tokens, OAuth flows). Include the header name or env variable convention when visible.
+
+6. **Coverage Summary** — Total endpoints found, breakdown by source tier, and any gaps compared to the Phase 1 research brief (e.g., "Brief mentions webhooks but no webhook endpoints found in community code").
+
+### If user declines crowd sniff
+
+Proceed with whatever spec source exists. If no spec was found, fall back to `--docs` or ask the user to provide a spec/HAR manually.
+
+---
+
## Phase 1.5: Ecosystem Absorb Gate
THIS IS A MANDATORY STOP GATE. Do not generate until this is complete and approved.
@@ -976,6 +1064,38 @@ printing-press generate \
# --client-pattern proxy-envelope
```
+Crowd-sniff-enriched (original spec + crowd-discovered spec):
+
+```bash
+printing-press generate \
+ --spec <original-spec-path-or-url> \
+ --spec "$RESEARCH_DIR/<api>-crowd-spec.yaml" \
+ --name <api> \
+ --output "$PRESS_LIBRARY/<api>-pp-cli" \
+ --force --lenient --validate
+```
+
+Crowd-sniff-only (no original spec, crowd sniff was the primary source):
+
+```bash
+printing-press generate \
+ --spec "$RESEARCH_DIR/<api>-crowd-spec.yaml" \
+ --output "$PRESS_LIBRARY/<api>-pp-cli" \
+ --force --lenient --validate
+```
+
+Both sniff + crowd-sniff (merged with original):
+
+```bash
+printing-press generate \
+ --spec <original-spec-path-or-url> \
+ --spec "$RESEARCH_DIR/<api>-sniff-spec.yaml" \
+ --spec "$RESEARCH_DIR/<api>-crowd-spec.yaml" \
+ --name <api> \
+ --output "$PRESS_LIBRARY/<api>-pp-cli" \
+ --force --lenient --validate
+```
+
Docs-only:
```bash
diff --git a/testdata/crowdsniff/github-code-search-response.json b/testdata/crowdsniff/github-code-search-response.json
new file mode 100644
index 00000000..40af39f9
--- /dev/null
+++ b/testdata/crowdsniff/github-code-search-response.json
@@ -0,0 +1,74 @@
+{
+ "total_count": 5,
+ "incomplete_results": false,
+ "items": [
+ {
+ "name": "client.js",
+ "path": "src/client.js",
+ "html_url": "https://github.com/alice/notion-tools/blob/main/src/client.js",
+ "repository": {
+ "full_name": "alice/notion-tools"
+ },
+ "text_matches": [
+ {
+ "fragment": "const resp = await fetch('https://api.notion.com/v1/users', { headers })"
+ },
+ {
+ "fragment": "await fetch('https://api.notion.com/v1/projects', { method: 'POST' })"
+ }
+ ]
+ },
+ {
+ "name": "api.py",
+ "path": "notion/api.py",
+ "html_url": "https://github.com/bob/notion-sdk-py/blob/main/notion/api.py",
+ "repository": {
+ "full_name": "bob/notion-sdk-py"
+ },
+ "text_matches": [
+ {
+ "fragment": "response = self.session.get('https://api.notion.com/v1/users/me')"
+ }
+ ]
+ },
+ {
+ "name": "index.ts",
+ "path": "src/index.ts",
+ "html_url": "https://github.com/charlie/notion-wrapper/blob/main/src/index.ts",
+ "repository": {
+ "full_name": "charlie/notion-wrapper"
+ },
+ "text_matches": [
+ {
+ "fragment": "this.http.get('/v1/databases')\nthis.http.post('/v1/pages')"
+ }
+ ]
+ },
+ {
+ "name": "util.js",
+ "path": "lib/util.js",
+ "html_url": "https://github.com/alice/notion-tools/blob/main/lib/util.js",
+ "repository": {
+ "full_name": "alice/notion-tools"
+ },
+ "text_matches": [
+ {
+ "fragment": "fetch('https://api.notion.com/v1/users')"
+ }
+ ]
+ },
+ {
+ "name": "handler.go",
+ "path": "pkg/handler.go",
+ "html_url": "https://github.com/dave/stale-notion/blob/main/pkg/handler.go",
+ "repository": {
+ "full_name": "dave/stale-notion"
+ },
+ "text_matches": [
+ {
+ "fragment": "http.Get(\"https://api.notion.com/v1/blocks\")"
+ }
+ ]
+ }
+ ]
+}
diff --git a/testdata/crowdsniff/github-repo-response.json b/testdata/crowdsniff/github-repo-response.json
new file mode 100644
index 00000000..bea1ceae
--- /dev/null
+++ b/testdata/crowdsniff/github-repo-response.json
@@ -0,0 +1,4 @@
+{
+ "full_name": "alice/notion-tools",
+ "pushed_at": "2026-02-15T10:30:00Z"
+}
diff --git a/testdata/crowdsniff/sample-sdk.js b/testdata/crowdsniff/sample-sdk.js
new file mode 100644
index 00000000..3ccdf160
--- /dev/null
+++ b/testdata/crowdsniff/sample-sdk.js
@@ -0,0 +1,41 @@
+// Sample SDK source code for testing pattern extraction.
+
+const BASE_URL = "https://api.example.com";
+
+class ExampleClient {
+ constructor(apiKey) {
+ this.baseUrl = "https://api.example.com/v2";
+ this.apiKey = apiKey;
+ }
+
+ async listUsers() {
+ return this.get("/v1/users");
+ }
+
+ async getUser(userId) {
+ return this.get(`/v1/users/${userId}`);
+ }
+
+ async createUser(data) {
+ return this.post("/v1/users", data);
+ }
+
+ async updateProject(projectId, data) {
+ return this.patch(`/v1/projects/${projectId}`, data);
+ }
+
+ async deleteItem(itemId) {
+ return this.delete(`/v1/items/${itemId}`);
+ }
+}
+
+// Fetch-based wrapper
+async function fetchProjects() {
+ return fetch("/v1/projects");
+}
+
+// Axios-style
+const result = axios.get("/v1/teams");
+
+// Request with explicit method
+api.request({method: "PUT", url: "/v1/settings"});
← 3093e11a feat(skills): read MCP source code during ecosystem absorb (
·
back to Cli Printing Press
·
fix(cli): generator improvements from postman-explore retro fba41deb →