← back to Cli Printing Press
feat(cli): surface per-endpoint evidence and confidence from browser-sniff (#1397)
f32214ea6e80bc7f01ae0a51c06bca809a3b9ed9 · 2026-05-14 18:22:23 -0700 · Matt Van Horn
* feat(cli): observed_auth per endpoint on sniffed specs and clusters
U1 of the browser-sniff evidence-surfaces plan. Adds a new per-endpoint
ObservedAuth []string field to spec.Endpoint and EndpointCluster, populated
during sniff-driven spec generation from the request headers seen across each
endpoint's captured entries.
The set is the lowercased, sorted union of auth-shaped header names observed
across the group: Authorization, Cookie, X-CSRF-Token, X-API-Key, etc., plus
contains-style matches on token, secret, signature, api-key. Header VALUES are
never inspected or recorded; only the NAME travels. Empty sets are omitted
from emitted YAML and JSON via the omitempty tags.
Phase 2 Free/Paid Tier Routing Enrichment in skills/printing-press/SKILL.md
gains a detection-signal note pointing at the new per-endpoint surface so
downstream routing can act on per-operation evidence rather than spec-wide
guesses.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): confidence + normalization flags on endpoint cluster
U2 of the browser-sniff evidence-surfaces plan. Extends EndpointCluster with
two new fields surfaced from signals AnalyzeTraffic already computes but
previously discarded:
- NormalizationFlags []string: stable set of shape anomalies — single-sample,
single-status, mixed-content-types, request-body-only-on-some-samples. The
divergent-response-shape value is reserved for a future signal but listed
in the documented vocabulary so writers and readers share one set of names.
- Confidence string: coarse low | medium | high derived from Count, Statuses,
and the flag set. "low" when Count<3 or any flag fires; "medium" for 3-9
samples without flags; "high" for 10+ samples with multi-status responses
and no flags. Coarse on purpose so future numeric tuning stays
backward-compatible.
Both fields are additive on EndpointCluster, marked omitempty, and round-trip
through WriteTrafficAnalysis / ReadTrafficAnalysis without a version bump
because the existing UnmarshalJSON tolerates unknown fields.
Golden refresh: testdata/golden/expected/browser-sniff-sample/traffic-analysis.json
gains "confidence": "low" plus "normalization_flags": ["single-sample",
"single-status"] on each cluster (the existing fixture happens to have one
sample per endpoint). Intentional behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): per-endpoint redacted samples directory next to spec
U3 of the browser-sniff evidence-surfaces plan. Writes one redacted JSON
file per endpoint group to <spec-stem>-samples/, named
<method>__<path-slug>__<hash>.json (8-char sha256 prefix over METHOD +
normalizedPath so the same group lands at the same filename across reruns).
A new redact.go handles credential stripping:
- Header names matching authorization, cookie, set-cookie, x-csrf-token,
x-xsrf-token, x-api-key, proxy-authorization, plus contains-style matches
on token/secret/signature/api-key. Values replaced with <redacted>;
header NAMES travel verbatim.
- JSON body keys (case-insensitive, separator-insensitive: api_key, apiKey,
api-key, API-Key all collapse to one normalized form). Default key set:
password, token, secret, api_key, accessToken, refreshToken, creditCard,
ssn.
- String values matching JWT, email, and conservative E.164 phone patterns.
- Each replacement records a path label like "request_headers.authorization"
or "response_body.api_key" or "response_body.pattern:jwt" in the
per-file redactions list so reviewers know what was stripped.
Sample selection per group: most-recent successful 2xx; falls back to
most-recent non-error (<500); falls back to most-recent entry. Bodies
larger than 16 KB are truncated with response_body_truncated: true.
CLI: new --samples-output flag defaulting to DefaultSamplesPath(outputPath).
writeBrowserSniffOutputs gains a directory atomic-rename-and-restore step
mirroring the existing file-publish pattern so a failure mid-publish does
not leave the samples directory in a half-state.
Out of scope here: HTML surfaces from discoverHTMLSurfaceEntries and
GraphQL operation samples. Covered as follow-up; today's WriteSamples
matches the api-entry endpoint groups.
Golden refresh: stdout.txt gains one line announcing the samples
directory write. Behavior intentional.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): --min-samples and --include flags on browser-sniff
U4 of the browser-sniff evidence-surfaces plan. Two operator-facing knobs
that tune what lands in the emitted spec without changing the
TrafficAnalysis sidecar (so dropped endpoints stay auditable).
--min-samples N (default 1, no-op): drops endpoints from the emitted spec
when their underlying capture cluster has fewer than N paired entries.
The dropped endpoints remain in TrafficAnalysis.endpoint_clusters with
their Confidence: "low" and NormalizationFlags: ["single-sample"]
markers, so an operator can see what was filtered. Recommended value
for production capture is 2.
--include "a,b,c" (default empty): comma-separated host or path
substrings that force a positive classification score, bypassing the
default analytics/telemetry blocklist and the static-asset suffix
demotion. Used to rescue a specific endpoint or host the default
heuristics would otherwise drop. Include match wins over an overlapping
--blocklist entry.
Implementation:
- New SetAdditionalIncludeList package state mirroring SetAdditionalBlocklist.
- scoreEntry short-circuits to a strong positive when a URL's host or path
contains any include pattern.
- New FilterEndpointsByMinSamples helper re-derives capture clusters and
drops spec endpoints whose underlying cluster falls below the threshold.
Resources left with no endpoints are dropped too. Untouched when
minSamples <= 1.
- GraphQL operations that share a single underlying cluster (e.g., many
operationName values all hitting /frontend/graphql) survive or drop
together based on the cluster total. Per-operation thresholding is
future work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Files touched
A docs/plans/2026-05-14-001-feat-browser-sniff-evidence-surfaces-plan.mdM internal/browsersniff/analysis.goM internal/browsersniff/analysis_test.goM internal/browsersniff/classifier.goM internal/browsersniff/classifier_test.goA internal/browsersniff/redact.goA internal/browsersniff/redact_test.goM internal/browsersniff/specgen.goM internal/browsersniff/specgen_test.goM internal/cli/browser_sniff.goM internal/cli/browser_sniff_test.goM internal/spec/spec.goM skills/printing-press/SKILL.mdM testdata/golden/expected/browser-sniff-sample/stdout.txtM testdata/golden/expected/browser-sniff-sample/traffic-analysis.json
Diff
commit f32214ea6e80bc7f01ae0a51c06bca809a3b9ed9
Author: Matt Van Horn <mvanhorn@users.noreply.github.com>
Date: Thu May 14 18:22:23 2026 -0700
feat(cli): surface per-endpoint evidence and confidence from browser-sniff (#1397)
* feat(cli): observed_auth per endpoint on sniffed specs and clusters
U1 of the browser-sniff evidence-surfaces plan. Adds a new per-endpoint
ObservedAuth []string field to spec.Endpoint and EndpointCluster, populated
during sniff-driven spec generation from the request headers seen across each
endpoint's captured entries.
The set is the lowercased, sorted union of auth-shaped header names observed
across the group: Authorization, Cookie, X-CSRF-Token, X-API-Key, etc., plus
contains-style matches on token, secret, signature, api-key. Header VALUES are
never inspected or recorded; only the NAME travels. Empty sets are omitted
from emitted YAML and JSON via the omitempty tags.
Phase 2 Free/Paid Tier Routing Enrichment in skills/printing-press/SKILL.md
gains a detection-signal note pointing at the new per-endpoint surface so
downstream routing can act on per-operation evidence rather than spec-wide
guesses.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): confidence + normalization flags on endpoint cluster
U2 of the browser-sniff evidence-surfaces plan. Extends EndpointCluster with
two new fields surfaced from signals AnalyzeTraffic already computes but
previously discarded:
- NormalizationFlags []string: stable set of shape anomalies — single-sample,
single-status, mixed-content-types, request-body-only-on-some-samples. The
divergent-response-shape value is reserved for a future signal but listed
in the documented vocabulary so writers and readers share one set of names.
- Confidence string: coarse low | medium | high derived from Count, Statuses,
and the flag set. "low" when Count<3 or any flag fires; "medium" for 3-9
samples without flags; "high" for 10+ samples with multi-status responses
and no flags. Coarse on purpose so future numeric tuning stays
backward-compatible.
Both fields are additive on EndpointCluster, marked omitempty, and round-trip
through WriteTrafficAnalysis / ReadTrafficAnalysis without a version bump
because the existing UnmarshalJSON tolerates unknown fields.
Golden refresh: testdata/golden/expected/browser-sniff-sample/traffic-analysis.json
gains "confidence": "low" plus "normalization_flags": ["single-sample",
"single-status"] on each cluster (the existing fixture happens to have one
sample per endpoint). Intentional behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): per-endpoint redacted samples directory next to spec
U3 of the browser-sniff evidence-surfaces plan. Writes one redacted JSON
file per endpoint group to <spec-stem>-samples/, named
<method>__<path-slug>__<hash>.json (8-char sha256 prefix over METHOD +
normalizedPath so the same group lands at the same filename across reruns).
A new redact.go handles credential stripping:
- Header names matching authorization, cookie, set-cookie, x-csrf-token,
x-xsrf-token, x-api-key, proxy-authorization, plus contains-style matches
on token/secret/signature/api-key. Values replaced with <redacted>;
header NAMES travel verbatim.
- JSON body keys (case-insensitive, separator-insensitive: api_key, apiKey,
api-key, API-Key all collapse to one normalized form). Default key set:
password, token, secret, api_key, accessToken, refreshToken, creditCard,
ssn.
- String values matching JWT, email, and conservative E.164 phone patterns.
- Each replacement records a path label like "request_headers.authorization"
or "response_body.api_key" or "response_body.pattern:jwt" in the
per-file redactions list so reviewers know what was stripped.
Sample selection per group: most-recent successful 2xx; falls back to
most-recent non-error (<500); falls back to most-recent entry. Bodies
larger than 16 KB are truncated with response_body_truncated: true.
CLI: new --samples-output flag defaulting to DefaultSamplesPath(outputPath).
writeBrowserSniffOutputs gains a directory atomic-rename-and-restore step
mirroring the existing file-publish pattern so a failure mid-publish does
not leave the samples directory in a half-state.
Out of scope here: HTML surfaces from discoverHTMLSurfaceEntries and
GraphQL operation samples. Covered as follow-up; today's WriteSamples
matches the api-entry endpoint groups.
Golden refresh: stdout.txt gains one line announcing the samples
directory write. Behavior intentional.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): --min-samples and --include flags on browser-sniff
U4 of the browser-sniff evidence-surfaces plan. Two operator-facing knobs
that tune what lands in the emitted spec without changing the
TrafficAnalysis sidecar (so dropped endpoints stay auditable).
--min-samples N (default 1, no-op): drops endpoints from the emitted spec
when their underlying capture cluster has fewer than N paired entries.
The dropped endpoints remain in TrafficAnalysis.endpoint_clusters with
their Confidence: "low" and NormalizationFlags: ["single-sample"]
markers, so an operator can see what was filtered. Recommended value
for production capture is 2.
--include "a,b,c" (default empty): comma-separated host or path
substrings that force a positive classification score, bypassing the
default analytics/telemetry blocklist and the static-asset suffix
demotion. Used to rescue a specific endpoint or host the default
heuristics would otherwise drop. Include match wins over an overlapping
--blocklist entry.
Implementation:
- New SetAdditionalIncludeList package state mirroring SetAdditionalBlocklist.
- scoreEntry short-circuits to a strong positive when a URL's host or path
contains any include pattern.
- New FilterEndpointsByMinSamples helper re-derives capture clusters and
drops spec endpoints whose underlying cluster falls below the threshold.
Resources left with no endpoints are dropped too. Untouched when
minSamples <= 1.
- GraphQL operations that share a single underlying cluster (e.g., many
operationName values all hitting /frontend/graphql) survive or drop
together based on the cluster total. Per-operation thresholding is
future work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
...01-feat-browser-sniff-evidence-surfaces-plan.md | 303 +++++++++++++++
internal/browsersniff/analysis.go | 109 +++++-
internal/browsersniff/analysis_test.go | 267 +++++++++++++
internal/browsersniff/classifier.go | 60 ++-
internal/browsersniff/classifier_test.go | 157 ++++++++
internal/browsersniff/redact.go | 188 +++++++++
internal/browsersniff/redact_test.go | 210 ++++++++++
internal/browsersniff/specgen.go | 318 ++++++++++++++-
internal/browsersniff/specgen_test.go | 432 +++++++++++++++++++++
internal/cli/browser_sniff.go | 125 +++++-
internal/cli/browser_sniff_test.go | 98 ++++-
internal/spec/spec.go | 10 +-
skills/printing-press/SKILL.md | 7 +
.../expected/browser-sniff-sample/stdout.txt | 1 +
.../browser-sniff-sample/traffic-analysis.json | 15 +
15 files changed, 2268 insertions(+), 32 deletions(-)
diff --git a/docs/plans/2026-05-14-001-feat-browser-sniff-evidence-surfaces-plan.md b/docs/plans/2026-05-14-001-feat-browser-sniff-evidence-surfaces-plan.md
new file mode 100644
index 00000000..4a1d3ffa
--- /dev/null
+++ b/docs/plans/2026-05-14-001-feat-browser-sniff-evidence-surfaces-plan.md
@@ -0,0 +1,303 @@
+---
+title: Surface per-endpoint evidence and confidence from browser-sniff
+status: active
+created: 2026-05-14
+type: feat
+depth: standard
+---
+
+# Surface per-endpoint evidence and confidence from browser-sniff
+
+Target repo: cli-printing-press
+
+## Summary
+
+PP's `internal/browsersniff` package already computes most of what this plan needs. `AnalyzeTraffic` produces a versioned `TrafficAnalysis` with `EndpointClusters`, an `AuthAnalysis` of `AuthCandidate`s with header names and evidence-back-references, and a numeric `Confidence` per observation. `buildEndpoint` in `specgen.go` already calls `detectAuth(group.Entries, "")` per endpoint group. None of that per-endpoint detail escapes in a form downstream skills can act on, and the spec YAML carries no per-operation evidence handles.
+
+This plan surfaces four signals without inventing new analysis:
+
+1. Per-endpoint `ObservedAuth` field added to `EndpointCluster` and to `spec.Endpoint`. Population reuses the existing `detectAuth(group.Entries, "")` call site in `buildEndpoint`; only header names travel to YAML, values are never written. Enables Phase 2 Free/Paid Tier Routing Enrichment to route by per-endpoint evidence rather than spec-level guesses.
+2. Per-endpoint confidence + normalization flags added as fields on the existing `EndpointCluster` struct, not a new sidecar file. `confidence: low | medium | high` and a small flag set (`single-sample`, `single-status`, `mixed-content-types`, `request-body-only-on-some-samples`, `divergent-response-shape`). The existing `<spec-stem>-traffic-analysis.json` becomes the canonical confidence surface; no second file.
+3. New `<spec-stem>-samples/<method>__<path-slug>__<hash>.json` directory of one redacted request/response sample per endpoint group. New redaction helper is required (browsersniff has no redactor today). Mirrors the `writeBrowserSniffOutputs` temp-then-atomic-rename pattern for directory output.
+4. New `--min-samples N` flag on the `browser-sniff` subcommand, and a new `--include` flag complementing the existing `--blocklist`. Below-threshold endpoints drop from emitted spec YAML but remain visible in `TrafficAnalysis.EndpointClusters` so the audit trail is preserved.
+
+Recommendation: ship in order U1 -> U2 -> U3 -> U4. U1 lands the biggest downstream payoff and is structurally simplest. U2 and U3 are independent of U1. U4 depends on U2 so dropped endpoints stay auditable.
+
+Explicitly deferred: HTML coverage report, multi-stage pipeline refactor, spec preview helper. See Scope Boundaries.
+
+---
+
+## Problem Frame
+
+`internal/browsersniff/analysis.go` already does the heavy lifting. The data exists; the surface does not.
+
+- `AnalyzeTraffic` produces `TrafficAnalysis{Auth, EndpointClusters, Protections, ...}` (analysis.go L420-453). `EndpointClusters` is a list of grouped endpoints. The struct does not currently carry sample counts, observed status codes, or any confidence bucket.
+- `detectTrafficAuth(capture, entries)` (analysis.go L714-785) produces `AuthCandidate`s with `HeaderNames`, `CookieNames`, `QueryNames`, numeric `Confidence`, and `Evidence []EvidenceRef`. `EvidenceRef.EntryIndex` points at the source `EnrichedEntry`. That's the join key from auth candidate to endpoint cluster - but nothing actually joins them.
+- `buildEndpoint(group EndpointGroup)` in specgen.go L559-601 already calls `detectAuth(nil, group.Entries, "")` to produce a `spec.Auth` it uses internally for query-param filtering. The function knows the auth headers for this endpoint group at the moment of spec emission. It throws that detail away after using it for filtering.
+- `spec.Endpoint` (the YAML emission target) has no field for observed auth and no extensions map. The information cannot land on the spec without a typed field addition.
+- Response bodies in `EnrichedEntry.ResponseBody` are stored unredacted. There is no redactor in `internal/browsersniff`. The only redaction-shaped helper is the auth-domain binding check in `internal/cli/browser_sniff.go`. A samples-on-disk feature needs a fresh redactor.
+- `writeBrowserSniffOutputs` (browser_sniff.go L95-138) writes the spec and the traffic-analysis JSON with a temp-then-atomic-rename + backup-restore pattern. A samples directory needs the same crash-safety treatment adapted for directory output.
+- The `browser-sniff` subcommand exposes `--har, --output, --analysis-output, --name, --blocklist, --auth-from`. There is no `--min-samples`, no `--include`. The `--blocklist` flag already feeds `SetAdditionalBlocklist` (classifier.go L53-58) so the existing extension pattern is hostname-list-based.
+
+The four units below build on top of these existing surfaces, not around them.
+
+---
+
+## Scope Boundaries
+
+### In scope
+
+- Additive field changes to `EndpointCluster` (analysis.go) and `spec.Endpoint` (internal/spec) that carry per-endpoint observed auth, sample count, status codes, normalization flags, and a coarse confidence bucket.
+- A redaction helper in `internal/browsersniff` covering header names, body keys, and a small regex set (JWT, email, phone).
+- A new `<spec-stem>-samples/` output directory with one redacted file per endpoint group.
+- Two new `browser-sniff` flags: `--min-samples N` and `--include <csv>`.
+- Phase 2 Auth Enrichment note in `skills/printing-press/SKILL.md` mentioning the new per-endpoint signal.
+- Golden harness updates (`testdata/golden/...`) for the new spec fields, traffic-analysis fields, and samples directory.
+- Test cases in `internal/cli/browser_sniff_test.go`, `internal/browsersniff/analysis_test.go`, `internal/browsersniff/specgen_test.go`, and `internal/browsersniff/classifier_test.go`.
+
+### Outside this product's identity
+
+- Cloud-hosted capture backends. PP capture stays local.
+- JS or non-Go client emission. PP outputs Go.
+- HTML coverage report as a primary deliverable. The existing markdown discovery report stays canonical for user-facing review.
+
+### Deferred to Follow-Up Work
+
+- Re-architecting `internal/browsersniff` into discrete load/filter/normalize/infer/emit stages with re-runnable sub-steps. Genuinely useful for debugging but a separate refactor plan.
+- Spec preview helper (`browser-sniff --preview` opening a local Swagger UI). Quality-of-life, not load-bearing.
+- A `--slug` variable type in `normalizeEntryPath` (varying-alpha-tokens segment detection). Today PP recognizes `{id}/{uuid}/{hash}`; slug heuristics would be a separate ask.
+- Hoisting structurally-identical schemas into `components.schemas`. Out of scope; deals with spec ergonomics, not evidence.
+
+---
+
+## Key Technical Decisions
+
+- The confidence and observed-auth signals live on the existing `EndpointCluster` type (analysis.go). A separate `confidence.json` sidecar would duplicate state that already belongs in `TrafficAnalysis`. The single sidecar file (`<spec-stem>-traffic-analysis.json`) becomes the canonical surface.
+- `EndpointCluster` gains four additive fields: `SampleCount int`, `StatusCodes []int`, `NormalizationFlags []string`, `Confidence string`, `ObservedAuth []string`. All `omitempty`. `trafficAnalysisVersion` stays at `"1"` because the existing `UnmarshalJSON` flow tolerates unknown fields. No legacy-compat hook needed; readers older than this change ignore the new fields, which is the right behavior.
+- `spec.Endpoint` gains one additive field: `ObservedAuth []string` with `yaml:"observed_auth,omitempty"`. Spec consumers older than this change ignore it. No version bump on the spec format.
+- Confidence bucket assignment lives in a small helper inside `buildEndpointClusters` (analysis.go around L435 in `AnalyzeTraffic`). Rule: `low` if `SampleCount < 3` OR any normalization flag is set; `medium` if `3 <= SampleCount <= 9` AND no flags; `high` if `SampleCount >= 10` AND distinct status codes >= 2 AND no flags. Coarse on purpose so future numeric-confidence tuning does not break downstream consumers.
+- Normalization flag names: `single-sample`, `single-status`, `mixed-content-types`, `request-body-only-on-some-samples`, `divergent-response-shape`. PP-specific flags may be added later (e.g. `proxy-envelope-detected`, `graphql-persisted-hash`) but those names should not collide.
+- Per-endpoint `ObservedAuth` is derived inside `buildEndpoint` (specgen.go L559) by extending the existing `detectAuth(nil, group.Entries, "")` call to also return the set of auth-shaped header names it observed. The set is lowercased, deduplicated, sorted. Empty sets are not written. Population on `EndpointCluster` happens in `buildEndpointClusters` by replaying the same per-group detection (or by passing the spec endpoints back into cluster building - the call site is small enough that duplicating the detection is cheaper than threading a shared map).
+- Samples directory uses path `<spec-stem>-samples/<method>__<path-slug>__<hash>.json`. Filename slug rules: method lowercased; path slug replaces `/` with `_` and removes `{` `}`; hash is the first 8 chars of `sha256(method + " " + normalizedPath)`. Same endpoint -> same filename across reruns. Collision probability with 8 hex chars is ~10^-9 at typical endpoint counts; acceptable.
+- Sample selection per endpoint: most-recent successful 2xx; fall back to most-recent non-error; fall back to most-recent entry. Ordering within `EndpointGroup.Entries` is capture-order (preserved through `classifyInCaptureOrder`).
+- Redaction is new code in `internal/browsersniff/redact.go`. Header names to redact (case-insensitive): `authorization, cookie, set-cookie, x-csrf-token, x-xsrf-token, x-api-key, proxy-authorization`, plus name patterns `*token*, *secret*, *signature*`. Body keys: `password, token, secret, api_key, apiKey, accessToken, refreshToken`. Body regexes: JWT (`^eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$`), email, phone (E.164-ish). Replacement is the literal string `<redacted>` so JSON structure is preserved for the reviewer. A per-file `redactions []string` field lists which header names and body keys were stripped, so the reviewer knows what is missing.
+- Samples directory write uses a `<spec-stem>-samples.tmp/` directory next to the spec, written first; on success the existing samples directory is renamed to a `.backup` sibling and the `.tmp` is renamed to the canonical name; on failure the backup is restored. Mirrors the existing `writeBrowserSniffOutputs` two-phase commit pattern adapted for directory atomicity. `os.Rename` on a directory is atomic on POSIX, which is sufficient.
+- `--min-samples` filtering happens at the spec emission boundary in `AnalyzeCapture` (specgen.go around L35) - endpoints whose `EndpointGroup.Entries` count is below the threshold are excluded from the emitted `apiSpec.Resources[].Endpoints[]`. They remain in `TrafficAnalysis.EndpointClusters` so the audit trail is intact.
+- `--include <csv>` adds a new precedence rule in `scoreEntry`: if the URL host or path matches any include pattern, the score is forced positive regardless of blocklist or static-asset suffix demotion. Include wins over blocklist.
+- `--blocklist` stays as the existing flag. Documentation in the SKILL.md note clarifies that `--blocklist` extends `DefaultBlocklist` and `--include` rescues.
+- All four units are independently mergeable. Recommended order is U1 -> U2 -> U3 -> U4 but no unit depends on another to compile or pass tests.
+
+---
+
+## Implementation Units
+
+### U1. Per-endpoint observed-auth on cluster and spec endpoint
+
+Goal: thread the auth-shaped header names that `detectAuth` already extracts per endpoint group up to both `EndpointCluster` and `spec.Endpoint`, so downstream consumers (Phase 2 Auth Enrichment, MCP routing, tier detection) can act on per-operation evidence.
+
+Requirements: per-endpoint auth visibility.
+
+Dependencies: none.
+
+Files:
+- internal/spec/ (the package containing `spec.Endpoint` definition - add `ObservedAuth []string` field with `yaml:"observed_auth,omitempty"` tag)
+- internal/browsersniff/analysis.go (add `ObservedAuth []string` to `EndpointCluster`; populate in `buildEndpointClusters` around L435)
+- internal/browsersniff/specgen.go (extend the existing `detectAuth(nil, group.Entries, "")` call in `buildEndpoint` L569 to also return observed header names; set `endpoint.ObservedAuth` before return)
+- internal/browsersniff/analysis_test.go (cluster-level observed-auth cases)
+- internal/browsersniff/specgen_test.go (spec-level observed-auth cases)
+- skills/printing-press/SKILL.md (Phase 2 Free/Paid Tier Routing Enrichment around L1707 - mention reading `endpoint.ObservedAuth` and `cluster.ObservedAuth` for per-endpoint signal)
+- testdata/golden/expected/ (refresh affected browser-sniff golden specs and traffic-analysis fixtures)
+
+Approach: the existing `detectAuth(capture *EnrichedCapture, entries []EnrichedEntry, ...)` signature already takes entries. Add a sibling helper or extend the return shape to also surface `[]string` of observed auth-shaped header names. Capture-level auth context (the `*EnrichedCapture` arg) is left alone. The header-name set is the union across `group.Entries` of names matching: case-insensitive `authorization` (Bearer-prefixed or not), case-insensitive `cookie`, names containing `api-key` / `api_key` / `x-auth-token`, plus the patterns matched by the existing `isAuthQueryName` filter for query-string auth (those go into a separate per-endpoint `observed_auth_query []string` if useful - see deferred items, not in this unit).
+
+Patterns to follow: the existing `detectTrafficAuth` accumulation pattern at analysis.go L714 (case switches on header name, dedup via `uniqueStrings`). Reuse `uniqueStrings` and the existing case predicates - do not invent new ones.
+
+Test scenarios:
+- Happy path: one sample with `Authorization: Bearer XYZ` -> cluster `ObservedAuth == ["authorization"]` and spec endpoint `ObservedAuth == ["authorization"]`.
+- Multiple auth headers across one sample: `Authorization` + `X-CSRF-Token` -> both present in alphabetical order.
+- Mixed across samples: sample A has `Authorization`, sample B does not -> union still emits `["authorization"]` (record presence across the group, not requirement per sample).
+- No auth headers anywhere on the endpoint -> field is omitted from the spec YAML and from the cluster JSON (omitempty respected).
+- Case-insensitive merge: `Authorization` in one sample and `AUTHORIZATION` in another collapse to a single `authorization` entry.
+- Header value redaction: a sample with `Authorization: Bearer eyJ...` produces `ObservedAuth == ["authorization"]` without leaking the token value into either the cluster or the spec.
+- GraphQL endpoints (built via `buildGraphQLOperationEndpoint` at specgen.go L228) also carry `ObservedAuth` on their spec endpoint.
+
+Verification: existing browser-sniff goldens refreshed; new auth-mixed golden fixture renders the field; `go test ./internal/browsersniff/... ./internal/cli/...` passes; `scripts/golden.sh verify` clean.
+
+### U2. Confidence bucket and normalization flags on `EndpointCluster`
+
+Goal: surface the per-cluster sample stats and a coarse `low | medium | high` bucket on the existing `EndpointCluster` type, so absorb / dogfood / novel-feature gating skills can reason about which endpoints are anecdotes vs staples.
+
+Requirements: machine-readable confidence signal for downstream gates.
+
+Dependencies: none. Independent of U1 even though both land on `EndpointCluster`.
+
+Files:
+- internal/browsersniff/analysis.go (extend `EndpointCluster` with `SampleCount int`, `StatusCodes []int`, `NormalizationFlags []string`, `Confidence string`; populate in `buildEndpointClusters` around L435)
+- internal/browsersniff/analysis_test.go (bucket boundary tests, flag tests)
+- testdata/golden/expected/ (refresh affected traffic-analysis goldens)
+- skills/printing-press/SKILL.md (absorb gate around L1241 region - mention reading `EndpointCluster.Confidence`)
+
+Approach: add a `bucketConfidence(samples int, statuses []int, flags []string) string` helper in analysis.go. Populate `SampleCount` from `len(group.Entries)`. Populate `StatusCodes` from the deduplicated sorted `ResponseStatus` values across the group's entries. Populate `NormalizationFlags`:
+- `single-sample` if `SampleCount == 1`
+- `single-status` if `len(StatusCodes) == 1`
+- `mixed-content-types` if the group's entries show more than one distinct `ResponseContentType` (case-insensitive prefix grouping: `application/json` vs `text/html` etc.)
+- `request-body-only-on-some-samples` if the group's method is one of `POST/PUT/PATCH` AND `0 < count(entries with non-empty RequestBody) < SampleCount`
+- `divergent-response-shape` - reserved; this signal is intended for the future case where pre-normalization paths collapsed but responses diverge structurally. Today PP keys clusters by `host + method + normalizedPath` so collapse is already prevented. Leave the flag name reserved; do not populate in this unit. Document in a comment.
+
+Bucket rule:
+- `low` if `SampleCount < 3` OR `len(NormalizationFlags) > 0`
+- `medium` if `3 <= SampleCount <= 9` AND `len(NormalizationFlags) == 0`
+- `high` if `SampleCount >= 10` AND `len(StatusCodes) >= 2` AND `len(NormalizationFlags) == 0`
+
+Patterns to follow: the existing `EndpointCluster` definition and writer (whatever shape it has today - the cluster type is built in `buildEndpointClusters`, sorted via `sortTrafficAnalysis`). The new fields are populated in the same builder, then sorted alongside existing fields by the existing sort code.
+
+Test scenarios:
+- Single-sample endpoint -> `Confidence: "low"`, `NormalizationFlags: ["single-sample"]`, `SampleCount: 1`.
+- 5 samples, all returning 200 -> `Confidence: "low"`, flags include `single-status`, `SampleCount: 5`, `StatusCodes: [200]`.
+- 12 samples, statuses 200 and 404, no flags -> `Confidence: "high"`, `StatusCodes: [200, 404]`, `SampleCount: 12`.
+- 4 samples, statuses 200 and 401, no flags -> `Confidence: "medium"`.
+- POST endpoint where 2 of 3 samples carry a request body -> flag `request-body-only-on-some-samples`, bucket `low`.
+- Endpoint with both `application/json` and `text/html` response content-types -> flag `mixed-content-types`.
+- GET endpoint with body samples (impossible in HTTP-correct terms, but seen in the wild) does NOT trigger `request-body-only-on-some-samples` because the method is not POST/PUT/PATCH.
+- Cluster sort order remains deterministic (existing sort by host/method/path is preserved).
+- The traffic-analysis JSON round-trips: write -> read -> all new fields preserved with version still `"1"`.
+
+Verification: traffic-analysis golden fixtures show the new fields; `go test ./internal/browsersniff/...` passes; `scripts/golden.sh verify` clean.
+
+### U3. Redacted per-endpoint samples directory
+
+Goal: write one redacted JSON file per endpoint group to `<spec-stem>-samples/`, named by method + path slug + hash, so absorb / dogfood / reviewer flows can point at concrete evidence per endpoint.
+
+Requirements: discoverable per-endpoint evidence.
+
+Dependencies: none (the redaction helper is built inside this unit; not reused from elsewhere).
+
+Files:
+- internal/browsersniff/redact.go (new file - redaction helper for header maps and JSON body trees)
+- internal/browsersniff/redact_test.go (new file - redactor unit tests)
+- internal/browsersniff/specgen.go (sample selection logic in or alongside `buildEndpoint`, then directory write triggered from a new `WriteSamples(apiSpec, capture, outputDir)` helper)
+- internal/cli/browser_sniff.go (extend `writeBrowserSniffOutputs` to also write the samples directory using the temp-then-rename + backup-restore pattern)
+- internal/cli/browser_sniff_test.go (extend `TestWriteBrowserSniffOutputsRestoresExistingFilesWhenSpecPublishFails` and add new test for samples-dir restore-on-failure)
+- internal/browsersniff/specgen_test.go (sample selection rules, filename hashing determinism)
+- skills/printing-press/references/browser-sniff-capture.md (add a short section pointing users at `<spec-stem>-samples/`)
+- testdata/golden/expected/ (add representative golden samples for an existing capture fixture)
+
+Approach: redaction helper API:
+- `RedactHeaders(headers map[string]string) (map[string]string, []string)` - returns redacted headers plus the list of header names that were redacted (lowercased, sorted, deduplicated).
+- `RedactJSONBody(body string) (string, []string)` - parses body as JSON; if parse succeeds, walks the object tree redacting sensitive keys and value-pattern matches, returning the re-serialized JSON and the list of redacted dotted paths. If body is not JSON, applies value-pattern regex sweep against the raw string and returns it with a list of pattern names that matched.
+
+Sample selection (per `EndpointGroup`): iterate `group.Entries` in capture order; pick the most-recent successful (200-299) entry. If none successful, pick the most-recent entry regardless of status. Capture order is preserved through `classifyInCaptureOrder`.
+
+Per-file format:
+
+```json
+{
+ "endpoint": "GET /v1/items/{id}",
+ "raw_url": "/v1/items/42?page=2",
+ "status": 200,
+ "method": "GET",
+ "request_headers": { "accept": "application/json" },
+ "request_body": null,
+ "response_headers": { "content-type": "application/json" },
+ "response_body": { "id": 42, "name": "Widget" },
+ "redactions": ["request_headers.authorization", "response_body.api_key"],
+ "response_body_known": true
+}
+```
+
+Filename rule: `<method>__<path-slug>__<hash>.json` where method is lowercased, path-slug replaces `/` with `_` and strips `{` `}`, hash is `sha256(METHOD + " " + normalizedPath)[:8]` hex.
+
+Directory write contract:
+1. Build the sample-file contents in memory for every endpoint group.
+2. Create `<spec-stem>-samples.tmp/` (clean if pre-existing).
+3. Write every sample file into the temp directory.
+4. If a `<spec-stem>-samples/` directory already exists, rename it to `<spec-stem>-samples.backup/`.
+5. Rename the `.tmp/` directory to `<spec-stem>-samples/`.
+6. On error at step 4 or 5, restore the backup.
+7. On success, remove the backup.
+
+Patterns to follow: `writeBrowserSniffOutputs` two-phase pattern in browser_sniff.go L95-138 (`siblingTempPath`, `backupFileForReplace`, `restoreFileBackup`). Mirror for directories using `os.Rename` on the directory.
+
+Test scenarios:
+- Two endpoints with identical templated path but different methods produce different filenames (different hash inputs).
+- Same endpoint captured twice across reruns produces the same filename (hash deterministic).
+- Sample selection: 5 entries (200, 200, 500, 200, 404) -> the most-recent 200 wins.
+- Sample selection: 3 entries all 500 -> the most-recent 500 wins.
+- Redaction header: `Authorization: Bearer XYZ` becomes `Authorization: <redacted>` and `redactions` contains `request_headers.authorization`.
+- Redaction body key: response body `{"api_key": "sk_live_..."}` becomes `{"api_key": "<redacted>"}` and `redactions` contains `response_body.api_key`.
+- Redaction value regex: response body containing a JWT-shaped string outside a known key still gets replaced with `<redacted>` token-by-token; `redactions` contains `pattern:jwt`.
+- Filename slugging: `POST /v1/orders/{orderId}/items/{itemId}` slugs to `post__v1_orders_orderId_items_itemId__<hash>.json`.
+- Empty response body: file is written with `response_body: null` and `response_body_known: false`.
+- HTML response (`groupLooksHTML` is true): sample still written; `response_body` is the raw HTML string truncated to a reasonable limit (e.g. 16 KB) with a `response_body_truncated: true` flag if truncation occurred. Truncation limit goes in a named constant.
+- Directory restore: if spec write fails after samples directory is in place, the previous samples directory is restored from the `.backup/` sibling (test extends the existing `TestWriteBrowserSniffOutputsRestoresExistingFilesWhenSpecPublishFails`).
+
+Verification: samples directory exists with one file per endpoint group; filenames pass `regexp.MustCompile("^[a-z]+__[a-zA-Z0-9_]+__[0-9a-f]{8}\\.json$")`; redaction tests pass; restore-on-failure test passes; golden fixture matches.
+
+### U4. `--min-samples` and `--include` flags
+
+Goal: give operators tunable knobs for noise reduction at the spec-emission boundary, and a rescue mechanism for cases where the default blocklist or static-asset demotion drops something they need.
+
+Requirements: operator-visible noise filtering and rescue.
+
+Dependencies: U2 (so dropped-by-min-samples endpoints remain visible in `TrafficAnalysis.EndpointClusters` with their `low` bucket and `single-sample` flag for audit).
+
+Files:
+- internal/cli/browser_sniff.go (`newBrowserSniffCmd` flag definitions; thread values to spec emission)
+- internal/browsersniff/classifier.go (add `SetAdditionalIncludeList` mirror of `SetAdditionalBlocklist`; extend `scoreEntry` to short-circuit positive on include match)
+- internal/browsersniff/classifier_test.go (include precedence, score short-circuit)
+- internal/browsersniff/specgen.go (apply `MinSamples` filter at the boundary where groups become spec endpoints in `AnalyzeCapture` - below-threshold groups skip endpoint emission but still appear in `TrafficAnalysis.EndpointClusters` because that path runs independently)
+- internal/browsersniff/specgen_test.go (min-samples emission cases)
+- internal/cli/browser_sniff_test.go (flag parsing + integration)
+- skills/printing-press/references/browser-sniff-capture.md (document `--min-samples`, `--include`, and the relationship to `--blocklist`)
+- testdata/golden/expected/ (help-text golden refresh; specific min-samples golden case)
+
+Approach:
+- Add `--min-samples` (int, default 1) and `--include` (string, comma-separated, default empty) to `newBrowserSniffCmd`.
+- Add a package-level `SetAdditionalIncludeList(patterns []string)` and `additionalIncludeList []string` mirror of the existing `SetAdditionalBlocklist` / `additionalBlocklist` machinery in classifier.go.
+- Modify `scoreEntry` to check the include list first. If the URL host or path matches any include pattern (substring match; mirror the simple `HasSuffix(host, blocked)` style used by blocklist), force the score to a high positive value and skip the rest of the scoring. This makes include unconditionally win over blocklist demotion AND static-asset suffix demotion.
+- Thread `--min-samples` through `AnalyzeCapture` to the point where endpoint groups become `apiSpec.Resources[].Endpoints[]`. Drop groups whose `len(Entries) < minSamples`. Do not touch `AnalyzeTraffic` - clusters with low sample counts still appear in `TrafficAnalysis` so the user can see what was filtered.
+- Document `--min-samples=2` as the recommended value for production capture in the SKILL.md reference.
+
+Patterns to follow: existing `SetAdditionalBlocklist` and `additionalBlocklist` machinery (classifier.go L22-23, L53-58). Existing `splitCSV` helper (browser_sniff.go L173). Existing `cmd.Flags().StringVar` pattern in `newBrowserSniffCmd`.
+
+Test scenarios:
+- Default behavior (`--min-samples=1`): existing capture fixtures produce identical spec output (regression guard).
+- `--min-samples=2`: an endpoint with one sample drops from the emitted spec but remains in `TrafficAnalysis.EndpointClusters` with `Confidence: "low"` and `NormalizationFlags: ["single-sample"]`.
+- `--min-samples=5`: endpoints with four or fewer samples drop from the spec; their samples files (from U3) still exist on disk so the audit trail is intact.
+- `--include "/track/important"`: a URL that the existing analytics blocklist would normally demote returns from `scoreEntry` with positive score, becomes an API entry, and lands in the emitted spec.
+- `--include "api.competitor.com"`: a host on the default blocklist is rescued; classifier returns it as API.
+- `--include` and `--blocklist` both set, overlapping pattern: include wins.
+- `--include` matches override the static-asset suffix demotion: a `.js` URL listed in include lands as API.
+- Help golden reflects the new flags with the recommended-value note in the description text.
+
+Verification: `printing-press browser-sniff --help` shows both new flags; default behavior on existing golden fixtures is unchanged; targeted goldens cover the new behavior; `scripts/golden.sh verify` clean.
+
+---
+
+## System-Wide Impact
+
+- Phase 2 Free/Paid Tier Routing Enrichment (skills/printing-press/SKILL.md L1707 region) and Public Parameter Name Enrichment (L1637 region) gain per-endpoint signal. Tier routing accuracy on sites that mix anonymous and authenticated endpoints improves. The SKILL.md change is a note pointing at `endpoint.ObservedAuth` and `cluster.ObservedAuth`; the enrichment logic itself is not in scope.
+- Absorb manifest gating (skills/printing-press/SKILL.md L1241 region) can read `TrafficAnalysis.EndpointClusters[].Confidence` to weight novel-feature suggestions and de-prioritize `low`-bucket endpoints. Plan emits the data; the absorb-side consumer is a separate change.
+- Dogfood `reimplementation_check` (per AGENTS.md "Anti-reimplementation") can read `<spec-stem>-samples/` to ground a "this endpoint actually exists in the trace" assertion. Consumer is out of scope.
+- Golden harness churn: every unit touches `testdata/golden/expected/...`. Land one unit per PR, refresh that unit's goldens, explain the diff in the PR body per AGENTS.md.
+- `crowd-sniff` (internal/cli/crowd_sniff.go) calls `browsersniff.WriteSpec` directly. The `ObservedAuth` field on `spec.Endpoint` becomes available to crowd-sniff too; crowd-sniff would emit empty arrays (no observed traffic to derive from). The omitempty tag keeps the YAML clean.
+- The `internal/spec` package gains a single field. Other consumers of `spec.Endpoint` (`internal/generator`, `internal/openapi`, ingestion paths) tolerate unknown fields and ignore the new one by default. No generator behavior changes until a future consumer wires `ObservedAuth` in.
+- `internal/browsersniff` package gains roughly 300-500 LoC across four units plus one new file (`redact.go`). No external dependency added; everything uses the Go standard library + existing PP packages.
+- Printed CLIs are not directly affected. They consume the spec; a new optional field on `spec.Endpoint` is invisible to them until a generator consumer is added.
+
+## Risks & Mitigations
+
+- Golden churn across all four units. Mitigation: one unit per PR, golden diffs explained in PR body.
+- `spec.Endpoint` field addition is a public-ish surface change because the YAML format is the contract for vendor-spec inputs and printed-CLI generation. Mitigation: field is additive with `omitempty`; older readers ignore it; no version bump required. The contract is preserved.
+- `--min-samples` default of 1 keeps behavior identical for existing callers. A future patch might raise the default to 2 once novel-feature ranking actually consumes confidence; that change is explicitly out of scope here.
+- Samples directory atomicity uses `os.Rename` on directories, which is atomic on POSIX. On Windows the rename may fail if the target exists; PP is already POSIX-first per AGENTS.md tooling and CI matrix. Mitigation: document POSIX assumption near the writer; defer Windows support.
+- Path-hash collisions at 8 hex chars are theoretically possible (~10^-9 birthday for typical endpoint counts). If two endpoints ever collide the failure is visible (two endpoints share a file, the second write overwrites the first). Mitigation if it ever fires: extend hash to 12 chars in a follow-up.
+- `ObservedAuth` is observation-only; it must not be confused with a security scheme declaration. Mitigation: field name, doc-comment on the spec field, and the SKILL.md consumer note all reinforce the distinction.
+- Redaction is best-effort. The default header/key/regex set covers common credentials but app-specific secrets may slip through. Mitigation: document this limitation in `browser-sniff-capture.md`; a follow-up unit could add a `--redact` flag to extend the defaults (deferred).
+
+## Verification (whole plan)
+
+- `go test ./internal/browsersniff/... ./internal/cli/... ./internal/spec/...` is green.
+- `scripts/golden.sh verify` is green after each unit's golden refresh.
+- `printing-press browser-sniff --har <existing-fixture>.har` produces: spec YAML where mixed-auth endpoints carry `observed_auth`; a `<spec-stem>-traffic-analysis.json` where each `EndpointCluster` has `sample_count`, `status_codes`, `normalization_flags`, `confidence`, `observed_auth`; a `<spec-stem>-samples/` directory with one redacted file per endpoint group; and `--min-samples=2` produces a spec with the long tail dropped while the traffic-analysis still lists every cluster.
+- One end-to-end run of `/printing-press <api>` on a site with mixed anonymous + authenticated endpoints shows the tier-routing enrichment phase referencing per-endpoint auth surface.
diff --git a/internal/browsersniff/analysis.go b/internal/browsersniff/analysis.go
index df651c55..2fb14d87 100644
--- a/internal/browsersniff/analysis.go
+++ b/internal/browsersniff/analysis.go
@@ -308,16 +308,34 @@ func (p *ProtectionObservation) UnmarshalJSON(data []byte) error {
}
type EndpointCluster struct {
- Host string `json:"host,omitempty"`
- Method string `json:"method"`
- Path string `json:"path"`
- Count int `json:"count"`
- Statuses []int `json:"statuses,omitempty"`
- ContentTypes []string `json:"content_types,omitempty"`
- SizeClass string `json:"size_class,omitempty"`
- RequestShape ShapeSummary `json:"request_shape"`
- ResponseShape ShapeSummary `json:"response_shape"`
- Evidence []EvidenceRef `json:"evidence,omitempty"`
+ Host string `json:"host,omitempty"`
+ Method string `json:"method"`
+ Path string `json:"path"`
+ Count int `json:"count"`
+ Statuses []int `json:"statuses,omitempty"`
+ ContentTypes []string `json:"content_types,omitempty"`
+ SizeClass string `json:"size_class,omitempty"`
+ RequestShape ShapeSummary `json:"request_shape"`
+ ResponseShape ShapeSummary `json:"response_shape"`
+ // ObservedAuth lists lowercased request header names observed on this
+ // cluster's entries that match common auth surfaces (Authorization,
+ // Cookie, X-API-Key, etc.). Observation-only — values are never recorded.
+ // Mirrors spec.Endpoint.ObservedAuth so downstream gates can read
+ // per-endpoint auth signal directly from the traffic-analysis sidecar.
+ ObservedAuth []string `json:"observed_auth,omitempty"`
+ // NormalizationFlags surfaces per-cluster shape anomalies that downstream
+ // confidence consumers (absorb gate, dogfood, novel-feature ranking) care
+ // about. Possible values: single-sample, single-status, mixed-content-types,
+ // request-body-only-on-some-samples, divergent-response-shape. Empty
+ // slice is omitted via omitempty.
+ NormalizationFlags []string `json:"normalization_flags,omitempty"`
+ // Confidence is a coarse bucket derived from Count, Statuses, and
+ // NormalizationFlags: "low" when Count<3 or any flag is set, "medium"
+ // for 3-9 samples with no flags, "high" for 10+ samples with multiple
+ // status codes and no flags. The bucket is intentionally coarse so
+ // future numeric-confidence refinements stay backward-compatible.
+ Confidence string `json:"confidence,omitempty"`
+ Evidence []EvidenceRef `json:"evidence,omitempty"`
}
type ShapeSummary struct {
@@ -1105,6 +1123,9 @@ func buildEndpointClusters(groups []EndpointGroup, entries []EnrichedEntry) []En
cluster.SizeClass = classifyBodySize(totalSize, len(group.Entries))
cluster.RequestShape = summarizeRequestShape(group.Entries, requestBodies)
cluster.ResponseShape = summarizeResponseShape(responseBodies)
+ cluster.ObservedAuth = observedAuthHeaders(group.Entries)
+ cluster.NormalizationFlags = computeNormalizationFlags(group.Method, cluster.Count, cluster.Statuses, cluster.ContentTypes, len(requestBodies))
+ cluster.Confidence = bucketConfidence(cluster.Count, cluster.Statuses, cluster.NormalizationFlags)
clusters = append(clusters, cluster)
}
sort.Slice(clusters, func(i, j int) bool {
@@ -1119,6 +1140,74 @@ func buildEndpointClusters(groups []EndpointGroup, entries []EnrichedEntry) []En
return clusters
}
+// computeNormalizationFlags returns the set of shape-anomaly flags for an
+// endpoint cluster. Flags surface signals downstream confidence consumers
+// care about — small samples, single-status responses, content-type drift,
+// inconsistent request-body presence on write methods. Order is stable so
+// the resulting JSON is golden-friendly.
+//
+// The divergent-response-shape flag from the plan is reserved here for a
+// future signal: it would fire when pre-normalization paths collapsed but
+// responses diverged structurally. Today's classifier already keys clusters
+// by host + method + normalizedPath so that collapse cannot happen, hence
+// the flag is never populated. Kept in the documented set so writers and
+// readers across PP and external review tooling share a single vocabulary.
+func computeNormalizationFlags(method string, sampleCount int, statuses []int, contentTypes []string, requestBodyCount int) []string {
+ flags := make([]string, 0, 4)
+ if sampleCount <= 1 {
+ flags = append(flags, "single-sample")
+ }
+ if len(statuses) == 1 {
+ flags = append(flags, "single-status")
+ }
+ if hasMixedContentTypes(contentTypes) {
+ flags = append(flags, "mixed-content-types")
+ }
+ switch strings.ToUpper(strings.TrimSpace(method)) {
+ case "POST", "PUT", "PATCH":
+ if requestBodyCount > 0 && requestBodyCount < sampleCount {
+ flags = append(flags, "request-body-only-on-some-samples")
+ }
+ }
+ if len(flags) == 0 {
+ return nil
+ }
+ return flags
+}
+
+// hasMixedContentTypes returns true when the cluster's content types span
+// more than one media type after stripping parameters. "application/json"
+// and "application/json; charset=utf-8" count as the same media type, since
+// the only difference is encoding metadata.
+func hasMixedContentTypes(contentTypes []string) bool {
+ seen := map[string]bool{}
+ for _, ct := range contentTypes {
+ head := strings.SplitN(ct, ";", 2)[0]
+ normalized := strings.ToLower(strings.TrimSpace(head))
+ if normalized == "" {
+ continue
+ }
+ seen[normalized] = true
+ if len(seen) > 1 {
+ return true
+ }
+ }
+ return false
+}
+
+// bucketConfidence maps Count + Statuses + flags onto a coarse low/medium/high
+// label. Coarse on purpose — future numeric refinements should not require
+// downstream consumers to learn new label values.
+func bucketConfidence(sampleCount int, statuses []int, flags []string) string {
+ if sampleCount < 3 || len(flags) > 0 {
+ return "low"
+ }
+ if sampleCount >= 10 && len(statuses) >= 2 {
+ return "high"
+ }
+ return "medium"
+}
+
func originalEntryIndexes(entries []EnrichedEntry) map[string][]int {
indexes := make(map[string][]int, len(entries))
for index, entry := range entries {
diff --git a/internal/browsersniff/analysis_test.go b/internal/browsersniff/analysis_test.go
index eda0ff05..e55feb0b 100644
--- a/internal/browsersniff/analysis_test.go
+++ b/internal/browsersniff/analysis_test.go
@@ -45,6 +45,273 @@ func TestAnalyzeTraffic_EmptyAndNilCapture(t *testing.T) {
assert.Contains(t, warningTypes(analysis.Warnings), "empty_capture")
}
+func TestComputeNormalizationFlags(t *testing.T) {
+ t.Parallel()
+
+ cases := []struct {
+ name string
+ method string
+ sampleCount int
+ statuses []int
+ contentTypes []string
+ requestBodyCount int
+ want []string
+ }{
+ {
+ name: "single sample triggers single-sample and single-status",
+ method: "GET",
+ sampleCount: 1,
+ statuses: []int{200},
+ contentTypes: []string{"application/json"},
+ want: []string{"single-sample", "single-status"},
+ },
+ {
+ name: "five samples all-200 still single-status",
+ method: "GET",
+ sampleCount: 5,
+ statuses: []int{200},
+ contentTypes: []string{"application/json"},
+ want: []string{"single-status"},
+ },
+ {
+ name: "ten samples multi-status clean",
+ method: "GET",
+ sampleCount: 10,
+ statuses: []int{200, 404},
+ contentTypes: []string{"application/json"},
+ want: nil,
+ },
+ {
+ name: "POST body inconsistent fires only when some samples have body",
+ method: "POST",
+ sampleCount: 3,
+ statuses: []int{200, 201},
+ contentTypes: []string{"application/json"},
+ requestBodyCount: 2,
+ want: []string{"request-body-only-on-some-samples"},
+ },
+ {
+ name: "POST all samples have body does not fire flag",
+ method: "POST",
+ sampleCount: 3,
+ statuses: []int{200, 201},
+ contentTypes: []string{"application/json"},
+ requestBodyCount: 3,
+ want: nil,
+ },
+ {
+ name: "GET with body samples does not fire request-body flag",
+ method: "GET",
+ sampleCount: 3,
+ statuses: []int{200, 304},
+ contentTypes: []string{"application/json"},
+ requestBodyCount: 2,
+ want: nil,
+ },
+ {
+ name: "mixed content types fires only on real media-type drift",
+ method: "GET",
+ sampleCount: 4,
+ statuses: []int{200, 404},
+ contentTypes: []string{"application/json", "text/html"},
+ want: []string{"mixed-content-types"},
+ },
+ {
+ name: "json charset variants do not count as mixed",
+ method: "GET",
+ sampleCount: 4,
+ statuses: []int{200, 404},
+ contentTypes: []string{"application/json", "application/json; charset=utf-8"},
+ want: nil,
+ },
+ {
+ name: "zero samples treated as single-sample edge",
+ method: "GET",
+ sampleCount: 0,
+ statuses: []int{},
+ contentTypes: nil,
+ want: []string{"single-sample"},
+ },
+ }
+
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ t.Parallel()
+ got := computeNormalizationFlags(tc.method, tc.sampleCount, tc.statuses, tc.contentTypes, tc.requestBodyCount)
+ assert.Equal(t, tc.want, got)
+ })
+ }
+}
+
+func TestBucketConfidence(t *testing.T) {
+ t.Parallel()
+
+ cases := []struct {
+ name string
+ sampleCount int
+ statuses []int
+ flags []string
+ want string
+ }{
+ {name: "single sample with flag -> low", sampleCount: 1, statuses: []int{200}, flags: []string{"single-sample"}, want: "low"},
+ {name: "two samples no flags -> low", sampleCount: 2, statuses: []int{200, 404}, flags: nil, want: "low"},
+ {name: "three samples no flags -> medium", sampleCount: 3, statuses: []int{200, 404}, flags: nil, want: "medium"},
+ {name: "nine samples no flags -> medium", sampleCount: 9, statuses: []int{200, 404}, flags: nil, want: "medium"},
+ {name: "ten samples multi-status no flags -> high", sampleCount: 10, statuses: []int{200, 404}, flags: nil, want: "high"},
+ {name: "ten samples single status -> medium not high", sampleCount: 10, statuses: []int{200}, flags: nil, want: "medium"},
+ {name: "twenty samples with flag -> low overrides count", sampleCount: 20, statuses: []int{200, 404}, flags: []string{"mixed-content-types"}, want: "low"},
+ }
+
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ t.Parallel()
+ got := bucketConfidence(tc.sampleCount, tc.statuses, tc.flags)
+ assert.Equal(t, tc.want, got)
+ })
+ }
+}
+
+func TestAnalyzeTraffic_PopulatesConfidenceAndFlagsOnClusters(t *testing.T) {
+ t.Parallel()
+
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/items",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ },
+ }
+
+ analysis, err := AnalyzeTraffic(capture)
+ require.NoError(t, err)
+ require.NotEmpty(t, analysis.EndpointClusters)
+
+ cluster := analysis.EndpointClusters[0]
+ assert.Equal(t, 1, cluster.Count)
+ assert.Equal(t, "low", cluster.Confidence)
+ assert.Contains(t, cluster.NormalizationFlags, "single-sample")
+ assert.Contains(t, cluster.NormalizationFlags, "single-status")
+}
+
+func TestAnalyzeTraffic_ClusterConfidenceRoundTripsThroughSidecar(t *testing.T) {
+ t.Parallel()
+
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/items",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ },
+ }
+
+ analysis, err := AnalyzeTraffic(capture)
+ require.NoError(t, err)
+
+ tmp := filepath.Join(t.TempDir(), "spec-traffic-analysis.json")
+ require.NoError(t, WriteTrafficAnalysis(analysis, tmp))
+
+ roundTrip, err := ReadTrafficAnalysis(tmp)
+ require.NoError(t, err)
+ require.NotEmpty(t, roundTrip.EndpointClusters)
+ assert.Equal(t, analysis.EndpointClusters[0].Confidence, roundTrip.EndpointClusters[0].Confidence)
+ assert.Equal(t, analysis.EndpointClusters[0].NormalizationFlags, roundTrip.EndpointClusters[0].NormalizationFlags)
+}
+
+func TestAnalyzeTraffic_PopulatesObservedAuthOnEndpointCluster(t *testing.T) {
+ t.Parallel()
+
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/items",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{
+ "Authorization": "Bearer eyJtoken",
+ "Cookie": "session=x",
+ },
+ },
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/public",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"ok":true}`,
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ },
+ }
+
+ analysis, err := AnalyzeTraffic(capture)
+ require.NoError(t, err)
+
+ var authedCluster *EndpointCluster
+ var publicCluster *EndpointCluster
+ for i := range analysis.EndpointClusters {
+ c := &analysis.EndpointClusters[i]
+ switch c.Path {
+ case "/v1/items":
+ authedCluster = c
+ case "/v1/public":
+ publicCluster = c
+ }
+ }
+ require.NotNil(t, authedCluster, "expected /v1/items cluster")
+ require.NotNil(t, publicCluster, "expected /v1/public cluster")
+
+ assert.Equal(t, []string{"authorization", "cookie"}, authedCluster.ObservedAuth)
+ assert.Nil(t, publicCluster.ObservedAuth, "public endpoint should omit observed_auth")
+}
+
+func TestAnalyzeTraffic_ObservedAuthCanonicalizesAcrossSamples(t *testing.T) {
+ t.Parallel()
+
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/me",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{"AUTHORIZATION": "Bearer a"},
+ },
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/me",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{"authorization": "Bearer b", "X-API-Key": "k"},
+ },
+ },
+ }
+
+ analysis, err := AnalyzeTraffic(capture)
+ require.NoError(t, err)
+
+ require.NotEmpty(t, analysis.EndpointClusters)
+ cluster := analysis.EndpointClusters[0]
+ assert.Equal(t, "/v1/me", cluster.Path)
+ assert.Equal(t, []string{"authorization", "x-api-key"}, cluster.ObservedAuth)
+}
+
func TestAnalyzeTraffic_RedactsAuthSignals(t *testing.T) {
t.Parallel()
diff --git a/internal/browsersniff/classifier.go b/internal/browsersniff/classifier.go
index a18b57d3..a04d43c6 100644
--- a/internal/browsersniff/classifier.go
+++ b/internal/browsersniff/classifier.go
@@ -21,6 +21,8 @@ var (
numericPattern = regexp.MustCompile(`^\d+$`)
blocklistMu sync.RWMutex
additionalBlocklist []string
+ includeListMu sync.RWMutex
+ additionalInclude []string
)
func ClassifyEntries(entries []EnrichedEntry) (api []EnrichedEntry, noise []EnrichedEntry) {
@@ -32,8 +34,9 @@ func ClassifyEntries(entries []EnrichedEntry) (api []EnrichedEntry, noise []Enri
blocklistMu.RUnlock()
blocklist := append(DefaultBlocklist(), extraBlocklist...)
+ include := includePatterns()
for _, entry := range entries {
- score := scoreEntry(entry, blocklist)
+ score := scoreEntry(entry, blocklist, include)
classified := entry
if score > 0 {
classified.Classification = "api"
@@ -57,6 +60,50 @@ func SetAdditionalBlocklist(domains []string) {
additionalBlocklist = append([]string(nil), domains...)
}
+// SetAdditionalIncludeList stores operator-supplied include patterns that
+// force a positive score in classification regardless of blocklist matches
+// or static-asset suffix demotion. Patterns are matched as case-insensitive
+// substrings against the URL's host and path. Include wins over blocklist.
+func SetAdditionalIncludeList(patterns []string) {
+ includeListMu.Lock()
+ defer includeListMu.Unlock()
+
+ additionalInclude = append([]string(nil), patterns...)
+}
+
+func includePatterns() []string {
+ includeListMu.RLock()
+ defer includeListMu.RUnlock()
+ if len(additionalInclude) == 0 {
+ return nil
+ }
+ out := make([]string, len(additionalInclude))
+ copy(out, additionalInclude)
+ return out
+}
+
+// matchesIncludePattern returns true when any include pattern is a
+// case-insensitive substring of host or path. Substring matching keeps the
+// flag friendly to operators: --include "/track/important" or
+// --include "api.partner.com" both work without quoting regex metacharacters.
+func matchesIncludePattern(host string, path string, patterns []string) bool {
+ if len(patterns) == 0 {
+ return false
+ }
+ lowerHost := strings.ToLower(host)
+ lowerPath := strings.ToLower(path)
+ for _, pattern := range patterns {
+ p := strings.ToLower(strings.TrimSpace(pattern))
+ if p == "" {
+ continue
+ }
+ if strings.Contains(lowerHost, p) || strings.Contains(lowerPath, p) {
+ return true
+ }
+ }
+ return false
+}
+
func DefaultBlocklist() []string {
return []string{
"google-analytics.com",
@@ -117,7 +164,7 @@ func DeduplicateEndpoints(entries []EnrichedEntry) []EndpointGroup {
return groups
}
-func scoreEntry(entry EnrichedEntry, blocklist []string) int {
+func scoreEntry(entry EnrichedEntry, blocklist []string, include []string) int {
score := 0
responseType := strings.ToLower(entry.ResponseContentType)
requestType := strings.ToLower(getHeaderValue(entry.RequestHeaders, "Content-Type"))
@@ -125,6 +172,15 @@ func scoreEntry(entry EnrichedEntry, blocklist []string) int {
host := strings.ToLower(extractHost(entry.URL))
urlLower := strings.ToLower(entry.URL)
+ // Operator-supplied include patterns short-circuit the rest of scoring:
+ // a match forces a strong positive score, bypassing blocklist demotion,
+ // static-asset suffix demotion, and the response-content-type penalty.
+ // Used to rescue a specific endpoint or host that default heuristics
+ // would otherwise drop.
+ if matchesIncludePattern(host, path, include) {
+ return 10
+ }
+
if strings.Contains(responseType, "application/json") {
score += 2
}
diff --git a/internal/browsersniff/classifier_test.go b/internal/browsersniff/classifier_test.go
index f51636c5..4f82ebed 100644
--- a/internal/browsersniff/classifier_test.go
+++ b/internal/browsersniff/classifier_test.go
@@ -234,3 +234,160 @@ func emptyStrings(values []string) []string {
return values
}
+
+func TestIncludeListRescuesBlockedHost(t *testing.T) {
+ // Not t.Parallel: mutates the package-level include-list state.
+ SetAdditionalIncludeList([]string{"google-analytics.com"})
+ defer SetAdditionalIncludeList(nil)
+
+ // google-analytics.com is on the DefaultBlocklist and would normally
+ // score negative. The include match should force a positive score.
+ entries := []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://www.google-analytics.com/collect?v=1",
+ ResponseContentType: "image/gif",
+ ResponseBody: "",
+ },
+ }
+ api, noise := ClassifyEntries(entries)
+ assert.Len(t, api, 1, "include match should rescue google-analytics endpoint")
+ assert.Empty(t, noise)
+}
+
+func TestIncludeListRescuesStaticAssetByPath(t *testing.T) {
+ SetAdditionalIncludeList([]string{"/track/important"})
+ defer SetAdditionalIncludeList(nil)
+
+ entries := []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://example.com/track/important.js",
+ ResponseContentType: "application/javascript",
+ ResponseBody: "",
+ },
+ }
+ api, noise := ClassifyEntries(entries)
+ assert.Len(t, api, 1, "include path match should rescue static asset")
+ assert.Empty(t, noise)
+}
+
+func TestIncludeListEmptyPreservesDefaultBehavior(t *testing.T) {
+ SetAdditionalIncludeList(nil)
+
+ entries := []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://www.google-analytics.com/collect",
+ ResponseContentType: "image/gif",
+ },
+ }
+ api, noise := ClassifyEntries(entries)
+ assert.Empty(t, api, "without include, analytics endpoint stays in noise")
+ assert.Len(t, noise, 1)
+}
+
+func TestIncludeListWinsOverBlocklistOverlap(t *testing.T) {
+ SetAdditionalBlocklist([]string{"api.partner.com"})
+ SetAdditionalIncludeList([]string{"api.partner.com"})
+ defer SetAdditionalBlocklist(nil)
+ defer SetAdditionalIncludeList(nil)
+
+ entries := []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.partner.com/v1/data",
+ ResponseContentType: "application/json",
+ ResponseBody: `{"ok":true}`,
+ },
+ }
+ api, _ := ClassifyEntries(entries)
+ assert.Len(t, api, 1, "include should win over overlapping blocklist entry")
+}
+
+func TestFilterEndpointsByMinSamples_DropsSingletons(t *testing.T) {
+ t.Parallel()
+
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/popular",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/popular",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":2}`,
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/rare",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ },
+ }
+
+ apiSpec, err := AnalyzeCapture(capture)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ dropped := FilterEndpointsByMinSamples(apiSpec, capture, 2)
+ assert.Equal(t, 1, dropped, "rare endpoint with single sample should drop")
+
+ // The rare endpoint should be gone from the spec, popular should remain.
+ found := map[string]bool{}
+ for _, resource := range apiSpec.Resources {
+ for _, endpoint := range resource.Endpoints {
+ found[endpoint.Path] = true
+ }
+ }
+ assert.True(t, found["/v1/popular"], "popular endpoint should remain")
+ assert.False(t, found["/v1/rare"], "rare endpoint should be filtered")
+}
+
+func TestFilterEndpointsByMinSamples_DefaultIsNoop(t *testing.T) {
+ t.Parallel()
+
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/items",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ },
+ }
+ apiSpec, err := AnalyzeCapture(capture)
+ if err != nil {
+ t.Fatal(err)
+ }
+ before := 0
+ for _, r := range apiSpec.Resources {
+ before += len(r.Endpoints)
+ }
+
+ dropped := FilterEndpointsByMinSamples(apiSpec, capture, 1)
+ assert.Zero(t, dropped, "--min-samples=1 (default) must be a no-op")
+
+ after := 0
+ for _, r := range apiSpec.Resources {
+ after += len(r.Endpoints)
+ }
+ assert.Equal(t, before, after, "endpoint count must not change with default min-samples")
+}
diff --git a/internal/browsersniff/redact.go b/internal/browsersniff/redact.go
new file mode 100644
index 00000000..5b4e273a
--- /dev/null
+++ b/internal/browsersniff/redact.go
@@ -0,0 +1,188 @@
+// Redaction helpers for samples written to <spec-stem>-samples/. Headers,
+// JSON body keys, and string values matching JWT / email / E.164 phone
+// patterns are replaced with the sentinel value below. The original
+// EnrichedCapture stays untouched — only sample-file output is redacted.
+package browsersniff
+
+import (
+ "encoding/json"
+ "fmt"
+ "regexp"
+ "strings"
+)
+
+// RedactedSentinel is the replacement string written in place of any
+// redacted header value or JSON value. Kept as a string (not nil) so the
+// surrounding structural type is preserved for schema inference and so
+// reviewers can tell at a glance that the field existed.
+const RedactedSentinel = "<redacted>"
+
+var (
+ redactHeaderExact = map[string]bool{
+ "authorization": true,
+ "cookie": true,
+ "set-cookie": true,
+ "x-csrf-token": true,
+ "x-xsrf-token": true,
+ "x-api-key": true,
+ "proxy-authorization": true,
+ }
+ redactHeaderContains = []string{"token", "secret", "signature", "api-key", "api_key"}
+ redactBodyKeys = map[string]bool{
+ "password": true,
+ "token": true,
+ "secret": true,
+ "apikey": true,
+ "accesstoken": true,
+ "refreshtoken": true,
+ "creditcard": true,
+ "ssn": true,
+ }
+ redactJWTPattern = regexp.MustCompile(`eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+`)
+ redactEmailPattern = regexp.MustCompile(`[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}`)
+ // Phone redaction is intentionally conservative: only matches numbers
+ // written in canonical E.164 form (leading `+`, country code, 7-14
+ // trailing digits). Plain long digit runs without a `+` are skipped to
+ // avoid false-positive redaction of timestamps, IDs, and order numbers.
+ redactPhonePattern = regexp.MustCompile(`\+[1-9]\d{6,14}`)
+)
+
+// RedactHeaders returns a redacted copy of headers plus the sorted set of
+// lowercased header names whose values were replaced. Headers not matching
+// the auth-shape patterns are copied through unchanged. Returns a non-nil
+// map even when no redactions fire so callers don't have to nil-check.
+func RedactHeaders(headers map[string]string) (map[string]string, []string) {
+ out := make(map[string]string, len(headers))
+ redacted := map[string]bool{}
+ for name, value := range headers {
+ lower := strings.ToLower(strings.TrimSpace(name))
+ if isRedactHeaderName(lower) {
+ out[name] = RedactedSentinel
+ redacted[lower] = true
+ continue
+ }
+ out[name] = value
+ }
+ if len(redacted) == 0 {
+ return out, nil
+ }
+ return out, sortedBoolKeys(redacted)
+}
+
+func isRedactHeaderName(lowerName string) bool {
+ if redactHeaderExact[lowerName] {
+ return true
+ }
+ for _, contains := range redactHeaderContains {
+ if strings.Contains(lowerName, contains) {
+ return true
+ }
+ }
+ return false
+}
+
+// RedactJSONBody returns the body with sensitive JSON keys and value
+// patterns replaced, plus a sorted list of dotted paths where redactions
+// occurred. If the body parses as JSON, the structure is preserved and
+// values are replaced in place. If the body is not JSON, the raw string is
+// regex-swept for JWT / email / phone patterns and the list contains the
+// pattern names that hit (`pattern:jwt`, `pattern:email`, `pattern:phone`).
+func RedactJSONBody(body string) (string, []string) {
+ trimmed := strings.TrimSpace(body)
+ if trimmed == "" {
+ return body, nil
+ }
+ var parsed any
+ if err := json.Unmarshal([]byte(trimmed), &parsed); err == nil {
+ paths := map[string]bool{}
+ redacted := redactJSONValue(parsed, "", paths)
+ marshaled, err := json.Marshal(redacted)
+ if err == nil {
+ return string(marshaled), sortedBoolKeys(paths)
+ }
+ }
+ return redactStringPatterns(body)
+}
+
+func redactJSONValue(value any, path string, paths map[string]bool) any {
+ switch v := value.(type) {
+ case map[string]any:
+ for key, child := range v {
+ childPath := joinPath(path, key)
+ if isRedactBodyKey(key) {
+ v[key] = RedactedSentinel
+ paths[childPath] = true
+ continue
+ }
+ v[key] = redactJSONValue(child, childPath, paths)
+ }
+ return v
+ case []any:
+ for i, child := range v {
+ childPath := fmt.Sprintf("%s[%d]", path, i)
+ v[i] = redactJSONValue(child, childPath, paths)
+ }
+ return v
+ case string:
+ if redacted, pattern := redactStringValue(v); pattern != "" {
+ paths[joinPath(path, "pattern:"+pattern)] = true
+ return redacted
+ }
+ return v
+ default:
+ return value
+ }
+}
+
+// isRedactBodyKey normalizes a key to lowercase with separators stripped
+// (so api_key, apiKey, and api-key all collapse to "apikey") and looks up
+// against the redact list.
+func isRedactBodyKey(name string) bool {
+ normalized := strings.ToLower(name)
+ normalized = strings.ReplaceAll(normalized, "_", "")
+ normalized = strings.ReplaceAll(normalized, "-", "")
+ return redactBodyKeys[normalized]
+}
+
+func redactStringValue(s string) (string, string) {
+ switch {
+ case redactJWTPattern.MatchString(s):
+ return RedactedSentinel, "jwt"
+ case redactEmailPattern.MatchString(s):
+ return RedactedSentinel, "email"
+ case redactPhonePattern.MatchString(s):
+ return RedactedSentinel, "phone"
+ }
+ return s, ""
+}
+
+// redactStringPatterns applies the same JWT / email / phone sweep against a
+// raw (non-JSON) body. Returns the body with matching substrings replaced
+// and a list of pattern names that fired.
+func redactStringPatterns(body string) (string, []string) {
+ patterns := map[string]bool{}
+ out := body
+ if redactJWTPattern.MatchString(out) {
+ out = redactJWTPattern.ReplaceAllString(out, RedactedSentinel)
+ patterns["pattern:jwt"] = true
+ }
+ if redactEmailPattern.MatchString(out) {
+ out = redactEmailPattern.ReplaceAllString(out, RedactedSentinel)
+ patterns["pattern:email"] = true
+ }
+ if redactPhonePattern.MatchString(out) {
+ out = redactPhonePattern.ReplaceAllString(out, RedactedSentinel)
+ patterns["pattern:phone"] = true
+ }
+ if len(patterns) == 0 {
+ return body, nil
+ }
+ return out, sortedBoolKeys(patterns)
+}
+
+func joinPath(parent string, child string) string {
+ if parent == "" {
+ return child
+ }
+ return parent + "." + child
+}
diff --git a/internal/browsersniff/redact_test.go b/internal/browsersniff/redact_test.go
new file mode 100644
index 00000000..4a6ffdad
--- /dev/null
+++ b/internal/browsersniff/redact_test.go
@@ -0,0 +1,210 @@
+package browsersniff
+
+import (
+ "encoding/json"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+func TestRedactHeaders(t *testing.T) {
+ t.Parallel()
+
+ cases := []struct {
+ name string
+ headers map[string]string
+ wantRedacted map[string]string
+ wantRedacList []string
+ }{
+ {
+ name: "authorization redacted",
+ headers: map[string]string{
+ "Authorization": "Bearer eyJtok.foo.bar",
+ "Accept": "application/json",
+ },
+ wantRedacted: map[string]string{
+ "Authorization": RedactedSentinel,
+ "Accept": "application/json",
+ },
+ wantRedacList: []string{"authorization"},
+ },
+ {
+ name: "no auth headers preserves map and returns nil list",
+ headers: map[string]string{"Accept": "application/json"},
+ wantRedacted: map[string]string{"Accept": "application/json"},
+ wantRedacList: nil,
+ },
+ {
+ name: "cookie set-cookie csrf api-key all redacted",
+ headers: map[string]string{
+ "Cookie": "session=x",
+ "Set-Cookie": "session=x; Path=/",
+ "X-CSRF-Token": "csrf",
+ "X-API-Key": "k",
+ },
+ wantRedacted: map[string]string{
+ "Cookie": RedactedSentinel,
+ "Set-Cookie": RedactedSentinel,
+ "X-CSRF-Token": RedactedSentinel,
+ "X-API-Key": RedactedSentinel,
+ },
+ wantRedacList: []string{"cookie", "set-cookie", "x-api-key", "x-csrf-token"},
+ },
+ {
+ name: "contains-token contains-secret contains-signature patterns",
+ headers: map[string]string{
+ "X-Auth-Token": "t",
+ "X-Hub-Secret": "s",
+ "X-Sig-Signature": "g",
+ "X-Trace-Id": "non-auth",
+ },
+ wantRedacted: map[string]string{
+ "X-Auth-Token": RedactedSentinel,
+ "X-Hub-Secret": RedactedSentinel,
+ "X-Sig-Signature": RedactedSentinel,
+ "X-Trace-Id": "non-auth",
+ },
+ wantRedacList: []string{"x-auth-token", "x-hub-secret", "x-sig-signature"},
+ },
+ {
+ name: "empty input returns empty output",
+ headers: map[string]string{},
+ wantRedacted: map[string]string{},
+ wantRedacList: nil,
+ },
+ }
+
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ t.Parallel()
+ gotHeaders, gotList := RedactHeaders(tc.headers)
+ assert.Equal(t, tc.wantRedacted, gotHeaders)
+ assert.Equal(t, tc.wantRedacList, gotList)
+ })
+ }
+}
+
+func TestRedactJSONBody_KeyBased(t *testing.T) {
+ t.Parallel()
+
+ body := `{"id":42,"name":"widget","api_key":"sk_live_x","nested":{"accessToken":"abc","ok":true}}`
+ redacted, paths := RedactJSONBody(body)
+
+ var parsed map[string]any
+ require.NoError(t, json.Unmarshal([]byte(redacted), &parsed))
+ assert.Equal(t, RedactedSentinel, parsed["api_key"])
+ assert.Equal(t, float64(42), parsed["id"])
+ assert.Equal(t, "widget", parsed["name"])
+
+ nested, ok := parsed["nested"].(map[string]any)
+ require.True(t, ok)
+ assert.Equal(t, RedactedSentinel, nested["accessToken"])
+ assert.Equal(t, true, nested["ok"])
+
+ assert.Contains(t, paths, "api_key")
+ assert.Contains(t, paths, "nested.accessToken")
+}
+
+func TestRedactJSONBody_KeyVariantsCollapse(t *testing.T) {
+ t.Parallel()
+
+ body := `{"api_key":"a","apiKey":"b","api-key":"c","API-Key":"d"}`
+ redacted, paths := RedactJSONBody(body)
+
+ var parsed map[string]any
+ require.NoError(t, json.Unmarshal([]byte(redacted), &parsed))
+ for _, key := range []string{"api_key", "apiKey", "api-key", "API-Key"} {
+ assert.Equal(t, RedactedSentinel, parsed[key], "key %q should be redacted", key)
+ }
+ assert.Len(t, paths, 4)
+}
+
+func TestRedactJSONBody_ValuePatterns(t *testing.T) {
+ t.Parallel()
+
+ body := `{"token_field":"eyJhbGc.eyJzdWI.sig","contact":"user@example.com","line":"+14155551212"}`
+ redacted, paths := RedactJSONBody(body)
+
+ var parsed map[string]any
+ require.NoError(t, json.Unmarshal([]byte(redacted), &parsed))
+ assert.Equal(t, RedactedSentinel, parsed["contact"])
+ assert.Equal(t, RedactedSentinel, parsed["line"])
+ // "token_field" key isn't in the canonical body-key set so the value
+ // itself triggers redaction via the JWT regex.
+ assert.Equal(t, RedactedSentinel, parsed["token_field"])
+
+ hasJWT, hasEmail, hasPhone := false, false, false
+ for _, p := range paths {
+ switch p {
+ case "token_field.pattern:jwt":
+ hasJWT = true
+ case "contact.pattern:email":
+ hasEmail = true
+ case "line.pattern:phone":
+ hasPhone = true
+ }
+ }
+ assert.True(t, hasJWT, "expected JWT pattern path")
+ assert.True(t, hasEmail, "expected email pattern path")
+ assert.True(t, hasPhone, "expected phone pattern path")
+}
+
+func TestRedactJSONBody_ArraysAndNesting(t *testing.T) {
+ t.Parallel()
+
+ body := `{"items":[{"password":"p1"},{"password":"p2","other":"keep"}]}`
+ redacted, paths := RedactJSONBody(body)
+
+ var parsed map[string]any
+ require.NoError(t, json.Unmarshal([]byte(redacted), &parsed))
+ items := parsed["items"].([]any)
+ require.Len(t, items, 2)
+ first := items[0].(map[string]any)
+ second := items[1].(map[string]any)
+ assert.Equal(t, RedactedSentinel, first["password"])
+ assert.Equal(t, RedactedSentinel, second["password"])
+ assert.Equal(t, "keep", second["other"])
+
+ assert.Contains(t, paths, "items[0].password")
+ assert.Contains(t, paths, "items[1].password")
+}
+
+func TestRedactJSONBody_NonJSONFallback(t *testing.T) {
+ t.Parallel()
+
+ body := "Token: eyJabc.def.ghi and email user@example.com submitted"
+ redacted, paths := RedactJSONBody(body)
+
+ assert.Contains(t, redacted, RedactedSentinel)
+ assert.NotContains(t, redacted, "eyJabc.def.ghi")
+ assert.NotContains(t, redacted, "user@example.com")
+ assert.Contains(t, paths, "pattern:jwt")
+ assert.Contains(t, paths, "pattern:email")
+}
+
+func TestRedactJSONBody_EmptyBodyReturnsEmpty(t *testing.T) {
+ t.Parallel()
+
+ got, paths := RedactJSONBody("")
+ assert.Equal(t, "", got)
+ assert.Nil(t, paths)
+
+ got2, paths2 := RedactJSONBody(" ")
+ assert.Equal(t, " ", got2)
+ assert.Nil(t, paths2)
+}
+
+func TestRedactJSONBody_PhoneRequiresPlusPrefix(t *testing.T) {
+ t.Parallel()
+
+ // Long digit run without leading + should not redact (avoid false positives
+ // on timestamps, IDs, order numbers).
+ body := `{"order_number":"15551234567","line":"+14155551212"}`
+ redacted, _ := RedactJSONBody(body)
+
+ var parsed map[string]any
+ require.NoError(t, json.Unmarshal([]byte(redacted), &parsed))
+ assert.Equal(t, "15551234567", parsed["order_number"])
+ assert.Equal(t, RedactedSentinel, parsed["line"])
+}
diff --git a/internal/browsersniff/specgen.go b/internal/browsersniff/specgen.go
index ca446ecf..fcac05a9 100644
--- a/internal/browsersniff/specgen.go
+++ b/internal/browsersniff/specgen.go
@@ -1,6 +1,8 @@
package browsersniff
import (
+ "crypto/sha256"
+ "encoding/hex"
"encoding/json"
"fmt"
"net/url"
@@ -235,9 +237,10 @@ func buildGraphQLOperationEndpoint(op graphQLOperationGroup, resourceName string
payloadParams := graphqlPayloadParams(op)
endpoint := spec.Endpoint{
- Method: op.Method,
- Path: op.Path,
- Description: graphQLBFFCommandDescription(resourceName, endpointName),
+ Method: op.Method,
+ Path: op.Path,
+ Description: graphQLBFFCommandDescription(resourceName, endpointName),
+ ObservedAuth: observedAuthHeaders(op.Entries),
Response: spec.ResponseDef{
Type: inferResponseType(responseBodies),
Item: safeGraphQLOperationName(op.OperationName),
@@ -578,11 +581,12 @@ func buildEndpoint(group EndpointGroup) spec.Endpoint {
}
endpoint := spec.Endpoint{
- Method: group.Method,
- Path: group.NormalizedPath,
- Description: fmt.Sprintf("%s %s", group.Method, group.NormalizedPath),
- Params: params,
- Body: body,
+ Method: group.Method,
+ Path: group.NormalizedPath,
+ Description: fmt.Sprintf("%s %s", group.Method, group.NormalizedPath),
+ Params: params,
+ Body: body,
+ ObservedAuth: observedAuthHeaders(group.Entries),
Response: spec.ResponseDef{
Type: responseType,
Item: deriveResponseItemName(group.NormalizedPath),
@@ -982,6 +986,45 @@ func detectAuth(capture *EnrichedCapture, entries []EnrichedEntry, name string)
return spec.AuthConfig{Type: "none"}
}
+// observedAuthHeaders returns the sorted set of lowercased request header
+// names observed across entries that match common auth surfaces
+// (Authorization, Cookie, X-CSRF-Token, X-API-Key, etc., plus contains-style
+// matches on token / secret / signature / api-key). Values are never
+// inspected or returned; only the header NAME travels. Returns nil when
+// nothing matched so consumers using `omitempty` can drop the field cleanly.
+func observedAuthHeaders(entries []EnrichedEntry) []string {
+ seen := map[string]bool{}
+ for _, entry := range entries {
+ for name := range entry.RequestHeaders {
+ lower := strings.ToLower(strings.TrimSpace(name))
+ if lower == "" {
+ continue
+ }
+ if isObservedAuthHeaderName(lower) {
+ seen[lower] = true
+ }
+ }
+ }
+ if len(seen) == 0 {
+ return nil
+ }
+ return sortedBoolKeys(seen)
+}
+
+func isObservedAuthHeaderName(lowerName string) bool {
+ switch lowerName {
+ case "authorization", "cookie", "set-cookie", "x-csrf-token", "x-xsrf-token", "x-api-key", "proxy-authorization":
+ return true
+ }
+ if strings.Contains(lowerName, "api-key") || strings.Contains(lowerName, "api_key") {
+ return true
+ }
+ if strings.Contains(lowerName, "token") || strings.Contains(lowerName, "secret") || strings.Contains(lowerName, "signature") {
+ return true
+ }
+ return false
+}
+
func detectCapturedAuth(capture *AuthCapture, envPrefix string) spec.AuthConfig {
if capture == nil {
return spec.AuthConfig{}
@@ -1165,3 +1208,262 @@ func deriveNameFromURL(raw string) string {
return strings.Join(labels, "-")
}
+
+// FilterEndpointsByMinSamples drops endpoints from apiSpec.Resources whose
+// underlying capture cluster carries fewer than minSamples paired entries.
+// Resources left with no endpoints are dropped too. Returns the number of
+// endpoints removed. A minSamples value <= 1 is a no-op (default behavior).
+//
+// Filtering happens against re-derived endpoint groups from the capture
+// rather than the spec itself, so the qualifying set matches exactly what
+// AnalyzeCapture would have seen. GraphQL operations that share one
+// underlying cluster (e.g., many distinct operationName values all hitting
+// /frontend/graphql) survive together or drop together with that cluster.
+// Per-operation thresholding is a future refinement.
+//
+// The TrafficAnalysis sidecar is intentionally untouched: dropped endpoints
+// remain visible there with their `low` confidence and `single-sample`
+// flag (or whichever applies) so an operator can audit what filtered out.
+func FilterEndpointsByMinSamples(apiSpec *spec.APISpec, capture *EnrichedCapture, minSamples int) int {
+ if apiSpec == nil || capture == nil || minSamples <= 1 {
+ return 0
+ }
+ apiEntries, _ := ClassifyEntries(capture.Entries)
+ groups := DeduplicateEndpoints(apiEntries)
+
+ qualifying := map[string]bool{}
+ for _, g := range groups {
+ if len(g.Entries) >= minSamples {
+ qualifying[endpointFilterKey(g.Method, g.NormalizedPath)] = true
+ }
+ }
+
+ dropped := 0
+ for resourceName, resource := range apiSpec.Resources {
+ for endpointName, endpoint := range resource.Endpoints {
+ key := endpointFilterKey(endpoint.Method, endpoint.Path)
+ if !qualifying[key] {
+ delete(resource.Endpoints, endpointName)
+ dropped++
+ }
+ }
+ if len(resource.Endpoints) == 0 {
+ delete(apiSpec.Resources, resourceName)
+ } else {
+ apiSpec.Resources[resourceName] = resource
+ }
+ }
+ return dropped
+}
+
+func endpointFilterKey(method string, path string) string {
+ return strings.ToUpper(strings.TrimSpace(method)) + " " + path
+}
+
+// SampleFile is the on-disk shape of one redacted endpoint sample written
+// to <spec-stem>-samples/<method>__<path-slug>__<hash>.json. Designed so a
+// reviewer can read a single file and see exactly what evidence backed the
+// emitted spec entry, with credentials stripped at write time.
+type SampleFile struct {
+ Endpoint string `json:"endpoint"`
+ Method string `json:"method"`
+ RawURL string `json:"raw_url"`
+ Status int `json:"status"`
+ RequestHeaders map[string]string `json:"request_headers,omitempty"`
+ RequestBody any `json:"request_body"`
+ ResponseHeaders map[string]string `json:"response_headers,omitempty"`
+ ResponseBody any `json:"response_body"`
+ ResponseBodyKnown bool `json:"response_body_known"`
+ ResponseBodyTruncated bool `json:"response_body_truncated,omitempty"`
+ Redactions []string `json:"redactions,omitempty"`
+}
+
+const sampleBodyMaxBytes = 16 * 1024
+
+// DefaultSamplesPath returns the canonical samples directory for a spec at
+// specPath: a sibling directory named <stem>-samples/. Mirrors
+// DefaultTrafficAnalysisPath's naming convention so artifacts cluster.
+func DefaultSamplesPath(specPath string) string {
+ dir := filepath.Dir(specPath)
+ base := filepath.Base(specPath)
+ ext := filepath.Ext(base)
+ stem := strings.TrimSuffix(base, ext)
+ if stem == "" || stem == "." {
+ stem = "spec"
+ }
+ return filepath.Join(dir, stem+"-samples")
+}
+
+// WriteSamples writes one redacted SampleFile per endpoint group to
+// outputDir. outputDir must already exist. Returns the number of files
+// written. Files are named method__path-slug__hash.json so the same
+// endpoint group produces the same filename across reruns.
+//
+// The capture's entries are re-classified and re-deduplicated locally
+// using the same ClassifyEntries + DeduplicateEndpoints path that
+// AnalyzeCapture uses, so the file set is a 1:1 reflection of what the
+// emitted spec would contain.
+func WriteSamples(capture *EnrichedCapture, outputDir string) (int, error) {
+ if capture == nil {
+ return 0, fmt.Errorf("capture is required")
+ }
+ if strings.TrimSpace(outputDir) == "" {
+ return 0, fmt.Errorf("output directory is required")
+ }
+
+ apiEntries, _ := ClassifyEntries(capture.Entries)
+ groups := DeduplicateEndpoints(apiEntries)
+
+ written := 0
+ for _, group := range groups {
+ sample := buildSampleFile(group)
+ filename := sampleFilename(group)
+ data, err := encodeSampleJSON(sample)
+ if err != nil {
+ return written, fmt.Errorf("marshaling sample %s: %w", filename, err)
+ }
+ path := filepath.Join(outputDir, filename)
+ if err := os.WriteFile(path, data, 0o600); err != nil {
+ return written, fmt.Errorf("writing sample %s: %w", filename, err)
+ }
+ written++
+ }
+ return written, nil
+}
+
+// encodeSampleJSON marshals a SampleFile with HTML-escaping disabled so the
+// `<redacted>` sentinel and other URL-safe characters travel verbatim
+// instead of becoming `<redacted>`. Reviewers read these files
+// directly; unicode-escaped content fights that.
+func encodeSampleJSON(sample SampleFile) ([]byte, error) {
+ var buf strings.Builder
+ encoder := json.NewEncoder(&buf)
+ encoder.SetEscapeHTML(false)
+ encoder.SetIndent("", " ")
+ if err := encoder.Encode(sample); err != nil {
+ return nil, err
+ }
+ return []byte(buf.String()), nil
+}
+
+// sampleFilename returns method__path-slug__hash.json. The hash is the
+// first 8 hex chars of sha256(METHOD + " " + normalizedPath) so the same
+// endpoint group always lands at the same filename across reruns.
+func sampleFilename(group EndpointGroup) string {
+ method := strings.ToLower(strings.TrimSpace(group.Method))
+ if method == "" {
+ method = "get"
+ }
+
+ slug := group.NormalizedPath
+ slug = strings.ReplaceAll(slug, "{", "")
+ slug = strings.ReplaceAll(slug, "}", "")
+ slug = strings.ReplaceAll(slug, "/", "_")
+ slug = strings.TrimLeft(slug, "_")
+ if slug == "" {
+ slug = "root"
+ }
+
+ h := sha256.Sum256([]byte(strings.ToUpper(method) + " " + group.NormalizedPath))
+ return fmt.Sprintf("%s__%s__%s.json", method, slug, hex.EncodeToString(h[:4]))
+}
+
+// buildSampleFile selects a representative entry from the group, redacts
+// it, and assembles the on-disk SampleFile struct. Redaction paths are
+// path-prefixed so a reviewer can locate exactly what was stripped.
+func buildSampleFile(group EndpointGroup) SampleFile {
+ entry := selectSampleEntry(group.Entries)
+
+ redactedReqHeaders, reqHeaderRedactions := RedactHeaders(entry.RequestHeaders)
+ redactedRespHeaders, respHeaderRedactions := RedactHeaders(entry.ResponseHeaders)
+
+ reqBody, _, reqBodyRedactions := preparedSampleBody(entry.RequestBody, "request_body")
+ respBody, respTruncated, respBodyRedactions := preparedSampleBody(entry.ResponseBody, "response_body")
+
+ redactions := make([]string, 0, len(reqHeaderRedactions)+len(respHeaderRedactions)+len(reqBodyRedactions)+len(respBodyRedactions))
+ for _, name := range reqHeaderRedactions {
+ redactions = append(redactions, "request_headers."+name)
+ }
+ for _, name := range respHeaderRedactions {
+ redactions = append(redactions, "response_headers."+name)
+ }
+ redactions = append(redactions, reqBodyRedactions...)
+ redactions = append(redactions, respBodyRedactions...)
+ sort.Strings(redactions)
+ if len(redactions) == 0 {
+ redactions = nil
+ }
+
+ return SampleFile{
+ Endpoint: fmt.Sprintf("%s %s", group.Method, group.NormalizedPath),
+ Method: group.Method,
+ RawURL: entry.URL,
+ Status: entry.ResponseStatus,
+ RequestHeaders: redactedReqHeaders,
+ RequestBody: reqBody,
+ ResponseHeaders: redactedRespHeaders,
+ ResponseBody: respBody,
+ ResponseBodyKnown: strings.TrimSpace(entry.ResponseBody) != "",
+ ResponseBodyTruncated: respTruncated,
+ Redactions: redactions,
+ }
+}
+
+// selectSampleEntry picks the most-recent successful (2xx) entry; falls
+// back to most-recent non-error (<500); falls back to the most-recent
+// entry. Capture order is preserved by ClassifyEntries, so iterating and
+// overwriting on each match yields "most-recent of category".
+func selectSampleEntry(entries []EnrichedEntry) EnrichedEntry {
+ if len(entries) == 0 {
+ return EnrichedEntry{}
+ }
+
+ var success, nonError EnrichedEntry
+ var hasSuccess, hasNonError bool
+ for _, entry := range entries {
+ if entry.ResponseStatus >= 200 && entry.ResponseStatus < 300 {
+ success = entry
+ hasSuccess = true
+ }
+ if entry.ResponseStatus > 0 && entry.ResponseStatus < 500 {
+ nonError = entry
+ hasNonError = true
+ }
+ }
+ if hasSuccess {
+ return success
+ }
+ if hasNonError {
+ return nonError
+ }
+ return entries[len(entries)-1]
+}
+
+// preparedSampleBody returns the body value (JSON-typed when parseable,
+// otherwise a string), a truncation flag, and the path-prefixed redaction
+// labels. pathPrefix is the dotted root applied to redaction labels.
+func preparedSampleBody(body string, pathPrefix string) (any, bool, []string) {
+ trimmed := strings.TrimSpace(body)
+ if trimmed == "" {
+ return nil, false, nil
+ }
+
+ truncated := false
+ if len(trimmed) > sampleBodyMaxBytes {
+ trimmed = trimmed[:sampleBodyMaxBytes]
+ truncated = true
+ }
+
+ redactedBody, redactedPaths := RedactJSONBody(trimmed)
+
+ prefixed := make([]string, 0, len(redactedPaths))
+ for _, p := range redactedPaths {
+ prefixed = append(prefixed, pathPrefix+"."+p)
+ }
+
+ var parsed any
+ if err := json.Unmarshal([]byte(redactedBody), &parsed); err == nil {
+ return parsed, truncated, prefixed
+ }
+ return redactedBody, truncated, prefixed
+}
diff --git a/internal/browsersniff/specgen_test.go b/internal/browsersniff/specgen_test.go
index 20950e13..a15da424 100644
--- a/internal/browsersniff/specgen_test.go
+++ b/internal/browsersniff/specgen_test.go
@@ -1,8 +1,10 @@
package browsersniff
import (
+ "encoding/json"
"os"
"path/filepath"
+ "strings"
"testing"
"github.com/mvanhorn/cli-printing-press/v4/internal/spec"
@@ -370,6 +372,436 @@ func TestDetectAuth_FallsBackToHeaderInference(t *testing.T) {
assert.Equal(t, "Authorization", auth.Header)
}
+func TestObservedAuthHeaders(t *testing.T) {
+ t.Parallel()
+
+ cases := []struct {
+ name string
+ entries []EnrichedEntry
+ want []string
+ }{
+ {
+ name: "single authorization header",
+ entries: []EnrichedEntry{
+ {RequestHeaders: map[string]string{"Authorization": "Bearer eyJabc"}},
+ },
+ want: []string{"authorization"},
+ },
+ {
+ name: "multiple distinct auth headers in one sample",
+ entries: []EnrichedEntry{
+ {RequestHeaders: map[string]string{
+ "Authorization": "Bearer t",
+ "X-CSRF-Token": "abc",
+ "Cookie": "session=x",
+ }},
+ },
+ want: []string{"authorization", "cookie", "x-csrf-token"},
+ },
+ {
+ name: "mixed across samples - union, presence not requirement",
+ entries: []EnrichedEntry{
+ {RequestHeaders: map[string]string{"Authorization": "Bearer t"}},
+ {RequestHeaders: map[string]string{"Accept": "application/json"}},
+ },
+ want: []string{"authorization"},
+ },
+ {
+ name: "no auth headers - nil result for omitempty",
+ entries: []EnrichedEntry{
+ {RequestHeaders: map[string]string{"Accept": "application/json", "User-Agent": "x"}},
+ },
+ want: nil,
+ },
+ {
+ name: "case-insensitive merge",
+ entries: []EnrichedEntry{
+ {RequestHeaders: map[string]string{"Authorization": "x"}},
+ {RequestHeaders: map[string]string{"AUTHORIZATION": "y"}},
+ {RequestHeaders: map[string]string{"authorization": "z"}},
+ },
+ want: []string{"authorization"},
+ },
+ {
+ name: "values never leak - only names emitted",
+ entries: []EnrichedEntry{
+ {RequestHeaders: map[string]string{"Authorization": "Bearer eyJ.SECRET.token"}},
+ },
+ want: []string{"authorization"},
+ },
+ {
+ name: "api-key and api_key variants",
+ entries: []EnrichedEntry{
+ {RequestHeaders: map[string]string{"X-Api-Key": "k1"}},
+ {RequestHeaders: map[string]string{"x_api_key": "k2"}},
+ },
+ want: []string{"x-api-key", "x_api_key"},
+ },
+ {
+ name: "contains-token contains-secret contains-signature patterns",
+ entries: []EnrichedEntry{
+ {RequestHeaders: map[string]string{
+ "X-Auth-Token": "t",
+ "X-Hub-Secret": "s",
+ "X-Sig-Signature": "g",
+ }},
+ },
+ want: []string{"x-auth-token", "x-hub-secret", "x-sig-signature"},
+ },
+ {
+ name: "empty entries",
+ entries: []EnrichedEntry{},
+ want: nil,
+ },
+ {
+ name: "non-auth headers ignored",
+ entries: []EnrichedEntry{
+ {RequestHeaders: map[string]string{
+ "Accept": "application/json",
+ "User-Agent": "ua/1.0",
+ "Referer": "https://example.com",
+ }},
+ },
+ want: nil,
+ },
+ }
+
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ t.Parallel()
+ got := observedAuthHeaders(tc.entries)
+ assert.Equal(t, tc.want, got)
+ })
+ }
+}
+
+func TestAnalyzeCapture_PopulatesObservedAuthOnSpecEndpoint(t *testing.T) {
+ t.Parallel()
+
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/items",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{
+ "Authorization": "Bearer eyJtoken",
+ "Accept": "application/json",
+ },
+ },
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/items",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":2}`,
+ RequestHeaders: map[string]string{
+ "Accept": "application/json",
+ },
+ },
+ },
+ }
+
+ apiSpec, err := AnalyzeCapture(capture)
+ require.NoError(t, err)
+
+ var found bool
+ for _, resource := range apiSpec.Resources {
+ for _, endpoint := range resource.Endpoints {
+ if endpoint.Method == "GET" && endpoint.Path == "/v1/items" {
+ assert.Equal(t, []string{"authorization"}, endpoint.ObservedAuth)
+ found = true
+ }
+ }
+ }
+ assert.True(t, found, "expected GET /v1/items endpoint")
+}
+
+func TestAnalyzeCapture_OmitsObservedAuthWhenAbsent(t *testing.T) {
+ t.Parallel()
+
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/public",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"ok":true}`,
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ },
+ }
+
+ apiSpec, err := AnalyzeCapture(capture)
+ require.NoError(t, err)
+
+ for _, resource := range apiSpec.Resources {
+ for _, endpoint := range resource.Endpoints {
+ assert.Nil(t, endpoint.ObservedAuth, "endpoint %s %s should have nil ObservedAuth", endpoint.Method, endpoint.Path)
+ }
+ }
+}
+
+func TestAnalyzeCapture_PopulatesObservedAuthOnGraphQLEndpoint(t *testing.T) {
+ t.Parallel()
+
+ postsEntry := graphqlBFFEntry("PostsToday", `{"date":"2026-04-22"}`, "aaa111")
+ postsEntry.RequestHeaders["Authorization"] = "Bearer eyJtoken"
+ launchesEntry := graphqlBFFEntry("ProductPageLaunches", `{"slug":"sample-product"}`, "bbb222")
+
+ capture := &EnrichedCapture{
+ TargetURL: "https://www.example.com",
+ Entries: []EnrichedEntry{postsEntry, launchesEntry, postsEntry},
+ }
+
+ apiSpec, err := AnalyzeCapture(capture)
+ require.NoError(t, err)
+
+ posts, ok := apiSpec.Resources["posts"]
+ require.True(t, ok, "expected posts resource from PostsToday operation")
+ postsToday, ok := posts.Endpoints["today"]
+ require.True(t, ok, "expected today endpoint from PostsToday operation")
+ assert.Equal(t, []string{"authorization"}, postsToday.ObservedAuth)
+
+ products, ok := apiSpec.Resources["products"]
+ require.True(t, ok, "expected products resource from ProductPageLaunches operation")
+ launches, ok := products.Endpoints["launches"]
+ require.True(t, ok, "expected launches endpoint")
+ assert.Nil(t, launches.ObservedAuth, "launches operation had no auth headers")
+}
+
+func TestSampleFilename_DeterministicAcrossReruns(t *testing.T) {
+ t.Parallel()
+
+ group := EndpointGroup{Method: "GET", NormalizedPath: "/v1/items/{id}"}
+ first := sampleFilename(group)
+ second := sampleFilename(group)
+ assert.Equal(t, first, second, "filename hash must be deterministic")
+ assert.Regexp(t, `^get__v1_items_id__[0-9a-f]{8}\.json$`, first)
+}
+
+func TestSampleFilename_MethodVariesHash(t *testing.T) {
+ t.Parallel()
+
+ getName := sampleFilename(EndpointGroup{Method: "GET", NormalizedPath: "/v1/items"})
+ postName := sampleFilename(EndpointGroup{Method: "POST", NormalizedPath: "/v1/items"})
+ assert.NotEqual(t, getName, postName, "method change must change the filename")
+}
+
+func TestSampleFilename_StripsBraces(t *testing.T) {
+ t.Parallel()
+
+ got := sampleFilename(EndpointGroup{Method: "POST", NormalizedPath: "/v1/orders/{orderId}/items/{itemId}"})
+ assert.NotContains(t, got, "{")
+ assert.NotContains(t, got, "}")
+ assert.Contains(t, got, "orders_orderId_items_itemId")
+}
+
+func TestSelectSampleEntry_PrefersMostRecentSuccess(t *testing.T) {
+ t.Parallel()
+
+ entries := []EnrichedEntry{
+ {URL: "/a", ResponseStatus: 200, ResponseBody: `{"first":true}`},
+ {URL: "/a", ResponseStatus: 200, ResponseBody: `{"second":true}`},
+ {URL: "/a", ResponseStatus: 500, ResponseBody: `{"oops":true}`},
+ {URL: "/a", ResponseStatus: 200, ResponseBody: `{"third":true}`},
+ {URL: "/a", ResponseStatus: 404, ResponseBody: `{"notfound":true}`},
+ }
+ got := selectSampleEntry(entries)
+ assert.Equal(t, `{"third":true}`, got.ResponseBody, "most-recent 2xx should win over later 4xx/5xx")
+}
+
+func TestSelectSampleEntry_FallsBackToNonError(t *testing.T) {
+ t.Parallel()
+
+ entries := []EnrichedEntry{
+ {URL: "/a", ResponseStatus: 500, ResponseBody: `{"oops":true}`},
+ {URL: "/a", ResponseStatus: 404, ResponseBody: `{"notfound":true}`},
+ {URL: "/a", ResponseStatus: 502, ResponseBody: `{"badgw":true}`},
+ }
+ got := selectSampleEntry(entries)
+ assert.Equal(t, `{"notfound":true}`, got.ResponseBody, "404 should be the most-recent non-5xx")
+}
+
+func TestSelectSampleEntry_FallsBackToMostRecentWhenAllErrors(t *testing.T) {
+ t.Parallel()
+
+ entries := []EnrichedEntry{
+ {URL: "/a", ResponseStatus: 500, ResponseBody: `{"first":true}`},
+ {URL: "/a", ResponseStatus: 502, ResponseBody: `{"second":true}`},
+ }
+ got := selectSampleEntry(entries)
+ assert.Equal(t, `{"second":true}`, got.ResponseBody)
+}
+
+func TestWriteSamples_WritesOneFilePerEndpointGroup(t *testing.T) {
+ t.Parallel()
+
+ dir := t.TempDir()
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/items/42",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":42,"api_key":"sk_secret"}`,
+ RequestHeaders: map[string]string{
+ "Authorization": "Bearer eyJtoken",
+ "Accept": "application/json",
+ },
+ },
+ {
+ Method: "POST",
+ URL: "https://api.example.com/v1/items",
+ ResponseStatus: 201,
+ ResponseContentType: "application/json",
+ RequestBody: `{"name":"widget","password":"p"}`,
+ ResponseBody: `{"id":43}`,
+ RequestHeaders: map[string]string{"Content-Type": "application/json"},
+ },
+ },
+ }
+
+ written, err := WriteSamples(capture, dir)
+ require.NoError(t, err)
+ assert.Equal(t, 2, written)
+
+ files, err := os.ReadDir(dir)
+ require.NoError(t, err)
+ require.Len(t, files, 2)
+
+ var foundGET, foundPOST bool
+ for _, f := range files {
+ assert.Regexp(t, `^[a-z]+__[a-zA-Z0-9_]+__[0-9a-f]{8}\.json$`, f.Name())
+ data, err := os.ReadFile(filepath.Join(dir, f.Name()))
+ require.NoError(t, err)
+
+ var sample SampleFile
+ require.NoError(t, json.Unmarshal(data, &sample))
+
+ switch sample.Method {
+ case "GET":
+ foundGET = true
+ assert.Equal(t, "GET /v1/items/{id}", sample.Endpoint)
+ assert.Equal(t, 200, sample.Status)
+ assert.True(t, sample.ResponseBodyKnown)
+ assert.Equal(t, RedactedSentinel, sample.RequestHeaders["Authorization"])
+ assert.Equal(t, "application/json", sample.RequestHeaders["Accept"])
+ body, ok := sample.ResponseBody.(map[string]any)
+ require.True(t, ok)
+ assert.Equal(t, RedactedSentinel, body["api_key"])
+ assert.Contains(t, sample.Redactions, "request_headers.authorization")
+ assert.Contains(t, sample.Redactions, "response_body.api_key")
+ case "POST":
+ foundPOST = true
+ assert.Equal(t, 201, sample.Status)
+ reqBody, ok := sample.RequestBody.(map[string]any)
+ require.True(t, ok)
+ assert.Equal(t, RedactedSentinel, reqBody["password"])
+ assert.Contains(t, sample.Redactions, "request_body.password")
+ }
+ }
+ assert.True(t, foundGET, "GET sample should be present")
+ assert.True(t, foundPOST, "POST sample should be present")
+}
+
+func TestWriteSamples_OmitsResponseBodyKnownWhenAbsent(t *testing.T) {
+ t.Parallel()
+
+ dir := t.TempDir()
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/no-body",
+ ResponseStatus: 204,
+ ResponseContentType: "application/json",
+ ResponseBody: "",
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ },
+ }
+
+ written, err := WriteSamples(capture, dir)
+ require.NoError(t, err)
+ require.Equal(t, 1, written)
+
+ files, err := os.ReadDir(dir)
+ require.NoError(t, err)
+ require.Len(t, files, 1)
+
+ data, err := os.ReadFile(filepath.Join(dir, files[0].Name()))
+ require.NoError(t, err)
+
+ var sample SampleFile
+ require.NoError(t, json.Unmarshal(data, &sample))
+ assert.False(t, sample.ResponseBodyKnown)
+ assert.Nil(t, sample.ResponseBody)
+}
+
+func TestWriteSamples_TruncatesOversizedBodies(t *testing.T) {
+ t.Parallel()
+
+ dir := t.TempDir()
+ // Build an oversized JSON-shaped body so the classifier keeps it as an
+ // API endpoint. HTML-shaped surfaces are discovered separately by
+ // AnalyzeCapture and are not yet covered by WriteSamples; that is
+ // follow-up work (see plan deferred section).
+ bigValue := strings.Repeat("x", sampleBodyMaxBytes+1000)
+ bigBody := `{"blob":"` + bigValue + `"}`
+ capture := &EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/blob",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: bigBody,
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ },
+ }
+
+ written, err := WriteSamples(capture, dir)
+ require.NoError(t, err)
+ require.Equal(t, 1, written)
+
+ files, err := os.ReadDir(dir)
+ require.NoError(t, err)
+ require.Len(t, files, 1)
+
+ data, err := os.ReadFile(filepath.Join(dir, files[0].Name()))
+ require.NoError(t, err)
+
+ var sample SampleFile
+ require.NoError(t, json.Unmarshal(data, &sample))
+ assert.True(t, sample.ResponseBodyTruncated, "oversized body should set truncated flag")
+ // After truncation the body is no longer parseable JSON, so it lands as
+ // a raw string in the sample.
+ body, ok := sample.ResponseBody.(string)
+ require.True(t, ok, "truncated body falls back to raw string")
+ assert.LessOrEqual(t, len(body), sampleBodyMaxBytes)
+}
+
+func TestDefaultSamplesPath(t *testing.T) {
+ t.Parallel()
+
+ got := DefaultSamplesPath("/tmp/cache/example-spec.yaml")
+ assert.Equal(t, "/tmp/cache/example-spec-samples", got)
+
+ got2 := DefaultSamplesPath("api.yml")
+ assert.Equal(t, "api-samples", got2)
+}
+
func TestWriteEnrichedCaptureUsesPrivatePermissions(t *testing.T) {
t.Parallel()
diff --git a/internal/cli/browser_sniff.go b/internal/cli/browser_sniff.go
index 1b762b11..6f9cacbe 100644
--- a/internal/cli/browser_sniff.go
+++ b/internal/cli/browser_sniff.go
@@ -17,8 +17,11 @@ func newBrowserSniffCmd() *cobra.Command {
var harPath string
var outputPath string
var analysisOutputPath string
+ var samplesOutputPath string
var name string
var blocklist string
+ var include string
+ var minSamples int
var authFrom string
cmd := &cobra.Command{
@@ -26,6 +29,7 @@ func newBrowserSniffCmd() *cobra.Command {
Short: "Analyze captured web traffic to discover API endpoints and generate a spec",
RunE: func(cmd *cobra.Command, args []string) error {
browsersniff.SetAdditionalBlocklist(splitCSV(blocklist))
+ browsersniff.SetAdditionalIncludeList(splitCSV(include))
capture, err := browsersniff.LoadCapture(harPath)
if err != nil {
@@ -59,13 +63,20 @@ func newBrowserSniffCmd() *cobra.Command {
if analysisOutputPath == "" {
analysisOutputPath = browsersniff.DefaultTrafficAnalysisPath(outputPath)
}
+ if samplesOutputPath == "" {
+ samplesOutputPath = browsersniff.DefaultSamplesPath(outputPath)
+ }
trafficAnalysis, err := browsersniff.AnalyzeTraffic(capture)
if err != nil {
return fmt.Errorf("analyzing traffic: %w", err)
}
browsersniff.ApplyReachabilityDefaults(apiSpec, trafficAnalysis)
- if err := writeBrowserSniffOutputs(apiSpec, trafficAnalysis, outputPath, analysisOutputPath); err != nil {
+
+ droppedEndpoints := browsersniff.FilterEndpointsByMinSamples(apiSpec, capture, minSamples)
+
+ samplesWritten, err := writeBrowserSniffOutputs(apiSpec, trafficAnalysis, capture, outputPath, analysisOutputPath, samplesOutputPath)
+ if err != nil {
return err
}
@@ -76,6 +87,12 @@ func newBrowserSniffCmd() *cobra.Command {
fmt.Printf("Spec written to %s (%d endpoints across %d resources)\n", outputPath, endpoints, len(apiSpec.Resources))
fmt.Printf("Traffic analysis written to %s\n", analysisOutputPath)
+ if samplesOutputPath != "" && samplesWritten > 0 {
+ fmt.Printf("Samples written to %s (%d endpoint%s)\n", samplesOutputPath, samplesWritten, plural(samplesWritten))
+ }
+ if droppedEndpoints > 0 {
+ fmt.Printf("Dropped %d endpoint%s below --min-samples=%d (still visible in %s)\n", droppedEndpoints, plural(droppedEndpoints), minSamples, analysisOutputPath)
+ }
fmt.Printf("Run 'printing-press generate --spec %s' to build the CLI\n", outputPath)
return nil
},
@@ -84,57 +101,145 @@ func newBrowserSniffCmd() *cobra.Command {
cmd.Flags().StringVar(&harPath, "har", "", "Path to HAR or enriched capture file")
cmd.Flags().StringVar(&outputPath, "output", "", "Output path for generated spec YAML")
cmd.Flags().StringVar(&analysisOutputPath, "analysis-output", "", "Output path for traffic analysis JSON (defaults beside the spec)")
+ cmd.Flags().StringVar(&samplesOutputPath, "samples-output", "", "Output directory for per-endpoint redacted samples (defaults to <spec-stem>-samples beside the spec; pass empty string to disable via --samples-output=\"\")")
cmd.Flags().StringVar(&name, "name", "", "Override the auto-detected API name")
- cmd.Flags().StringVar(&blocklist, "blocklist", "", "Comma-separated additional domains to filter")
+ cmd.Flags().StringVar(&blocklist, "blocklist", "", "Comma-separated additional hostnames to filter (extends the default analytics/telemetry blocklist)")
+ cmd.Flags().StringVar(&include, "include", "", "Comma-separated host or path substrings to rescue from default filtering; matches win over --blocklist and the static-asset suffix demotion")
+ cmd.Flags().IntVar(&minSamples, "min-samples", 1, "Drop endpoints with fewer than N paired samples from the emitted spec; the dropped endpoints remain in the traffic-analysis sidecar for audit. Default 1 leaves behavior unchanged; 2+ is recommended for production capture")
cmd.Flags().StringVar(&authFrom, "auth-from", "", "Path to an enriched capture file to import auth from")
_ = cmd.MarkFlagRequired("har")
return cmd
}
-func writeBrowserSniffOutputs(apiSpec *spec.APISpec, trafficAnalysis *browsersniff.TrafficAnalysis, outputPath string, analysisOutputPath string) error {
+func plural(n int) string {
+ if n == 1 {
+ return ""
+ }
+ return "s"
+}
+
+func writeBrowserSniffOutputs(apiSpec *spec.APISpec, trafficAnalysis *browsersniff.TrafficAnalysis, capture *browsersniff.EnrichedCapture, outputPath string, analysisOutputPath string, samplesOutputPath string) (int, error) {
specTmp := siblingTempPath(outputPath, "spec")
analysisTmp := siblingTempPath(analysisOutputPath, "traffic-analysis")
defer func() { _ = os.Remove(specTmp) }()
defer func() { _ = os.Remove(analysisTmp) }()
if err := browsersniff.WriteSpec(apiSpec, specTmp); err != nil {
- return fmt.Errorf("writing spec: %w", err)
+ return 0, fmt.Errorf("writing spec: %w", err)
}
if err := browsersniff.WriteTrafficAnalysis(trafficAnalysis, analysisTmp); err != nil {
- return fmt.Errorf("writing traffic analysis: %w", err)
+ return 0, fmt.Errorf("writing traffic analysis: %w", err)
}
+ samplesWritten := 0
+ samplesTmp := ""
+ if samplesOutputPath != "" && capture != nil {
+ samplesTmp = samplesOutputPath + ".tmp"
+ _ = os.RemoveAll(samplesTmp)
+ if err := os.MkdirAll(samplesTmp, 0o755); err != nil {
+ return 0, fmt.Errorf("preparing samples temp dir: %w", err)
+ }
+ written, err := browsersniff.WriteSamples(capture, samplesTmp)
+ if err != nil {
+ _ = os.RemoveAll(samplesTmp)
+ return 0, fmt.Errorf("writing samples: %w", err)
+ }
+ samplesWritten = written
+ }
+ defer func() {
+ if samplesTmp != "" {
+ _ = os.RemoveAll(samplesTmp)
+ }
+ }()
+
analysisBackup, analysisHadBackup, err := backupFileForReplace(analysisOutputPath)
if err != nil {
- return fmt.Errorf("preparing traffic analysis publish: %w", err)
+ return 0, fmt.Errorf("preparing traffic analysis publish: %w", err)
}
specBackup, specHadBackup, err := backupFileForReplace(outputPath)
if err != nil {
restoreFileBackup(analysisOutputPath, analysisBackup, analysisHadBackup)
- return fmt.Errorf("preparing spec publish: %w", err)
+ return 0, fmt.Errorf("preparing spec publish: %w", err)
+ }
+ samplesBackup := ""
+ samplesHadBackup := false
+ if samplesTmp != "" {
+ samplesBackup, samplesHadBackup, err = backupDirForReplace(samplesOutputPath)
+ if err != nil {
+ restoreFileBackup(analysisOutputPath, analysisBackup, analysisHadBackup)
+ restoreFileBackup(outputPath, specBackup, specHadBackup)
+ return 0, fmt.Errorf("preparing samples publish: %w", err)
+ }
}
cleanupBackups := true
defer func() {
if cleanupBackups {
_ = os.Remove(analysisBackup)
_ = os.Remove(specBackup)
+ if samplesBackup != "" {
+ _ = os.RemoveAll(samplesBackup)
+ }
}
}()
if err := os.Rename(analysisTmp, analysisOutputPath); err != nil {
restoreFileBackup(analysisOutputPath, analysisBackup, analysisHadBackup)
restoreFileBackup(outputPath, specBackup, specHadBackup)
- return fmt.Errorf("publishing traffic analysis: %w", err)
+ restoreDirBackup(samplesOutputPath, samplesBackup, samplesHadBackup)
+ return 0, fmt.Errorf("publishing traffic analysis: %w", err)
}
if err := os.Rename(specTmp, outputPath); err != nil {
_ = os.Remove(analysisOutputPath)
restoreFileBackup(analysisOutputPath, analysisBackup, analysisHadBackup)
restoreFileBackup(outputPath, specBackup, specHadBackup)
- return fmt.Errorf("publishing spec: %w", err)
+ restoreDirBackup(samplesOutputPath, samplesBackup, samplesHadBackup)
+ return 0, fmt.Errorf("publishing spec: %w", err)
+ }
+ if samplesTmp != "" {
+ if err := os.Rename(samplesTmp, samplesOutputPath); err != nil {
+ _ = os.Remove(analysisOutputPath)
+ _ = os.Remove(outputPath)
+ restoreFileBackup(analysisOutputPath, analysisBackup, analysisHadBackup)
+ restoreFileBackup(outputPath, specBackup, specHadBackup)
+ restoreDirBackup(samplesOutputPath, samplesBackup, samplesHadBackup)
+ return 0, fmt.Errorf("publishing samples: %w", err)
+ }
}
- return nil
+ return samplesWritten, nil
+}
+
+func backupDirForReplace(path string) (string, bool, error) {
+ info, err := os.Stat(path)
+ if err != nil && !errors.Is(err, os.ErrNotExist) {
+ return "", false, err
+ }
+ if err == nil && !info.IsDir() {
+ return "", false, fmt.Errorf("%s is not a directory", path)
+ }
+
+ backup := path + ".backup"
+ _ = os.RemoveAll(backup)
+ if err := os.Rename(path, backup); err != nil {
+ if errors.Is(err, os.ErrNotExist) {
+ return backup, false, nil
+ }
+ return backup, false, err
+ }
+ return backup, true, nil
+}
+
+func restoreDirBackup(path string, backup string, hadBackup bool) {
+ if backup == "" {
+ return
+ }
+ _ = os.RemoveAll(path)
+ if hadBackup {
+ _ = os.Rename(backup, path)
+ return
+ }
+ _ = os.RemoveAll(backup)
}
func siblingTempPath(path string, suffix string) string {
diff --git a/internal/cli/browser_sniff_test.go b/internal/cli/browser_sniff_test.go
index d6ac55a3..8da735d9 100644
--- a/internal/cli/browser_sniff_test.go
+++ b/internal/cli/browser_sniff_test.go
@@ -110,7 +110,7 @@ func TestWriteBrowserSniffOutputsRestoresExistingFilesWhenSpecPublishFails(t *te
Types: map[string]spec.TypeDef{},
}
- err := writeBrowserSniffOutputs(apiSpec, &browsersniff.TrafficAnalysis{Version: "1"}, blockingDir, analysisPath)
+ _, err := writeBrowserSniffOutputs(apiSpec, &browsersniff.TrafficAnalysis{Version: "1"}, nil, blockingDir, analysisPath, "")
require.Error(t, err)
assert.Contains(t, err.Error(), "preparing spec publish:")
@@ -121,6 +121,102 @@ func TestWriteBrowserSniffOutputsRestoresExistingFilesWhenSpecPublishFails(t *te
assert.FileExists(t, filepath.Join(blockingDir, "marker"))
}
+func TestWriteBrowserSniffOutputsWritesSamplesDirectory(t *testing.T) {
+ t.Parallel()
+
+ dir := t.TempDir()
+ specPath := filepath.Join(dir, "out-spec.yaml")
+ analysisPath := browsersniff.DefaultTrafficAnalysisPath(specPath)
+ samplesPath := browsersniff.DefaultSamplesPath(specPath)
+
+ capture := &browsersniff.EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []browsersniff.EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/items",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{
+ "Authorization": "Bearer eyJ.t.x",
+ "Accept": "application/json",
+ },
+ },
+ },
+ }
+
+ apiSpec, err := browsersniff.AnalyzeCapture(capture)
+ require.NoError(t, err)
+ trafficAnalysis, err := browsersniff.AnalyzeTraffic(capture)
+ require.NoError(t, err)
+
+ written, err := writeBrowserSniffOutputs(apiSpec, trafficAnalysis, capture, specPath, analysisPath, samplesPath)
+ require.NoError(t, err)
+ assert.Positive(t, written, "at least one sample file should be written")
+
+ assert.FileExists(t, specPath)
+ assert.FileExists(t, analysisPath)
+ assert.DirExists(t, samplesPath)
+
+ entries, err := os.ReadDir(samplesPath)
+ require.NoError(t, err)
+ assert.NotEmpty(t, entries, "samples directory should have at least one file")
+ for _, entry := range entries {
+ data, err := os.ReadFile(filepath.Join(samplesPath, entry.Name()))
+ require.NoError(t, err)
+ assert.Contains(t, string(data), browsersniff.RedactedSentinel, "Authorization should be redacted")
+ assert.NotContains(t, string(data), "eyJ.t.x", "raw token must not leak")
+ }
+}
+
+func TestWriteBrowserSniffOutputsRestoresSamplesDirOnSpecFailure(t *testing.T) {
+ t.Parallel()
+
+ dir := t.TempDir()
+ specPath := filepath.Join(dir, "out-spec.yaml")
+ analysisPath := browsersniff.DefaultTrafficAnalysisPath(specPath)
+ samplesPath := browsersniff.DefaultSamplesPath(specPath)
+
+ // Pre-existing samples directory with a marker file. A subsequent failure
+ // during spec publish must restore the pre-existing samples directory
+ // intact (no half-overwritten state).
+ require.NoError(t, os.MkdirAll(samplesPath, 0o755))
+ markerPath := filepath.Join(samplesPath, "pre-existing.txt")
+ require.NoError(t, os.WriteFile(markerPath, []byte("keep me"), 0o600))
+
+ // Block spec publish by creating a non-empty directory at outputPath.
+ require.NoError(t, os.MkdirAll(specPath, 0o755))
+ require.NoError(t, os.WriteFile(filepath.Join(specPath, "blocker"), []byte("x"), 0o600))
+
+ capture := &browsersniff.EnrichedCapture{
+ TargetURL: "https://api.example.com",
+ Entries: []browsersniff.EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/v1/items",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"id":1}`,
+ RequestHeaders: map[string]string{"Accept": "application/json"},
+ },
+ },
+ }
+ apiSpec, err := browsersniff.AnalyzeCapture(capture)
+ require.NoError(t, err)
+ trafficAnalysis, err := browsersniff.AnalyzeTraffic(capture)
+ require.NoError(t, err)
+
+ _, err = writeBrowserSniffOutputs(apiSpec, trafficAnalysis, capture, specPath, analysisPath, samplesPath)
+ require.Error(t, err, "should fail because outputPath is a non-empty directory")
+
+ // Pre-existing samples directory must be restored intact.
+ assert.DirExists(t, samplesPath)
+ preserved, readErr := os.ReadFile(markerPath)
+ require.NoError(t, readErr)
+ assert.Equal(t, "keep me", string(preserved))
+}
+
// newRootCmdForTest mirrors Execute()'s command tree construction for test-level
// command dispatch assertions.
func newRootCmdForTest() *cobra.Command {
diff --git a/internal/spec/spec.go b/internal/spec/spec.go
index a7dd4729..074225cd 100644
--- a/internal/spec/spec.go
+++ b/internal/spec/spec.go
@@ -1128,7 +1128,15 @@ type Endpoint struct {
Meta map[string]string `yaml:"meta,omitempty" json:"meta,omitempty"` // per-endpoint metadata (e.g., source_tier, source_count from crowd-sniff)
HeaderOverrides []RequiredHeader `yaml:"header_overrides,omitempty" json:"header_overrides,omitempty"` // per-endpoint header overrides (e.g., different api-version)
NoAuth bool `yaml:"no_auth,omitempty" json:"no_auth,omitempty"` // true when the endpoint does not require authentication
- Tier string `yaml:"tier,omitempty" json:"tier,omitempty"`
+ // ObservedAuth lists the lowercased request header names observed on this
+ // endpoint during browser-sniff capture that match common auth surfaces
+ // (Authorization, Cookie, X-API-Key, etc.). Observation-only — header
+ // values are never recorded. Populated only by sniffed specs; vendor specs
+ // and crowd-sniff leave it empty. Consumers (Phase 2 tier routing, MCP
+ // surface routing) may use it as per-endpoint auth evidence rather than
+ // inferring from spec-level signals.
+ ObservedAuth []string `yaml:"observed_auth,omitempty" json:"observed_auth,omitempty"`
+ Tier string `yaml:"tier,omitempty" json:"tier,omitempty"`
// IDField is the resolved primary-key field name for items returned by this
// endpoint, populated either by a path-item-level `x-resource-id` extension
// or, for OpenAPI specs, by walking the response schema (id → name → first
diff --git a/skills/printing-press/SKILL.md b/skills/printing-press/SKILL.md
index 4db82b1e..e860807d 100644
--- a/skills/printing-press/SKILL.md
+++ b/skills/printing-press/SKILL.md
@@ -1760,6 +1760,13 @@ Detection signals:
documented as paid, partner, premium, or quota-gated.
- Browser/crowd sniffing found public endpoints and SDK/MCP research found a
separate credential for expanded coverage.
+- Sniffed specs carry per-endpoint `observed_auth` (a list of lowercased request
+ header names observed during capture, e.g. `[authorization]` or `[x-api-key]`).
+ An empty or missing `observed_auth` on an endpoint is evidence that the request
+ went anonymously; a populated list is evidence the endpoint required auth. The
+ same per-endpoint signal is mirrored on `TrafficAnalysis.endpoint_clusters[].observed_auth`
+ in the traffic-analysis sidecar. Treat both as observation-only — not a security
+ scheme declaration — and corroborate with documentation before declaring a tier.
Action:
- Internal YAML: add `tier_routing` plus `tier` on the affected resource or
diff --git a/testdata/golden/expected/browser-sniff-sample/stdout.txt b/testdata/golden/expected/browser-sniff-sample/stdout.txt
index 54df07e0..f51220c3 100644
--- a/testdata/golden/expected/browser-sniff-sample/stdout.txt
+++ b/testdata/golden/expected/browser-sniff-sample/stdout.txt
@@ -1,3 +1,4 @@
Spec written to <ARTIFACT_DIR>/browser-sniff-sample/spec.yaml (3 endpoints across 3 resources)
Traffic analysis written to <ARTIFACT_DIR>/browser-sniff-sample/traffic-analysis.json
+Samples written to <ARTIFACT_DIR>/browser-sniff-sample/spec-samples (3 endpoints)
Run 'printing-press generate --spec <ARTIFACT_DIR>/browser-sniff-sample/spec.yaml' to build the CLI
diff --git a/testdata/golden/expected/browser-sniff-sample/traffic-analysis.json b/testdata/golden/expected/browser-sniff-sample/traffic-analysis.json
index 2950265e..0a2baff6 100644
--- a/testdata/golden/expected/browser-sniff-sample/traffic-analysis.json
+++ b/testdata/golden/expected/browser-sniff-sample/traffic-analysis.json
@@ -55,6 +55,7 @@
],
"endpoint_clusters": [
{
+ "confidence": "low",
"content_types": [
"application/json"
],
@@ -72,6 +73,10 @@
],
"host": "httpbin.org",
"method": "GET",
+ "normalization_flags": [
+ "single-sample",
+ "single-status"
+ ],
"path": "/get",
"request_shape": {},
"response_shape": {},
@@ -81,6 +86,7 @@
]
},
{
+ "confidence": "low",
"content_types": [
"application/json"
],
@@ -98,6 +104,10 @@
],
"host": "httpbin.org",
"method": "GET",
+ "normalization_flags": [
+ "single-sample",
+ "single-status"
+ ],
"path": "/headers",
"request_shape": {},
"response_shape": {},
@@ -107,6 +117,7 @@
]
},
{
+ "confidence": "low",
"content_types": [
"application/json"
],
@@ -124,6 +135,10 @@
],
"host": "httpbin.org",
"method": "GET",
+ "normalization_flags": [
+ "single-sample",
+ "single-status"
+ ],
"path": "/ip",
"request_shape": {},
"response_shape": {},
← 770b65a6 docs(cli): clarify issue ownership workflow (#1414)
·
back to Cli Printing Press
·
docs(cli): document sync --resources parent-keyed dependent d4b173d6 →