← back to Cli Printing Press
fix(cli): detect path params in browser-sniff URL grouper (#1657)
2c3271fcf95dc9ede77864f3c78bfde9bb4b6476 · 2026-05-19 03:03:36 -0700 · Trevin Chow
* fix(cli): detect path params in browser-sniff URL grouper
HAR-sniffed specs collapsed variable path segments into literal endpoint
names because the normalizer only recognized UUID v4, lowercase hex
hashes, and pure-numeric segments. Application IDs that ship a short
type prefix (`t_xxx`, `cus_xxx`), long opaque base62 / ULID / nanoid
tails, or colon-composite discriminators (`create-image:reference:gpt-
image-2`) all landed in the spec as literal paths, producing per-ID
command files that worked for the exact rows captured and nothing else.
This change:
- Extends `normalizeEntryPath` with prefixed-ID, long-opaque-alnum, and
colon-composite heuristics, each guarded by a non-trivial-token or
mixed-case-or-digit floor so route literals like `subscriptions` or
`host:80` don't get parametrized.
- Derives the placeholder name from the parent path segment
(`tables/t_xxx` -> `tables/{table_id}`) using a conservative
snake-case + trailing-`s` singularizer, with `{id}`/`{uuid}`/`{hash}`
as the fallback when the parent isn't usable as an identifier root.
- Adds a `collapseVariantGroups` post-pass to `DeduplicateEndpoints`
and `DeduplicateTrafficEndpoints` that merges groups whose paths
differ in exactly one segment when at least one member's value
matches a strong ID shape, so cross-entry variance backstops shape
heuristics that miss.
Tests cover the OpenArt history-ID, Clay prefixed-ID, OpenArt colon-
composite form-ID, and the issue's positive/negative acceptance cases.
Closes #1386
* fix(cli): address Greptile findings on browser-sniff path-param detection
Two issues flagged in PR #1657 review:
1. (P1) `collapseVariantGroups` was unreachable as written. Its
`anyOpaque` gate called `looksLikeIDShape`, which uses the exact same
six regex patterns as `normalizeEntryPath` — so any segment that
would satisfy the gate had already been replaced by a placeholder in
the per-segment pass, putting it in a different bucket from the
literal-shaped peer it was supposed to merge with. Replace the gate
with `looksParameterizable`, a weaker check that's TRUE when a
segment has a digit, mixed case, or any strong-ID shape. Also
require ALL members at the varying position to satisfy this gate
(not just one), so a real literal like `health` paired with a
data-shaped sibling stays separate.
Adds a test covering the case Greptile flagged:
`/api/messages/abc123` and `/api/messages/xyz456` (6-char opaque
IDs, no prefix) now collapse to `/api/messages/{message_id}`.
2. (P2) When two ID segments sit directly under the same parent
(`/resources/123/456`), the second one walked back past the
freshly-emitted `{resource_id}` placeholder, found `resources`
again, and emitted a second `{resource_id}` — producing a path with
duplicate parameter names that OpenAPI rejects. Track per-path
placeholder name use and append a counter suffix on collision, so
the path normalizes to `/resources/{resource_id}/{resource_id_2}`.
Adds dedicated single-entry and two-entry tests for the consecutive-
ID case.
* fix(cli): address Greptile follow-up findings on variance pass
Two new P1 findings on the previous fix:
1. `collapseVariantGroups` could still emit duplicate placeholder names
when a per-segment-placed `{resource_id}` lived earlier in the same
path. The variance pass walked back through the existing placeholder,
found `resources` again, and emitted a second `{resource_id}`. Build
a per-target use-count from the existing path's placeholders and
route the variance emission through `disambiguatePlaceholder`, so the
collision becomes `{resource_id_2}` just like the per-segment path.
2. `collapseVariantGroups` was host-blind. `DeduplicateTrafficEndpoints`
keys its dedup map on host but `EndpointGroup` carried no host
field, so the variance pass bucketed entries from different hosts
together and could merge unrelated services into one group. Add a
`Host` field to `EndpointGroup`, populate it in
`DeduplicateTrafficEndpoints`, and include it in the variance
skeleton key.
Adds tests for both: a two-entry case where per-segment normalization
plants a placeholder before the variance position; and a multi-host
case asserting that two distinct hosts stay in separate groups even
when their path shapes coincide.
* fix(cli): narrow variance-pass gate to digit-only widening
P1 follow-up: a pure mixed-case branch (hasUpper && hasLower) in
looksParameterizable was matching PascalCase route literals like
/api/CreateDocument and /api/ListDocuments, which are common in
action-style REST routes (ASP.NET, gRPC-HTTP transcoding). Two such
distinct endpoints would land in the same skeleton bucket and merge into
/api/{api_id}, destroying both routes.
Drop the mixed-case branch entirely. The has-digit branch alone is the
correct widening for the variance pass's target — short opaque IDs like
abc123/xyz456 all carry a digit. The rare digit-free opaque ID is left
for users to parametrize manually rather than risk false-positive
collapses on PascalCase action names.
The variance-pass disambiguation test now exercises digit-bearing
short-opaque IDs (abc123/xyz456 under /resources/) so it stays correct
under the narrowed gate. Adds an explicit negative case: PascalCase
route literals must stay separate.
* fix(cli): make disambiguator scan past suffixed placeholders
P1 follow-up: when a path already carries multiple suffixed
placeholders (`/resources/{resource_id}/{resource_id_2}/...`) and the
variance pass tries to emit a colliding `{resource_id}`, the previous
disambiguator pre-populated `used` with full placeholder strings and
returned `{resource_id_2}` — a name already in the path. Result:
`/resources/{resource_id}/{resource_id_2}/{resource_id_2}`, invalid
OpenAPI.
Rework `disambiguatePlaceholder` to loop on candidate names: start at
the base name, then `_2`, `_3`, ... until a name not already in `used`
is found. Marks every emitted name in `used` so chains stay unique
regardless of how the variance pass pre-populates the map.
Adds a triple-consecutive test asserting that
`/resources/123/456/abc123` and `/resources/789/012/xyz456` normalize
to `/resources/{resource_id}/{resource_id_2}/{resource_id_3}`.
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Files touched
M internal/browsersniff/analysis.goM internal/browsersniff/classifier.goM internal/browsersniff/classifier_test.goM internal/browsersniff/specgen_test.go
Diff
commit 2c3271fcf95dc9ede77864f3c78bfde9bb4b6476
Author: Trevin Chow <trevin@trevinchow.com>
Date: Tue May 19 03:03:36 2026 -0700
fix(cli): detect path params in browser-sniff URL grouper (#1657)
* fix(cli): detect path params in browser-sniff URL grouper
HAR-sniffed specs collapsed variable path segments into literal endpoint
names because the normalizer only recognized UUID v4, lowercase hex
hashes, and pure-numeric segments. Application IDs that ship a short
type prefix (`t_xxx`, `cus_xxx`), long opaque base62 / ULID / nanoid
tails, or colon-composite discriminators (`create-image:reference:gpt-
image-2`) all landed in the spec as literal paths, producing per-ID
command files that worked for the exact rows captured and nothing else.
This change:
- Extends `normalizeEntryPath` with prefixed-ID, long-opaque-alnum, and
colon-composite heuristics, each guarded by a non-trivial-token or
mixed-case-or-digit floor so route literals like `subscriptions` or
`host:80` don't get parametrized.
- Derives the placeholder name from the parent path segment
(`tables/t_xxx` -> `tables/{table_id}`) using a conservative
snake-case + trailing-`s` singularizer, with `{id}`/`{uuid}`/`{hash}`
as the fallback when the parent isn't usable as an identifier root.
- Adds a `collapseVariantGroups` post-pass to `DeduplicateEndpoints`
and `DeduplicateTrafficEndpoints` that merges groups whose paths
differ in exactly one segment when at least one member's value
matches a strong ID shape, so cross-entry variance backstops shape
heuristics that miss.
Tests cover the OpenArt history-ID, Clay prefixed-ID, OpenArt colon-
composite form-ID, and the issue's positive/negative acceptance cases.
Closes #1386
* fix(cli): address Greptile findings on browser-sniff path-param detection
Two issues flagged in PR #1657 review:
1. (P1) `collapseVariantGroups` was unreachable as written. Its
`anyOpaque` gate called `looksLikeIDShape`, which uses the exact same
six regex patterns as `normalizeEntryPath` — so any segment that
would satisfy the gate had already been replaced by a placeholder in
the per-segment pass, putting it in a different bucket from the
literal-shaped peer it was supposed to merge with. Replace the gate
with `looksParameterizable`, a weaker check that's TRUE when a
segment has a digit, mixed case, or any strong-ID shape. Also
require ALL members at the varying position to satisfy this gate
(not just one), so a real literal like `health` paired with a
data-shaped sibling stays separate.
Adds a test covering the case Greptile flagged:
`/api/messages/abc123` and `/api/messages/xyz456` (6-char opaque
IDs, no prefix) now collapse to `/api/messages/{message_id}`.
2. (P2) When two ID segments sit directly under the same parent
(`/resources/123/456`), the second one walked back past the
freshly-emitted `{resource_id}` placeholder, found `resources`
again, and emitted a second `{resource_id}` — producing a path with
duplicate parameter names that OpenAPI rejects. Track per-path
placeholder name use and append a counter suffix on collision, so
the path normalizes to `/resources/{resource_id}/{resource_id_2}`.
Adds dedicated single-entry and two-entry tests for the consecutive-
ID case.
* fix(cli): address Greptile follow-up findings on variance pass
Two new P1 findings on the previous fix:
1. `collapseVariantGroups` could still emit duplicate placeholder names
when a per-segment-placed `{resource_id}` lived earlier in the same
path. The variance pass walked back through the existing placeholder,
found `resources` again, and emitted a second `{resource_id}`. Build
a per-target use-count from the existing path's placeholders and
route the variance emission through `disambiguatePlaceholder`, so the
collision becomes `{resource_id_2}` just like the per-segment path.
2. `collapseVariantGroups` was host-blind. `DeduplicateTrafficEndpoints`
keys its dedup map on host but `EndpointGroup` carried no host
field, so the variance pass bucketed entries from different hosts
together and could merge unrelated services into one group. Add a
`Host` field to `EndpointGroup`, populate it in
`DeduplicateTrafficEndpoints`, and include it in the variance
skeleton key.
Adds tests for both: a two-entry case where per-segment normalization
plants a placeholder before the variance position; and a multi-host
case asserting that two distinct hosts stay in separate groups even
when their path shapes coincide.
* fix(cli): narrow variance-pass gate to digit-only widening
P1 follow-up: a pure mixed-case branch (hasUpper && hasLower) in
looksParameterizable was matching PascalCase route literals like
/api/CreateDocument and /api/ListDocuments, which are common in
action-style REST routes (ASP.NET, gRPC-HTTP transcoding). Two such
distinct endpoints would land in the same skeleton bucket and merge into
/api/{api_id}, destroying both routes.
Drop the mixed-case branch entirely. The has-digit branch alone is the
correct widening for the variance pass's target — short opaque IDs like
abc123/xyz456 all carry a digit. The rare digit-free opaque ID is left
for users to parametrize manually rather than risk false-positive
collapses on PascalCase action names.
The variance-pass disambiguation test now exercises digit-bearing
short-opaque IDs (abc123/xyz456 under /resources/) so it stays correct
under the narrowed gate. Adds an explicit negative case: PascalCase
route literals must stay separate.
* fix(cli): make disambiguator scan past suffixed placeholders
P1 follow-up: when a path already carries multiple suffixed
placeholders (`/resources/{resource_id}/{resource_id_2}/...`) and the
variance pass tries to emit a colliding `{resource_id}`, the previous
disambiguator pre-populated `used` with full placeholder strings and
returned `{resource_id_2}` — a name already in the path. Result:
`/resources/{resource_id}/{resource_id_2}/{resource_id_2}`, invalid
OpenAPI.
Rework `disambiguatePlaceholder` to loop on candidate names: start at
the base name, then `_2`, `_3`, ... until a name not already in `used`
is found. Marks every emitted name in `used` so chains stay unique
regardless of how the variance pass pre-populates the map.
Adds a triple-consecutive test asserting that
`/resources/123/456/abc123` and `/resources/789/012/xyz456` normalize
to `/resources/{resource_id}/{resource_id_2}/{resource_id_3}`.
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
---
internal/browsersniff/analysis.go | 3 +-
internal/browsersniff/classifier.go | 432 ++++++++++++++++++++++++++++++-
internal/browsersniff/classifier_test.go | 189 +++++++++++++-
internal/browsersniff/specgen_test.go | 2 +-
4 files changed, 609 insertions(+), 17 deletions(-)
diff --git a/internal/browsersniff/analysis.go b/internal/browsersniff/analysis.go
index f914266c..3cdab09d 100644
--- a/internal/browsersniff/analysis.go
+++ b/internal/browsersniff/analysis.go
@@ -555,13 +555,14 @@ func DeduplicateTrafficEndpoints(entries []EnrichedEntry) []EndpointGroup {
indexByKey[key] = len(groups)
groups = append(groups, EndpointGroup{
+ Host: host,
Method: method,
NormalizedPath: normalizedPath,
Entries: []EnrichedEntry{entry},
})
}
- return groups
+ return collapseVariantGroups(groups)
}
func classifyInCaptureOrder(entries []EnrichedEntry, apiEntries []EnrichedEntry, noiseEntries []EnrichedEntry) []EnrichedEntry {
diff --git a/internal/browsersniff/classifier.go b/internal/browsersniff/classifier.go
index a04d43c6..34936b74 100644
--- a/internal/browsersniff/classifier.go
+++ b/internal/browsersniff/classifier.go
@@ -5,24 +5,50 @@ import (
"net"
"net/url"
"regexp"
+ "sort"
+ "strconv"
"strings"
"sync"
)
type EndpointGroup struct {
+ // Host is the originating URL host for this group (lowercased), set by
+ // host-aware dedup paths like DeduplicateTrafficEndpoints. Empty when the
+ // caller doesn't care about host separation (DeduplicateEndpoints flattens
+ // across hosts). The cross-entry variance pass keys on this field so two
+ // groups sharing method/path-shape but coming from different hosts stay
+ // separate.
+ Host string
Method string
NormalizedPath string
Entries []EnrichedEntry
}
var (
- uuidSegmentPattern = regexp.MustCompile(`(?i)^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`)
- hashSegmentPattern = regexp.MustCompile(`(?i)^[0-9a-f]{32,}$`)
- numericPattern = regexp.MustCompile(`^\d+$`)
- blocklistMu sync.RWMutex
- additionalBlocklist []string
- includeListMu sync.RWMutex
- additionalInclude []string
+ uuidSegmentPattern = regexp.MustCompile(`(?i)^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`)
+ hashSegmentPattern = regexp.MustCompile(`(?i)^[0-9a-f]{32,}$`)
+ numericPattern = regexp.MustCompile(`^\d+$`)
+ // prefixedIDPattern matches application-issued IDs that ship a short
+ // type-prefix and a long alphanumeric tail, such as Clay's t_/r_/c_, Stripe's
+ // cus_/sub_, or OpenAI's run_/asst_. The tail floor of 8 chars keeps short
+ // literal segments like "v1" or two-letter language codes from matching.
+ prefixedIDPattern = regexp.MustCompile(`^[a-z]{1,5}_[A-Za-z0-9]{8,}$`)
+ // longAlnumIDPattern matches opaque application IDs without separators that
+ // are long enough and mixed enough to be implausible as literal route names:
+ // nanoid (21 chars), ULID (26 chars), short base62 IDs like OpenArt's
+ // `Zu2uNCmGDnmNCel8gbFQ` (20 chars). The mixed-case-or-digit floor below
+ // rules out long lowercase words (e.g. "subscriptions", "notifications").
+ longAlnumIDPattern = regexp.MustCompile(`^[A-Za-z0-9]{20,}$`)
+ // colonCompositePattern matches IDs that carry an embedded type discriminator
+ // via colon segments, like OpenArt form_ids
+ // (`create-image:reference:gpt-image-2`) or Stripe price tier IDs. Requires
+ // at least two colons total so simple key:value or port-style values
+ // (e.g. `host:80`) don't get pulled in as composite IDs.
+ colonCompositePattern = regexp.MustCompile(`^[A-Za-z0-9._-]+:[A-Za-z0-9._-]+:[A-Za-z0-9._:-]+$`)
+ blocklistMu sync.RWMutex
+ additionalBlocklist []string
+ includeListMu sync.RWMutex
+ additionalInclude []string
)
func ClassifyEntries(entries []EnrichedEntry) (api []EnrichedEntry, noise []EnrichedEntry) {
@@ -161,7 +187,7 @@ func DeduplicateEndpoints(entries []EnrichedEntry) []EndpointGroup {
})
}
- return groups
+ return collapseVariantGroups(groups)
}
func scoreEntry(entry EnrichedEntry, blocklist []string, include []string) int {
@@ -308,15 +334,35 @@ func extractPath(rawURL string) string {
func normalizeEntryPath(rawURL string) string {
path := extractPath(rawURL)
segments := strings.Split(path, "/")
+ // Track per-path placeholder name use so consecutive IDs that resolve to
+ // the same parent (e.g. /resources/123/456) don't both emit
+ // {resource_id}; the second collision gets a counter suffix.
+ nameCounts := make(map[string]int)
for i, segment := range segments {
+ if segment == "" {
+ continue
+ }
+ // Skip framing segments (api, /v1) so the placeholder name reflects the
+ // resource the ID belongs to, not the version prefix.
+ parent := previousMeaningfulSegment(segments, i)
+ var placeholder string
switch {
case numericPattern.MatchString(segment):
- segments[i] = "{id}"
+ placeholder = idPlaceholder(parent, "id")
case uuidSegmentPattern.MatchString(segment):
- segments[i] = "{uuid}"
+ placeholder = idPlaceholder(parent, "uuid")
case hashSegmentPattern.MatchString(segment):
- segments[i] = "{hash}"
+ placeholder = idPlaceholder(parent, "hash")
+ case prefixedIDPattern.MatchString(segment):
+ placeholder = idPlaceholder(parent, "id")
+ case colonCompositePattern.MatchString(segment) && hasNonTrivialToken(segment):
+ placeholder = idPlaceholder(parent, "id")
+ case longAlnumIDPattern.MatchString(segment) && looksOpaqueID(segment):
+ placeholder = idPlaceholder(parent, "id")
+ default:
+ continue
}
+ segments[i] = disambiguatePlaceholder(placeholder, nameCounts)
}
normalized := strings.Join(segments, "/")
@@ -326,3 +372,367 @@ func normalizeEntryPath(rawURL string) string {
return normalized
}
+
+// disambiguatePlaceholder appends a counter suffix when the same placeholder
+// name has already appeared earlier in the current path. For
+// /resources/123/456 the first segment yields {resource_id}; the second walks
+// back past the freshly-emitted placeholder, sees `resources` again, and would
+// emit a second {resource_id} — instead it becomes {resource_id_2}. This keeps
+// downstream spec generation safe: OpenAPI rejects duplicate path-parameter
+// names within a single path template.
+//
+// The `used` map tracks every concrete placeholder string already emitted in
+// the current path (including suffixed variants). The walk advances the
+// counter as long as the candidate name collides, so chains like
+// {resource_id}/{resource_id_2}/{resource_id_3} stay unique even when the
+// variance pass populates `used` from a path the per-segment normalizer
+// already disambiguated.
+func disambiguatePlaceholder(placeholder string, used map[string]int) string {
+ if used[placeholder] == 0 {
+ used[placeholder] = 1
+ return placeholder
+ }
+ inner := strings.TrimSuffix(strings.TrimPrefix(placeholder, "{"), "}")
+ // Find the next free suffix. Start at 2 (the first collision becomes _2)
+ // and advance until the candidate isn't already in `used`.
+ for n := 2; ; n++ {
+ candidate := "{" + inner + "_" + strconv.Itoa(n) + "}"
+ if used[candidate] == 0 {
+ used[candidate] = 1
+ return candidate
+ }
+ }
+}
+
+// previousMeaningfulSegment walks backwards from index i looking for a segment
+// that isn't empty, a placeholder, or a routing framing segment (api, vN). The
+// placeholder name (table_id, widget_id) reflects the resource the ID belongs
+// to, not the version prefix it happens to live under.
+func previousMeaningfulSegment(segments []string, i int) string {
+ for j := i - 1; j >= 0; j-- {
+ s := segments[j]
+ if s == "" {
+ continue
+ }
+ if strings.HasPrefix(s, "{") && strings.HasSuffix(s, "}") {
+ continue
+ }
+ if s == "api" || isVersionSegment(s) {
+ continue
+ }
+ return s
+ }
+ return ""
+}
+
+// isVersionSegment is the local mirror of discovery.isVersionSegment kept here
+// to avoid a package import cycle through the discovery package's spec import.
+func isVersionSegment(segment string) bool {
+ if len(segment) < 2 || segment[0] != 'v' {
+ return false
+ }
+ for _, r := range segment[1:] {
+ if r < '0' || r > '9' {
+ return false
+ }
+ }
+ return true
+}
+
+// idPlaceholder returns "{parent_id}" when parent is a useful resource segment
+// (drops trailing 's', forces snake_case), falling back to "{fallback}" when
+// the parent is empty or not safely Go-identifier shaped.
+func idPlaceholder(parent string, fallback string) string {
+ name := placeholderNameFromParent(parent)
+ if name == "" {
+ return "{" + fallback + "}"
+ }
+ return "{" + name + "_id}"
+}
+
+func placeholderNameFromParent(parent string) string {
+ parent = strings.TrimSpace(parent)
+ if parent == "" {
+ return ""
+ }
+ // Normalize to snake_case so "user-groups" -> "user_group_id" and an
+ // already-singular parent like "user" stays "user_id". Reject parents that
+ // aren't safely shaped as a Go identifier root (e.g. all-digit, empty).
+ lower := strings.ToLower(parent)
+ lower = strings.ReplaceAll(lower, "-", "_")
+ for _, r := range lower {
+ if r == '_' {
+ continue
+ }
+ if r >= 'a' && r <= 'z' {
+ continue
+ }
+ if r >= '0' && r <= '9' {
+ continue
+ }
+ return ""
+ }
+ // Drop a trailing 's' on plurals; keep "status"/"address" intact via simple
+ // "ss" guard. Mirrors the conservative singularizer used elsewhere; complex
+ // irregulars (entries -> entry) are left to downstream tooling that owns
+ // command naming.
+ if strings.HasSuffix(lower, "ies") && len(lower) > 3 {
+ lower = lower[:len(lower)-3] + "y"
+ } else if strings.HasSuffix(lower, "s") && !strings.HasSuffix(lower, "ss") && len(lower) > 1 {
+ lower = lower[:len(lower)-1]
+ }
+ return lower
+}
+
+// hasNonTrivialToken guards colon-composite detection: at least one
+// colon-separated token must have 3+ chars so plain port-style values like
+// "host:80" aren't mistaken for composite IDs.
+func hasNonTrivialToken(segment string) bool {
+ for token := range strings.SplitSeq(segment, ":") {
+ if len(token) >= 3 {
+ return true
+ }
+ }
+ return false
+}
+
+// looksLikeIDShape returns true when a segment matches any of the strong ID
+// heuristics (UUID, hex hash, numeric, prefixed application id, colon
+// composite, or long opaque alphanumeric). These are the same shapes the
+// per-segment normalizer recognizes; kept as a named helper so future variance
+// callers can ask the question without inlining the regex list.
+func looksLikeIDShape(segment string) bool {
+ if numericPattern.MatchString(segment) {
+ return true
+ }
+ if uuidSegmentPattern.MatchString(segment) {
+ return true
+ }
+ if hashSegmentPattern.MatchString(segment) {
+ return true
+ }
+ if prefixedIDPattern.MatchString(segment) {
+ return true
+ }
+ if colonCompositePattern.MatchString(segment) && hasNonTrivialToken(segment) {
+ return true
+ }
+ if longAlnumIDPattern.MatchString(segment) && looksOpaqueID(segment) {
+ return true
+ }
+ return false
+}
+
+// looksParameterizable is the weaker gate used by the cross-entry variance
+// pass. By construction, any segment that satisfies looksLikeIDShape would
+// already have been replaced by normalizeEntryPath, so gating on that shape
+// alone makes the variance pass unreachable. The pass exists to catch IDs
+// that the per-segment patterns can't classify in isolation — short opaque
+// tokens like `abc123` or `xyz456` that only stand out as IDs when two
+// entries land at the same position with different literal values.
+//
+// The widening is deliberately narrow: presence of a digit. Pure mixed-case
+// alone (`hasUpper && hasLower`) is not enough — PascalCase is a normal
+// shape for action-style REST routes (`/api/CreateDocument` vs
+// `/api/ListDocuments`, common in ASP.NET / gRPC-HTTP transcoding), and a
+// pure-letter mixed-case check would merge those into `/api/{id}`. The
+// short-opaque IDs the variance pass exists to catch all carry at least one
+// digit; the rare digit-free opaque ID is left for users to parametrize
+// manually rather than risk destroying real route names.
+func looksParameterizable(segment string) bool {
+ if looksLikeIDShape(segment) {
+ return true
+ }
+ for _, r := range segment {
+ if r >= '0' && r <= '9' {
+ return true
+ }
+ }
+ return false
+}
+
+// looksOpaqueID applies the mixed-case-or-digit floor for the long-alphanumeric
+// shape so route literals like "subscriptions" (all-lowercase, no digits) aren't
+// flagged. A long segment carrying any digit, or mixing case, is treated as an
+// opaque application ID.
+func looksOpaqueID(segment string) bool {
+ hasDigit := false
+ hasUpper := false
+ hasLower := false
+ for _, r := range segment {
+ switch {
+ case r >= '0' && r <= '9':
+ hasDigit = true
+ case r >= 'A' && r <= 'Z':
+ hasUpper = true
+ case r >= 'a' && r <= 'z':
+ hasLower = true
+ }
+ }
+ return hasDigit || (hasUpper && hasLower)
+}
+
+// collapseVariantGroups folds endpoint groups whose paths differ in exactly
+// one segment into a single parameterized group. Two HAR captures of
+// /widgets/abc-a and /widgets/abc-b produce two raw groups; the variance pass
+// detects they share the same method and identical path prefix/suffix, and
+// merges them into /widgets/{widget_id}. Only runs when neither path already
+// carries a placeholder at the diverging segment (preserves explicit hits from
+// the per-segment normalizer).
+func collapseVariantGroups(groups []EndpointGroup) []EndpointGroup {
+ if len(groups) < 2 {
+ return groups
+ }
+
+ // Bucket by (host, method, segment count, positions of existing
+ // placeholders) so only same-shape paths from the same host can collapse
+ // together. Two groups from different hosts must stay separate even when
+ // their path shapes coincide — DeduplicateTrafficEndpoints keys by host
+ // upstream, and the variance pass must preserve that separation.
+ type skeletonKey struct {
+ host string
+ method string
+ length int
+ placeholder string // bitmask of positions already holding placeholders
+ }
+ buckets := make(map[skeletonKey][]int)
+ skeletons := make(map[int][]string)
+ for idx, group := range groups {
+ segments := strings.Split(group.NormalizedPath, "/")
+ skeletons[idx] = segments
+ var placeholderMask strings.Builder
+ for _, s := range segments {
+ if strings.HasPrefix(s, "{") && strings.HasSuffix(s, "}") {
+ placeholderMask.WriteByte('1')
+ } else {
+ placeholderMask.WriteByte('0')
+ }
+ }
+ key := skeletonKey{
+ host: group.Host,
+ method: group.Method,
+ length: len(segments),
+ placeholder: placeholderMask.String(),
+ }
+ buckets[key] = append(buckets[key], idx)
+ }
+
+ // Map from group index to merge-target index. Targets stay; non-targets get
+ // merged into the target and dropped from output.
+ mergeInto := make(map[int]int)
+
+ for _, members := range buckets {
+ if len(members) < 2 {
+ continue
+ }
+ segLen := len(skeletons[members[0]])
+
+ // Find positions that vary across the bucket. We only promote one
+ // position per bucket pass; if multiple positions vary, the paths are
+ // genuinely different resources and shouldn't collapse.
+ varyingPositions := make([]int, 0)
+ for pos := range segLen {
+ seen := make(map[string]struct{})
+ for _, idx := range members {
+ seen[skeletons[idx][pos]] = struct{}{}
+ if len(seen) > 1 {
+ break
+ }
+ }
+ if len(seen) > 1 {
+ varyingPositions = append(varyingPositions, pos)
+ }
+ }
+ if len(varyingPositions) != 1 {
+ continue
+ }
+ pos := varyingPositions[0]
+
+ // The diverging segment must look like a parameter candidate at every
+ // member: not a placeholder (filtered already), not a routing keyword.
+ // Use the weaker looksParameterizable check (digit present, mixed case,
+ // or strong-ID shape) rather than looksLikeIDShape — by construction
+ // any strong-ID-shaped segment was already replaced by
+ // normalizeEntryPath, so gating on that alone makes the pass
+ // unreachable. The looksParameterizable widening catches short opaque
+ // tokens (`abc123` vs `xyz456`) that only stand out as IDs once two
+ // entries land at the same position with different values, while still
+ // rejecting plain route literals like `health`/`version`.
+ anyParameterizable := false
+ allParameterizable := true
+ for _, idx := range members {
+ s := skeletons[idx][pos]
+ if s == "" || s == "api" || isVersionSegment(s) {
+ allParameterizable = false
+ continue
+ }
+ if looksParameterizable(s) {
+ anyParameterizable = true
+ } else {
+ allParameterizable = false
+ }
+ }
+ if !anyParameterizable || !allParameterizable {
+ continue
+ }
+
+ // Pick the lowest-index member as the merge target and rewrite its
+ // path. Other members fold into it.
+ target := members[0]
+ for _, idx := range members[1:] {
+ if idx < target {
+ target = idx
+ }
+ }
+ parent := previousMeaningfulSegment(skeletons[target], pos)
+ placeholder := idPlaceholder(parent, "id")
+ newSegments := append([]string(nil), skeletons[target]...)
+ // Build the in-path placeholder use count from the existing segments so
+ // the variance-pass emission disambiguates against placeholders the
+ // per-segment normalizer already placed. Without this, a path like
+ // /resources/{resource_id}/AbcDef would have its variance position
+ // resolved by walking back through the existing {resource_id} to find
+ // `resources` again, producing a duplicate `{resource_id}`.
+ used := make(map[string]int)
+ for _, s := range newSegments {
+ if strings.HasPrefix(s, "{") && strings.HasSuffix(s, "}") {
+ used[s]++
+ }
+ }
+ newSegments[pos] = disambiguatePlaceholder(placeholder, used)
+ groups[target].NormalizedPath = strings.Join(newSegments, "/")
+
+ for _, idx := range members {
+ if idx == target {
+ continue
+ }
+ mergeInto[idx] = target
+ }
+ }
+
+ if len(mergeInto) == 0 {
+ return groups
+ }
+
+ // Two-pass to keep the merge order deterministic: first fold every merged
+ // group's Entries into its target in source-index order (map iteration is
+ // random and would scramble Entries between runs), then emit survivors in
+ // original order.
+ sourceIdxs := make([]int, 0, len(mergeInto))
+ for idx := range mergeInto {
+ sourceIdxs = append(sourceIdxs, idx)
+ }
+ sort.Ints(sourceIdxs)
+ for _, idx := range sourceIdxs {
+ target := mergeInto[idx]
+ groups[target].Entries = append(groups[target].Entries, groups[idx].Entries...)
+ }
+ out := make([]EndpointGroup, 0, len(groups)-len(mergeInto))
+ for idx := range groups {
+ if _, merged := mergeInto[idx]; merged {
+ continue
+ }
+ out = append(out, groups[idx])
+ }
+ return out
+}
diff --git a/internal/browsersniff/classifier_test.go b/internal/browsersniff/classifier_test.go
index 4f82ebed..02c45cea 100644
--- a/internal/browsersniff/classifier_test.go
+++ b/internal/browsersniff/classifier_test.go
@@ -4,6 +4,7 @@ import (
"testing"
"github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
)
func TestClassifyEntries(t *testing.T) {
@@ -157,23 +158,143 @@ func TestDeduplicateEndpoints(t *testing.T) {
wantGroupSizes []int
}{
{
- name: "numeric ids normalize to id placeholder",
+ name: "numeric ids normalize to parent-derived id placeholder",
entries: []EnrichedEntry{
{Method: "GET", URL: "https://example.com/users/123?expand=true"},
{Method: "GET", URL: "https://example.com/users/456"},
},
wantMethods: []string{"GET"},
- wantNormalizedPaths: []string{"/users/{id}"},
+ wantNormalizedPaths: []string{"/users/{user_id}"},
wantGroupSizes: []int{2},
},
{
- name: "uuid segment normalizes to uuid placeholder",
+ name: "uuid segment normalizes to parent-derived id placeholder",
entries: []EnrichedEntry{
{Method: "GET", URL: "https://example.com/orders/550e8400-e29b-41d4-a716-446655440000"},
{Method: "GET", URL: "https://example.com/orders/123e4567-e89b-12d3-a456-426614174000?include=items"},
},
wantMethods: []string{"GET"},
- wantNormalizedPaths: []string{"/orders/{uuid}"},
+ wantNormalizedPaths: []string{"/orders/{order_id}"},
+ wantGroupSizes: []int{2},
+ },
+ {
+ name: "prefixed application ids normalize to parent-derived placeholder",
+ entries: []EnrichedEntry{
+ {Method: "GET", URL: "https://example.com/api/v1/tables/t_0t3vswhpKogASf2XZpW"},
+ {Method: "GET", URL: "https://example.com/api/v1/tables/t_zzAAbbCC11223344"},
+ },
+ wantMethods: []string{"GET"},
+ wantNormalizedPaths: []string{"/api/v1/tables/{table_id}"},
+ wantGroupSizes: []int{2},
+ },
+ {
+ name: "colon-composite form ids collapse to parent-derived placeholder",
+ entries: []EnrichedEntry{
+ {Method: "POST", URL: "https://example.com/suite/api/forms/creations/create-image:reference:gpt-image-2"},
+ {Method: "POST", URL: "https://example.com/suite/api/forms/creations/create-image:reference:gpt-image-3"},
+ },
+ wantMethods: []string{"POST"},
+ wantNormalizedPaths: []string{"/suite/api/forms/creations/{creation_id}"},
+ wantGroupSizes: []int{2},
+ },
+ {
+ name: "long opaque application ids normalize via cross-entry variance",
+ entries: []EnrichedEntry{
+ {Method: "GET", URL: "https://example.com/suite/api/history/Zu2uNCmGDnmNCel8gbFQ"},
+ {Method: "GET", URL: "https://example.com/suite/api/history/AbcDefGhi1234567890X"},
+ },
+ wantMethods: []string{"GET"},
+ wantNormalizedPaths: []string{"/suite/api/history/{history_id}"},
+ wantGroupSizes: []int{2},
+ },
+ {
+ name: "static endpoints with distinct literal segments stay separate",
+ entries: []EnrichedEntry{
+ {Method: "GET", URL: "https://example.com/api/health"},
+ {Method: "GET", URL: "https://example.com/api/version"},
+ },
+ wantMethods: []string{"GET", "GET"},
+ wantNormalizedPaths: []string{"/api/health", "/api/version"},
+ wantGroupSizes: []int{1, 1},
+ },
+ {
+ // Short opaque IDs (`abc123`, `xyz456`) fail every per-segment ID
+ // regex in isolation — they're too short for the long-alnum or
+ // hash patterns and have no type prefix. The cross-entry variance
+ // pass is the only thing that can identify them, because their
+ // id-ness becomes obvious only when two entries land at the same
+ // position with different data-shaped values.
+ name: "short opaque ids parametrize via cross-entry variance",
+ entries: []EnrichedEntry{
+ {Method: "GET", URL: "https://example.com/api/messages/abc123"},
+ {Method: "GET", URL: "https://example.com/api/messages/xyz456"},
+ },
+ wantMethods: []string{"GET"},
+ wantNormalizedPaths: []string{"/api/messages/{message_id}"},
+ wantGroupSizes: []int{2},
+ },
+ {
+ // Consecutive numeric ID segments share the same parent
+ // (`resources` for both). Without disambiguation the resulting
+ // path would carry two `{resource_id}` placeholders, which breaks
+ // downstream OpenAPI generation (duplicate path-parameter names
+ // within a single path template are rejected).
+ name: "consecutive ids under same parent get disambiguated names",
+ entries: []EnrichedEntry{
+ {Method: "GET", URL: "https://example.com/resources/123/456"},
+ {Method: "GET", URL: "https://example.com/resources/789/012"},
+ },
+ wantMethods: []string{"GET"},
+ wantNormalizedPaths: []string{"/resources/{resource_id}/{resource_id_2}"},
+ wantGroupSizes: []int{2},
+ },
+ {
+ // Variance pass must disambiguate against placeholders the
+ // per-segment normalizer already placed. Position 2 (`123`/`456`)
+ // becomes `{resource_id}` in per-segment normalization; position
+ // 3 (`abc123`/`xyz456`) is short-opaque-with-digit, not a strong
+ // per-segment ID shape but promoted by the variance pass. The
+ // walker for position 3 would find `resources` again — placeholder
+ // emission must see the existing `{resource_id}` and disambiguate
+ // to `{resource_id_2}` rather than emit a duplicate.
+ name: "variance pass disambiguates against per-segment placeholder",
+ entries: []EnrichedEntry{
+ {Method: "GET", URL: "https://example.com/resources/123/abc123"},
+ {Method: "GET", URL: "https://example.com/resources/456/xyz456"},
+ },
+ wantMethods: []string{"GET"},
+ wantNormalizedPaths: []string{"/resources/{resource_id}/{resource_id_2}"},
+ wantGroupSizes: []int{2},
+ },
+ {
+ // PascalCase route literals (common in ASP.NET, gRPC-HTTP
+ // transcoding) must NOT collapse via the variance pass.
+ // `CreateDocument` and `ListDocuments` are distinct action-style
+ // endpoints, both digit-free; the parameterizability check
+ // requires a digit specifically to avoid this false positive.
+ name: "pascal-case route literals stay separate",
+ entries: []EnrichedEntry{
+ {Method: "GET", URL: "https://example.com/api/CreateDocument"},
+ {Method: "GET", URL: "https://example.com/api/ListDocuments"},
+ },
+ wantMethods: []string{"GET", "GET"},
+ wantNormalizedPaths: []string{"/api/CreateDocument", "/api/ListDocuments"},
+ wantGroupSizes: []int{1, 1},
+ },
+ {
+ // Three consecutive same-parent IDs: per-segment normalization
+ // produces /resources/{resource_id}/{resource_id_2}/abc123. The
+ // variance pass then promotes position 4. Naive counter-only
+ // disambiguation would re-emit {resource_id_2} (already in the
+ // path); the implementation must keep advancing the counter past
+ // every existing suffix and land on {resource_id_3}.
+ name: "triple consecutive ids disambiguate to _3",
+ entries: []EnrichedEntry{
+ {Method: "GET", URL: "https://example.com/resources/123/456/abc123"},
+ {Method: "GET", URL: "https://example.com/resources/789/012/xyz456"},
+ },
+ wantMethods: []string{"GET"},
+ wantNormalizedPaths: []string{"/resources/{resource_id}/{resource_id_2}/{resource_id_3}"},
wantGroupSizes: []int{2},
},
}
@@ -191,6 +312,66 @@ func TestDeduplicateEndpoints(t *testing.T) {
}
}
+// TestDeduplicateEndpoints_SingleEntryHeuristics covers the per-segment ID
+// shapes that must parametrize even when only one HAR entry is available, so
+// the cross-entry variance pass has nothing to compare against.
+func TestDeduplicateEndpoints_SingleEntryHeuristics(t *testing.T) {
+ t.Parallel()
+
+ tests := []struct {
+ name string
+ url string
+ wantPath string
+ }{
+ {"uuid v4", "https://example.com/widgets/550e8400-e29b-41d4-a716-446655440000", "/widgets/{widget_id}"},
+ {"long hex hash", "https://example.com/blobs/0123456789abcdef0123456789abcdef", "/blobs/{blob_id}"},
+ {"numeric id under resource", "https://example.com/workspaces/125600", "/workspaces/{workspace_id}"},
+ {"prefixed application id under resource", "https://example.com/records/r_0tf32xmAhEgGSh3TDWR", "/records/{record_id}"},
+ {"colon-composite under resource", "https://example.com/forms/creations/create-image:reference:gpt-image-2", "/forms/creations/{creation_id}"},
+ {"long base62 id under resource", "https://example.com/history/Zu2uNCmGDnmNCel8gbFQ", "/history/{history_id}"},
+ {"literal short segments remain literal", "https://example.com/api/health", "/api/health"},
+ {"version segment retains literal v1 framing", "https://example.com/api/v1/users", "/api/v1/users"},
+ {"consecutive ids under same parent disambiguate", "https://example.com/resources/123/456", "/resources/{resource_id}/{resource_id_2}"},
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ t.Parallel()
+
+ groups := DeduplicateEndpoints([]EnrichedEntry{{Method: "GET", URL: tt.url}})
+ require.Len(t, groups, 1)
+ assert.Equal(t, tt.wantPath, groups[0].NormalizedPath)
+ })
+ }
+}
+
+// TestDeduplicateTrafficEndpoints_VariancePassRespectsHost verifies that the
+// cross-entry variance pass cannot merge groups from different hosts. The
+// upstream host-aware key keeps them separate at dedup time, but
+// collapseVariantGroups originally bucketed only by (method, length, mask) and
+// would collapse them into a single mixed-host group. The skeleton key now
+// includes Host so this no longer happens.
+func TestDeduplicateTrafficEndpoints_VariancePassRespectsHost(t *testing.T) {
+ t.Parallel()
+
+ entries := []EnrichedEntry{
+ {Method: "GET", URL: "https://api.service1.com/orders/ORD123"},
+ {Method: "GET", URL: "https://api.service2.com/orders/SHP456"},
+ }
+
+ groups := DeduplicateTrafficEndpoints(entries)
+
+ require.Len(t, groups, 2, "different hosts must stay in separate groups")
+ hosts := map[string]struct{}{}
+ for _, g := range groups {
+ hosts[g.Host] = struct{}{}
+ }
+ assert.Equal(t, map[string]struct{}{
+ "api.service1.com": {},
+ "api.service2.com": {},
+ }, hosts)
+}
+
func entryURLs(entries []EnrichedEntry) []string {
urls := make([]string, 0, len(entries))
for _, entry := range entries {
diff --git a/internal/browsersniff/specgen_test.go b/internal/browsersniff/specgen_test.go
index a15da424..38950f70 100644
--- a/internal/browsersniff/specgen_test.go
+++ b/internal/browsersniff/specgen_test.go
@@ -689,7 +689,7 @@ func TestWriteSamples_WritesOneFilePerEndpointGroup(t *testing.T) {
switch sample.Method {
case "GET":
foundGET = true
- assert.Equal(t, "GET /v1/items/{id}", sample.Endpoint)
+ assert.Equal(t, "GET /v1/items/{item_id}", sample.Endpoint)
assert.Equal(t, 200, sample.Status)
assert.True(t, sample.ResponseBodyKnown)
assert.Equal(t, RedactedSentinel, sample.RequestHeaders["Authorization"])
← 17cd2774 fix(cli): dogfood prefers bundled <dir>/spec.yaml over calle
·
back to Cli Printing Press
·
fix(skills): stop phase 4.95 code review from being skipped 5cb39a69 →