← back to Cli Printing Press
feat(cli): promote html_scrape reachability mode when captcha-tier protection blocks JSON + SSR sibling carries state blob (#1065)
d7e2d74f32fcefbb730366c40bb4c33974bc3ead · 2026-05-11 08:19:35 -0700 · Trevin Chow
* feat(cli): extend SSR state-blob detector with new signatures + signature passing (U1, #953)
Adds state-view, __NUXT__, and __APP_INITIAL_STATE__ to the existing
isSSREmbeddedData detector. Enforces a 2xx status floor and a
10 KB body-size floor to reject challenge pages and empty templates.
Refactors the helper to return the matched signature label, surfaced
on the ssr_embedded_data protocol observation's Details["signature"]
field so downstream consumers (mode selector in U2, specgen in U3)
can pick the right CSS selector.
Test fixture helper makeSSRPage builds 10KB+ HTML bodies for each
signature; the existing test that asserts ssr_embedded_data +
browser_rendered now uses the helper. Adds 13 unit tests for the
detector covering each signature, the size+status floors, the
priority-on-multi-match rule, and the non-HTML / no-signature
negative cases.
* feat(cli): wire html_scrape mode + emit html_extract: embedded-json from sniff (U2-U4, #953)
U2: Reachability mode selector promotes html_scrape over browser_required
when an API-classified entry carries captcha-tier protection (captcha,
bot_challenge, aws_waf, vercel_challenge) AND an HTML entry on the same
registered domain (eTLD+1) emits the ssr_embedded_data protocol with a
matched signature. Cross-subdomain Yandex-shape captures qualify via
eTLD+1; cloudflare/akamai stay on lighter clearance modes per design.
Protection-to-entry attribution uses ProtectionObservation.Evidence[].
EntryIndex so a Cloudflare-fronted SSR HTML page emitting a cloudflare
signal from its own cf-ray header does not satisfy the captcha-tier
predicate. The HTML side uses content-type (not Classification) to
identify HTML entries — the classifier returns "noise" for text/html
so Classification is not a usable filter there.
U3: ApplyReachabilityDefaults extended to rewrite HTMLExtract on
HTML-shaped endpoints when Mode == "html_scrape". Sets Mode to
embedded-json and ScriptSelector based on the matched signature
(__NEXT_DATA__ → script#__NEXT_DATA__, state-view → script.state-view,
__NUXT__ → script#__NUXT__, etc.). The window.__ signature gets an
empty selector so the runtime falls back to DefaultEmbeddedJSONScript
Selector and the operator hand-tunes.
U4: scripts/golden.sh verify passes — no fixtures retroactively match
the new two-signal pattern. ReachabilityAnalysis adds two omitempty
fields (HTMLExtractMode + HTMLExtractSignature) for downstream
consumption; the new sameRegisteredDomain helper uses
golang.org/x/net/publicsuffix (already transitively pulled in,
promoted to a direct dep by go mod tidy).
Tests: 3531 pass (was 3505 before U1). New coverage:
- 13 detector cases (signatures, size/status floors, priority order)
- 8 reachability promotion cases (yandex shape, same-host, eTLD+1,
cloudflare baseline, captcha without sibling, aws_waf, etc.)
- 8 same-eTLD+1 host-pairing cases
- 9 specgen-rewrite cases (each signature mapping + non-HTML endpoint
preservation + signature-empty baseline)
* refactor(cli): apply /simplify findings to html_scrape detector
Drop redundant HTMLExtractMode field (callers only need the matched
signature label; mode is always embedded-json for SSR detection).
Export SSR signature constants so reachability.go can map signatures
to script selectors without string duplication. Trim narrative
comments per AGENTS.md hygiene rules.
Test rename `expectMode`/`expectSignature` → `expectedMode`/`expectedSignature`
for consistency. Correct misleading `accepts-304-cached` test name
(behavior is rejection) and drop speculative future-work comment.
Plan: correct 304 wording to match 2xx-only gate.
* fix(cli): address /ce-code-review findings on html_scrape detector
P0: re-detect SSR signature per entry in
findSSRStateBlobEntryOnRegisteredDomain. The protocol observation's
Details["signature"] collapses to last-write-wins across multi-entry
input, so the signature returned from the protocol map could mismatch
the returned entry index when multiple SSR HTML entries exist on
different domains.
P1: tighten window.__ and state-view substring matchers. Bare
"window.__" matched analytics globals like window.__gtag; bare
"state-view" matched CSS class names like state-view-port. Replace
with shape-bearing variants:
- window.__initial_state / window.__app_state / window.__apollo_state / window.__data__
- "state-view" (quoted, double + single).
P2: preserve operator-set ScriptSelector when our signature mapping
has no concrete selector (e.g. window.__ → ""), instead of clobbering
the hand-tuned value.
Add control fixtures so future regressions surface:
- rejects-window-prefix-on-analytics-globals
- rejects-state-view-as-css-fragment
- multi-ssr-signature-attributed-to-matching-entry (P0 ordering guard)
* fix(cli): address pr-review-toolkit findings on html_scrape detector
P1: preserve upstream confidence when promoting to html_scrape — was
hardcoded to 0.85, now takes max(confidence, 0.85). Reason string
claimed "preferred" while numerically downgrading.
P1: tighten sameRegisteredDomain comment so it accurately describes
the literal-equality fast path (runs before publicsuffix) rather than
suggesting a fallback that fires after publicsuffix errors.
P2: rewrite applyHTMLScrapeExtractionDefaults doc as why-not-what.
Add coverage for gaps surfaced by pr-test-analyzer:
- TestApplyReachabilityDefaults_HTMLScrapePromotesNestedSubResources:
promote a nested SubResources resource so the recursion in
promoteHTMLExtractInResource has explicit coverage.
- E2E cases for vercel_challenge and bot_challenge captcha-tier
protections so the captchaTierProtections allowlist has at least
one end-to-end case per entry.
Files touched
A docs/plans/2026-05-10-003-feat-html-scrape-mode-selector-plan.mdM go.modM internal/browsersniff/analysis.goM internal/browsersniff/analysis_test.goM internal/browsersniff/reachability.go
Diff
commit d7e2d74f32fcefbb730366c40bb4c33974bc3ead
Author: Trevin Chow <trevin@trevinchow.com>
Date: Mon May 11 08:19:35 2026 -0700
feat(cli): promote html_scrape reachability mode when captcha-tier protection blocks JSON + SSR sibling carries state blob (#1065)
* feat(cli): extend SSR state-blob detector with new signatures + signature passing (U1, #953)
Adds state-view, __NUXT__, and __APP_INITIAL_STATE__ to the existing
isSSREmbeddedData detector. Enforces a 2xx status floor and a
10 KB body-size floor to reject challenge pages and empty templates.
Refactors the helper to return the matched signature label, surfaced
on the ssr_embedded_data protocol observation's Details["signature"]
field so downstream consumers (mode selector in U2, specgen in U3)
can pick the right CSS selector.
Test fixture helper makeSSRPage builds 10KB+ HTML bodies for each
signature; the existing test that asserts ssr_embedded_data +
browser_rendered now uses the helper. Adds 13 unit tests for the
detector covering each signature, the size+status floors, the
priority-on-multi-match rule, and the non-HTML / no-signature
negative cases.
* feat(cli): wire html_scrape mode + emit html_extract: embedded-json from sniff (U2-U4, #953)
U2: Reachability mode selector promotes html_scrape over browser_required
when an API-classified entry carries captcha-tier protection (captcha,
bot_challenge, aws_waf, vercel_challenge) AND an HTML entry on the same
registered domain (eTLD+1) emits the ssr_embedded_data protocol with a
matched signature. Cross-subdomain Yandex-shape captures qualify via
eTLD+1; cloudflare/akamai stay on lighter clearance modes per design.
Protection-to-entry attribution uses ProtectionObservation.Evidence[].
EntryIndex so a Cloudflare-fronted SSR HTML page emitting a cloudflare
signal from its own cf-ray header does not satisfy the captcha-tier
predicate. The HTML side uses content-type (not Classification) to
identify HTML entries — the classifier returns "noise" for text/html
so Classification is not a usable filter there.
U3: ApplyReachabilityDefaults extended to rewrite HTMLExtract on
HTML-shaped endpoints when Mode == "html_scrape". Sets Mode to
embedded-json and ScriptSelector based on the matched signature
(__NEXT_DATA__ → script#__NEXT_DATA__, state-view → script.state-view,
__NUXT__ → script#__NUXT__, etc.). The window.__ signature gets an
empty selector so the runtime falls back to DefaultEmbeddedJSONScript
Selector and the operator hand-tunes.
U4: scripts/golden.sh verify passes — no fixtures retroactively match
the new two-signal pattern. ReachabilityAnalysis adds two omitempty
fields (HTMLExtractMode + HTMLExtractSignature) for downstream
consumption; the new sameRegisteredDomain helper uses
golang.org/x/net/publicsuffix (already transitively pulled in,
promoted to a direct dep by go mod tidy).
Tests: 3531 pass (was 3505 before U1). New coverage:
- 13 detector cases (signatures, size/status floors, priority order)
- 8 reachability promotion cases (yandex shape, same-host, eTLD+1,
cloudflare baseline, captcha without sibling, aws_waf, etc.)
- 8 same-eTLD+1 host-pairing cases
- 9 specgen-rewrite cases (each signature mapping + non-HTML endpoint
preservation + signature-empty baseline)
* refactor(cli): apply /simplify findings to html_scrape detector
Drop redundant HTMLExtractMode field (callers only need the matched
signature label; mode is always embedded-json for SSR detection).
Export SSR signature constants so reachability.go can map signatures
to script selectors without string duplication. Trim narrative
comments per AGENTS.md hygiene rules.
Test rename `expectMode`/`expectSignature` → `expectedMode`/`expectedSignature`
for consistency. Correct misleading `accepts-304-cached` test name
(behavior is rejection) and drop speculative future-work comment.
Plan: correct 304 wording to match 2xx-only gate.
* fix(cli): address /ce-code-review findings on html_scrape detector
P0: re-detect SSR signature per entry in
findSSRStateBlobEntryOnRegisteredDomain. The protocol observation's
Details["signature"] collapses to last-write-wins across multi-entry
input, so the signature returned from the protocol map could mismatch
the returned entry index when multiple SSR HTML entries exist on
different domains.
P1: tighten window.__ and state-view substring matchers. Bare
"window.__" matched analytics globals like window.__gtag; bare
"state-view" matched CSS class names like state-view-port. Replace
with shape-bearing variants:
- window.__initial_state / window.__app_state / window.__apollo_state / window.__data__
- "state-view" (quoted, double + single).
P2: preserve operator-set ScriptSelector when our signature mapping
has no concrete selector (e.g. window.__ → ""), instead of clobbering
the hand-tuned value.
Add control fixtures so future regressions surface:
- rejects-window-prefix-on-analytics-globals
- rejects-state-view-as-css-fragment
- multi-ssr-signature-attributed-to-matching-entry (P0 ordering guard)
* fix(cli): address pr-review-toolkit findings on html_scrape detector
P1: preserve upstream confidence when promoting to html_scrape — was
hardcoded to 0.85, now takes max(confidence, 0.85). Reason string
claimed "preferred" while numerically downgrading.
P1: tighten sameRegisteredDomain comment so it accurately describes
the literal-equality fast path (runs before publicsuffix) rather than
suggesting a fallback that fires after publicsuffix errors.
P2: rewrite applyHTMLScrapeExtractionDefaults doc as why-not-what.
Add coverage for gaps surfaced by pr-test-analyzer:
- TestApplyReachabilityDefaults_HTMLScrapePromotesNestedSubResources:
promote a nested SubResources resource so the recursion in
promoteHTMLExtractInResource has explicit coverage.
- E2E cases for vercel_challenge and bot_challenge captcha-tier
protections so the captchaTierProtections allowlist has at least
one end-to-end case per entry.
---
...5-10-003-feat-html-scrape-mode-selector-plan.md | 324 +++++++++++
go.mod | 2 +-
internal/browsersniff/analysis.go | 199 ++++++-
internal/browsersniff/analysis_test.go | 631 ++++++++++++++++++++-
internal/browsersniff/reachability.go | 56 ++
5 files changed, 1201 insertions(+), 11 deletions(-)
diff --git a/docs/plans/2026-05-10-003-feat-html-scrape-mode-selector-plan.md b/docs/plans/2026-05-10-003-feat-html-scrape-mode-selector-plan.md
new file mode 100644
index 00000000..f34eceb6
--- /dev/null
+++ b/docs/plans/2026-05-10-003-feat-html-scrape-mode-selector-plan.md
@@ -0,0 +1,324 @@
+---
+title: 'feat: Promote html_scrape end-to-end (selector + specgen) when SSR state-blob detected alongside captcha-protected JSON'
+type: feat
+status: active
+date: 2026-05-10
+---
+
+# feat: Promote html_scrape end-to-end (selector + specgen) when SSR state-blob detected alongside captcha-protected JSON
+
+## Summary
+
+Wire the browser-sniff pipeline to emit `html_extract.mode: embedded-json` specs when a HAR shows captcha-protected JSON endpoints alongside an SSR state-blob HTML sibling on the same registered domain. The change spans three layers: the existing `isSSREmbeddedData` detector is extended to recognize Yandex's `state-view` and Nuxt's `__NUXT__` / `__APP_INITIAL_STATE__` signatures (today it only covers `__NEXT_DATA__` / `window.__` / `application/ld+json`); the reachability mode selector adds a fifth recommendation that overrides `browser_required` when the two-signal conjunction holds; and the spec-generator's `inferHTMLExtract` learns to emit `embedded-json` mode with the script selector keyed to the matched signature. Today's sniff pipeline only ever emits `page` or `links` modes — `embedded-json` is reachable only via hand-authored or catalog-supplied specs.
+
+---
+
+## Problem Frame
+
+Issue [#953](https://github.com/mvanhorn/cli-printing-press/issues/953) (P1, retro). The original framing proposed a 470-LOC new `internal/scrape/` generator-template family. Audit (posted to #953) ruled that out — the spec layer already accepts `html_extract.mode: embedded-json` (added by the food52 retro), and the runtime correctly handles the resulting CLI shape. Re-audit during this plan's doc-review pass revealed a second misconception: the sniff pipeline DOES NOT emit `embedded-json` mode today. The spec layer's acceptance of that mode is reachable only via operator-authored or catalog-supplied specs — sniff-derived specs go through [`internal/browsersniff/specgen.go`](../../internal/browsersniff/specgen.go) `inferHTMLExtract` which only produces `HTMLExtractModePage` or `HTMLExtractModeLinks`.
+
+Current state by layer:
+
+| Layer | Status | Reference |
+|---|---|---|
+| Catalog `integration_mode` enum | ✓ `html-scrape` accepted (wrapper-library metadata) | [`internal/catalog/catalog.go:107`](../../internal/catalog/catalog.go#L107) |
+| Spec `html_extract.mode` parsing | ✓ `embedded-json` mode defined | [`internal/spec/spec.go:48-54`](../../internal/spec/spec.go#L48) |
+| Runtime spec consumption | ✓ generator emits correct CLI shape when spec has embedded-json | (food52 retro work) |
+| HAR protocol detection | ✓ `ssr_embedded_data` emitted at 0.85 confidence | [`internal/browsersniff/analysis.go:677-680`](../../internal/browsersniff/analysis.go#L677) |
+| State-blob signature coverage | ⚠ only `__NEXT_DATA__` / `window.__` / `application/ld+json` | [`internal/browsersniff/analysis.go:1331-1337`](../../internal/browsersniff/analysis.go#L1331) |
+| Reachability mode selector | ✗ chooses among 4 modes; never `html_scrape` | [`internal/browsersniff/analysis.go:902-928`](../../internal/browsersniff/analysis.go#L902) |
+| **Sniff → spec emission of `embedded-json`** | ✗ `inferHTMLExtract` only emits `page` / `links` | [`internal/browsersniff/specgen.go:771-782`](../../internal/browsersniff/specgen.go#L771) |
+
+The Yandex Maps retro evidence: JSON `/maps/api/*` endpoints were captcha-gated, but consumer HTML pages on the same registered domain were cold-fetchable and carried a `<script type="application/json" class="state-view">` blob containing the same structured data. Today the selector routes this to `browser_required`; specgen emits `mode: page`. The right outcome is `html_scrape` mode + `mode: embedded-json` spec, which generates a goquery-based extraction CLI instead of a Playwright shell.
+
+**Honest framing on impact.** The original issue claimed "15-25% of curated catalog." Audit shows `kayak` is the only direct fit in the curated catalog today; `google-flights` is wrapper-only; remaining 19 entries are clean JSON. Accurate framing: second occurrence of the pattern in the operator-driven print stream — food52 motivated the spec-level support, yandex motivates analyzer + specgen completion. No quantified catalog-percent claim.
+
+---
+
+## Requirements
+
+- **R1.** The HAR state-blob detector (`isSSREmbeddedData`) recognizes the three signatures the rescope cited that it does not currently match: `state-view`, `__NUXT__`, `__APP_INITIAL_STATE__`. Existing matches (`__NEXT_DATA__`, `window.__`, `application/ld+json`) stay.
+- **R2.** State-blob detection requires HTTP status `2xx` (`200-299`) — `304 Not Modified` and similar cached responses are accepted — and body ≥ 10 KB. Smaller bodies likely represent empty templates or challenge pages; the floor rejects those. (Calibration note: original 50 KB threshold was derived from Yandex captures of 540 KB+; lowered to 10 KB after doc review surfaced asymmetric risk — false negatives are silent, false positives are noisy and self-correcting.)
+- **R3.** The detector surfaces which state-blob signature matched (one of `__NEXT_DATA__`, `__NUXT__`, `__APP_INITIAL_STATE__`, `state-view`, `application/ld+json`, `window.__`). Downstream consumers need this to pick the right CSS selector for spec emission — `script#__NEXT_DATA__` for Next.js, `script.state-view` for Yandex, etc.
+- **R4.** The reachability mode selector adds a fifth recommendation: `html_scrape`. Promotes when both signals are present in the HAR:
+ - At least one entry with `Classification == "api"` carries a captcha-tier protection signal (`captcha`, `bot_challenge`, `aws_waf`, `vercel_challenge`) — and that protection signal is attributed to the API entry itself (via `ProtectionObservation.Evidence[i].EntryIndex`), not just present anywhere in the HAR. Cloudflare / Akamai / etc. keep their existing routes (lighter modes that can still reach JSON).
+ - At least one HTML response (detected by `Content-Type` containing `html` — the classifier returns `noise` for HTML, so we cannot use Classification here) on the **same registered domain (eTLD+1)** as the protected API entry emits the `ssr_embedded_data` protocol observation. Subdomain-split shapes like `api.example.com` + `www.example.com` qualify.
+- **R5.** Priority: `html_scrape` mode overrides `browser_required` when the two-signal conjunction holds. Other modes (`standard_http`, `browser_http`, `browser_clearance_http`) are unchanged because the captcha-tier protection set that triggers `html_scrape` is a strict subset of what triggers `browser_required` today — by construction, this addition affects only the `browser_required` slice of the cascade.
+- **R6.** When mode is `html_scrape`, the emitted spec carries `html_extract.mode: embedded-json` with a `script_selector` derived from the matched signature. The mapping:
+ - `__NEXT_DATA__` → `script#__NEXT_DATA__`
+ - `__NUXT__` → `script#__NUXT__`
+ - `__APP_INITIAL_STATE__` → `script#__APP_INITIAL_STATE__`
+ - `state-view` → `script.state-view`
+ - `application/ld+json` → `script[type="application/ld+json"]`
+ - `window.__` → no script-tag selector; emit `mode: embedded-json` with a comment noting hand-tuning needed
+- **R7.** Regression: a HAR without the two-signal conjunction routes through the existing four-mode cascade unchanged. Clean JSON HARs still go to `standard_http`; captcha-only HARs (no SSR sibling) still go to `browser_required`.
+
+---
+
+## Key Technical Decisions
+
+### KTD-1. Extend the existing detector and protocol; do not introduce new labels
+
+`ssr_embedded_data` is already in the protocol enum at 0.85 confidence. Extend `isSSREmbeddedData` to cover the three new signatures rather than adding a new protocol label. Keeps the enum stable; downstream consumers (existing test fixtures, scoring) don't need updates.
+
+### KTD-2. Surface the matched signature on the protocol observation
+
+The selector and specgen both need to know which signature matched (so the script-selector can be picked correctly per R6). Two options:
+- Store the signature in `ProtocolObservation.Details` (existing map field) under a key like `signature`.
+- Add a new `Signature` field to `ProtocolObservation` (more invasive).
+
+Pick the first — `Details` is the right home for protocol-specific data, no struct surgery, and the field is already part of the JSON payload. Selector reads `protocols[ssr_embedded_data].Details["signature"]`; specgen reads the same.
+
+### KTD-3. Same-registered-domain (eTLD+1) matching, not literal hostname
+
+Use the Go ecosystem's `golang.org/x/net/publicsuffix` (or `net/url` if a sufficient helper exists) to compute eTLD+1. The literal-hostname check rejects the canonical Yandex-shape capture where API and HTML live on different subdomains; the eTLD+1 check accepts it. The cross-domain false-positive risk (captcha-protected API on one origin + unrelated marketing SSR on a different origin) is tolerated because (a) the matched pair must share the registered domain — coincidence across organizations is implausible — and (b) the captcha-tier protection requirement is itself a strong filter against benign captures.
+
+### KTD-4. Captcha-tier protection only; not cloudflare/akamai/etc.
+
+The promotion fires only when an API entry carries `captcha`, `bot_challenge`, `aws_waf`, or `vercel_challenge`. These are "JSON is unreachable without a browser" signals. Cloudflare / Akamai / Datadome / PerimeterX / protected_web / login_redirect can frequently be cleared with bearer tokens or session cookies, so those entries should continue routing to `browser_clearance_http` or `browser_http` (lighter modes than `html_scrape`). This means the new branch never pulls a HAR away from a lighter mode — it only pulls from `browser_required` toward `html_scrape`.
+
+### KTD-5. Protection-to-entry attribution via EvidenceRef.EntryIndex
+
+`hasProtection` walks the analysis-level Protections list (one observation per detected protection). Each observation carries `Evidence[i].EntryIndex` linking back to the entry that surfaced the signal. To require "the API entry itself is protected," the new check looks up evidence entries' indices and tests their `Classification`. Naive `hasProtection(captcha) && hasSSREmbeddedData()` would also fire on benign HARs where the SSR HTML page is itself behind Cloudflare (emitting a `cloudflare` protection signal from its own `cf-ray` header) and the JSON is unprotected — the entry-level join prevents that.
+
+### KTD-6. Content-type-based check for the HTML side; not Classification
+
+The classifier returns `"api"` or `"noise"` only — text/html entries earn -2 and land in `"noise"`. R3's "HTML-classified entry" is not a real category. The HTML side of the same-eTLD+1 join walks raw `entries`, filters by `strings.Contains(strings.ToLower(entry.ResponseContentType), "html")`, and tests for the `ssr_embedded_data` protocol on that entry's `EntryIndex`.
+
+### KTD-7. Cascade insertion: after browser_required, conditional override
+
+The existing selector at [`analysis.go:902-928`](../../internal/browsersniff/analysis.go#L902) is a mix of fall-through and guarded branches. The captcha branch (line 907) sets `browser_required` with no guard against override. The new `html_scrape` branch is inserted after the captcha/bot/aws_waf/vercel_challenge branches. It guards on the explicit two-signal conjunction (KTD-5 + KTD-6) and overrides `browser_required` when fired. Other branches that gate on `mode == "standard_http"` (lines 917, 922) are unaffected — they never run when `mode == "html_scrape"` because the override happens after them in real terms (the new branch fires only when captcha/bot has already promoted to `browser_required`).
+
+### KTD-8. Signature passing into specgen: read from ReachabilityAnalysis
+
+`specgen.inferHTMLExtract` is per-EndpointGroup; ReachabilityAnalysis is HAR-level. Plumb the signature down by:
+- Adding the signature to `ReachabilityAnalysis` (one field — the recommended html_extract mode and matched script_selector if any).
+- `AnalyzeCapture` (specgen.go) calls `classifyReachability` and threads the result into `inferHTMLExtract` via a new parameter or via `EndpointGroup` metadata set during capture analysis.
+
+The implementer picks the exact threading mechanism during work; the plan-time decision is "the signature must reach specgen" not "via this specific struct field."
+
+### KTD-9. Selector emits a script_selector hint; specgen finalizes it
+
+The selector (analyzer side) emits which signature matched. `inferHTMLExtract` consumes that and maps to the canonical CSS selector per R6's table. The mapping table lives in spec.go or specgen.go (closer to specgen since it produces the spec). Keeps the analyzer free of CSS-selector vocabulary; keeps specgen as the single source of truth for what selectors map to.
+
+### KTD-10. No new dependencies except `golang.org/x/net/publicsuffix`
+
+eTLD+1 resolution requires the public suffix list. Go's ecosystem has `golang.org/x/net/publicsuffix` which is the standard choice and likely already pulled in transitively. If not, this is the one new dependency the plan requires. No other new packages.
+
+---
+
+## Implementation Units
+
+### U1. Extend SSR state-blob detector + surface matched signature
+
+**Goal:** Update `isSSREmbeddedData` to detect the three new signatures, add status-class and body-size floors, and return *which* signature matched (not just a bool). Store the signature on the protocol observation's `Details` map.
+
+**Requirements:** R1, R2, R3.
+
+**Dependencies:** none.
+
+**Files:**
+- `internal/browsersniff/analysis.go` (modify `isSSREmbeddedData` at line 1331 — refactor signature to return `(matched bool, signature string)`)
+- `internal/browsersniff/analysis.go` (modify the detector call site at line 677-680 — pass the signature into the protocol observation's `Details` map)
+- `internal/browsersniff/analysis_test.go` (extend tests)
+
+**Approach:**
+- Refactor `isSSREmbeddedData(entry EnrichedEntry) bool` → `detectSSREmbeddedData(entry EnrichedEntry) (matched bool, signature string)`. Returns the signature label (one of the six per R3) when matched, empty string otherwise.
+- Add status-class check (`entry.ResponseStatus >= 200 && entry.ResponseStatus < 300`) at the top of the function. Rejects challenge pages (403/429/503) and error templates.
+- Add body-size check (`len(entry.ResponseBody) >= 10_000`) — calibrated lower than the original 50 KB per doc-review. Named const `ssrEmbeddedDataMinBodySize`.
+- Signature priority for the case of multiple matches in one body: pick the first matched in the order listed in R3 (the order is roughly framework-specific → generic, so Next.js wins over `window.__` when both present, which is the typical Next.js shape).
+- Update the call site at line 677-680: receive the signature and store as `Details["signature"]` on the `ssr_embedded_data` protocol observation.
+
+**Patterns to follow:** Existing detector body shape at [`analysis.go:1331-1337`](../../internal/browsersniff/analysis.go#L1331); the `Details` map convention used by other detectors (search for `Details: map[string]string{` in analysis.go for existing examples).
+
+**Test scenarios:**
+- HTML response with `state-view` script tag, status 200, 20 KB body → matched, signature == `state-view`.
+- HTML response with `__NUXT__` script tag, status 200, 20 KB body → matched, signature == `__NUXT__`.
+- HTML response with `__APP_INITIAL_STATE__`, status 200, 20 KB body → matched, signature == `__APP_INITIAL_STATE__`.
+- HTML response with `__NEXT_DATA__`, status 200, 20 KB body → matched, signature == `__NEXT_DATA__` (regression).
+- HTML response with `application/ld+json`, status 200, 20 KB body → matched, signature == `application/ld+json` (regression).
+- HTML response with `window.__INITIAL_STATE__`, status 200, 20 KB body → matched, signature == `window.__` (regression).
+- HTML response with both `__NEXT_DATA__` and `application/ld+json`, status 200, 20 KB body → matched, signature == `__NEXT_DATA__` (priority).
+- HTML response with `__NEXT_DATA__`, status 200, 5 KB body → NOT matched (body floor).
+- HTML response with `__NEXT_DATA__`, status 403, 20 KB body → NOT matched (status floor — challenge page).
+- HTML response with `__NEXT_DATA__`, status 304, 20 KB body → NOT matched (status floor — only 2xx promotes; 304 is treated as a non-fresh response we cannot inspect reliably).
+- Non-HTML response (JSON, XML) with state-blob marker substring → NOT matched (existing content-type guard).
+- Call-site verification: a fixture HAR with one SSR HTML entry produces a protocol observation with `Details["signature"]` populated.
+
+**Verification:** `go test ./internal/browsersniff/ -run SSR` passes. `go vet` clean.
+
+---
+
+### U2. Mode selector adds `html_scrape` on captcha-tier + same-eTLD+1 SSR sibling
+
+**Goal:** Teach the reachability selector to override `browser_required` with `html_scrape` when an API entry carries captcha-tier protection AND an HTML entry on the same eTLD+1 emits the `ssr_embedded_data` protocol.
+
+**Requirements:** R4, R5, R7.
+
+**Dependencies:** U1.
+
+**Files:**
+- `internal/browsersniff/analysis.go` (modify the reachability cascade at line 902-928 — add the new branch)
+- `internal/browsersniff/analysis.go` (add helper functions: `entryHasCaptchaTierProtection`, `entryEmitsSSREmbeddedData`, `sameRegisteredDomain`)
+- `internal/browsersniff/analysis_test.go` (extend reachability tests)
+- `go.mod` / `go.sum` (add `golang.org/x/net/publicsuffix` if not already pulled in)
+
+**Approach:**
+- Define a helper `apiEntryHasCaptchaTierProtection(entries, protections) (entryIndex int, ok bool)`. Walks `protections`, filters to label in `{captcha, bot_challenge, aws_waf, vercel_challenge}`, joins back to entries via `Evidence[i].EntryIndex`, and requires the entry's `Classification == "api"`. Returns the index of the first matching API entry (or `-1, false`).
+- Define a helper `htmlEntryHasSSRStateBlobOnRegisteredDomain(entries, protocols, refHost string) (entryIndex int, signature string, ok bool)`. Walks entries, filters to those whose `Content-Type` contains `html` and whose host's eTLD+1 matches `refHost`'s eTLD+1, and checks if any of those entries has an `ssr_embedded_data` protocol observation in `protocols`. Returns the entry index, the signature label (read from `Details["signature"]`), and a match flag.
+- In the cascade at line 902-928, after the `browser_required`-setting branches (line 902-911, captcha and browser_rendered), insert a new branch:
+ - If `apiEntryHasCaptchaTierProtection` matched: let `refHost = extractHost(entries[apiIdx].URL)`.
+ - If `htmlEntryHasSSRStateBlobOnRegisteredDomain(entries, protocols, refHost)` matched: set `mode = "html_scrape"`, confidence = 0.85, reason = `"captcha-tier protection on API + same-registered-domain SSR state blob (signature: <sig>); html_scrape preferred over browser_required"`.
+- Same-registered-domain helper: `sameRegisteredDomain(hostA, hostB string) bool` — uses `publicsuffix.EffectiveTLDPlusOne` on both, returns true on match (case-insensitive). Fall-back: if either lookup fails (private TLD, malformed host), require literal hostname equality.
+- The new branch must come AFTER the captcha/bot_challenge branches so it can read the already-set `browser_required` mode and override it. It must come BEFORE the cloudflare branch (line 917) which guards on `mode == "standard_http"` — so the cloudflare branch won't run when html_scrape is set.
+
+**Patterns to follow:** Existing cascade structure at [`analysis.go:902-928`](../../internal/browsersniff/analysis.go#L902); existing `hasProtection` / `hasProtocol` / `hasAuth` helpers (they walk top-level lists; the new helpers walk per-entry); the `extractHost` / URL parsing already used throughout the file.
+
+**Test scenarios:**
+- HAR with API JSON entry on `api.example.com` returning 403 with captcha protection on that entry + HTML entry on `www.example.com` returning 200 with state-view blob (20 KB) → mode is `html_scrape`. Reason names the signature. (Yandex-shape positive — cross-subdomain.)
+- HAR with API JSON on `example.com/api` returning 403/captcha + HTML on `example.com/` with `__NEXT_DATA__` (20 KB) → mode is `html_scrape`. (Same-host positive.)
+- HAR with API JSON on `example.com` returning 403/captcha + HTML on `other-site.com` with state blob → mode is `browser_required` (different registered domain). KTD-3.
+- HAR with HTML page on `example.com` carrying cloudflare protection (cf-ray on the HTML response) + JSON on `example.com/api` returning 200 (no protection) + SSR state blob on the HTML → mode is `standard_http`. The protection isn't on the API entry; KTD-5 must filter this out.
+- HAR with API JSON returning 200 (no protection) + HTML entry with state blob → mode is `standard_http`. No protection, no promotion.
+- HAR with API JSON returning 403/cloudflare (NOT captcha-tier) + HTML state-blob sibling → mode is `browser_clearance_http` (cloudflare's existing route). KTD-4.
+- HAR with API JSON returning 403/aws_waf + HTML state-blob sibling on same eTLD+1 → mode is `html_scrape`. (aws_waf is in the captcha-tier set.)
+- HAR with API JSON returning 403/captcha + HTML entry without state blob → mode is `browser_required`. Baseline preserved.
+- HAR with only state-blob HTML, no API entries → mode is `standard_http`. Degenerate case.
+- HAR with API JSON returning 403/captcha + HTML state-blob on same registered domain BUT under a different subdomain that's actually a different organization's site (theoretical: same eTLD+1 but different organizational ownership — e.g., a co.uk shared registrar) → not a realistic case in practice; the test asserts the rule's behavior, not a contrived false-positive scenario.
+- Regression: all existing reachability tests at the four prior modes continue to pass.
+
+**Verification:** `go test ./internal/browsersniff/` passes including new reachability tests. Existing `TestReachability*` suite green.
+
+---
+
+### U3. Wire specgen to consume the signal and emit `html_extract.mode: embedded-json`
+
+**Goal:** Update `inferHTMLExtract` in `specgen.go` to consume the new selector signal (matched signature from ReachabilityAnalysis) and emit `html_extract.mode: embedded-json` with the correct script selector when the signal is present.
+
+**Requirements:** R6.
+
+**Dependencies:** U2.
+
+**Files:**
+- `internal/browsersniff/specgen.go` (modify `inferHTMLExtract` at line 771-782 — add the new mode branch; thread the signature signal in)
+- `internal/browsersniff/specgen.go` (modify `AnalyzeCapture` at the entry point to surface ReachabilityAnalysis into `inferHTMLExtract` — either as a new parameter or via an enrichment field on `EndpointGroup`)
+- `internal/spec/spec.go` if a new helper is needed for the script_selector mapping (likely not — the mapping per R6 is small enough to live in specgen)
+- `internal/browsersniff/specgen_test.go` (add tests)
+
+**Approach:**
+- Add a parameter or field that carries the signature down from `ReachabilityAnalysis` into `inferHTMLExtract`. The implementer chooses the threading shape — pass it as an explicit param on `inferHTMLExtract`, OR attach it to the `EndpointGroup` during the capture-analysis phase before `inferHTMLExtract` is called per-group.
+- In `inferHTMLExtract`: if the ReachabilityAnalysis says `Mode == "html_scrape"` AND a signature is present, emit `HTMLExtract{Mode: HTMLExtractModeEmbeddedJSON, ScriptSelector: <mapped>}` instead of the current `page`/`links` logic.
+- Mapping table (KTD-9):
+ - `__NEXT_DATA__` → `script#__NEXT_DATA__`
+ - `__NUXT__` → `script#__NUXT__`
+ - `__APP_INITIAL_STATE__` → `script#__APP_INITIAL_STATE__`
+ - `state-view` → `script.state-view`
+ - `application/ld+json` → `script[type="application/ld+json"]`
+ - `window.__` → empty selector with a comment noting hand-tuning needed
+- For endpoint groups outside the SSR-bearing host's eTLD+1, fall back to the existing `page`/`links` logic — the html_scrape mode applies per-group, not globally. (Most browser-sniff captures center on a single target; cross-domain endpoints in the same HAR are rare and the existing logic handles them.)
+
+**Patterns to follow:** Existing `inferHTMLExtract` shape at [`internal/browsersniff/specgen.go:771-782`](../../internal/browsersniff/specgen.go#L771); existing usage of `HTMLExtractModePage` and `HTMLExtractModeLinks` for the conventional emission path.
+
+**Test scenarios:**
+- End-to-end positive: HAR fixture matching U2's Yandex-shape case (captcha-protected API + same-eTLD+1 SSR `state-view` HTML) → run the full capture-analysis flow → assert the emitted spec carries `html_extract.mode: embedded-json` with `script_selector: "script.state-view"`.
+- End-to-end positive: HAR with `__NEXT_DATA__` signature → assert emitted spec has `script_selector: "script#__NEXT_DATA__"`.
+- End-to-end positive: HAR with `__NUXT__` signature → `script#__NUXT__`.
+- End-to-end positive: HAR with `application/ld+json` signature → `script[type="application/ld+json"]`.
+- End-to-end control: HAR without the two-signal conjunction → emitted spec has no `html_extract` block (or has `mode: page` / `mode: links` per existing logic).
+- End-to-end regression: HAR currently producing `mode: page` → still produces `mode: page` (the new branch is gated on the html_scrape Reachability.Mode signal; absent that signal, the existing logic runs).
+- Edge case: HAR with `window.__` signature → emitted spec has `mode: embedded-json` with empty `script_selector` and a comment in the spec or warning in the output noting hand-tuning is required.
+
+**Verification:** `go test ./internal/browsersniff/` passes including new specgen tests. Manual smoke: run `printing-press` against a Yandex-shape HAR fixture, inspect the emitted spec, confirm `html_extract.mode: embedded-json` + the right script_selector.
+
+---
+
+### U4. Test fixtures + golden verification
+
+**Goal:** Add HAR test fixtures for the two-signal positive case + the negative regressions. Update goldens if any existing fixture retroactively matches the new condition.
+
+**Requirements:** R7.
+
+**Dependencies:** U1, U2, U3.
+
+**Files:**
+- `internal/browsersniff/testdata/` or wherever existing HAR fixtures live — add minimal Yandex-shape and Next.js-shape HARs.
+- `testdata/golden/expected/` — re-run `scripts/golden.sh verify` after U3 lands. If any existing fixture retroactively matches the new two-signal pattern, document the diff in the PR per AGENTS.md golden-update rule.
+
+**Approach:**
+- Construct two minimal HARs:
+ - **Yandex-shape:** one entry on `api.example.com` returning 403 with captcha headers (cookie redirect to `/showcaptcha`), one entry on `www.example.com` returning 200 with a 20 KB body containing `<script type="application/json" class="state-view">{...}</script>`.
+ - **Next.js-shape (same-host):** one entry on `example.com/api/foo` returning 403/captcha, one entry on `example.com/foo` returning 200 with a 20 KB body containing `<script id="__NEXT_DATA__" type="application/json">{...}</script>`.
+- Run `scripts/golden.sh verify`. If existing fixtures shift mode, inspect whether the new mode is actually correct for them (the prior `browser_required` may have been a too-heavy recommendation that html_scrape correctly replaces). Update goldens per the golden-update workflow if so.
+
+**Test scenarios:** N/A — verification unit. Test expectation: `scripts/golden.sh verify` passes after any fixture updates; documented in PR description.
+
+**Verification:** `scripts/golden.sh verify` green. PR description names any fixtures touched and the rationale.
+
+---
+
+## Scope Boundaries
+
+### In scope
+- Detector extension: add `state-view`, `__NUXT__`, `__APP_INITIAL_STATE__` + status/size floors (U1).
+- Signature surfacing on the protocol observation's `Details` map (U1).
+- Reachability mode selector promotion (U2) — captcha-tier protection per-API-entry + same-eTLD+1 SSR HTML sibling.
+- Specgen emission of `html_extract.mode: embedded-json` with the right `script_selector` per matched signature (U3).
+- Golden-fixture verification (U4).
+
+### Deferred to Follow-Up Work
+- New `internal/scrape/` generator template family — the rescope already ruled this out. The runtime + spec + (now) sniff layers handle html_scrape end-to-end without a parallel scrape package.
+- Multi-page pagination for state-blob extraction — already handled at the spec layer (food52 retro work).
+- Anti-bot fallback messaging in generated CLIs — generator concern.
+- Tightening the body-size floor based on real-world n>1 evidence — the 10 KB threshold is a conservative starting point; if real captures produce many false positives, retune in a follow-up.
+- Operator-facing override (forcing html_scrape mode despite a no-state-blob HAR, or vice versa) — out of scope; if needed, exposes as a flag in a follow-up.
+- Detection of inline JSON in HTML attributes (not script tags) — narrow shape, low signal density.
+
+### Outside this work's identity (tracked elsewhere)
+- Generator-template work for novel scraping patterns — out of scope; surface a new issue if a printed CLI needs it.
+- Public-library catalog re-evaluation of which entries should switch to html_scrape — operator-driven, not analyzer-driven.
+
+---
+
+## System-Wide Impact
+
+- **HAR analyzer:** signature now surfaced on `ProtocolObservation.Details`. New cascade branch in the mode selector. Two new helpers (`apiEntryHasCaptchaTierProtection`, `htmlEntryHasSSRStateBlobOnRegisteredDomain`, `sameRegisteredDomain`).
+- **Spec emission:** `inferHTMLExtract` gains a third branch (embedded-json) alongside the existing page/links logic. ReachabilityAnalysis now threaded into specgen.
+- **Generated CLIs:** for HARs matching the new pattern, the emitted spec now produces a goquery-based CLI shape instead of a Playwright shell. For all other HARs, no change.
+- **Existing tests:** the four prior reachability-mode tests + existing specgen page/links tests must continue to pass. Additions are guarded on the new signal; absent the signal, the existing paths run.
+- **Golden fixtures:** may shift if any existing test HAR retroactively matches the new two-signal pattern. U4 covers this.
+- **Dependencies:** adds `golang.org/x/net/publicsuffix` if not already pulled in.
+
+---
+
+## Risks
+
+| Risk | Likelihood | Impact | Mitigation |
+|---|---|---|---|
+| 10 KB body-size floor too low — captcha pages with embedded `__NEXT_DATA__` carrying ~10 KB of challenge content pass the floor | Medium | False-positive html_scrape promotion on captcha pages; CLI emits empty extraction | Captcha-page bodies are typically < 5 KB; 10 KB is above the typical challenge-page size. If real captures show false positives, raise the floor or add a semantic check (parse the embedded JSON and require >N keys) |
+| Signature priority when multiple signatures present | Low | Wrong script_selector picked when an HTML page has both `__NEXT_DATA__` and `application/ld+json` | KTD-1 / R3 fix the order; the typical Next.js shape has both but `__NEXT_DATA__` is the right pick |
+| eTLD+1 false positives across organizations sharing a registrar (`*.co.uk` shared hosts) | Very low | Cross-org false positive triggering html_scrape | The two-signal conjunction (captcha-tier API + state-blob HTML) is implausible across organizations; risk is theoretical, not practical |
+| Cross-host capture: API on `api-cdn.example.org` (different eTLD+1 from main site `example.com`) — Yandex-shape captures could still split this way for some sites | Low | False negative: routes to browser_required | Documented in Scope Boundaries as a known limitation; operator can hand-edit the spec |
+| Test fixtures coincidentally match the new pattern | Low | Existing reachability tests fail | U4 reviews + updates goldens per the AGENTS.md golden-update rule |
+| `EvidenceRef.EntryIndex` not consistently populated across all protection detectors | Low | New helper fails to find the originating entry | Trace through `detectProtections` during U2 implementation; if any path doesn't set EntryIndex, fix it as part of U2 |
+| `publicsuffix` dependency surfaces a build concern (not already vendored) | Low | go.mod / go.sum change needed | KTD-10 notes this; verify before commit |
+
+---
+
+## Verification Strategy
+
+- Unit tests per implementation unit (table-driven with `testify/assert` per AGENTS.md convention).
+- Integration test in U2: HAR fixture → analyzer → assert recommended mode.
+- End-to-end test in U3: HAR fixture → analyzer → spec emission → assert `html_extract.mode: embedded-json` + correct `script_selector`.
+- `scripts/golden.sh verify` after U3 (and again after U4 if fixtures shift).
+- `go test ./...` clean before PR.
+- `go vet ./...` and `golangci-lint run ./...` clean.
+
+---
+
+## Notes for the implementer
+
+- **AGENTS.md "machine vs printed CLI":** machine change. Analyzer + specgen are press internals; printed-CLI templates are not touched.
+- **AGENTS.md commit style:** scope is `cli` (the Go binary). Type is `feat`. Suggested title: `feat(cli): emit html_extract.mode: embedded-json from sniff pipeline on captcha + SSR state blob (#953)`.
+- **Issue claim is already in place** on #953 (assignment + "evaluating" comment + rescope comment posted by the planner).
+- **The rescope** documented at the top of #953's comment thread captures the architectural shift this plan now reflects (analyzer + specgen wiring is the real gap, not a new generator package).
+- **U3 is the largest unit by code volume.** Don't underestimate it because it's listed third. The threading from `ReachabilityAnalysis` into per-EndpointGroup `inferHTMLExtract` is the load-bearing change; U1 and U2 are preconditions that enable it.
diff --git a/go.mod b/go.mod
index 36c2ff64..34c174a1 100644
--- a/go.mod
+++ b/go.mod
@@ -10,6 +10,7 @@ require (
github.com/spf13/cobra v1.10.2
github.com/stretchr/testify v1.11.1
golang.org/x/mod v0.35.0
+ golang.org/x/net v0.53.0
golang.org/x/sync v0.20.0
golang.org/x/sys v0.43.0
golang.org/x/text v0.36.0
@@ -54,6 +55,5 @@ require (
github.com/xyproto/randomstring v1.2.0 // indirect
go.uber.org/mock v0.6.0 // indirect
golang.org/x/crypto v0.50.0 // indirect
- golang.org/x/net v0.53.0 // indirect
golang.org/x/tools v0.44.0 // indirect
)
diff --git a/internal/browsersniff/analysis.go b/internal/browsersniff/analysis.go
index 8e288039..df651c55 100644
--- a/internal/browsersniff/analysis.go
+++ b/internal/browsersniff/analysis.go
@@ -10,6 +10,8 @@ import (
"strings"
"time"
+ "golang.org/x/net/publicsuffix"
+
"github.com/mvanhorn/cli-printing-press/v4/internal/discovery"
"github.com/mvanhorn/cli-printing-press/v4/internal/spec"
)
@@ -264,6 +266,12 @@ type ReachabilityAnalysis struct {
Confidence float64 `json:"confidence"`
Reasons []string `json:"reasons,omitempty"`
Evidence []EvidenceRef `json:"evidence,omitempty"`
+
+ // HTMLExtractSignature is set when Mode == "html_scrape" and carries
+ // which SSR state-blob signature triggered the promotion (one of
+ // SSRSignature*). Downstream spec emission maps it to a script
+ // selector. Empty otherwise.
+ HTMLExtractSignature string `json:"html_extract_signature,omitempty"`
}
type ProtectionObservation struct {
@@ -674,8 +682,8 @@ func detectProtocols(entries []EnrichedEntry) []ProtocolObservation {
if strings.Contains(host, "firebase") || strings.Contains(path, "firestore") || strings.Contains(path, "google.firestore") {
addProtocol("firebase", 0.75, entry, index, "firebase/firestore host or path", nil)
}
- if isSSREmbeddedData(entry) {
- addProtocol("ssr_embedded_data", 0.85, entry, index, "HTML contains embedded structured data", nil)
+ if signature := detectSSREmbeddedData(entry); signature != "" {
+ addProtocol("ssr_embedded_data", 0.85, entry, index, "HTML contains embedded structured data", map[string]string{"signature": signature})
} else if strings.Contains(respType, "text/html") && strings.TrimSpace(entry.ResponseBody) != "" {
addProtocol("html_scrape", 0.55, entry, index, "HTML response observed", nil)
}
@@ -927,12 +935,129 @@ func classifyReachability(analysis *TrafficAnalysis, entries []EnrichedEntry) *R
}
}
+ // html_scrape overrides browser_required when an API entry carries
+ // a captcha-tier signal AND a same-eTLD+1 HTML sibling emits an SSR
+ // state blob — cheaper than spinning up a browser when the same data
+ // is reachable from a cold HTML fetch.
+ htmlExtractSignature := ""
+ if apiIdx, ok := findCaptchaTierProtectedAPIEntry(entries, analysis.Protections); ok {
+ refHost := extractHost(entries[apiIdx].URL)
+ if _, signature, ok := findSSRStateBlobEntryOnRegisteredDomain(entries, analysis.Protocols, refHost); ok {
+ mode = "html_scrape"
+ if confidence < 0.85 {
+ confidence = 0.85
+ }
+ reasons = []string{fmt.Sprintf("captcha-tier protection on API + same-registered-domain SSR state blob (signature: %s); html_scrape preferred over browser_required", signature)}
+ htmlExtractSignature = signature
+ }
+ }
+
return &ReachabilityAnalysis{
- Mode: mode,
- Confidence: confidence,
- Reasons: reasons,
- Evidence: evidence,
+ Mode: mode,
+ Confidence: confidence,
+ Reasons: reasons,
+ Evidence: evidence,
+ HTMLExtractSignature: htmlExtractSignature,
+ }
+}
+
+// captchaTierProtections are the labels that signal "JSON is unreachable
+// without a browser" — the html_scrape promotion fires only on these
+// (not on cloudflare/akamai/datadome/perimeterx, which can usually be
+// cleared with bearer tokens or session cookies via lighter modes).
+var captchaTierProtections = map[string]bool{
+ "captcha": true,
+ "bot_challenge": true,
+ "aws_waf": true,
+ "vercel_challenge": true,
+}
+
+// findCaptchaTierProtectedAPIEntry returns the index of the first
+// API-classified entry that itself surfaces a captcha-tier protection
+// signal. Walking via EvidenceRef.EntryIndex ensures the protection is
+// attributed to the API entry — a Cloudflare-fronted SSR HTML page
+// emitting a cloudflare signal from its own response headers does not
+// satisfy this check.
+func findCaptchaTierProtectedAPIEntry(entries []EnrichedEntry, protections []ProtectionObservation) (int, bool) {
+ for _, p := range protections {
+ if !captchaTierProtections[p.Label] {
+ continue
+ }
+ for _, ev := range p.Evidence {
+ idx := ev.EntryIndex
+ if idx < 0 || idx >= len(entries) {
+ continue
+ }
+ if entries[idx].Classification == "api" {
+ return idx, true
+ }
+ }
+ }
+ return -1, false
+}
+
+// findSSRStateBlobEntryOnRegisteredDomain returns the index and matched
+// signature of the first HTML entry on the same registered domain
+// (eTLD+1) as refHost that emits the ssr_embedded_data protocol. Uses
+// content-type to identify HTML entries because the classifier marks
+// HTML as "noise". Signature is re-detected per entry because the
+// protocol observation collapses multi-entry details, so the per-entry
+// signature is the only reliable source.
+func findSSRStateBlobEntryOnRegisteredDomain(entries []EnrichedEntry, protocols []ProtocolObservation, refHost string) (int, string, bool) {
+ var ssr *ProtocolObservation
+ for i := range protocols {
+ if protocols[i].Label == "ssr_embedded_data" {
+ ssr = &protocols[i]
+ break
+ }
+ }
+ if ssr == nil {
+ return -1, "", false
+ }
+ for _, ev := range ssr.Evidence {
+ idx := ev.EntryIndex
+ if idx < 0 || idx >= len(entries) {
+ continue
+ }
+ entry := entries[idx]
+ if !strings.Contains(strings.ToLower(entry.ResponseContentType), "html") {
+ continue
+ }
+ if !sameRegisteredDomain(extractHost(entry.URL), refHost) {
+ continue
+ }
+ signature := detectSSREmbeddedData(entry)
+ if signature == "" {
+ continue
+ }
+ return idx, signature, true
}
+ return -1, "", false
+}
+
+// sameRegisteredDomain compares two hosts at the eTLD+1 level so
+// subdomain splits like api.example.com / www.example.com qualify as
+// "same site." Literal-equality is checked first so private or unknown
+// TLDs (intranet hosts, raw IPs, .test/.local) still match themselves
+// even when publicsuffix can't resolve them.
+func sameRegisteredDomain(hostA, hostB string) bool {
+ a := strings.ToLower(strings.TrimSpace(hostA))
+ b := strings.ToLower(strings.TrimSpace(hostB))
+ if a == "" || b == "" {
+ return false
+ }
+ if a == b {
+ return true
+ }
+ aETLD, err := publicsuffix.EffectiveTLDPlusOne(a)
+ if err != nil {
+ return false
+ }
+ bETLD, err := publicsuffix.EffectiveTLDPlusOne(b)
+ if err != nil {
+ return false
+ }
+ return aETLD == bETLD
}
func hasAPIBrowserRenderedEntry(entries []EnrichedEntry) bool {
@@ -1328,12 +1453,68 @@ func containsJSONRPC(body string) bool {
return ok
}
-func isSSREmbeddedData(entry EnrichedEntry) bool {
+// ssrEmbeddedDataMinBodySize is the body-size floor below which an HTML
+// response with a state-blob marker is treated as an empty template or
+// challenge page rather than a real SSR payload.
+const ssrEmbeddedDataMinBodySize = 10_000
+
+// SSR state-blob signature labels surfaced by detectSSREmbeddedData and
+// consumed by spec emission to pick the right script selector. Exported
+// so the producer (this file) and consumer (reachability.go) share the
+// symbol set rather than duplicating string literals.
+const (
+ SSRSignatureNextData = "__NEXT_DATA__"
+ SSRSignatureNuxt = "__NUXT__"
+ SSRSignatureAppInitialState = "__APP_INITIAL_STATE__"
+ SSRSignatureStateView = "state-view"
+ SSRSignatureLDJSON = "application/ld+json"
+ SSRSignatureWindowPrefix = "window.__"
+)
+
+// ssrEmbeddedDataSignatures lists each substring the detector matches
+// against alongside its signature label. Order matters — earlier
+// entries win on multi-match because framework-specific signatures
+// imply a known DOM shape the generic markers don't. The state-view
+// and window.__ entries require shape-bearing context (quoted attr
+// value, state-named global) so analytics globals like window.__gtag
+// and CSS classes like state-view-port don't promote benign pages.
+var ssrEmbeddedDataSignatures = []struct {
+ substring string
+ label string
+}{
+ {"__next_data__", SSRSignatureNextData},
+ {"__nuxt__", SSRSignatureNuxt},
+ {"__app_initial_state__", SSRSignatureAppInitialState},
+ {`"state-view"`, SSRSignatureStateView},
+ {`'state-view'`, SSRSignatureStateView},
+ {"application/ld+json", SSRSignatureLDJSON},
+ {"window.__initial_state", SSRSignatureWindowPrefix},
+ {"window.__app_state", SSRSignatureWindowPrefix},
+ {"window.__apollo_state", SSRSignatureWindowPrefix},
+ {"window.__data__", SSRSignatureWindowPrefix},
+}
+
+// detectSSREmbeddedData returns the matched signature label when an
+// HTML response carries a server-rendered state blob, or "" when no
+// signature matches. Requires HTTP 2xx and body >= the size floor so
+// challenge pages and empty templates do not promote to html_scrape.
+func detectSSREmbeddedData(entry EnrichedEntry) string {
if !strings.Contains(strings.ToLower(entry.ResponseContentType), "html") {
- return false
+ return ""
+ }
+ if entry.ResponseStatus < 200 || entry.ResponseStatus >= 300 {
+ return ""
+ }
+ if len(entry.ResponseBody) < ssrEmbeddedDataMinBodySize {
+ return ""
}
body := strings.ToLower(entry.ResponseBody)
- return strings.Contains(body, "__next_data__") || strings.Contains(body, "application/ld+json") || strings.Contains(body, "window.__")
+ for _, sig := range ssrEmbeddedDataSignatures {
+ if strings.Contains(body, sig.substring) {
+ return sig.label
+ }
+ }
+ return ""
}
func looksBrowserRendered(entry EnrichedEntry) bool {
diff --git a/internal/browsersniff/analysis_test.go b/internal/browsersniff/analysis_test.go
index 9408a2cc..eda0ff05 100644
--- a/internal/browsersniff/analysis_test.go
+++ b/internal/browsersniff/analysis_test.go
@@ -124,7 +124,7 @@ func TestAnalyzeTraffic_DetectsProtocolProtectionAndWarningCategories(t *testing
URL: "https://app.example.com/explore",
ResponseStatus: 200,
ResponseContentType: "text/html",
- ResponseBody: `<html><script id="__NEXT_DATA__" type="application/json">{"props":{}}</script><div id="__next"></div></html>`,
+ ResponseBody: makeSSRPageWithNextData(`{"props":{}}`),
},
{
Method: "GET",
@@ -639,3 +639,632 @@ func TestEvidenceRef_MixedArrayInTrafficAnalysis(t *testing.T) {
assert.Contains(t, string(out), `"entry_index":0`,
"object-form evidence should re-emit as a JSON object")
}
+
+func TestDetectSSREmbeddedData(t *testing.T) {
+ tests := []struct {
+ name string
+ entry EnrichedEntry
+ expect string
+ }{
+ {
+ name: "next-data",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: makeSSRPage("__NEXT_DATA__", `{"props":{}}`),
+ },
+ expect: "__NEXT_DATA__",
+ },
+ {
+ name: "nuxt",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: makeSSRPage("__NUXT__", `{"state":{}}`),
+ },
+ expect: "__NUXT__",
+ },
+ {
+ name: "app-initial-state",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: makeSSRPage("__APP_INITIAL_STATE__", `{"data":{}}`),
+ },
+ expect: "__APP_INITIAL_STATE__",
+ },
+ {
+ name: "state-view-yandex",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: makeSSRPage("state-view", `{"map":{}}`),
+ },
+ expect: "state-view",
+ },
+ {
+ name: "ld-json",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: makeSSRPage("application/ld+json", `{"@type":"Product"}`),
+ },
+ expect: "application/ld+json",
+ },
+ {
+ name: "window-prefix",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: makeSSRPage("window.__", `{"initial":{}}`),
+ },
+ expect: "window.__",
+ },
+ {
+ name: "priority-next-data-wins-over-ld-json",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: `<html><head></head><body>` +
+ `<script id="__NEXT_DATA__" type="application/json">{}</script>` +
+ `<script type="application/ld+json">{}</script>` +
+ strings.Repeat("<!-- pad -->\n", 1000) + `</body></html>`,
+ },
+ expect: "__NEXT_DATA__",
+ },
+ {
+ name: "rejects-below-size-floor",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: `<html><script id="__NEXT_DATA__" type="application/json">{}</script></html>`,
+ },
+ expect: "",
+ },
+ {
+ name: "rejects-non-2xx-status",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 403,
+ ResponseBody: makeSSRPage("__NEXT_DATA__", `{"challenge":true}`),
+ },
+ expect: "",
+ },
+ {
+ name: "rejects-304-cached",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 304,
+ ResponseBody: makeSSRPage("__NEXT_DATA__", `{"cached":true}`),
+ },
+ expect: "",
+ },
+ {
+ name: "rejects-non-html-content-type",
+ entry: EnrichedEntry{
+ ResponseContentType: "application/json",
+ ResponseStatus: 200,
+ ResponseBody: makeSSRPage("__NEXT_DATA__", `{"shape":{}}`),
+ },
+ expect: "",
+ },
+ {
+ name: "no-signature-no-match",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: `<html><head></head><body><p>just a normal page</p>` +
+ strings.Repeat("<!-- pad -->\n", 1000) + `</body></html>`,
+ },
+ expect: "",
+ },
+ {
+ // window.__ matchers must require a state-shaped identifier so
+ // analytics globals like window.__gtag don't promote benign
+ // pages into the html_scrape path.
+ name: "rejects-window-prefix-on-analytics-globals",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: `<html><head><script>window.__gtag=function(){};window.__ga=1;</script></head><body><p>analytics page</p>` +
+ strings.Repeat("<!-- pad -->\n", 1000) + `</body></html>`,
+ },
+ expect: "",
+ },
+ {
+ // state-view matcher must require the script/attribute shape
+ // so CSS class names like state-view-port don't promote.
+ name: "rejects-state-view-as-css-fragment",
+ entry: EnrichedEntry{
+ ResponseContentType: "text/html",
+ ResponseStatus: 200,
+ ResponseBody: `<html><head></head><body><div class="state-view-port-container"></div>` +
+ strings.Repeat("<!-- pad -->\n", 1000) + `</body></html>`,
+ },
+ expect: "",
+ },
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ got := detectSSREmbeddedData(tt.entry)
+ assert.Equal(t, tt.expect, got)
+ })
+ }
+}
+
+func TestApplyReachabilityDefaults_HTMLScrapeEmitsEmbeddedJSON(t *testing.T) {
+ tests := []struct {
+ name string
+ signature string
+ expectedSelector string
+ }{
+ {"next-data", "__NEXT_DATA__", "script#__NEXT_DATA__"},
+ {"nuxt", "__NUXT__", "script#__NUXT__"},
+ {"app-initial-state", "__APP_INITIAL_STATE__", "script#__APP_INITIAL_STATE__"},
+ {"state-view", "state-view", "script.state-view"},
+ {"ld-json", "application/ld+json", `script[type="application/ld+json"]`},
+ {"window-prefix-leaves-selector-empty", "window.__", ""},
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ apiSpec := &spec.APISpec{
+ Resources: map[string]spec.Resource{
+ "pages": {
+ Endpoints: map[string]spec.Endpoint{
+ "get_index": {
+ Method: "GET",
+ Path: "/",
+ HTMLExtract: &spec.HTMLExtract{
+ Mode: spec.HTMLExtractModePage,
+ LinkPrefixes: []string{"/listing/"},
+ Limit: 50,
+ },
+ },
+ },
+ },
+ },
+ }
+ analysis := &TrafficAnalysis{
+ Reachability: &ReachabilityAnalysis{
+ Mode: "html_scrape",
+ HTMLExtractSignature: tt.signature,
+ },
+ }
+ ApplyReachabilityDefaults(apiSpec, analysis)
+
+ ep := apiSpec.Resources["pages"].Endpoints["get_index"]
+ require.NotNil(t, ep.HTMLExtract)
+ assert.Equal(t, spec.HTMLExtractModeEmbeddedJSON, ep.HTMLExtract.Mode)
+ assert.Equal(t, tt.expectedSelector, ep.HTMLExtract.ScriptSelector)
+ assert.Nil(t, ep.HTMLExtract.LinkPrefixes, "link prefixes should clear when promoting to embedded-json")
+ })
+ }
+}
+
+func TestApplyReachabilityDefaults_HTMLScrapeLeavesNonHTMLEndpointsAlone(t *testing.T) {
+ apiSpec := &spec.APISpec{
+ Resources: map[string]spec.Resource{
+ "api": {
+ Endpoints: map[string]spec.Endpoint{
+ "get_foo": {
+ Method: "GET",
+ Path: "/api/foo",
+ // No HTMLExtract — this is a JSON endpoint.
+ },
+ },
+ },
+ },
+ }
+ analysis := &TrafficAnalysis{
+ Reachability: &ReachabilityAnalysis{
+ Mode: "html_scrape",
+ HTMLExtractSignature: "__NEXT_DATA__",
+ },
+ }
+ ApplyReachabilityDefaults(apiSpec, analysis)
+
+ ep := apiSpec.Resources["api"].Endpoints["get_foo"]
+ assert.Nil(t, ep.HTMLExtract, "non-HTML endpoints stay untouched")
+}
+
+func TestApplyReachabilityDefaults_HTMLScrapePromotesNestedSubResources(t *testing.T) {
+ apiSpec := &spec.APISpec{
+ Resources: map[string]spec.Resource{
+ "pages": {
+ Endpoints: map[string]spec.Endpoint{
+ "get_index": {
+ Method: "GET",
+ Path: "/",
+ HTMLExtract: &spec.HTMLExtract{
+ Mode: spec.HTMLExtractModePage,
+ LinkPrefixes: []string{"/parent/"},
+ },
+ },
+ },
+ SubResources: map[string]spec.Resource{
+ "items": {
+ Endpoints: map[string]spec.Endpoint{
+ "get_item": {
+ Method: "GET",
+ Path: "/parent/{id}",
+ HTMLExtract: &spec.HTMLExtract{
+ Mode: spec.HTMLExtractModePage,
+ LinkPrefixes: []string{"/parent/"},
+ },
+ },
+ },
+ },
+ },
+ },
+ },
+ }
+ analysis := &TrafficAnalysis{
+ Reachability: &ReachabilityAnalysis{
+ Mode: "html_scrape",
+ HTMLExtractSignature: "__NEXT_DATA__",
+ },
+ }
+ ApplyReachabilityDefaults(apiSpec, analysis)
+
+ parent := apiSpec.Resources["pages"].Endpoints["get_index"]
+ assert.Equal(t, spec.HTMLExtractModeEmbeddedJSON, parent.HTMLExtract.Mode)
+ assert.Equal(t, "script#__NEXT_DATA__", parent.HTMLExtract.ScriptSelector)
+
+ child := apiSpec.Resources["pages"].SubResources["items"].Endpoints["get_item"]
+ require.NotNil(t, child.HTMLExtract, "sub-resource endpoint should still have HTMLExtract")
+ assert.Equal(t, spec.HTMLExtractModeEmbeddedJSON, child.HTMLExtract.Mode, "sub-resource endpoint must also promote")
+ assert.Equal(t, "script#__NEXT_DATA__", child.HTMLExtract.ScriptSelector)
+ assert.Nil(t, child.HTMLExtract.LinkPrefixes, "sub-resource link prefixes should clear")
+}
+
+func TestApplyReachabilityDefaults_HTMLScrapeNotAppliedWhenSignatureEmpty(t *testing.T) {
+ apiSpec := &spec.APISpec{
+ Resources: map[string]spec.Resource{
+ "pages": {
+ Endpoints: map[string]spec.Endpoint{
+ "get_index": {
+ Method: "GET",
+ Path: "/",
+ HTMLExtract: &spec.HTMLExtract{
+ Mode: spec.HTMLExtractModePage,
+ },
+ },
+ },
+ },
+ },
+ }
+ analysis := &TrafficAnalysis{
+ Reachability: &ReachabilityAnalysis{
+ Mode: "html_scrape",
+ // Signature empty — should not promote
+ },
+ }
+ ApplyReachabilityDefaults(apiSpec, analysis)
+
+ ep := apiSpec.Resources["pages"].Endpoints["get_index"]
+ assert.Equal(t, spec.HTMLExtractModePage, ep.HTMLExtract.Mode, "without a signature, mode stays page")
+}
+
+func TestClassifyReachability_HTMLScrapePromotion(t *testing.T) {
+ tests := []struct {
+ name string
+ entries []EnrichedEntry
+ expectedMode string
+ expectedSignature string
+ }{
+ {
+ name: "yandex-shape-cross-subdomain-promotes",
+ entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.yandex.example.com/maps/api/search",
+ ResponseStatus: 403,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"error":"captcha required","redirect":"/showcaptcha?retpath=/"}`,
+ },
+ {
+ Method: "GET",
+ URL: "https://www.yandex.example.com/maps/org/foo/12345/",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("state-view", `{"org":{"name":"Cafe Bistro"}}`),
+ },
+ },
+ expectedMode: "html_scrape",
+ expectedSignature: "state-view",
+ },
+ {
+ name: "same-host-promotes-with-next-data",
+ entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://example.com/api/foo",
+ ResponseStatus: 403,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"error":"captcha required"}`,
+ },
+ {
+ Method: "GET",
+ URL: "https://example.com/foo",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("__NEXT_DATA__", `{"foo":1}`),
+ },
+ },
+ expectedMode: "html_scrape",
+ expectedSignature: "__NEXT_DATA__",
+ },
+ {
+ name: "different-registered-domain-does-not-promote",
+ entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/foo",
+ ResponseStatus: 403,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"error":"captcha required"}`,
+ },
+ {
+ Method: "GET",
+ URL: "https://other-site.com/foo",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("__NEXT_DATA__", `{}`),
+ },
+ },
+ expectedMode: "browser_required",
+ expectedSignature: "",
+ },
+ {
+ name: "cloudflare-only-stays-on-clearance-mode",
+ entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/foo",
+ ResponseStatus: 403,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"error":"blocked"}`,
+ ResponseHeaders: map[string]string{"Server": "cloudflare", "CF-Ray": "abc"},
+ },
+ {
+ Method: "GET",
+ URL: "https://www.example.com/foo",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("__NEXT_DATA__", `{}`),
+ },
+ },
+ // Cloudflare is not in the captcha tier — promotion does
+ // not fire. The existing browser_http branch handles this
+ // case (cloudflare on the API entry routes to browser_http).
+ expectedMode: "browser_http",
+ expectedSignature: "",
+ },
+ {
+ name: "no-protection-no-promotion",
+ entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://example.com/api/foo",
+ ResponseStatus: 200,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"foo":1}`,
+ },
+ {
+ Method: "GET",
+ URL: "https://example.com/foo",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("__NEXT_DATA__", `{}`),
+ },
+ },
+ expectedMode: "standard_http",
+ expectedSignature: "",
+ },
+ {
+ name: "captcha-without-ssr-sibling-stays-browser-required",
+ entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://example.com/api/foo",
+ ResponseStatus: 403,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"error":"captcha required"}`,
+ },
+ },
+ expectedMode: "browser_required",
+ expectedSignature: "",
+ },
+ {
+ name: "aws-waf-captcha-tier-promotes",
+ entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/foo",
+ ResponseStatus: 403,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"error":"blocked by AWS WAF","captcha":"required"}`,
+ ResponseHeaders: map[string]string{"x-amzn-RequestId": "abc"},
+ },
+ {
+ Method: "GET",
+ URL: "https://www.example.com/foo",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("__NUXT__", `{}`),
+ },
+ },
+ expectedMode: "html_scrape",
+ expectedSignature: "__NUXT__",
+ },
+ {
+ name: "vercel-challenge-captcha-tier-promotes",
+ entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/foo",
+ ResponseStatus: 403,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"error":"vercel challenge"}`,
+ ResponseHeaders: map[string]string{"x-vercel-mitigated": "challenge"},
+ },
+ {
+ Method: "GET",
+ URL: "https://www.example.com/foo",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("__APP_INITIAL_STATE__", `{}`),
+ },
+ },
+ expectedMode: "html_scrape",
+ expectedSignature: "__APP_INITIAL_STATE__",
+ },
+ {
+ name: "bot-challenge-captcha-tier-promotes",
+ entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/foo",
+ ResponseStatus: 403,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"error":"managed challenge"}`,
+ ResponseHeaders: map[string]string{"cf-mitigated": "challenge"},
+ },
+ {
+ Method: "GET",
+ URL: "https://www.example.com/foo",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("application/ld+json", `{"@type":"Product"}`),
+ },
+ },
+ expectedMode: "html_scrape",
+ expectedSignature: "application/ld+json",
+ },
+ {
+ // Ordering guard: matching SSR entry is iterated before a
+ // non-matching cross-domain SSR entry. The selected signature
+ // must come from the entry that satisfies same-eTLD+1, not
+ // from the last SSR entry seen by the protocol scanner.
+ name: "multi-ssr-signature-attributed-to-matching-entry",
+ entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://api.example.com/foo",
+ ResponseStatus: 403,
+ ResponseContentType: "application/json",
+ ResponseBody: `{"error":"captcha required"}`,
+ },
+ {
+ Method: "GET",
+ URL: "https://www.example.com/foo",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("__NEXT_DATA__", `{}`),
+ },
+ {
+ Method: "GET",
+ URL: "https://unrelated-site.com/bar",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("__NUXT__", `{}`),
+ },
+ },
+ expectedMode: "html_scrape",
+ expectedSignature: "__NEXT_DATA__",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ capture := &EnrichedCapture{Entries: tt.entries}
+ analysis, err := AnalyzeTraffic(capture)
+ require.NoError(t, err)
+ require.NotNil(t, analysis.Reachability)
+ assert.Equal(t, tt.expectedMode, analysis.Reachability.Mode)
+ assert.Equal(t, tt.expectedSignature, analysis.Reachability.HTMLExtractSignature)
+ })
+ }
+}
+
+func TestSameRegisteredDomain(t *testing.T) {
+ tests := []struct {
+ a, b string
+ want bool
+ }{
+ {"example.com", "example.com", true},
+ {"api.example.com", "www.example.com", true},
+ {"api.example.com", "example.com", true},
+ {"example.com", "other-site.com", false},
+ {"api.example.co.uk", "www.example.co.uk", true},
+ {"example.co.uk", "example.com", false},
+ {"", "example.com", false},
+ {"EXAMPLE.com", "example.COM", true},
+ }
+ for _, tt := range tests {
+ t.Run(tt.a+"_"+tt.b, func(t *testing.T) {
+ assert.Equal(t, tt.want, sameRegisteredDomain(tt.a, tt.b))
+ })
+ }
+}
+
+func TestDetectProtocols_SSREmbeddedDataSurfacesSignatureInDetails(t *testing.T) {
+ capture := &EnrichedCapture{
+ Entries: []EnrichedEntry{
+ {
+ Method: "GET",
+ URL: "https://example.com/maps/org/foo/12345/",
+ ResponseStatus: 200,
+ ResponseContentType: "text/html",
+ ResponseBody: makeSSRPage("state-view", `{"org":{"name":"Cafe Bistro"}}`),
+ },
+ },
+ }
+ analysis, err := AnalyzeTraffic(capture)
+ require.NoError(t, err)
+
+ var ssr *ProtocolObservation
+ for i := range analysis.Protocols {
+ if analysis.Protocols[i].Label == "ssr_embedded_data" {
+ ssr = &analysis.Protocols[i]
+ break
+ }
+ }
+ require.NotNil(t, ssr, "ssr_embedded_data protocol must be observed")
+ assert.Equal(t, "state-view", ssr.Details["signature"])
+}
+
+// makeSSRPage builds an HTML body carrying the requested SSR signature
+// marker and pads past ssrEmbeddedDataMinBodySize. Real SSR captures are
+// 10KB+; the filler simulates that without embedding a real page. The
+// __NEXT_DATA__ shape also includes the `id="__next"` mount node so
+// `looksBrowserRendered` fires on the same fixture (matches typical
+// Next.js output).
+func makeSSRPage(signature, payload string) string {
+ var inner string
+ switch signature {
+ case "__NEXT_DATA__":
+ inner = `<script id="__NEXT_DATA__" type="application/json">` + payload + `</script><div id="__next"></div>`
+ case "__NUXT__":
+ inner = `<script>window.__NUXT__=` + payload + `</script>`
+ case "__APP_INITIAL_STATE__":
+ inner = `<script id="__APP_INITIAL_STATE__" type="application/json">` + payload + `</script>`
+ case "state-view":
+ inner = `<script type="application/json" class="state-view">` + payload + `</script>`
+ case "application/ld+json":
+ inner = `<script type="application/ld+json">` + payload + `</script>`
+ case "window.__":
+ inner = `<script>window.__INITIAL_STATE__=` + payload + `</script>`
+ default:
+ inner = payload
+ }
+ filler := strings.Repeat("<!-- ssr -->\n", 1000)
+ return `<html><head></head><body>` + inner + filler + `</body></html>`
+}
+
+// makeSSRPageWithNextData is the most common shorthand for tests that
+// only need a Next.js-shape body.
+func makeSSRPageWithNextData(payload string) string {
+ return makeSSRPage("__NEXT_DATA__", payload)
+}
diff --git a/internal/browsersniff/reachability.go b/internal/browsersniff/reachability.go
index 0509a5fc..9f3bc7e9 100644
--- a/internal/browsersniff/reachability.go
+++ b/internal/browsersniff/reachability.go
@@ -13,6 +13,10 @@ func ApplyReachabilityDefaults(apiSpec *spec.APISpec, analysis *TrafficAnalysis)
return
}
+ if analysis.Reachability.Mode == "html_scrape" && analysis.Reachability.HTMLExtractSignature != "" {
+ applyHTMLScrapeExtractionDefaults(apiSpec, analysis.Reachability.HTMLExtractSignature)
+ }
+
if analysis.Reachability.Mode == "browser_http" || analysis.Reachability.Mode == "browser_clearance_http" || analysis.Reachability.Mode == "browser_required" {
if apiSpec.HTTPTransport == "" {
switch analysis.Reachability.Mode {
@@ -145,3 +149,55 @@ func hasRequiredInput(endpoint spec.Endpoint) bool {
}
return false
}
+
+// scriptSelectorForSignature maps the SSR state-blob signature to the
+// runtime extractor's "tag" / "tag#id" / "tag.class" grammar. Empty
+// for window.__-style inline state — the runtime falls back to
+// DefaultEmbeddedJSONScriptSelector and the operator hand-tunes.
+func scriptSelectorForSignature(signature string) string {
+ switch signature {
+ case SSRSignatureNextData:
+ return spec.DefaultEmbeddedJSONScriptSelector
+ case SSRSignatureNuxt:
+ return "script#__NUXT__"
+ case SSRSignatureAppInitialState:
+ return "script#__APP_INITIAL_STATE__"
+ case SSRSignatureStateView:
+ return "script.state-view"
+ case SSRSignatureLDJSON:
+ return `script[type="application/ld+json"]`
+ default:
+ return ""
+ }
+}
+
+// applyHTMLScrapeExtractionDefaults promotes endpoints that already
+// declared HTMLExtract to embedded-json mode with a signature-derived
+// selector. Endpoints without HTMLExtract are skipped because the
+// signature alone is not enough to fabricate extraction config for
+// endpoints that never asked for HTML extraction.
+func applyHTMLScrapeExtractionDefaults(apiSpec *spec.APISpec, signature string) {
+ selector := scriptSelectorForSignature(signature)
+ for resourceName, resource := range apiSpec.Resources {
+ updated := promoteHTMLExtractInResource(resource, selector)
+ apiSpec.Resources[resourceName] = updated
+ }
+}
+
+func promoteHTMLExtractInResource(resource spec.Resource, selector string) spec.Resource {
+ for name, endpoint := range resource.Endpoints {
+ if endpoint.HTMLExtract == nil {
+ continue
+ }
+ endpoint.HTMLExtract.Mode = spec.HTMLExtractModeEmbeddedJSON
+ if selector != "" {
+ endpoint.HTMLExtract.ScriptSelector = selector
+ }
+ endpoint.HTMLExtract.LinkPrefixes = nil
+ resource.Endpoints[name] = endpoint
+ }
+ for subName, sub := range resource.SubResources {
+ resource.SubResources[subName] = promoteHTMLExtractInResource(sub, selector)
+ }
+ return resource
+}
← 424199a7 fix(ci): drop cancel-in-progress from pr-title workflow (#10
·
back to Cli Printing Press
·
fix(cli): preserve unknown-shape records in compactListField dc6fe879 →