← back to Cli Printing Press
feat(cli): add probe-reachability for no-browser challenge classification (#331)
09dccfb9e3124cac54be3ad40603a36b5da9079b · 2026-04-27 00:05:14 -0700 · Trevin Chow
* feat(cli): add probe-reachability subcommand
Classify a URL's reachability with stdlib HTTP and Surf (Chrome TLS
fingerprint) probes, emitting one of standard_http | browser_http |
browser_clearance_http | unknown — the same vocabulary as
internal/browsersniff. Lets the printing-press skill decide silently
whether a printed CLI needs Surf transport vs a Chrome attach for
clearance-cookie capture, instead of routing the user through
transport-tier menus.
Diagnostic only; exit 0 regardless of classification. Flags: --json,
--timeout, --probe-only stdlib|surf for debug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(skills): route browser-sniff escalation through probe-reachability
Phase 1.7's Direct HTTP challenge rule used to jump straight from a
bot-protection 429 to "open or attach to Chrome" prompts, even when a
Surf-shaped HTTP request would have cleared the challenge with no setup.
Run probe-reachability silently first; classify mode; settle runtime
without exposing transport tiers as a peer menu to the user. Phase 1.9's
reachability gate gets a fallback probe and a matrix row for
"Surf cleared → PASS." Cardinal Rule 4 in browser-sniff-capture.md
mirrors the framing: intent menus are fine, transport-tier menus
("Surf-only vs clearance-cookie vs HAR") are forbidden because the user
has no context to pick. Chrome consent fires at the moment of attach,
not bundled into the offer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Files touched
M go.modM go.sumA internal/cli/probe_reachability.goA internal/cli/probe_reachability_test.goM internal/cli/root.goA internal/reachability/detect.goA internal/reachability/detect_test.goA internal/reachability/probe.goA internal/reachability/probe_test.goA internal/reachability/render.goA internal/reachability/render_test.goA internal/reachability/types.goM skills/printing-press/SKILL.mdM skills/printing-press/references/browser-sniff-capture.md
Diff
commit 09dccfb9e3124cac54be3ad40603a36b5da9079b
Author: Trevin Chow <trevin@trevinchow.com>
Date: Mon Apr 27 00:05:14 2026 -0700
feat(cli): add probe-reachability for no-browser challenge classification (#331)
* feat(cli): add probe-reachability subcommand
Classify a URL's reachability with stdlib HTTP and Surf (Chrome TLS
fingerprint) probes, emitting one of standard_http | browser_http |
browser_clearance_http | unknown — the same vocabulary as
internal/browsersniff. Lets the printing-press skill decide silently
whether a printed CLI needs Surf transport vs a Chrome attach for
clearance-cookie capture, instead of routing the user through
transport-tier menus.
Diagnostic only; exit 0 regardless of classification. Flags: --json,
--timeout, --probe-only stdlib|surf for debug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(skills): route browser-sniff escalation through probe-reachability
Phase 1.7's Direct HTTP challenge rule used to jump straight from a
bot-protection 429 to "open or attach to Chrome" prompts, even when a
Surf-shaped HTTP request would have cleared the challenge with no setup.
Run probe-reachability silently first; classify mode; settle runtime
without exposing transport tiers as a peer menu to the user. Phase 1.9's
reachability gate gets a fallback probe and a matrix row for
"Surf cleared → PASS." Cardinal Rule 4 in browser-sniff-capture.md
mirrors the framing: intent menus are fine, transport-tier menus
("Surf-only vs clearance-cookie vs HAR") are forbidden because the user
has no context to pick. Chrome consent fires at the moment of attach,
not bundled into the offer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
go.mod | 15 ++
go.sum | 34 ++++
internal/cli/probe_reachability.go | 69 +++++++
internal/cli/probe_reachability_test.go | 41 ++++
internal/cli/root.go | 1 +
internal/reachability/detect.go | 145 ++++++++++++++
internal/reachability/detect_test.go | 160 +++++++++++++++
internal/reachability/probe.go | 219 +++++++++++++++++++++
internal/reachability/probe_test.go | 191 ++++++++++++++++++
internal/reachability/render.go | 109 ++++++++++
internal/reachability/render_test.go | 90 +++++++++
internal/reachability/types.go | 87 ++++++++
skills/printing-press/SKILL.md | 40 +++-
.../references/browser-sniff-capture.md | 2 +-
14 files changed, 1198 insertions(+), 5 deletions(-)
diff --git a/go.mod b/go.mod
index ed932adb..3faa3192 100644
--- a/go.mod
+++ b/go.mod
@@ -4,6 +4,7 @@ go 1.26.1
require (
github.com/dave/dst v0.27.4
+ github.com/enetx/surf v1.0.198
github.com/getkin/kin-openapi v0.133.0
github.com/mark3labs/mcp-go v0.47.0
github.com/spf13/cobra v1.10.2
@@ -15,22 +16,36 @@ require (
)
require (
+ github.com/andybalholm/brotli v1.2.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
+ github.com/enetx/g v1.0.223 // indirect
+ github.com/enetx/http v1.0.28 // indirect
+ github.com/enetx/http2 v1.0.26 // indirect
+ github.com/enetx/http3 v1.0.7 // indirect
+ github.com/enetx/iter v0.0.0-20250912135656-f1583323588f // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/google/jsonschema-go v0.4.2 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
+ github.com/klauspost/compress v1.18.5 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/oasdiff/yaml v0.0.0-20250309154309-f31be36b4037 // indirect
github.com/oasdiff/yaml3 v0.0.0-20250309153720-d2182401db90 // indirect
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
+ github.com/quic-go/qpack v0.6.0 // indirect
+ github.com/quic-go/quic-go v0.59.0 // indirect
+ github.com/refraction-networking/utls v1.8.3-0.20260301010127-aa6edf4b11af // indirect
github.com/spf13/cast v1.7.1 // indirect
github.com/spf13/pflag v1.0.9 // indirect
github.com/woodsbury/decimal128 v1.3.0 // indirect
+ github.com/wzshiming/socks5 v0.7.0 // indirect
github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
+ golang.org/x/crypto v0.48.0 // indirect
+ golang.org/x/net v0.50.0 // indirect
+ golang.org/x/sys v0.41.0 // indirect
golang.org/x/tools v0.42.0 // indirect
)
diff --git a/go.sum b/go.sum
index 372bc271..a40557a4 100644
--- a/go.sum
+++ b/go.sum
@@ -1,3 +1,5 @@
+github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
+github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/dave/dst v0.27.4 h1:d+EVnOZmphH+lUEXq9rit4GjsFSKJ3AhfRWf7eobTps=
github.com/dave/dst v0.27.4/go.mod h1:jHh6EOibnHgcUW3WjKHisiooEkYwqpHLBSX1iOBhEyc=
@@ -5,6 +7,18 @@ github.com/dave/jennifer v1.5.0 h1:HmgPN93bVDpkQyYbqhCHj5QlgvUkvEOzMyEvKLgCRrg=
github.com/dave/jennifer v1.5.0/go.mod h1:4MnyiFIlZS3l5tSDn8VnzE6ffAhYBMB2SZntBsZGUok=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/enetx/g v1.0.223 h1:9J8y76uiBLvNlMDyaLyCBUWELKNaHsRST+93Zxm2+No=
+github.com/enetx/g v1.0.223/go.mod h1:dxOHnNkhdZkwwOvgbJKcniq187TqyLO3DuTJpmk1tLQ=
+github.com/enetx/http v1.0.28 h1:IaNSSDFlAVVdHnYhNIR9wAN7GY4TWL/kkvYC3jOaueY=
+github.com/enetx/http v1.0.28/go.mod h1:1f4mytfF/SfjATEJnynpwGS6aa1ALjb8DtmYgFVblY0=
+github.com/enetx/http2 v1.0.26 h1:wy3lYGVwnIUY4Q+gyPPQCJ1a+BMXD1B7Unpyc/Csrxc=
+github.com/enetx/http2 v1.0.26/go.mod h1:t54ex5HIS8V1+2j6cvEOv6umlrHsbUPFKQ54nYB58Nk=
+github.com/enetx/http3 v1.0.7 h1:daFhveKBtv8rRallCjaHErzzSHIrq07ovoSvVkvhcMM=
+github.com/enetx/http3 v1.0.7/go.mod h1:sqpVGZ9F1/wCiW6sjBUS2errKAh3SUYn6VlWE7LL6KM=
+github.com/enetx/iter v0.0.0-20250912135656-f1583323588f h1:GUW+4AWfECIEJ9oAxgEAVGCpaozMCjRiUYnuR6Q0bCQ=
+github.com/enetx/iter v0.0.0-20250912135656-f1583323588f/go.mod h1:oMZN8hGLUpi7QBlMEUqailocNy0NFAO/7Lu+Nwh9HMM=
+github.com/enetx/surf v1.0.198 h1:TJkyEyy5M+GnLZlGKmByeXwG7K2vv7F5L+0SgwlDu7g=
+github.com/enetx/surf v1.0.198/go.mod h1:BtLmZDYAny66azybFr9UdFVnwy8WRV4FTAzElsd7bvE=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/getkin/kin-openapi v0.133.0 h1:pJdmNohVIJ97r4AUFtEXRXwESr8b0bD721u/Tz6k8PQ=
@@ -25,6 +39,8 @@ github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
+github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
+github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -43,6 +59,12 @@ github.com/perimeterx/marshmallow v1.1.5 h1:a2LALqQ1BlHM8PZblsDdidgv1mWi1DgC2UmX
github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
+github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
+github.com/quic-go/quic-go v0.59.0 h1:OLJkp1Mlm/aS7dpKgTc6cnpynnD2Xg7C1pwL6vy/SAw=
+github.com/quic-go/quic-go v0.59.0/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU=
+github.com/refraction-networking/utls v1.8.3-0.20260301010127-aa6edf4b11af h1:er2acxbi3N1nvEq6HXHUAR1nTWEJmQfqiGR8EVT9rfs=
+github.com/refraction-networking/utls v1.8.3-0.20260301010127-aa6edf4b11af/go.mod h1:jkSOEkLqn+S/jtpEHPOsVv/4V4EVnelwbMQl4vCWXAM=
github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -60,13 +82,25 @@ github.com/ugorji/go/codec v1.2.7 h1:YPXUKf7fYbp/y8xloBqZOw2qaVggbfwMlI8WM3wZUJ0
github.com/ugorji/go/codec v1.2.7/go.mod h1:WGN1fab3R1fzQlVQTkfxVtIBhWDRqOviHU95kRgeqEY=
github.com/woodsbury/decimal128 v1.3.0 h1:8pffMNWIlC0O5vbyHWFZAt5yWvWcrHA+3ovIIjVWss0=
github.com/woodsbury/decimal128 v1.3.0/go.mod h1:C5UTmyTjW3JftjUFzOVhC20BEQa2a4ZKOB5I6Zjb+ds=
+github.com/wzshiming/socks5 v0.7.0 h1:euJ+U48WrvVngi+opC8vAnpZ5sK12y1C2hPvb1f48Rg=
+github.com/wzshiming/socks5 v0.7.0/go.mod h1:BvCAqlzocQN5xwLjBZDBbvWlrx8sCYSSbHEOf2wZgT0=
+github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
+github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4=
github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT040gmv5BQTMR2HP4=
+go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
+go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
+golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
+golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
golang.org/x/mod v0.33.0 h1:tHFzIWbBifEmbwtGz65eaWyGiGZatSrT9prnU8DbVL8=
golang.org/x/mod v0.33.0/go.mod h1:swjeQEj+6r7fODbD2cqrnje9PnziFuw4bmLbBZFrQ5w=
+golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60=
+golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
+golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
golang.org/x/tools v0.42.0 h1:uNgphsn75Tdz5Ji2q36v/nsFSfR/9BRFvqhGBaJGd5k=
diff --git a/internal/cli/probe_reachability.go b/internal/cli/probe_reachability.go
new file mode 100644
index 00000000..8ee3fad7
--- /dev/null
+++ b/internal/cli/probe_reachability.go
@@ -0,0 +1,69 @@
+package cli
+
+import (
+ "context"
+ "fmt"
+ "time"
+
+ "github.com/mvanhorn/cli-printing-press/v2/internal/reachability"
+ "github.com/spf13/cobra"
+)
+
+// newProbeReachabilityCmd classifies a URL's reachability without opening
+// a browser. Used by the printing-press skill (Phase 1.7) to decide
+// whether browser-sniff needs to escalate to a real Chrome attach for
+// clearance-cookie capture, or whether Surf alone is enough.
+//
+// Diagnostic only — exit 0 regardless of classification. Exit non-zero
+// only on input-validation failure.
+func newProbeReachabilityCmd() *cobra.Command {
+ var (
+ asJSON bool
+ timeout time.Duration
+ probeOnly string
+ )
+
+ cmd := &cobra.Command{
+ Use: "probe-reachability <url>",
+ Short: "Classify a URL's reachability with no-browser probes",
+ Long: `Runs stdlib HTTP and Surf (Chrome TLS fingerprint) probes against the URL
+and emits a reachability mode matching internal/browsersniff vocabulary:
+standard_http, browser_http, browser_clearance_http, or unknown.
+
+Diagnostic only. Exit 0 regardless of result; exit non-zero only on
+input-validation failure.`,
+ Example: ` printing-press probe-reachability https://food52.com/
+ printing-press probe-reachability https://api.github.com/zen --json
+ printing-press probe-reachability https://example.com --probe-only surf --timeout 5s`,
+ Args: cobra.ExactArgs(1),
+ RunE: func(cmd *cobra.Command, args []string) error {
+ only := reachability.ProbeOnly(probeOnly)
+ switch only {
+ case reachability.ProbeOnlyNone, reachability.ProbeOnlyStdlib, reachability.ProbeOnlySurf:
+ default:
+ return &ExitError{Code: ExitInputError, Err: fmt.Errorf("invalid --probe-only value: %q (want stdlib or surf)", probeOnly)}
+ }
+
+ ctx, cancel := context.WithTimeout(cmd.Context(), timeout*2+5*time.Second)
+ defer cancel()
+
+ result, err := reachability.Probe(ctx, args[0], reachability.Options{
+ Timeout: timeout,
+ ProbeOnly: only,
+ })
+ if err != nil {
+ return &ExitError{Code: ExitInputError, Err: err}
+ }
+
+ if asJSON {
+ return reachability.RenderJSON(cmd.OutOrStdout(), result)
+ }
+ return reachability.RenderTable(cmd.OutOrStdout(), result)
+ },
+ }
+
+ cmd.Flags().BoolVar(&asJSON, "json", false, "emit JSON instead of a human-readable table")
+ cmd.Flags().DurationVar(&timeout, "timeout", 15*time.Second, "per-probe timeout")
+ cmd.Flags().StringVar(&probeOnly, "probe-only", "", "restrict the ladder to one rung (stdlib|surf); debug only")
+ return cmd
+}
diff --git a/internal/cli/probe_reachability_test.go b/internal/cli/probe_reachability_test.go
new file mode 100644
index 00000000..8a815566
--- /dev/null
+++ b/internal/cli/probe_reachability_test.go
@@ -0,0 +1,41 @@
+package cli
+
+import (
+ "bytes"
+ "strings"
+ "testing"
+)
+
+func TestProbeReachabilityCmd_InvalidProbeOnlyValue(t *testing.T) {
+ cmd := newProbeReachabilityCmd()
+ var out bytes.Buffer
+ cmd.SetOut(&out)
+ cmd.SetErr(&out)
+ cmd.SetArgs([]string{"--probe-only", "garbage", "https://example.com"})
+
+ err := cmd.Execute()
+ if err == nil {
+ t.Fatalf("expected error for invalid --probe-only value")
+ }
+ if !strings.Contains(err.Error(), "probe-only") {
+ t.Errorf("expected error to mention probe-only; got: %v", err)
+ }
+}
+
+func TestProbeReachabilityCmd_HelpFlags(t *testing.T) {
+ cmd := newProbeReachabilityCmd()
+ var out bytes.Buffer
+ cmd.SetOut(&out)
+ cmd.SetErr(&out)
+ cmd.SetArgs([]string{"--help"})
+
+ if err := cmd.Execute(); err != nil {
+ t.Fatalf("execute --help: %v", err)
+ }
+ help := out.String()
+ for _, want := range []string{"--json", "--timeout", "--probe-only"} {
+ if !strings.Contains(help, want) {
+ t.Errorf("help output missing %q:\n%s", want, help)
+ }
+ }
+}
diff --git a/internal/cli/root.go b/internal/cli/root.go
index bf9836ee..3244f8ad 100644
--- a/internal/cli/root.go
+++ b/internal/cli/root.go
@@ -60,6 +60,7 @@ func Execute() error {
rootCmd.AddCommand(newWorkflowVerifyCmd())
rootCmd.AddCommand(newLockCmd())
rootCmd.AddCommand(newMCPAuditCmd())
+ rootCmd.AddCommand(newProbeReachabilityCmd())
rootCmd.AddCommand(newSchemaCmd())
return rootCmd.Execute()
diff --git a/internal/reachability/detect.go b/internal/reachability/detect.go
new file mode 100644
index 00000000..b2eeb213
--- /dev/null
+++ b/internal/reachability/detect.go
@@ -0,0 +1,145 @@
+package reachability
+
+import (
+ "net/http"
+ "strings"
+)
+
+// Protection is one detected bot-protection signal in a probe response.
+// Label vocabulary matches internal/browsersniff/analysis.go's
+// ProtectionObservation labels so probe and post-capture analyzer agree.
+type Protection struct {
+ Label string
+ Evidence string
+}
+
+// classifyResponse returns protection signals detected in the response.
+// bodySnippet is expected to be a small (~4KB) lowercased read of the
+// response body — full body reads would be wasteful for large pages.
+func classifyResponse(status int, headers http.Header, bodySnippet string) []Protection {
+ var out []Protection
+ body := strings.ToLower(bodySnippet)
+ h := lowerHeaders(headers)
+ server := h["server"]
+
+ if v := h["cf-mitigated"]; v == "challenge" {
+ out = append(out, Protection{Label: "bot_challenge", Evidence: "cf-mitigated: challenge"})
+ }
+ if v := h["x-vercel-mitigated"]; v == "challenge" {
+ out = append(out, Protection{Label: "bot_challenge", Evidence: "x-vercel-mitigated: challenge"})
+ out = append(out, Protection{Label: "vercel_challenge", Evidence: "x-vercel-mitigated: challenge"})
+ }
+ if h["x-vercel-challenge-token"] != "" {
+ out = append(out, Protection{Label: "vercel_challenge", Evidence: "x-vercel-challenge-token present"})
+ }
+ if strings.Contains(body, "vercel security checkpoint") {
+ out = append(out, Protection{Label: "vercel_challenge", Evidence: "Vercel Security Checkpoint page"})
+ }
+
+ if h["aws-waf-token"] != "" || hasHeaderPrefix(h, "x-amzn-waf") ||
+ strings.Contains(body, "awswaf") || strings.Contains(body, "aws-waf") {
+ out = append(out, Protection{Label: "aws_waf", Evidence: "AWS WAF marker"})
+ }
+
+ // CDN fingerprints (cf-ray, server: cloudflare, x-akamai-transformed)
+ // are NOT protection signals on their own — Cloudflare and Akamai front
+ // huge swaths of the internet, and a normal 200 served through their
+ // edge looks identical to a challenge response except for the body and
+ // status. Only fire CDN-as-protection when:
+ // (a) status is 4xx/5xx — error response from the CDN, OR
+ // (b) body contains a challenge marker (cf-chl, "just a moment",
+ // "checking your browser", "ddos protection by cloudflare").
+ // DataDome and PerimeterX header presence stays a strong signal — those
+ // products only ship as bot mitigation, not as plain CDN.
+ cfFingerprint := strings.Contains(server, "cloudflare") || h["cf-ray"] != ""
+ cfChallengeBody := strings.Contains(body, "cf-chl") ||
+ strings.Contains(body, "just a moment") ||
+ strings.Contains(body, "checking your browser") ||
+ strings.Contains(body, "ddos protection by cloudflare")
+ akamaiFingerprint := h["x-akamai-transformed"] != ""
+
+ switch {
+ case cfChallengeBody:
+ out = append(out, Protection{Label: "cloudflare", Evidence: "Cloudflare challenge marker in body"})
+ case cfFingerprint && status >= 400:
+ out = append(out, Protection{Label: "cloudflare", Evidence: "Cloudflare error response"})
+ case akamaiFingerprint && status >= 400:
+ out = append(out, Protection{Label: "akamai", Evidence: "Akamai error response"})
+ case h["x-datadome"] != "" || strings.Contains(body, "datadome"):
+ out = append(out, Protection{Label: "datadome", Evidence: "DataDome marker"})
+ case strings.Contains(body, "perimeterx") || strings.Contains(body, "_px"):
+ out = append(out, Protection{Label: "perimeterx", Evidence: "PerimeterX marker"})
+ }
+
+ if strings.Contains(body, "recaptcha") || strings.Contains(body, "hcaptcha") ||
+ strings.Contains(body, "g-recaptcha") {
+ out = append(out, Protection{Label: "captcha", Evidence: "CAPTCHA widget"})
+ }
+
+ if (status == 403 || status == 429) && len(out) == 0 {
+ ct := strings.ToLower(headers.Get("Content-Type"))
+ if strings.Contains(ct, "html") || strings.Contains(body, "access denied") ||
+ strings.Contains(body, "too many requests") {
+ out = append(out, Protection{Label: "protected_web", Evidence: "403/429 HTML or access-denied"})
+ }
+ }
+
+ return out
+}
+
+// isClear returns true when the response is a successful, non-protected
+// reach. 2xx, 3xx, and 401/403/404 without protection markers all count —
+// they prove the URL is reachable; auth and routing are downstream concerns.
+func isClear(status int, protections []Protection) bool {
+ if len(protections) > 0 {
+ return false
+ }
+ if status >= 200 && status < 400 {
+ return true
+ }
+ // 401 and 404 are reachable: server reached us, transport works,
+ // the response is just auth-gated or wrong path. 403 without
+ // protection markers is the ambiguous one — we treat it as clear,
+ // matching analysis.go's logic that 403 + no protection is just an
+ // authz gate rather than a transport problem.
+ if status == 401 || status == 403 || status == 404 {
+ return true
+ }
+ return false
+}
+
+func lowerHeaders(h http.Header) map[string]string {
+ out := make(map[string]string, len(h))
+ for k, v := range h {
+ if len(v) == 0 {
+ continue
+ }
+ out[strings.ToLower(k)] = strings.ToLower(v[0])
+ }
+ return out
+}
+
+func hasHeaderPrefix(h map[string]string, prefix string) bool {
+ for k := range h {
+ if strings.HasPrefix(k, prefix) {
+ return true
+ }
+ }
+ return false
+}
+
+func protectionsToEvidence(protections []Protection) []string {
+ if len(protections) == 0 {
+ return nil
+ }
+ seen := map[string]bool{}
+ var out []string
+ for _, p := range protections {
+ if seen[p.Evidence] {
+ continue
+ }
+ seen[p.Evidence] = true
+ out = append(out, p.Evidence)
+ }
+ return out
+}
diff --git a/internal/reachability/detect_test.go b/internal/reachability/detect_test.go
new file mode 100644
index 00000000..904bdcf4
--- /dev/null
+++ b/internal/reachability/detect_test.go
@@ -0,0 +1,160 @@
+package reachability
+
+import (
+ "net/http"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+)
+
+func TestClassifyResponse(t *testing.T) {
+ tests := []struct {
+ name string
+ status int
+ headers http.Header
+ body string
+ wantLabels []string
+ emptyResult bool
+ }{
+ {
+ name: "plain 200",
+ status: 200,
+ headers: http.Header{"Content-Type": {"application/json"}},
+ body: `{"ok": true}`,
+ emptyResult: true,
+ },
+ {
+ name: "vercel challenge header",
+ status: 429,
+ headers: http.Header{
+ "X-Vercel-Mitigated": {"challenge"},
+ "Content-Type": {"text/html"},
+ },
+ body: "<html>Vercel Security Checkpoint</html>",
+ wantLabels: []string{"bot_challenge", "vercel_challenge", "vercel_challenge"},
+ },
+ {
+ name: "cloudflare cf-mitigated",
+ status: 403,
+ headers: http.Header{
+ "Cf-Mitigated": {"challenge"},
+ "Cf-Ray": {"abc123"},
+ "Server": {"cloudflare"},
+ },
+ body: "<html>Just a moment...</html>",
+ wantLabels: []string{"bot_challenge", "cloudflare"},
+ },
+ {
+ // Regression: cf-ray + server:cloudflare on a normal 200 means
+ // Cloudflare CDN, not a challenge. allrecipes.com (and tens of
+ // millions of other sites) front through Cloudflare; treating
+ // every CDN-fronted 200 as a protection signal would force
+ // every Cloudflare-protected site through the cookie-clearance
+ // flow even when Surf has already cleared the actual challenge.
+ name: "cloudflare CDN on 200 is not a protection signal",
+ status: 200,
+ headers: http.Header{
+ "Cf-Ray": {"abc123"},
+ "Server": {"cloudflare"},
+ "Content-Type": {"text/html"},
+ },
+ body: "<html>recipe content with no challenge markers</html>",
+ emptyResult: true,
+ },
+ {
+ name: "cloudflare CDN on error response is a protection signal",
+ status: 503,
+ headers: http.Header{
+ "Cf-Ray": {"abc123"},
+ "Server": {"cloudflare"},
+ "Content-Type": {"text/html"},
+ },
+ body: "<html>error</html>",
+ wantLabels: []string{"cloudflare"},
+ },
+ {
+ name: "aws waf token",
+ status: 403,
+ headers: http.Header{
+ "X-Amzn-Waf-Action": {"challenge"},
+ "Content-Type": {"text/html"},
+ },
+ body: "<html>access denied</html>",
+ wantLabels: []string{"aws_waf"},
+ },
+ {
+ name: "datadome via header",
+ status: 403,
+ headers: http.Header{
+ "X-Datadome": {"protected"},
+ },
+ body: "datadome challenge",
+ wantLabels: []string{"datadome"},
+ },
+ {
+ name: "captcha widget",
+ status: 200,
+ headers: http.Header{
+ "Content-Type": {"text/html"},
+ },
+ body: `<div class="g-recaptcha"></div>`,
+ wantLabels: []string{"captcha"},
+ },
+ {
+ name: "403 html with no markers becomes protected_web",
+ status: 403,
+ headers: http.Header{
+ "Content-Type": {"text/html"},
+ },
+ body: "<html>access denied</html>",
+ wantLabels: []string{"protected_web"},
+ },
+ {
+ name: "401 no markers does not classify as protection",
+ status: 401,
+ headers: http.Header{
+ "Content-Type": {"application/json"},
+ "Www-Authenticate": {"Bearer"},
+ },
+ body: `{"error": "unauthorized"}`,
+ emptyResult: true,
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ got := classifyResponse(tt.status, tt.headers, tt.body)
+ if tt.emptyResult {
+ assert.Empty(t, got, "expected no protections")
+ return
+ }
+ labels := make([]string, len(got))
+ for i, p := range got {
+ labels[i] = p.Label
+ }
+ assert.ElementsMatch(t, tt.wantLabels, labels)
+ })
+ }
+}
+
+func TestIsClear(t *testing.T) {
+ tests := []struct {
+ status int
+ protections []Protection
+ want bool
+ }{
+ {200, nil, true},
+ {301, nil, true},
+ {401, nil, true},
+ {403, nil, true}, // 403 without protection markers — assumed authz, not bot
+ {404, nil, true},
+ {429, nil, false},
+ {500, nil, false},
+ {200, []Protection{{Label: "vercel_challenge"}}, false},
+ {403, []Protection{{Label: "cloudflare"}}, false},
+ }
+ for _, tt := range tests {
+ got := isClear(tt.status, tt.protections)
+ assert.Equal(t, tt.want, got, "status=%d protections=%v", tt.status, tt.protections)
+ }
+}
diff --git a/internal/reachability/probe.go b/internal/reachability/probe.go
new file mode 100644
index 00000000..460bb62b
--- /dev/null
+++ b/internal/reachability/probe.go
@@ -0,0 +1,219 @@
+package reachability
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "io"
+ "net/http"
+ "strings"
+ "time"
+
+ "github.com/enetx/surf"
+ "github.com/mvanhorn/cli-printing-press/v2/internal/version"
+)
+
+const (
+ defaultTimeout = 15 * time.Second
+ bodyReadCap = 4 * 1024 // bytes scanned for protection markers
+)
+
+// Options configures Probe.
+type Options struct {
+ // Timeout per individual probe rung. Default 15s.
+ Timeout time.Duration
+
+ // ProbeOnly restricts the ladder to a single rung. Debug only — leave
+ // empty for normal use. When set, Result.Partial is true.
+ ProbeOnly ProbeOnly
+
+ // UserAgent overrides the stdlib probe's User-Agent. Default is
+ // "printing-press/<version> (probe-reachability)". Surf probes always
+ // use the Chrome UA from the impersonation profile and ignore this.
+ UserAgent string
+
+ // HTTPClientFactory is a test seam. When non-nil, both probe rungs
+ // resolve their *http.Client through this function instead of building
+ // a real stdlib or Surf client. Production callers leave it nil.
+ HTTPClientFactory func(Transport, time.Duration) (*http.Client, error)
+}
+
+// Probe runs the reachability ladder against url and returns a Result.
+// Returns a non-nil error only on input-validation failure (empty URL,
+// invalid ProbeOnly value); transport errors are reported per-rung in
+// Result.Probes and folded into Mode classification.
+func Probe(ctx context.Context, url string, opts Options) (*Result, error) {
+ if strings.TrimSpace(url) == "" {
+ return nil, errors.New("url is required")
+ }
+ switch opts.ProbeOnly {
+ case ProbeOnlyNone, ProbeOnlyStdlib, ProbeOnlySurf:
+ default:
+ return nil, fmt.Errorf("invalid probe-only value: %q", opts.ProbeOnly)
+ }
+ if opts.Timeout <= 0 {
+ opts.Timeout = defaultTimeout
+ }
+ if opts.UserAgent == "" {
+ opts.UserAgent = "printing-press/" + version.Get() + " (probe-reachability)"
+ }
+
+ result := &Result{URL: url, Probes: []ProbeResult{}}
+ partial := opts.ProbeOnly != ProbeOnlyNone
+
+ runStdlib := opts.ProbeOnly == ProbeOnlyNone || opts.ProbeOnly == ProbeOnlyStdlib
+ runSurf := opts.ProbeOnly == ProbeOnlyNone || opts.ProbeOnly == ProbeOnlySurf
+
+ if runStdlib {
+ pr := runRung(ctx, url, TransportStdlib, opts)
+ result.Probes = append(result.Probes, pr)
+ if pr.Status > 0 && len(pr.Evidence) == 0 && statusIsClear(pr.Status) {
+ result.Mode = ModeStandardHTTP
+ result.Confidence = 0.95
+ result.Recommendation = Recommendation{
+ Runtime: ModeStandardHTTP,
+ Rationale: "Plain stdlib HTTP returned a non-error response; no special transport needed.",
+ }
+ result.Partial = partial
+ return result, nil
+ }
+ }
+
+ if runSurf {
+ pr := runRung(ctx, url, TransportSurfChrome, opts)
+ result.Probes = append(result.Probes, pr)
+ if pr.Status > 0 && len(pr.Evidence) == 0 && statusIsClear(pr.Status) {
+ result.Mode = ModeBrowserHTTP
+ result.Confidence = 0.85
+ result.Recommendation = Recommendation{
+ Runtime: ModeBrowserHTTP,
+ Rationale: "Surf with Chrome TLS fingerprint cleared the protection that blocked plain stdlib HTTP.",
+ }
+ result.Partial = partial
+ return result, nil
+ }
+ }
+
+ classifyFailure(result, partial)
+ return result, nil
+}
+
+// statusIsClear is the status-only half of isClear. Used after evidence
+// has already been ruled empty by the caller.
+func statusIsClear(status int) bool {
+ return isClear(status, nil)
+}
+
+// classifyFailure mutates result to set Mode/Confidence/Recommendation when
+// no rung returned a clear pass.
+func classifyFailure(result *Result, partial bool) {
+ result.Partial = partial
+
+ hasProtection := false
+ allTransportErrors := true
+ for _, pr := range result.Probes {
+ if len(pr.Evidence) > 0 {
+ hasProtection = true
+ }
+ if pr.Status > 0 {
+ allTransportErrors = false
+ }
+ }
+
+ switch {
+ case allTransportErrors:
+ result.Mode = ModeUnknown
+ result.Confidence = 0
+ result.Recommendation = Recommendation{
+ Runtime: ModeUnknown,
+ Rationale: "All probes failed at the transport layer (DNS, timeout, connection). Cannot recommend a runtime without reaching the URL.",
+ }
+ case hasProtection:
+ // Stdlib + Surf both saw bot-protection signals. The probe cannot
+ // distinguish "needs clearance cookie" from "needs page-context
+ // execution" — both produce identical bare-HTTP signatures.
+ // Recommend escalation to a real browser-sniff capture, which
+ // will produce the authoritative classification.
+ result.Mode = ModeBrowserClearanceHTTP
+ result.Confidence = 0.6
+ result.Recommendation = Recommendation{
+ Runtime: ModeBrowserClearanceHTTP,
+ NeedsBrowserCapture: true,
+ NeedsClearanceCookie: true,
+ Rationale: "Stdlib and Surf both received bot-protection signals. A browser-sniff capture is needed to determine whether a clearance cookie alone suffices or whether live page-context execution is required.",
+ }
+ default:
+ // 5xx on both rungs, or other non-clear, non-protected responses.
+ result.Mode = ModeUnknown
+ result.Confidence = 0.3
+ result.Recommendation = Recommendation{
+ Runtime: ModeUnknown,
+ Rationale: "Probes returned non-clear, non-protected responses (e.g., 5xx). Site may be down or behind an unrecognized gate.",
+ }
+ }
+}
+
+// runRung executes one probe rung and returns its ProbeResult.
+func runRung(ctx context.Context, url string, transport Transport, opts Options) ProbeResult {
+ pr := ProbeResult{Transport: transport}
+ start := time.Now()
+ defer func() {
+ pr.ElapsedMS = time.Since(start).Milliseconds()
+ }()
+
+ client, err := buildClient(transport, opts)
+ if err != nil {
+ pr.Error = fmt.Sprintf("building %s client: %v", transport, err)
+ return pr
+ }
+
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+ if err != nil {
+ pr.Error = fmt.Sprintf("building request: %v", err)
+ return pr
+ }
+ if transport == TransportStdlib {
+ req.Header.Set("User-Agent", opts.UserAgent)
+ req.Header.Set("Accept", "text/html,application/json;q=0.9,*/*;q=0.8")
+ }
+
+ resp, err := client.Do(req)
+ if err != nil {
+ pr.Error = fmt.Sprintf("request: %v", err)
+ pr.ElapsedMS = time.Since(start).Milliseconds()
+ return pr
+ }
+ defer func() { _ = resp.Body.Close() }()
+
+ pr.Status = resp.StatusCode
+ pr.ContentType = resp.Header.Get("Content-Type")
+
+ body, _ := io.ReadAll(io.LimitReader(resp.Body, bodyReadCap))
+ protections := classifyResponse(resp.StatusCode, resp.Header, string(body))
+ pr.Evidence = protectionsToEvidence(protections)
+ pr.ElapsedMS = time.Since(start).Milliseconds()
+ return pr
+}
+
+// buildClient constructs the *http.Client for one probe rung. Tests
+// override via opts.HTTPClientFactory; production callers get the real
+// stdlib client or Surf with Chrome impersonation.
+func buildClient(transport Transport, opts Options) (*http.Client, error) {
+ if opts.HTTPClientFactory != nil {
+ return opts.HTTPClientFactory(transport, opts.Timeout)
+ }
+ switch transport {
+ case TransportStdlib:
+ return &http.Client{Timeout: opts.Timeout}, nil
+ case TransportSurfChrome:
+ client, err := surf.NewClient().Builder().Impersonate().Chrome().Timeout(opts.Timeout).Build().Result()
+ if err != nil {
+ return nil, fmt.Errorf("build surf client: %w", err)
+ }
+ std := client.Std()
+ std.Timeout = opts.Timeout
+ return std, nil
+ default:
+ return nil, fmt.Errorf("unknown transport: %q", transport)
+ }
+}
diff --git a/internal/reachability/probe_test.go b/internal/reachability/probe_test.go
new file mode 100644
index 00000000..44567879
--- /dev/null
+++ b/internal/reachability/probe_test.go
@@ -0,0 +1,191 @@
+package reachability
+
+import (
+ "context"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+// fakeFactory returns a per-transport http.Client whose RoundTripper
+// dispatches to a fake response builder. Lets tests simulate
+// "stdlib gets challenged, surf clears" without a real network.
+func fakeFactory(stdlib, surf func(*http.Request) (*http.Response, error)) func(Transport, time.Duration) (*http.Client, error) {
+ return func(t Transport, _ time.Duration) (*http.Client, error) {
+ switch t {
+ case TransportStdlib:
+ return &http.Client{Transport: roundTripFn(stdlib)}, nil
+ case TransportSurfChrome:
+ return &http.Client{Transport: roundTripFn(surf)}, nil
+ }
+ return nil, nil
+ }
+}
+
+type roundTripFn func(*http.Request) (*http.Response, error)
+
+func (f roundTripFn) RoundTrip(req *http.Request) (*http.Response, error) {
+ return f(req)
+}
+
+func vercelChallenge(_ *http.Request) (*http.Response, error) {
+ return &http.Response{
+ StatusCode: 429,
+ Header: http.Header{
+ "X-Vercel-Mitigated": {"challenge"},
+ "Content-Type": {"text/html"},
+ "Server": {"Vercel"},
+ },
+ Body: respBody("<html>Vercel Security Checkpoint</html>"),
+ }, nil
+}
+
+func ok200(_ *http.Request) (*http.Response, error) {
+ return &http.Response{
+ StatusCode: 200,
+ Header: http.Header{"Content-Type": {"text/html"}},
+ Body: respBody("<html>recipe content</html>"),
+ }, nil
+}
+
+func cloudflareChallenge(_ *http.Request) (*http.Response, error) {
+ return &http.Response{
+ StatusCode: 403,
+ Header: http.Header{
+ "Cf-Mitigated": {"challenge"},
+ "Cf-Ray": {"abc123"},
+ "Server": {"cloudflare"},
+ "Content-Type": {"text/html"},
+ },
+ Body: respBody("<html>Just a moment...</html>"),
+ }, nil
+}
+
+func respBody(s string) *bodyCloser {
+ return &bodyCloser{r: strings.NewReader(s)}
+}
+
+type bodyCloser struct {
+ r *strings.Reader
+}
+
+func (b *bodyCloser) Read(p []byte) (int, error) { return b.r.Read(p) }
+func (b *bodyCloser) Close() error { return nil }
+
+func TestProbe_StdlibPasses(t *testing.T) {
+ result, err := Probe(context.Background(), "https://example.com", Options{
+ HTTPClientFactory: fakeFactory(ok200, vercelChallenge),
+ })
+ require.NoError(t, err)
+ assert.Equal(t, ModeStandardHTTP, result.Mode)
+ assert.Len(t, result.Probes, 1, "should stop at first clear pass")
+ assert.Equal(t, TransportStdlib, result.Probes[0].Transport)
+ assert.False(t, result.Partial)
+ assert.False(t, result.Recommendation.NeedsBrowserCapture)
+}
+
+func TestProbe_StdlibChallengedSurfClears_FoodS52Pattern(t *testing.T) {
+ // The exact case from the bug report: Vercel passive challenge that
+ // stdlib trips on but Surf with Chrome TLS clears. No clearance cookie
+ // needed.
+ result, err := Probe(context.Background(), "https://food52.com", Options{
+ HTTPClientFactory: fakeFactory(vercelChallenge, ok200),
+ })
+ require.NoError(t, err)
+ assert.Equal(t, ModeBrowserHTTP, result.Mode)
+ assert.Len(t, result.Probes, 2, "stdlib fails, surf passes — both rungs ran")
+ assert.False(t, result.Recommendation.NeedsBrowserCapture, "surf alone is enough — no Chrome attach needed")
+ assert.False(t, result.Recommendation.NeedsClearanceCookie)
+ assert.NotEmpty(t, result.Probes[0].Evidence, "stdlib probe should record vercel evidence")
+ assert.Empty(t, result.Probes[1].Evidence, "surf probe should be clean")
+}
+
+func TestProbe_BothChallenged_RecommendsBrowserCapture(t *testing.T) {
+ result, err := Probe(context.Background(), "https://example.com", Options{
+ HTTPClientFactory: fakeFactory(cloudflareChallenge, cloudflareChallenge),
+ })
+ require.NoError(t, err)
+ assert.Equal(t, ModeBrowserClearanceHTTP, result.Mode)
+ assert.True(t, result.Recommendation.NeedsBrowserCapture)
+ assert.True(t, result.Recommendation.NeedsClearanceCookie)
+}
+
+func TestProbe_TransportFailure_Unknown(t *testing.T) {
+ failTransport := func(_ *http.Request) (*http.Response, error) {
+ return nil, &net500Error{}
+ }
+ result, err := Probe(context.Background(), "https://nope.invalid", Options{
+ HTTPClientFactory: fakeFactory(failTransport, failTransport),
+ })
+ require.NoError(t, err)
+ assert.Equal(t, ModeUnknown, result.Mode)
+ assert.NotEmpty(t, result.Probes[0].Error)
+}
+
+type net500Error struct{}
+
+func (net500Error) Error() string { return "simulated DNS/network failure" }
+
+func TestProbe_ProbeOnlyStdlib_PartialFlag(t *testing.T) {
+ result, err := Probe(context.Background(), "https://example.com", Options{
+ ProbeOnly: ProbeOnlyStdlib,
+ HTTPClientFactory: fakeFactory(ok200, vercelChallenge),
+ })
+ require.NoError(t, err)
+ assert.Equal(t, ModeStandardHTTP, result.Mode)
+ assert.True(t, result.Partial, "--probe-only must mark result as partial")
+ assert.Len(t, result.Probes, 1)
+ assert.Equal(t, TransportStdlib, result.Probes[0].Transport)
+}
+
+func TestProbe_ProbeOnlySurf_SkipsStdlib(t *testing.T) {
+ stdlibCalled := false
+ stdlib := func(_ *http.Request) (*http.Response, error) {
+ stdlibCalled = true
+ return ok200(nil)
+ }
+ result, err := Probe(context.Background(), "https://example.com", Options{
+ ProbeOnly: ProbeOnlySurf,
+ HTTPClientFactory: fakeFactory(stdlib, ok200),
+ })
+ require.NoError(t, err)
+ assert.False(t, stdlibCalled, "--probe-only surf must not run stdlib probe")
+ assert.Equal(t, ModeBrowserHTTP, result.Mode)
+ assert.True(t, result.Partial)
+}
+
+func TestProbe_InvalidProbeOnly(t *testing.T) {
+ _, err := Probe(context.Background(), "https://example.com", Options{
+ ProbeOnly: ProbeOnly("nonsense"),
+ })
+ require.Error(t, err)
+}
+
+func TestProbe_EmptyURL(t *testing.T) {
+ _, err := Probe(context.Background(), "", Options{})
+ require.Error(t, err)
+}
+
+func TestProbe_RealHTTPTestServer(t *testing.T) {
+ // End-to-end against a real local HTTP server, exercising the full
+ // stdlib path (no factory override).
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(200)
+ _, _ = w.Write([]byte(`{"ok":true}`))
+ }))
+ defer srv.Close()
+
+ result, err := Probe(context.Background(), srv.URL, Options{
+ ProbeOnly: ProbeOnlyStdlib, // skip surf for hermetic test (no real TLS cert)
+ Timeout: 5 * time.Second,
+ })
+ require.NoError(t, err)
+ assert.Equal(t, ModeStandardHTTP, result.Mode)
+ assert.Equal(t, 200, result.Probes[0].Status)
+}
diff --git a/internal/reachability/render.go b/internal/reachability/render.go
new file mode 100644
index 00000000..bc9b17a1
--- /dev/null
+++ b/internal/reachability/render.go
@@ -0,0 +1,109 @@
+package reachability
+
+import (
+ "encoding/json"
+ "fmt"
+ "io"
+ "strings"
+)
+
+// RenderJSON writes the result to w as indented JSON.
+func RenderJSON(w io.Writer, result *Result) error {
+ enc := json.NewEncoder(w)
+ enc.SetIndent("", " ")
+ return enc.Encode(result)
+}
+
+// RenderTable writes a human-readable table summarizing the probe rungs and
+// final classification. Columns: Transport, Status, Elapsed, Evidence/Error.
+func RenderTable(w io.Writer, result *Result) error {
+ headers := []string{"Transport", "Status", "Elapsed", "Notes"}
+ rows := make([][]string, 0, len(result.Probes))
+ for _, pr := range result.Probes {
+ status := "-"
+ if pr.Status > 0 {
+ status = fmt.Sprintf("%d", pr.Status)
+ }
+ notes := pr.Error
+ if notes == "" {
+ if len(pr.Evidence) == 0 {
+ notes = "no protection signals"
+ } else {
+ notes = strings.Join(pr.Evidence, "; ")
+ }
+ }
+ rows = append(rows, []string{
+ string(pr.Transport),
+ status,
+ fmt.Sprintf("%dms", pr.ElapsedMS),
+ notes,
+ })
+ }
+
+ widths := make([]int, len(headers))
+ for i, h := range headers {
+ widths[i] = len(h)
+ }
+ for _, row := range rows {
+ for i, cell := range row {
+ if len(cell) > widths[i] {
+ widths[i] = len(cell)
+ }
+ }
+ }
+
+ if err := writeRow(w, widths, headers); err != nil {
+ return err
+ }
+ sep := make([]string, len(headers))
+ for i := range sep {
+ sep[i] = strings.Repeat("-", widths[i])
+ }
+ if err := writeRow(w, widths, sep); err != nil {
+ return err
+ }
+ for _, row := range rows {
+ if err := writeRow(w, widths, row); err != nil {
+ return err
+ }
+ }
+
+ partial := ""
+ if result.Partial {
+ partial = " (partial — --probe-only restricted the ladder; ignore mode for skill decisions)"
+ }
+ if _, err := fmt.Fprintf(w, "\nMode: %s (confidence %.2f)%s\n", result.Mode, result.Confidence, partial); err != nil {
+ return err
+ }
+ if _, err := fmt.Fprintf(w, "Recommendation: runtime=%s, needs_browser_capture=%v, needs_clearance_cookie=%v\n",
+ result.Recommendation.Runtime,
+ result.Recommendation.NeedsBrowserCapture,
+ result.Recommendation.NeedsClearanceCookie); err != nil {
+ return err
+ }
+ if result.Recommendation.Rationale != "" {
+ if _, err := fmt.Fprintf(w, "Rationale: %s\n", result.Recommendation.Rationale); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+func writeRow(w io.Writer, widths []int, cells []string) error {
+ var b strings.Builder
+ for i, cell := range cells {
+ if i > 0 {
+ b.WriteString(" ")
+ }
+ b.WriteString(cell)
+ if i < len(cells)-1 {
+ pad := widths[i] - len(cell)
+ if pad > 0 {
+ b.WriteString(strings.Repeat(" ", pad))
+ }
+ }
+ }
+ b.WriteString("\n")
+ _, err := io.WriteString(w, b.String())
+ return err
+}
diff --git a/internal/reachability/render_test.go b/internal/reachability/render_test.go
new file mode 100644
index 00000000..0091e90f
--- /dev/null
+++ b/internal/reachability/render_test.go
@@ -0,0 +1,90 @@
+package reachability
+
+import (
+ "bytes"
+ "encoding/json"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+func TestRenderJSON(t *testing.T) {
+ r := &Result{
+ URL: "https://food52.com",
+ Mode: ModeBrowserHTTP,
+ Confidence: 0.85,
+ Probes: []ProbeResult{
+ {Transport: TransportStdlib, Status: 429, ElapsedMS: 412, Evidence: []string{"x-vercel-mitigated: challenge"}},
+ {Transport: TransportSurfChrome, Status: 200, ElapsedMS: 387},
+ },
+ Recommendation: Recommendation{
+ Runtime: ModeBrowserHTTP,
+ Rationale: "Surf cleared",
+ },
+ }
+ var buf bytes.Buffer
+ require.NoError(t, RenderJSON(&buf, r))
+
+ var decoded map[string]any
+ require.NoError(t, json.Unmarshal(buf.Bytes(), &decoded))
+ assert.Equal(t, "browser_http", decoded["mode"])
+ assert.Equal(t, "https://food52.com", decoded["url"])
+ probes, ok := decoded["probes"].([]any)
+ require.True(t, ok)
+ assert.Len(t, probes, 2)
+}
+
+func TestRenderTable_FullPath(t *testing.T) {
+ r := &Result{
+ URL: "https://food52.com",
+ Mode: ModeBrowserHTTP,
+ Confidence: 0.85,
+ Probes: []ProbeResult{
+ {Transport: TransportStdlib, Status: 429, ElapsedMS: 412, Evidence: []string{"x-vercel-mitigated: challenge"}},
+ {Transport: TransportSurfChrome, Status: 200, ElapsedMS: 387},
+ },
+ Recommendation: Recommendation{Runtime: ModeBrowserHTTP, Rationale: "Surf cleared"},
+ }
+ var buf bytes.Buffer
+ require.NoError(t, RenderTable(&buf, r))
+ out := buf.String()
+ assert.Contains(t, out, "Transport")
+ assert.Contains(t, out, "stdlib")
+ assert.Contains(t, out, "surf-chrome")
+ assert.Contains(t, out, "browser_http")
+ assert.Contains(t, out, "Surf cleared")
+ assert.NotContains(t, out, "partial")
+}
+
+func TestRenderTable_PartialNotice(t *testing.T) {
+ r := &Result{
+ URL: "https://example.com",
+ Mode: ModeStandardHTTP,
+ Partial: true,
+ Probes: []ProbeResult{
+ {Transport: TransportStdlib, Status: 200, ElapsedMS: 100},
+ },
+ Recommendation: Recommendation{Runtime: ModeStandardHTTP},
+ }
+ var buf bytes.Buffer
+ require.NoError(t, RenderTable(&buf, r))
+ out := buf.String()
+ assert.True(t, strings.Contains(out, "partial"), "table must surface partial flag for debug clarity")
+}
+
+func TestRenderTable_TransportError(t *testing.T) {
+ r := &Result{
+ URL: "https://nope.invalid",
+ Mode: ModeUnknown,
+ Probes: []ProbeResult{
+ {Transport: TransportStdlib, Error: "DNS lookup failed", ElapsedMS: 50},
+ },
+ Recommendation: Recommendation{Runtime: ModeUnknown},
+ }
+ var buf bytes.Buffer
+ require.NoError(t, RenderTable(&buf, r))
+ out := buf.String()
+ assert.Contains(t, out, "DNS lookup failed")
+}
diff --git a/internal/reachability/types.go b/internal/reachability/types.go
new file mode 100644
index 00000000..54662080
--- /dev/null
+++ b/internal/reachability/types.go
@@ -0,0 +1,87 @@
+// Package reachability runs no-browser probes against a URL to classify
+// whether a printed CLI can reach it with plain HTTP, with browser-compatible
+// HTTP (Surf/uTLS Chrome fingerprint), or whether it needs a real browser
+// session to capture a clearance cookie.
+//
+// The classification vocabulary mirrors internal/browsersniff/analysis.go's
+// Reachability.Mode so both flows speak one language: a Surf-pass during
+// pre-capture probing emits the same `browser_http` mode the post-capture
+// analyzer would have produced.
+package reachability
+
+// Mode classifies how a URL is reachable.
+type Mode string
+
+const (
+ // ModeStandardHTTP — plain stdlib HTTP works. Generated CLI ships with
+ // Go's default net/http client.
+ ModeStandardHTTP Mode = "standard_http"
+
+ // ModeBrowserHTTP — bot-protection signals exist but a Chrome-shaped
+ // HTTP client (Surf/uTLS) clears them without cookies. Generated CLI
+ // ships with Surf transport. No Chrome attach during generation.
+ ModeBrowserHTTP Mode = "browser_http"
+
+ // ModeBrowserClearanceHTTP — neither stdlib nor Surf cleared the
+ // protection. Recommendation, not a verdict: the skill should escalate
+ // to a real browser-sniff capture, which will produce the authoritative
+ // classification. The probe cannot tell whether the site needs a
+ // clearance cookie or live page-context execution.
+ ModeBrowserClearanceHTTP Mode = "browser_clearance_http"
+
+ // ModeUnknown — probes failed for non-protection reasons (DNS, timeout,
+ // connection refused, 5xx). Not enough evidence to recommend a runtime.
+ ModeUnknown Mode = "unknown"
+)
+
+// Transport names a probe rung.
+type Transport string
+
+const (
+ TransportStdlib Transport = "stdlib"
+ TransportSurfChrome Transport = "surf-chrome"
+)
+
+// ProbeOnly restricts the probe ladder to a single rung. Debug only — the
+// skill never sets this. With ProbeOnly set, Result.Partial is true and
+// the mode reflects only what the run rung proved.
+type ProbeOnly string
+
+const (
+ ProbeOnlyNone ProbeOnly = ""
+ ProbeOnlyStdlib ProbeOnly = "stdlib"
+ ProbeOnlySurf ProbeOnly = "surf"
+)
+
+// Result is the probe-reachability output. Stable JSON field names so the
+// printing-press skill (and any future scripts) can consume this without
+// depending on Go struct ordering.
+type Result struct {
+ URL string `json:"url"`
+ Mode Mode `json:"mode"`
+ Confidence float64 `json:"confidence"`
+ Probes []ProbeResult `json:"probes"`
+ Recommendation Recommendation `json:"recommendation"`
+ // Partial is true when --probe-only restricted the ladder to one rung.
+ // Skill consumers should ignore Mode when Partial is true.
+ Partial bool `json:"partial,omitempty"`
+}
+
+// ProbeResult is one rung of the probe ladder.
+type ProbeResult struct {
+ Transport Transport `json:"transport"`
+ Status int `json:"status,omitempty"`
+ ElapsedMS int64 `json:"elapsed_ms"`
+ Evidence []string `json:"evidence,omitempty"`
+ ContentType string `json:"content_type,omitempty"`
+ Error string `json:"error,omitempty"`
+}
+
+// Recommendation tells the caller what runtime to ship and what consent
+// gates remain.
+type Recommendation struct {
+ Runtime Mode `json:"runtime"`
+ NeedsBrowserCapture bool `json:"needs_browser_capture"`
+ NeedsClearanceCookie bool `json:"needs_clearance_cookie"`
+ Rationale string `json:"rationale"`
+}
diff --git a/skills/printing-press/SKILL.md b/skills/printing-press/SKILL.md
index d1af9802..db4c9b87 100644
--- a/skills/printing-press/SKILL.md
+++ b/skills/printing-press/SKILL.md
@@ -791,12 +791,35 @@ These are the only cases where Phase 1.7 is bypassed as a whole (not just skippe
### Direct HTTP challenge rule
-If a reachability probe during Phase 1 research returns bot-protection evidence (`403`, `429`, `cf-mitigated: challenge`, `x-vercel-mitigated: challenge`, `x-vercel-challenge-token`, AWS WAF, DataDome, PerimeterX, CAPTCHA, "Just a moment", "access denied"), treat it as a **browser-sniff escalation signal**, not as a browser-sniff failure.
+If a reachability probe during Phase 1 research returns bot-protection evidence (`403`, `429`, `cf-mitigated: challenge`, `x-vercel-mitigated: challenge`, `x-vercel-challenge-token`, AWS WAF, DataDome, PerimeterX, CAPTCHA, "Just a moment", "access denied"), **run the no-browser reachability probe before announcing any browser escalation**:
-When browser-sniff is approved or pre-approved:
+```bash
+printing-press probe-reachability "<url>" --json
+```
+
+This is non-negotiable. **Do not present transport tiers as a peer menu for the user to choose between.** Phrases like "Browser-sniff + clearance cookie", "Browser-sniff with Surf-only", "Try without browser at all", or "Browser-sniff, prefer Surf" route the user through implementation choices (Surf vs cookie vs full browser) they don't have context to make. The classifier is `probe-reachability`; the agent runs it and decides. Intent-level menus are fine — "Browser-sniff or HOLD?", "Browser-sniff or pick a different API?", or the standard yes/no browser-sniff offers below all ask about goals, not transport, and remain available.
+
+Escalate consent in the order the agent actually needs it, not bundled up-front:
+
+1. **Runtime probe (silent)** — `probe-reachability` runs without prompting. The user already opted into "the website itself" or equivalent in Phase 0; running an HTTP request needs no further consent.
+2. **Browser-sniff offer (intent prompt)** — Phase 1.7's normal "Browser-Sniff as enrichment" / "Browser-Sniff as primary" prompts ask whether to do browser-sniff at all. These are intent-level. Show them when the discovery matrix says to.
+3. **Chrome attach (separate consent if escalation happens)** — when the agent actually needs to open or attach to Chrome (because the discovery flow requires a real browser, or because `mode: browser_clearance_http` means the runtime needs cookie capture), surface that as its own moment so the user knows they may need to solve a challenge or sign in. The user-facing prompts at lines below already disclose Chrome attach as a possibility; that is the right place to confirm. Do not pre-announce Chrome attach when the probe has already settled the runtime as `browser_http` and the spec is complete enough to skip discovery — there is no Chrome attach to announce in that path.
+
+Two concerns are decided here, separately:
+
+- **Runtime** (does the printed CLI need browser-compatible HTTP, a clearance cookie, or live page-context execution?) — settled entirely by `probe-reachability`.
+- **Discovery** (does Phase 1.7 need to capture XHR traffic via a real browser to learn endpoints?) — settled by Phase 1.7's normal "When to offer browser-sniff" decision matrix above. Independent of runtime.
+
+The probe runs stdlib HTTP, then Surf with a Chrome TLS fingerprint, and emits one of `standard_http | browser_http | browser_clearance_http | unknown`. Apply `mode` to the **runtime** decision:
+
+- **`mode: standard_http`** — runtime is plain HTTP (the original probe was transient). Continue Phase 1.7 normally; the discovery decision is unchanged.
+- **`mode: browser_http`** — **runtime is settled: ship Surf transport** (`UsesBrowserHTTPTransport` will be set in the generator's traffic-analysis hints). The printed CLI will not include `auth login --chrome` for clearance cookies — Surf alone clears the challenge. Continue Phase 1.7's discovery decision normally; the existing "Browser-Sniff as enrichment" / "Browser-Sniff as primary" prompts (above) are framed around endpoint discovery and are correct as-written. Do **not** add clearance-cookie language to those prompts.
+- **`mode: browser_clearance_http`** — both probes hit protection signals. The runtime needs more than Surf (clearance cookie or live page-context execution; the probe cannot distinguish), so a real browser capture is required to find out. Proceed through Phase 1.7's normal browser-sniff offer (intent-level yes/no). The consent for Chrome attach happens at the moment the agent actually opens/attaches, where the user-facing prompts in `references/browser-sniff-capture.md` already disclose what's about to happen and may ask the user to solve a challenge. Note in the brief that runtime is provisionally `browser_clearance_http` pending capture results.
+- **`mode: unknown`** — probes failed at the transport layer (DNS/timeout/5xx). Fall through to the existing browser-sniff offer; the user decides whether to retry or pivot.
+
+When browser-sniff is approved or pre-approved AND the probe says `browser_clearance_http` or `unknown`:
- Do **not** offer alternate CLI shapes (RSS-first, official API, docs-only, narrower scope, "try anyway") before a real browser capture has been attempted.
- Do **not** write the brief as if browser-sniff is complete after only curl/direct HTTP probes.
-- Proceed to "If user approves browser-sniff" and explicitly tell the user: "Direct HTTP is blocked by `<protection>`, so I need a real browser capture. I will open or attach to Chrome; please solve the challenge and navigate the target flow."
- If browser automation tooling is unavailable, offer the user a manual HAR path before offering any scope pivot.
Only after the browser capture attempt fails by the criteria in `references/browser-sniff-capture.md` may you ask whether to pivot to RSS, official API, docs-only, or a smaller CLI scope.
@@ -1294,15 +1317,24 @@ curl -s -o /dev/null -w "%{http_code}" -m 10 "<base_url>/<simplest_get_path>" 2>
Or use `WebFetch` if curl is unavailable. The goal is one real response code.
+**If the check returns 403/429 with bot-protection evidence and `probe-reachability` has not already run for this URL during Phase 1.7's Direct HTTP challenge rule, run it now before consulting the decision matrix:**
+
+```bash
+printing-press probe-reachability "<base_url>" --json
+```
+
+The matrix below references `probe-reachability` `mode` for the bot-detection rows. If the probe already ran in Phase 1.7, reuse that result; do not re-probe.
+
### Decision Matrix
| Result | Browser capture result | Traffic-analysis reachability | Action |
|--------|------------------------|-------------------------------|--------|
| 2xx/3xx | Any | Any | **PASS** - proceed to Phase 2 |
| 401 (no key provided) | Any | Any | **PASS** - expected when API needs auth and user declined key gate |
+| 403/429 with HTML/bot detection | `probe-reachability` returned `browser_http` | runtime is `browser_http` (Surf) | **PASS** - the printed CLI will ship Surf transport which clears the protection. No clearance cookie capture in the printed CLI, regardless of whether browser-sniff also ran for endpoint discovery |
| 403/429 with HTML/bot detection | Successful useful capture | `browser_http` or `browser_clearance_http` | **PASS** - proceed with browser-compatible HTTP / clearance strategy |
| Any | Capture only works through a live page context | `browser_required` | **HOLD** - find a lighter replayable surface before Phase 2 |
-| 403/429 with HTML/bot detection | No browser capture attempted but browser-sniff approved/pre-approved | Any | **RETURN TO PHASE 1.7** - attempt cleared-browser capture before pivoting scope |
+| 403/429 with HTML/bot detection | No browser capture attempted but browser-sniff approved/pre-approved AND `probe-reachability` returned `browser_clearance_http` or `unknown` | Any | **RETURN TO PHASE 1.7** - attempt cleared-browser capture before pivoting scope |
| 403/429 with HTML/bot detection | Capture contains only challenge/error pages | Any | **HARD STOP** |
| 403 | No successful useful capture | Research found 403 issues | **HARD STOP** |
| 403 | No successful useful capture | No 403 research issues | **WARN** - ask user |
diff --git a/skills/printing-press/references/browser-sniff-capture.md b/skills/printing-press/references/browser-sniff-capture.md
index b0c8025b..73142825 100644
--- a/skills/printing-press/references/browser-sniff-capture.md
+++ b/skills/printing-press/references/browser-sniff-capture.md
@@ -15,7 +15,7 @@
3. **Use click-based SPA navigation after installing interceptors.** `browser-use open` triggers a full page reload which resets the JS context and destroys fetch/XHR interceptors. After installing interceptors, navigate by clicking links (`browser-use eval "document.querySelector('a[href*=account]').click()"` or `browser-use click`). Only use `browser-use open` for the first page load or when you need to re-install interceptors.
-4. **Direct HTTP challenges require a cleared browser attempt before scope pivots.** If research or preflight saw Cloudflare/Vercel/WAF/DataDome/PerimeterX/CAPTCHA evidence, tell the user that direct HTTP is blocked and proceed with a real browser capture. Do NOT replace the target with RSS/docs/official API or ask for a smaller CLI shape until after browser capture has failed by the criteria below.
+4. **Run `printing-press probe-reachability` before announcing any browser escalation, and don't expose transport tiers to the user as peer choices.** If research or preflight saw Cloudflare/Vercel/WAF/DataDome/PerimeterX/CAPTCHA evidence, the *first* action is the no-browser probe — not a Chrome-attach prompt and not a transport-tier menu like "Browser-sniff + clearance cookie / Browser-sniff Surf-only / HAR / Hold". Intent menus are fine (yes/no, browser-sniff or pivot, etc.); the wrong shape is forcing the user to pick between Surf vs cookie vs full browser, which is the classifier's job. Many passive challenges (Vercel TLS-fingerprint mitigation, lighter Cloudflare gates) clear with Surf alone, no cookie, no setup. If `probe-reachability` returns `mode: browser_http`, the printed CLI will ship Surf transport with zero clearance-cookie capture — runtime is settled silently. (Browser-sniff for endpoint *discovery* is a separate decision handled by Phase 1.7's normal matrix; if that matrix says to ask, the existing intent-level prompts already disclose Chrome attach as a possibility — that's the right place for that consent, not bundled into a transport-tier menu.) Only when the probe returns `browser_clearance_http` or `unknown` should you tell the user direct HTTP is blocked and proceed with a real browser capture. Do NOT replace the target with RSS/docs/official API or ask for a smaller CLI shape until after browser capture has failed by the criteria below.
5. **Replayability is the success criterion.** A browser capture succeeds only when it produces a shippable surface: replayable API calls, persisted-query registry entries, browser-clearance cookies that can be imported and replayed, or structured HTML/SSR/RSS/JSON-LD extraction targets. If the only observed path requires live page-context execution, report HOLD or return to discovery for a lighter surface. Do not continue as if resident browser transport is acceptable.
← 5966298e feat(skills): reconcile prior novel features on reprint (#32
·
back to Cli Printing Press
·
fix(cli): score auth prefixes from config (#332) 8bad2efd →