[object Object]

← back to Homesonspec

fix(live-fetch): broaden Accept header so D.R. Horton WAF stops 406ing

aa61dfd426515bca2c7878135015f021a9e509ea · 2026-07-23 01:04:59 -0700 · Steve Abrams

Narrow 'text/html,application/json' failed DRH's strict content
negotiation (406 Not Acceptable) from the datacenter IP. Added the
standard */*;q=0.8 fallback (kept the honest SpecHomesBot UA — no
browser spoofing). */* still covers Pulte's JSON API + XML sitemaps.
Fixes DRH = 0 published; helps every adapter through the one choke-point.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

Diff

commit aa61dfd426515bca2c7878135015f021a9e509ea
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Thu Jul 23 01:04:59 2026 -0700

    fix(live-fetch): broaden Accept header so D.R. Horton WAF stops 406ing
    
    Narrow 'text/html,application/json' failed DRH's strict content
    negotiation (406 Not Acceptable) from the datacenter IP. Added the
    standard */*;q=0.8 fallback (kept the honest SpecHomesBot UA — no
    browser spoofing). */* still covers Pulte's JSON API + XML sitemaps.
    Fixes DRH = 0 published; helps every adapter through the one choke-point.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 collectors/common/src/live-fetch.ts | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/collectors/common/src/live-fetch.ts b/collectors/common/src/live-fetch.ts
index 325c2f9..2b8ba1d 100644
--- a/collectors/common/src/live-fetch.ts
+++ b/collectors/common/src/live-fetch.ts
@@ -28,7 +28,15 @@ export class LiveFetcher {
 
     const response = await fetch(url, {
       method: "GET",
-      headers: { "User-Agent": USER_AGENT, Accept: "text/html,application/json" },
+      // Broad Accept with a */* fallback. A narrow "text/html,application/json"
+      // 406s on picky WAFs (D.R. Horton) that do strict content negotiation and
+      // reject the absence of a wildcard fallback. The honest bot UA is kept —
+      // this is not disguise, just a standards-compliant Accept header. */*
+      // still covers the JSON-API builders (Pulte) and XML sitemaps.
+      headers: {
+        "User-Agent": USER_AGENT,
+        Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,application/json;q=0.9,*/*;q=0.8",
+      },
       redirect: "follow",
       signal: AbortSignal.timeout(25_000),
     });

← f66f9ae Add Tri Pointe adapter (5th builder) — Next.js RSC (unescape  ·  back to Homesonspec  ·  fix(collectors): raise DRH+Lennar default PAGE_LIMIT 8→300 f 08c6d17 →