docs(toll): wave-2 recon — ~298 per-community fetches not 11k pages; homes load via amplify SPA XHR (browser-trace = fresh-session task)

f98e35a7afa1e56780ffd2c1d660e9724d9c8015 · 2026-08-03 07:29:05 -0700 · Steve Abrams

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

A collectors/toll-brothers/WAVE2-RECON.md

Diff

commit f98e35a7afa1e56780ffd2c1d660e9724d9c8015
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Mon Aug 3 07:29:05 2026 -0700

    docs(toll): wave-2 recon — ~298 per-community fetches not 11k pages; homes load via amplify SPA XHR (browser-trace = fresh-session task)
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 collectors/toll-brothers/WAVE2-RECON.md | 44 +++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/collectors/toll-brothers/WAVE2-RECON.md b/collectors/toll-brothers/WAVE2-RECON.md
new file mode 100644
index 00000000..f5db3700
--- /dev/null
+++ b/collectors/toll-brothers/WAVE2-RECON.md
@@ -0,0 +1,44 @@
+# Toll Brothers wave-2 (per-home QMI ingestion) — recon 2026-08-03
+
+**Goal:** ingest Toll's individual quick-delivery/spec homes (QMI). Wave-1 emits
+COMMUNITIES only (6,432 staged, all COMMUNITY) → InventoryHome = 0 today. This is
+a DELIBERATE deferral, not a bug (see collectors/SPECS-AUDIT.md).
+
+## Key finding: it's ~298 per-community fetches, NOT an 11k-page crawl
+The community feed `https://www.tollbrothers.com/api/v1/search/homesearchV5.json`
+returns `{perfect:[372], partial:[], count:610}` communities. Each community
+object carries **`numQDH`** (quick-delivery-home count), `comingSoonQMICount`,
+`prePlannedCount`, `hasQdh`, and an (empty in the summary feed) `qdhList`.
+- **298 of 372 communities have numQDH > 0** (e.g. 7, 5, 11 homes each).
+- So wave-2 = fetch the QMI homes for those ~298 communities only. Small.
+
+## What did NOT work (probed 2026-08-03)
+- Dedicated QMI search endpoints all 404: `qmisearchV5.json`,
+  `inventorysearchV5.json`, `quickmoveinsearchV5.json`, `/api/v1/qmi/search`.
+- `homesearchV5.json?communityId=<id>` returns 101KB of valid JSON but its 17
+  `qdhList` arrays are **empty** — that param doesn't hydrate the homes.
+- The community `url` points to an **Amplify SPA**
+  (`apps-tbcomamplify-prod.tollwebservices.com/luxury-homes-for-sale/<State>/<slug>`,
+  337KB). It has NO embedded homes JSON, NO `__NEXT_DATA__`, and NO visible
+  `/api/` references — homes load via a **runtime XHR** after hydration.
+
+## The remaining crack (the fresh-session task)
+Browser-trace ONE Toll community detail page (Playwright/Browserbase network
+capture) to catch the XHR that loads its QDH homes. Community `3131 Camino`
+(communityId **14614**, California, numQDH **7**) is a good probe target.
+Once the endpoint + params are known, build a Pulte-style JSON adapter:
+1. `fetch()`: get homesearchV5.json → filter communities to numQDH>0 → for each,
+   call the (traced) QMI endpoint.
+2. `extract()`: emit `entityType:"community"` (as today) PLUS `inventory_home`
+   records from the QMI payload. Capture core fields + a `specs` map (Toll should
+   expose was-price/incentives/plan — follow the DR-Horton/Pulte specs pattern,
+   commit `7c3609c`).
+3. Respect robots: `/siteplans /nse /compare /new-homes /sitesearch` are
+   disallowed; `/api/` is allowed. NO bot-wall circumvention (standing rule).
+
+## Watch-outs
+- Toll Akamai bot-blocks the Kamatera **datacenter** IP under load (Taylor
+  Morrison hit this). Trace + dev from Mac2 / residential egress; if prod crawl
+  gets challenged, that's the allowlisted-egress problem, not a code bug.
+- Publisher requires each home to hang off a Community (FK). Wave-1 already
+  publishes the communities, so extract communities BEFORE their homes.

← d6741529 docs(specs): 6-builder batch outcome — 4 capture rich per-ho · back to Homesonspec · docs(toll): wave-2 SOLVED — QMI homes are plain-fetch via si acd6564e →