← back to Homesonspec
feat(toll): wave-2 QMI home ingestion (adapter v2.0.0) — plain-fetch sitemap /Quick-Move-In pages
1f01a75d8b4c59c9ac56599072d903c1d2dacf75 · 2026-08-03 07:47:57 -0700 · Steve Abrams
fetch() yields QMI pages from the sitemap (dedupes MLS- aliases, TOLL_QMI_PAGE_LIMIT
volume cap); extract() parses __NEXT_DATA__ into inventory_home records + a specs map
(master-bed floor, collection, biddable). Verified against a real page (lot 91 Kruse:
$729k 4bd/3.5ba 2620sqft) and confirmed the communityName FK-matches the wave-1 feed
exactly, so homes publish instead of zeroing. Typecheck clean. Live crawl + prod
deploy stay Steve-gated (safe code-only path only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Files touched
M collectors/toll-brothers/WAVE2-RECON.mdM collectors/toll-brothers/src/index.ts
Diff
commit 1f01a75d8b4c59c9ac56599072d903c1d2dacf75
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Mon Aug 3 07:47:57 2026 -0700
feat(toll): wave-2 QMI home ingestion (adapter v2.0.0) — plain-fetch sitemap /Quick-Move-In pages
fetch() yields QMI pages from the sitemap (dedupes MLS- aliases, TOLL_QMI_PAGE_LIMIT
volume cap); extract() parses __NEXT_DATA__ into inventory_home records + a specs map
(master-bed floor, collection, biddable). Verified against a real page (lot 91 Kruse:
$729k 4bd/3.5ba 2620sqft) and confirmed the communityName FK-matches the wave-1 feed
exactly, so homes publish instead of zeroing. Typecheck clean. Live crawl + prod
deploy stay Steve-gated (safe code-only path only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
collectors/toll-brothers/WAVE2-RECON.md | 25 +++++-
collectors/toll-brothers/src/index.ts | 130 +++++++++++++++++++++++++++++++-
2 files changed, 151 insertions(+), 4 deletions(-)
diff --git a/collectors/toll-brothers/WAVE2-RECON.md b/collectors/toll-brothers/WAVE2-RECON.md
index bcccf705..6676577a 100644
--- a/collectors/toll-brothers/WAVE2-RECON.md
+++ b/collectors/toll-brothers/WAVE2-RECON.md
@@ -1,5 +1,15 @@
# Toll Brothers wave-2 (per-home QMI ingestion) — recon 2026-08-03
+> **STATUS: BUILT (adapter v2.0.0) + verified locally, 2026-08-03.** `fetch()` now
+> yields QMI pages from the sitemap; `extract()` parses `__NEXT_DATA__` into
+> inventory_home records. Proven against a real page (lot 91 / Kruse / Hosford Farms:
+> $729k, 4bd/3.5ba, 2620sqft, specs). Community-name FK match to the wave-1 feed
+> VERIFIED exact (homes will publish, not zero out). Typecheck clean.
+> **REMAINING (Steve-gated):** first live crawl (fetches ~400 QMI pages/run against
+> tollbrothers.com) + prod deploy via the SAFE code-only path (NOT deploy-kamatera.sh —
+> it drops the prod DB; see the specs deploy in session history). Watch Akamai
+> datacenter-IP blocking on the prod crawl (dev/first-run from Mac2).
+
**Goal:** ingest Toll's individual quick-delivery/spec homes (QMI). Wave-1 emits
COMMUNITIES only (6,432 staged, all COMMUNITY) → InventoryHome = 0 today. This is
a DELIBERATE deferral, not a bug (see collectors/SPECS-AUDIT.md).
@@ -34,8 +44,19 @@ But the INDIVIDUAL QMI home pages are plain-fetch with clean structured data:
`floorSize.value`→sqft, `address.{streetAddress,addressLocality,addressRegion,postalCode}`,
`geo`→lat/lon, `numberOfBedrooms`, `numberOfFullBathrooms`+`numberOfPartialBathrooms`,
`containedIn`→community, `image`, `description`. **NOTE: price is NOT in the JSON-LD.**
- - **`__NEXT_DATA__`** — carries `price` + `lotNumber` (+ likely status, incentives,
- was-price for the `specs` map). Pin the exact props path during the build.
+ - **`__NEXT_DATA__`** — the COMPLETE single source (verified against a real page,
+ lot 91 / Kruse / Hosford Farms OR). Find the object with a `lotNumber` key; its fields:
+ - price → **`pricedFrom`** (729000) · beds → `minBed` · baths → `minBath` + `minHalfBath`
+ · sqft → `minSqft` · address → `street`/`city`/`state`/`zip`/`county` · geo → `lat`/`lon`
+ · stories → `stories` · garage → `minGarage` · plan → `name` (+ `modelName`/`collectionName`)
+ · lot → `lotNumber` · completion/move-in → `moveInDate` · community → `communityName`/`communityId`/`communityUrl`
+ - status flags → `isQMI`, `isComingSoon`, `isBiddableQMI`, `isBiddableLot`, `isDecoratedModel`
+ (isComingSoon→PLANNED; else moveInDate past→MOVE_IN_READY else UNDER_CONSTRUCTION)
+ - `specs` map candidates → `masterBedroomLocation` ("3rd Floor"), `collectionName`,
+ `isBiddableQMI`/`isBiddableLot` (accepts-offers), `moveInDate`, `homeType`
+ - media (facts-only, drop per policy) → `gallery`, `floorplans`
+ - The JSON-LD `SingleFamilyResidence` block is a redundant backup (has address/beds/
+ sqft/geo but NO price) — **use `__NEXT_DATA__` as the primary; it has everything.**
### Build plan (Pulte-style adapter, plain-fetch)
1. `fetch()`: GET Toll's sitemap → collect `/Quick-Move-In/<numeric-id>` URLs, drop
diff --git a/collectors/toll-brothers/src/index.ts b/collectors/toll-brothers/src/index.ts
index af613548..33548f63 100644
--- a/collectors/toll-brothers/src/index.ts
+++ b/collectors/toll-brothers/src/index.ts
@@ -86,9 +86,108 @@ function extractCommunity(community: TbCommunity, sourceUrl: string): ExtractedR
};
}
+// ---- wave 2: per-home QMI ingestion (plain-fetch, see WAVE2-RECON.md) ----
+const SITEMAP_URL = "https://www.tollbrothers.com/sitemap.xml";
+// Volume control: cap QMI pages per run. contentHash snapshot-dedup makes an
+// unchanged page a fast no-op, so a low cap surfaces the head and later runs
+// walk further — the full ~11k is reached cheaply over recurring runs.
+const QMI_PAGE_LIMIT = Number(process.env.TOLL_QMI_PAGE_LIMIT ?? 400);
+
+const qmiNum = (v: unknown): number | null => {
+ const n = typeof v === "number" ? v : typeof v === "string" ? Number(v.replace(/[^0-9.]/g, "")) : NaN;
+ return Number.isFinite(n) && n > 0 ? n : null;
+};
+const qmiStr = (v: unknown): string | null => (typeof v === "string" && v.trim() ? v.trim() : null);
+const qmiBool = (v: unknown): boolean | null =>
+ typeof v === "boolean" ? v : v === "true" ? true : v === "false" ? false : null;
+
+/** Pull the __NEXT_DATA__ home object (the one carrying `lotNumber`) out of a QMI page. */
+function parseQmiHome(html: string): Record<string, unknown> | null {
+ const m = html.match(/<script id="__NEXT_DATA__"[^>]*>([\s\S]*?)<\/script>/);
+ if (!m) return null;
+ let root: unknown;
+ try { root = JSON.parse(m[1]!); } catch { return null; }
+ let home: Record<string, unknown> | null = null;
+ (function walk(o: unknown, depth: number) {
+ if (home || !o || typeof o !== "object" || depth > 12) return;
+ const rec = o as Record<string, unknown>;
+ if (rec.lotNumber !== undefined && rec.pricedFrom !== undefined) { home = rec; return; }
+ for (const k of Object.keys(rec)) walk(rec[k], depth + 1);
+ })(root, 0);
+ return home;
+}
+
+function qmiStatus(h: Record<string, unknown>): "MOVE_IN_READY" | "UNDER_CONSTRUCTION" | "PLANNED" {
+ if (qmiBool(h.isComingSoon)) return "PLANNED";
+ const md = qmiStr(h.moveInDate);
+ if (md && !Number.isNaN(Date.parse(md)) && Date.parse(md) <= Date.now()) return "MOVE_IN_READY";
+ return "UNDER_CONSTRUCTION";
+}
+
+/** A QMI home page → one inventory_home ExtractedRecord (community published in wave 1). */
+function extractQmiHome(html: string, sourceUrl: string): ExtractedRecord | null {
+ const h = parseQmiHome(html);
+ if (!h) return null;
+ const street = qmiStr(h.street);
+ const communityName = qmiStr(h.communityName);
+ if (!street || !communityName) return null; // no address / no FK-able community → skip honestly
+ const fullB = qmiNum(h.minBath);
+ const halfB = qmiNum(h.minHalfBath) ?? 0;
+ const baths = fullB !== null ? fullB + halfB * 0.5 : null;
+ const lat = (() => { const n = Number(h.lat); return Number.isFinite(n) && n !== 0 ? n : null; })();
+ const lon = (() => { const n = Number(h.lon); return Number.isFinite(n) && n !== 0 ? n : null; })();
+ const plan = qmiStr(h.name) ?? qmiStr(h.modelName);
+ const lot = qmiStr(h.lotNumber);
+
+ const specs: Record<string, unknown> = {
+ masterBedroomLocation: qmiStr(h.masterBedroomLocation),
+ collection: qmiStr(h.collectionName),
+ acceptsOffers: qmiBool(h.isBiddableQMI) ?? qmiBool(h.isBiddableLot),
+ homeType: qmiStr(h.homeType),
+ };
+ for (const k of Object.keys(specs)) if (specs[k] == null || specs[k] === false) delete specs[k];
+
+ const price = qmiNum(h.pricedFrom);
+ const moveIn = qmiStr(h.moveInDate);
+ return {
+ entityType: "inventory_home",
+ canonicalHints: {
+ builderSlug: BUILDER_SLUG,
+ communityName,
+ address: street,
+ builderInventoryId: lot,
+ lat: lat ?? undefined,
+ lon: lon ?? undefined,
+ planName: plan ?? undefined,
+ },
+ fields: {
+ street: fv(street, street, sourceUrl),
+ city: fv(qmiStr(h.city), qmiStr(h.city), sourceUrl),
+ state: fv(normalizeStateCode(qmiStr(h.state)), qmiStr(h.state), sourceUrl),
+ zip: fv(qmiStr(h.zip), qmiStr(h.zip), sourceUrl),
+ price: fv(price, price === null ? null : String(price), sourceUrl, price ? `pricedFrom ${price}` : null),
+ beds: fv(qmiNum(h.minBed), null, sourceUrl),
+ bathsTotal: fv(baths, fullB === null ? null : `${fullB} full + ${halfB} half`, sourceUrl),
+ sqft: fv(qmiNum(h.minSqft), null, sourceUrl),
+ stories: fv(qmiNum(h.stories), null, sourceUrl),
+ garageSpaces: fv(qmiNum(h.minGarage), null, sourceUrl),
+ lat: fv(lat, lat === null ? null : String(lat), sourceUrl),
+ lon: fv(lon, lon === null ? null : String(lon), sourceUrl),
+ homeType: fv("SINGLE_FAMILY" as const, null, sourceUrl, "Toll Brothers QMI home"),
+ constructionStatus: fv(qmiStatus(h), null, sourceUrl),
+ estCompletionDate: fv(moveIn, moveIn, sourceUrl),
+ lotNumber: fv(lot, lot, sourceUrl),
+ builderInventoryId: fv(lot, lot, sourceUrl),
+ planName: fv(plan, plan, sourceUrl),
+ images: fv<string[]>([], null, sourceUrl), // facts-only per docs/data-rights/POLICY.md
+ specs: fv(Object.keys(specs).length ? specs : null, null, sourceUrl, "toll QMI extras (master-bed floor, collection, biddable)"),
+ },
+ };
+}
+
export const tollBrothersAdapter: SourceAdapter = {
key: "toll-brothers-site",
- version: "1.0.0",
+ version: "2.0.0",
async *fetch(ctx: FetchContext): AsyncIterable<RawPage> {
if (ctx.mode === "fixture") {
@@ -96,10 +195,37 @@ export const tollBrothersAdapter: SourceAdapter = {
return;
}
const fetcher = new LiveFetcher(ctx.registry);
- yield await fetcher.fetch(FEED_URL);
+ yield await fetcher.fetch(FEED_URL); // wave 1: communities
+
+ // wave 2: QMI home pages from the sitemap (plain-fetch). Prefer the numeric-id
+ // URL; drop the MLS-<n> alias that points at the same home.
+ try {
+ const sitemap = await fetcher.fetch(SITEMAP_URL);
+ const urls = [...sitemap.body.toString("utf8").matchAll(/<loc>([^<]+\/Quick-Move-In\/[^<]+)<\/loc>/g)]
+ .map((m) => m[1]!)
+ .filter((u) => !/\/Quick-Move-In\/MLS-/i.test(u));
+ const seen = new Set<string>();
+ let n = 0;
+ for (const url of urls) {
+ if (seen.has(url)) continue;
+ seen.add(url);
+ if (n++ >= QMI_PAGE_LIMIT) break;
+ try { yield await fetcher.fetch(url); }
+ catch (error) { console.warn(` skip QMI ${url}: ${error instanceof Error ? error.message : String(error)}`); }
+ }
+ } catch (error) {
+ console.warn(` sitemap fetch failed: ${error instanceof Error ? error.message : String(error)}`);
+ }
},
extract(page: RawPage): ExtractionOutput {
+ // wave 2: a QMI home page → one inventory_home record.
+ if (/\/Quick-Move-In\//i.test(page.url)) {
+ const rec = extractQmiHome(page.body.toString("utf8"), page.url);
+ return rec
+ ? { records: [rec], errors: [] }
+ : { records: [], errors: [{ url: page.url, reason: "QMI page: no parseable __NEXT_DATA__ home" }] };
+ }
try {
const data = JSON.parse(page.body.toString("utf8")) as {
perfect?: TbCommunity[];
← acd6564e docs(toll): wave-2 SOLVED — QMI homes are plain-fetch via si
·
back to Homesonspec
·
feat(web): HomesOnSpec attribution on builder contact — 'men 576365d9 →