`collectors/BUILDERS.md`

147 lines
# Builder adapter recon (CA inventory) — 2026-07-23

All 6 top CA builders recon'd. **All are plain-curl, HTTP 200, no anti-bot, no browser needed.**
Each carries a per-community sales phone. Adapters are CA-scoped via an env var.

| Builder | key | status | source shape |
|---|---|---|---|
| Lennar | lennar-site | ✅ built | per-community `__NEXT_DATA__` Apollo (LENNAR_STATE_FILTER) |
| D.R. Horton | dr-horton-site | ✅ built+verified | per-community `var model = {…}` blob (DRHORTON_STATE) |
| KB Home | kb-home-site | ✅ built+verified | ONE feed `/move-in-ready` → `var allMIRs=JSON.parse("…")` double-decode |
| Pulte | pultegroup-site | ✅ built+live | clean JSON API (has lat/lon!) |
| Tri Pointe | tri-pointe-site | ✅ built+live | Next.js RSC `__next_f` `"display_price"` objects |
| Taylor Morrison | taylor-morrison-site | ⚠️ built+verified, BLOCKED at egress | Sitecore fedmodel (scan 13 blocks for `availableHomesList`); adapter correct (3 recs/0 err on fixture) but Akamai bot-blocks the Kamatera datacenter IP (37KB html "404" challenge on /ca/sitemap.xml under load). Needs residential/allowlisted egress — NOT circumvented per standing rule. Source `active=false, health=DEGRADED`. |
| Meritage | meritage-homes-site | ⛔ NEEDS BROWSER-TRACE RECON | Sitecore XM Cloud headless — inventory (QMIs) is NOT in server HTML (state/community/search pages all render empty shells; only plan ranges + community lat/lon in `__NEXT_DATA__`). QMI spec homes load via a runtime XHR (`community-qmi-filters` → XM Cloud edge API). NEXT STEP: capture the XHR endpoint+params via Playwright/Browserbase, then build a Pulte-style JSON-API adapter. Egress is CLEAN (200s) — only the endpoint is unknown. |

**Live builder counts (2026-07-23):** Lennar 38,800+ (TX drain running) · D.R. Horton 513 (full CA) · PulteGroup 180 · KB Home 143 · Tri Pointe 98. **32 states, 7,464 TX homes** (TX-zero fixed).

## Pulte (pultegroup-site) — JSON API, CLEANEST (do next)
- Communities: `GET https://www.pulte.com/api/community/getcommunities?brand=Pulte&state=California&pageSize=1000000&pageNumber=0`
- Homes per region: `POST https://www.pulte.com/api/qmi/inventorysearch` body `{"StateName":"California","RegionName":"<region>","BrandName":"Pulte"}`. CA regions: Bay Area, Los Angeles, Orange County, Palm Springs, Riverside County, Sacramento, San Bernardino County.
- Home fields: `price/priceDiscount/finalPrice, bedrooms, bathrooms/halfBaths/totalBaths, squareFeet, garages, floors, address.{street1,city,state,zipCode} (strip whitespace), dateAvailable, salesPhoneNumber, communityName, communityId, communityLatitude/Longitude, inventoryPageURL`.
- Status: `dateAvailable` "Available Now" → MOVE_IN_READY. HAS lat/lon (community-level). Adapter fetch() should POST per region (LiveFetcher may need a POST variant — or fetch the getcommunities GET + per-community `GET /api/plan/qmiplans?communityId=`).

## Tri Pointe (tri-pointe-site) — Next.js RSC
- Sitemap index `https://www.tripointehomes.com/sitemap.xml` → `sitemap-communities.xml`; CA communities = `<loc>` paths starting `/ca/` (73). URL: `/ca/{county}/{community}`.
- Per community page: unescape RSC (`.replace('\\"','"')`), brace-match each object containing `"display_price":\d+`.
- Fields: `display_price/min_price, min_bedrooms, min_bathrooms, min_sq_feet, min_garage, min_stories, home_status/availability_status/tpg_status, move_in_date, floor_plan, homesite, plan_type, address, cities[], zip_code, submarket, state_alpha_2_code`.
- Phone: community `"phone":"949.432.9284"` or JSON-LD `LocalBusiness.telephone`.

## Taylor Morrison (taylor-morrison-site) — Sitecore fedmodel
- Sitemap index `https://www.taylormorrison.com/sitemap-index.xml` → CA `https://www.taylormorrison.com/ca/sitemap.xml` (601). Community = 4-seg `/ca/{metro}/{city}/{community}`; drop suffix pages + `*-community-group-page`.
- Per community: fetch `{community}/available-homes`, regex `data-fed-ref="fedmodel">(\{…\})</script>`, JSON.parse, walk `availableHomesList.sections[*].homes[]`.
- Fields: `address, city, state, zip, homeSite, price, wasPrice, bed, fullBath, halfBath, totalBath, garages, sqft, availabilityStatus ("1"=available), readyDate, isComingSoon, floorPlan, viewHomeLink.Url, photo.Src`.
- Phone: community JSON-LD `telephone`. (SoCal grouped communities 302-redirect → resolve leaf.)

## Meritage (meritage-homes-site) — Next.js __NEXT_DATA__
- Sitemap `https://www.meritagehomes.com/sitemap.xml`; CA = `/state/ca/` (403), 3 regions. 2-seg `/state/ca/{region}/{slug}`: slug=street-address → HOME page; slug=name → COMMUNITY page.
- Home page `__NEXT_DATA__` path `props.pageProps.componentProps.{guid}.data.lotSearch.lotData` (find the componentProps child with `.data.lotSearch.lotData`).
- Home fields: `priceSpec, monthlyPrice, bedrooms, fullBathrooms, halfBathrooms, squareFootage, garages, stories, address1/city/state/zip, latitude/longitude, status ("Inventory"), constructionStage, completionEstimated, lot_number, imageUrls, communities.targetItems[].communityId`. HAS lat/lon.
- Community page path `...data.communitySearch.communityData` → `metroPhoneNumber` (sales phone), lat/long, price/bed/bath ranges, totalQmis.

## Notes
- Register each in `apps/workers/src/cli.ts` ADAPTERS + add workspace dep in `apps/workers/package.json`.
- SourceRegistry rows exist for all (inactive). To ingest: activate + set collectionMethod=CRAWL, deploy adapters to Kamatera, run `pnpm --filter @homesonspec/workers pipeline -- --adapter=<key>` with the state env.
- Pulte/Meritage give lat/lon → homes appear on the map. KB/DR-Horton/TriPointe/TaylorMorrison lack per-home geo in v1 (search/lists only) — geocoding is a follow-up.

---

# Regional-builder feasibility recon — 2026-07-27 (plain-fetch pass)

Probed 8 registered "adapter pending" regional builders for **plain-curl, facts-only** inventory
feasibility (respecting robots; no bot-wall circumvention). Finding: modern regional builders
overwhelmingly render per-home inventory **client-side (XHR)**, so plain-fetch adapters like
Discovery/Pulte are the exception, not the rule. Classifications written to `SourceRegistry.notes`.

| Builder | key | verdict | evidence |
|---|---|---|---|
| **Highland Homes** | highland-homes-site | **XHR-TRACE — best next** | Fully-open robots (no disallows) + rich sitemap (`/{metro}/{city}/{community}/homesites`, `/quick-move-in/{metro}`). jQuery + svg-pan-zoom site plan; facts load via AJAX (no facts/JSON-LD/`__NEXT_DATA__` in server HTML). Large TX/AZ/CO inventory. **Next: browser-trace the homesites/QMI AJAX endpoint → Pulte-style JSON-API adapter.** |
| Signature Homes | signature-homes-site | FEED-EXISTS-BUT-THIN | WordPress; permitted WP REST CPT `/wp/v2/move-in-home` (X-WP-Total=4). ACF facts NOT `show_in_rest`; move-in-home page HTML carries no price/beds/sqft (image-baked). Only breadcrumb JSON-LD community name + homesite label extractable. Revisit if catalog grows or ACF is REST-exposed. |
| Van Daele | van-daele-homes-site | SPA | WordPress, open robots, `/{region}-available-homes/` pages JS-hydrated (inline `$` values are filter dropdowns, not listings). Next: trace available-homes XHR / admin-ajax action. |
| SummerHill | summerhill-homes-site | SPA | AngularJS (`ng-app="app"`, `HomesNowReadyCtrl`); `/quick-move-in` facts client-side. Next: trace QMI XHR. |
| DeNova | denova-homes-site | ROBOTS-LIMITED | Per-lot inventory hydrates from robots-DISALLOWED `/*sitesearch.json*`; community pages are server-rendered but show only plan RANGES. Per-home OFF-LIMITS w/o permission → partner-feed outreach. Community-level metadata OK. |
| Landsea | landsea-homes-site | BOT-WALL | Cloudflare bot challenge (redirects → risewellhomes.com "Sorry you have been blocked"). Datacenter egress blocked — NOT circumvented. Needs allowlisted egress or partner feed. |
| Fulton | fulton-homes-site | thin/defer | 93KB homepage, no inventory-feed markers on landing; likely SPA/XHR. Defer behind Highland. |

**Buildable-now shortlist (plain-fetch):** none in this batch. **Best ROI next:** Highland Homes
(clean robots + big inventory) via a browser-trace of its AJAX endpoint — the same proven next-step
as Meritage. SPA/XHR builders (Highland, SummerHill, Van Daele) all converge on the same tool:
capture the inventory XHR once (Playwright/Browserbase), then a JSON-API adapter.

---

# 2026-07-27 (PASS 2) — Highland RE-VERDICT + David Weekley adapter BUILT

**Highland Homes — DOWNGRADED from "best next" to BOT-WALLED (not plain-fetch buildable).**
Browser-traced the QMI page (`/quick-move-in/{metro}`) with headless Playwright: the Algolia
inventory search **never fires** — Cloudflare `challenge-platform` intervenes before the Vue app
runs its query (0 `*.algolia.net` requests captured across a 13s load + scroll). The only 32-hex
key embedded in the server HTML (`dd383f7c31fb03968005570eff61eff7`, appId `RYAJNPEGOZ`) is the
**Insights** key — it returns `403 "Index not allowed"` on index `highland` and 14 name guesses, so
the real search key is fetched at runtime *behind* the CF challenge. Net: Highland needs a
challenge-solving browser (Browserbase/2captcha) to reach both the runtime key and the index —
same class as Landsea, NOT the clean XHR-trace PASS 1 hoped for. Not circumvented (standing rule).

**David Weekley (`david-weekley-site`) — ADAPTER BUILT (this pass).** Pivoted to a national
builder whose 2026-07-22 recon was "HTML_PARSE diff-2 clean" and re-verified it holds:
- Sitemap `https://www.davidweekleyhomes.com/sitemap.xml` -> ~855 real
  `/new-homes/{st}/{market}/homes-ready-soon` list pages (drop `%7B...%7D` template rows).
- robots.txt only blocks querystring dupes (`?planId=`/`?communityId=`) — **list pages are allowed**.
- Each page is **fully server-rendered** (no browser). Per-home QMI card = `.single-tabbed-listing`
  showcase block: url `/homes-ready-soon/{st}/{mkt}/{city}/{community}/{jobNumber}` (jobNumber =
  builderInventoryId), `span.address` (street+city+ST+zip), `.starting-price .value` (price),
  `.square-footage .value` (sqft), `.feature.story-count/.bedroom-count/.number-baths/.garage-count
  .value`, `.plan-title`+`.plan-id`, `.label.ready` -> "Ready Now" (MOVE_IN_READY) or "Ready m/d/yyyy"
  (UNDER_CONSTRUCTION + est completion date).
- **Live-verified read-only:** 3 TX pages -> 131 communities + **284 inventory_home records, 0 errors**,
  full facts (e.g. `840 Perry Pass, Round Rock TX 78664 · $397,990 · 3bd/2ba · 1713sqft · MOVE_IN_READY`).
  Fixture-parse matches source exactly (The Heron job 19370084: 3bd/3ba/2gar/2story/$362,990/2190sqft).
- Env scope: `DAVIDWEEKLEY_STATE` (2-letter), `DAVIDWEEKLEY_PAGE_LIMIT`. No per-home geo in v1
  (list pages carry no lat/lon — geocoding is the same follow-up as KB/DR-Horton/TriPointe).
- Facts-only (images omitted, mediaRights=NONE). 6/6 vitest + typecheck green. Registered in
  `apps/workers/src/cli.ts`. **To ingest:** activate `david-weekley-site` + set collectionMethod=CRAWL,
  deploy, run `pnpm --filter @homesonspec/workers pipeline -- --adapter=david-weekley-site`.

**National adapter-pending backlog (all HEALTHY recon, plain-fetch, higher ROI than the SPA regionals):**
~~ashton-woods (ld+json QMI sitemaps, diff-2)~~ **✅ BUILT (PASS 3, below)** · beazer (open robots,
`__next_f` prices, diff-3) · mattamy (Sitecore JSS state, richest fields, diff-2) · ryan-homes
(data-* cards, diff-3) · lgi-homes (`__NEXT_DATA__` plan-priced) · mi-homes (data-price cards,
diff-2/3). Do these next before the bot-walled regionals.

---

# 2026-07-28 (PASS 3) — Ashton Woods adapter BUILT (JSON-LD QMI, HAS geo)

**Ashton Woods (`ashton-woods-site`) — ADAPTER BUILT (this pass).** The next national
plain-fetch target off the backlog. Confirmed the 2026-07-22 "ld+json QMI sitemaps" recon holds
and it is the RICHEST national source yet (per-home geo → homes appear on the map):
- Sitemap index `https://www.ashtonwoods.com/sitemap.xml` → 13 per-metro `{metro}-sitemap-qmi.xml`
  files (atlanta, austin, charleston, dallas, houston, jacksonville, myrtle-beach, nashville,
  orlando, phoenix, raleigh, san-antonio, tampa) → each lists per-home QMI URLs
  `/{metro}/{community}/{plan}/{street-address}`.
- robots.txt disallows only /content/ /input/ /walk-in/ /sales_center/ /interactive-floorplan
  /realtor-program/ /welcome — **the metro QMI pages we read are allowed.**
- Each QMI page is **fully server-rendered** (no browser) with a clean JSON-LD pair:
  - `SingleFamilyResidence`: name, address{streetAddress/addressLocality/addressRegion/postalCode},
    `ContainedIn` (community), `Floorsize` ("2260FTK" → sqft), `AmenityFeature`
    ["4 Bedrooms","3 Baths | 1 Half Baths","2 Stories","2 Car Garage"], `geo{latitude,longitude}`,
    `telephone` (community sales phone).
  - `Product.offers`: `price` (number), `availability` (schema.org/InStock → for-sale gate).
- One page == one `inventory_home` + its `community` (community carries the same per-home geo so it
  also maps). Baths = full + half·0.5 ("3 Baths | 1 Half Baths" → 3.5). **coord() helper preserves
  negative longitudes** (US lon < 0 — same class of bug caught overnight in the Discovery adapter;
  a naive num() would strip the minus). builderInventoryId = the stable per-home page URL (AW has no
  numeric job id). Extract skips sold/unpriced pages (no price or availability≠InStock).
- **Live-verified read-only** (3 metros/states through the real adapter `extract()`, 0 errors):
  `5712 Terrace Drive, Watauga TX 76148 · $340,200 · 3bd/2.5ba · 2050sqft · geo(32.879,-97.260)`;
  Houston TX `$274,990 · 4bd/2ba`; Queen Creek AZ `$599,990 · 3bd/2.5ba`. Fixture parse matches
  source exactly (321 Possumhaw Lane, San Marcos TX: 4bd/3.5ba/2gar/2story/$369,636/2260sqft).
- Env scope: `ASHTONWOODS_METRO` (comma-list of metros, default all), `ASHTONWOODS_PAGE_LIMIT`.
- Facts-only (image omitted, mediaRights=NONE). 6/6 vitest + typecheck green (workers CLI typecheck
  green too). Registered in `apps/workers/src/cli.ts`. **To ingest (GATED go-live):** activate
  `ashton-woods-site` + set collectionMethod=CRAWL, deploy, run
  `pnpm --filter @homesonspec/workers pipeline -- --adapter=ashton-woods-site`.

**Remaining national backlog (do next):** beazer · mattamy (Sitecore JSS, richest) · ryan-homes ·
lgi-homes · mi-homes — then the bot-walled regionals (Highland/Landsea) which need allowlisted egress.