`collectors/SPECS-AUDIT.md`

129 lines
# All-specs capture audit — "capture every spec/info each builder offers"

**Goal (Steve, 2026-08-02):** stop capturing only the ~18 core columns; capture
**everything each builder exposes** and surface it on the home/community detail
pages. Approved scope: **all 28 active builders, audit-first**.

## Method (proven, $0, offline)
Raw source bytes are already stored per fetch at
`var/snapshots/<builder-key>/<hash>.html` (Kamatera), indexed by the
`RawSnapshot` table (`storagePath`, `httpStatus`, `url`). So we can enumerate
the **full field universe** each builder exposes without re-hitting their site:
extract the structured blob (`var model={…}` / `__NEXT_DATA__` / JSON API /
RSC `__next_f`, per `BUILDERS.md`), union all keys, diff against captured set,
classify each key **SPEC** (capture) vs **UI-NOISE** (drop). No external fetch,
no cost.

## Storage foundation (needed before expansion)
`InventoryHome` has no extensible field. Add **`specs Json?`** (grab-bag for
everything non-core: features, options, monthly payment, was-price, plan code,
lifecycle flags, purchase URL, promo, energy, virtual tour, etc.). Core
filter/sort fields stay as typed columns. Publisher writes `specs` from a
reserved payload key; detail page renders the full map. *(Prod schema migration
is a gated deploy.)*

## Audit results

### dr-horton-site — DONE (reference pattern)
Raw payload: **55 distinct keys; ~10 captured.** Real specs we currently drop:
- `OriginalPrice` / `TotalSalePrice` → **was-price / price-drop** signal
- `EstimatedMontlyPayment` [sic] → monthly payment estimate
- `MultiGen` / `MultiGenImageSrc` → multi-gen floor-plan flag
- `PlanCode` / `PlanName` → floor-plan identity (populate the idle `FloorPlan` table)
- `QmiAvailable` `Available` `IsPending` `JdeIsUnderContract` `JdeIsSold` `JdeIsSpec` `JdeIsClosed` `StageCode` `Status` → granular lifecycle
- `PurchaseUrl` `Url` → direct builder listing link
- `AllowOffer` → accepts offers
- `RequiresPromo` `PromoQueryString` `ModalText` → promo / incentive
- UI-noise dropped: FontColor, Background, BadgeImage, BrandImage, BlockSize, FontColor, SortingType, Sortorder, HtmlItems, IsNull, IsGlobalNullId…

### Fleet audit pass 1 (scripts/audit-fields.mjs, offline, all 28) — 2026-08-02
Ran the generic enumerator against each builder's newest 200-OK snapshot.

**Capture-ready (rich, real specs confirmed):**
| Builder | fields | standout specs to capture |
|---|---|---|
| pultegroup | 166 | apr, adjustableRate, basementType, averageDiscountRate, almostSoldOutFlag, comingPriceRange |
| toll-brothers | 147 | (large — needs noise trim) |
| perry-homes | 73 | baths, completionDate, designNumber, description |
| landsea | 46 | base_price, basements, cars_spaces, discount_percentage (Algolia index) |
| meritage | 43 | construction_stage, completion_actual/estimated, floorplan_description |
| dr-horton | 28 | OriginalPrice, EstimatedMontlyPayment, MultiGen, PlanCode (reference) |
| fischer | 11 | formattedPrice/Beds/Baths/Sqft/Floors |
| discovery | 6 | baths, plan, priceRaw |

**Noise-inflated — auditor grabbed the wrong blob, needs targeted parser:**
lennar (292, Auth0/Agentforce config), lgi (154, i18n labels), mattamy (18, Sitecore layout), holt (15, WordPress plugin), fulton (3, AJAX envelope).

**Zero — newest snapshot was an index/sitemap, NOT a home-detail page (re-audit against a detail-URL snapshot):**
ashton-woods, beazer, century-communities, david-weekley, dream-finders,
drees-homes, gl-homes, highland, kb-home, khovnanian, ryan-homes, shea-homes,
taylor-morrison, tri-pointe.

**Pass-2 fixes:** (1) select snapshot by URL matching each builder's home-detail
pattern (not just newest); (2) tighten NOISE denylist for config/i18n blobs;
(3) for RSC builders (tri-pointe) widen the __next_f home-object matcher.

### Fleet audit pass 2 (largest-snapshot re-run of the 14 zeros) — 2026-08-02
All 14 stayed at 0 **despite large content pages** (david-weekley 7.8MB,
kb-home 5.7MB, shea 4.2MB, khovnanian 2.3MB, ryan 1.8MB). The data IS in
these snapshots — the generic auditor just doesn't parse their blob shapes:
- taylor-morrison → `data-fed-ref="fedmodel">{…}` (Sitecore) — not handled
- tri-pointe → nested RSC `__next_f` — inner `[^{}]` matcher fails on nesting
- kb-home → `var allMIRs=JSON.parse("…")` giant double-encoded string — match broke
- shea / ryan / beazer / khovnanian / century / drees / dream-finders /
  ashton-woods / david-weekley / gl-homes / highland → bespoke shapes

**Conclusion:** a generic auditor tops out at ~4 blob shapes. For these 14,
"audit" == "extend the extractor" (the builder's own extractor already walks
its structure). So they fold directly into the per-builder fan-out below —
each worker reads that builder's extractor, enumerates the object it iterates,
and adds the SPEC fields. No separate audit step for them.

### Proving slice — dr-horton end-to-end (schema→extract→publish→render) — 2026-08-02
DONE + typechecks clean across schemas/collector/publisher/web. Verified against a
real snapshot: DRH home Items carry `PlanCode` (real, e.g. "X475"), `OriginalPrice`
(often 0 sentinel → dropped), `MultiGen` (bool). Extractor emits a pruned `specs`
map (nulls + false-flags dropped); publisher persists it; detail page shows an
"Additional details" block + builder-listing link.
**Follow-up:** `EstimatedMontlyPayment` / `TotalSalePrice` / `PurchaseUrl` are NOT on
the home Items — they live elsewhere in DRH's `var model` tree (plan-level?). A
second extraction pass is needed to reach them. Item-level specs work today.

### Capture-ready extension — 6-builder batch outcome — 2026-08-03
Extended the proving slice to the capture-ready builders. Verified each against a
REAL home object (the DRH lesson held every time — audit-flagged fields were often
community-level, not on the home). All committed, all typecheck clean.

**✅ Rich per-home specs captured (4):**
- **dr-horton** — planCode, multiGen, originalPrice (Item-level)
- **pulte** — priceDiscount ($ off list), mortgagePayment, hasBasement, isActiveAdult, discountExpiration, virtualTour, planDescription
- **meritage** — monthlyPayment (respects hide_monthly_payment), previous_mhdc_price (was-price), moveInTimeframe, floorplanName, incentiveAvailable, virtualTour, mlsNumber
- **perry** — discountedPrice (drop signal), promotion, description, elevation, market

**⏸️ Deferred — no rich per-home extras in what we crawl (verified, not skipped blindly):**
- **landsea** — Cloudflare bot-walled/DEGRADED; stored snapshots don't yield a clean Algolia home to verify against. Won't add unverified field names.
- **fischer** — CombinedHome/DetailFacts carry only core + formatted duplicates; ld+json description consumed for community name only. Extra capture needs parseDetail surgery for 45 homes (low ROI).
- **discovery** — per-home data is scraped from HTML qmi-cards (core only). NOTE: its COMMUNITY feed is very rich (tax_rate, active_adult, amenities, schools, promos, brochure PDFs) — a future COMMUNITY-enrichment opportunity, not home-specs.

### GATED — capture requires prod changes
Beyond read-only auditing, actually CAPTURING specs needs: (a) `specs Json?`
schema migration on the live Kamatera DB, (b) per-builder extractor edits, and
(c) a prod deploy of the worker + web. All Steve-gated. The 8 capture-ready
builders can be done first as the proving slice.

### Remaining 27 — TODO (fan-out, one worker per builder)
lennar · pultegroup · meritage-homes · century-communities · taylor-morrison ·
lgi-homes · perry-homes · david-weekley · mattamy · dream-finders · kb-home ·
ryan-homes · tri-pointe · khovnanian · drees-homes · beazer · fulton-homes ·
highland-homes · shea-homes · gl-homes · holt-homes · landsea-homes ·
ashton-woods · fischer-homes · discovery-homes · toll-brothers · brookfield

Each worker: sample newest 200-OK snapshot → enumerate keys → classify
SPEC/NOISE → append its table here → extend that builder's extractor to emit
the SPEC fields into `payload` (core→typed, extras→`specs`).

## Downstream (after capture)
- Publisher: persist `specs` + populate `FloorPlan` from PlanCode/PlanName.
- Detail page: render full spec map, grouped (pricing, layout, availability, links).
- New facets worth adding once populated: price-drop (was-price), monthly payment band, multi-gen, under-contract filter.