docs(specs): all-specs capture audit — method + dr-horton reference (55 fields, ~10 captured)

478e1c73e162066a7985670b428e4ddf5eecb8dd · 2026-08-02 08:10:32 -0700 · Steve Abrams

Audit-first plan for capturing every spec each builder exposes. Proven $0
offline method via var/snapshots RawSnapshot. dr-horton reference done:
was-price, monthly payment, multi-gen, plan code, lifecycle flags, purchase
URL, promo all currently dropped. Remaining 27 builders = fan-out work-list.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

A collectors/SPECS-AUDIT.md

Diff

commit 478e1c73e162066a7985670b428e4ddf5eecb8dd
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Sun Aug 2 08:10:32 2026 -0700

    docs(specs): all-specs capture audit — method + dr-horton reference (55 fields, ~10 captured)
    
    Audit-first plan for capturing every spec each builder exposes. Proven $0
    offline method via var/snapshots RawSnapshot. dr-horton reference done:
    was-price, monthly payment, multi-gen, plan code, lifecycle flags, purchase
    URL, promo all currently dropped. Remaining 27 builders = fan-out work-list.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 collectors/SPECS-AUDIT.md | 53 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/collectors/SPECS-AUDIT.md b/collectors/SPECS-AUDIT.md
new file mode 100644
index 00000000..26a031b2
--- /dev/null
+++ b/collectors/SPECS-AUDIT.md
@@ -0,0 +1,53 @@
+# All-specs capture audit — "capture every spec/info each builder offers"
+
+**Goal (Steve, 2026-08-02):** stop capturing only the ~18 core columns; capture
+**everything each builder exposes** and surface it on the home/community detail
+pages. Approved scope: **all 28 active builders, audit-first**.
+
+## Method (proven, $0, offline)
+Raw source bytes are already stored per fetch at
+`var/snapshots/<builder-key>/<hash>.html` (Kamatera), indexed by the
+`RawSnapshot` table (`storagePath`, `httpStatus`, `url`). So we can enumerate
+the **full field universe** each builder exposes without re-hitting their site:
+extract the structured blob (`var model={…}` / `__NEXT_DATA__` / JSON API /
+RSC `__next_f`, per `BUILDERS.md`), union all keys, diff against captured set,
+classify each key **SPEC** (capture) vs **UI-NOISE** (drop). No external fetch,
+no cost.
+
+## Storage foundation (needed before expansion)
+`InventoryHome` has no extensible field. Add **`specs Json?`** (grab-bag for
+everything non-core: features, options, monthly payment, was-price, plan code,
+lifecycle flags, purchase URL, promo, energy, virtual tour, etc.). Core
+filter/sort fields stay as typed columns. Publisher writes `specs` from a
+reserved payload key; detail page renders the full map. *(Prod schema migration
+is a gated deploy.)*
+
+## Audit results
+
+### dr-horton-site — DONE (reference pattern)
+Raw payload: **55 distinct keys; ~10 captured.** Real specs we currently drop:
+- `OriginalPrice` / `TotalSalePrice` → **was-price / price-drop** signal
+- `EstimatedMontlyPayment` [sic] → monthly payment estimate
+- `MultiGen` / `MultiGenImageSrc` → multi-gen floor-plan flag
+- `PlanCode` / `PlanName` → floor-plan identity (populate the idle `FloorPlan` table)
+- `QmiAvailable` `Available` `IsPending` `JdeIsUnderContract` `JdeIsSold` `JdeIsSpec` `JdeIsClosed` `StageCode` `Status` → granular lifecycle
+- `PurchaseUrl` `Url` → direct builder listing link
+- `AllowOffer` → accepts offers
+- `RequiresPromo` `PromoQueryString` `ModalText` → promo / incentive
+- UI-noise dropped: FontColor, Background, BadgeImage, BrandImage, BlockSize, FontColor, SortingType, Sortorder, HtmlItems, IsNull, IsGlobalNullId…
+
+### Remaining 27 — TODO (fan-out, one worker per builder)
+lennar · pultegroup · meritage-homes · century-communities · taylor-morrison ·
+lgi-homes · perry-homes · david-weekley · mattamy · dream-finders · kb-home ·
+ryan-homes · tri-pointe · khovnanian · drees-homes · beazer · fulton-homes ·
+highland-homes · shea-homes · gl-homes · holt-homes · landsea-homes ·
+ashton-woods · fischer-homes · discovery-homes · toll-brothers · brookfield
+
+Each worker: sample newest 200-OK snapshot → enumerate keys → classify
+SPEC/NOISE → append its table here → extend that builder's extractor to emit
+the SPEC fields into `payload` (core→typed, extras→`specs`).
+
+## Downstream (after capture)
+- Publisher: persist `specs` + populate `FloorPlan` from PlanCode/PlanName.
+- Detail page: render full spec map, grouped (pricing, layout, availability, links).
+- New facets worth adding once populated: price-drop (was-price), monthly payment band, multi-gen, under-contract filter.

← 49800740 feat(web): gated Google AdSense banner slots (top + footer) · back to Homesonspec · auto-save: 2026-08-02T17:46:59 (1 files) — scripts/audit-fie 3c8b733c →