auto-save: 2026-07-29T09:37:47 (1 files) — collectors/richmond-american/

63d56d74733350504654ba1e0b52ccc127ac3b82 · 2026-07-29 09:37:54 -0700 · Steve Abrams
Files touched

A collectors/richmond-american/RECON.md
Diff

commit 63d56d74733350504654ba1e0b52ccc127ac3b82
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Wed Jul 29 09:37:54 2026 -0700

    auto-save: 2026-07-29T09:37:47 (1 files) — collectors/richmond-american/
---
 collectors/richmond-american/RECON.md | 113 ++++++++++++++++++++++++++++++++++
 1 file changed, 113 insertions(+)

diff --git a/collectors/richmond-american/RECON.md b/collectors/richmond-american/RECON.md
new file mode 100644
index 00000000..7c50c741
--- /dev/null
+++ b/collectors/richmond-american/RECON.md
@@ -0,0 +1,113 @@
+# Richmond American Homes — RECON (2026-07-29, TK-10001)
+
+**Status: DEFERRED — architecture-fit blocker (NOT a robots/bot blocker, NOT bucket-D).**
+The inventory data is fully reachable, robots-ALLOWED, and extracts cleanly under an
+honest UA. It is NOT collectible by the current HTTP-only collector framework because
+the site's facts only materialize after a **real-browser render**. This adapter was
+intentionally NOT built; see the DTD verdict below. This file documents the exact
+verified path so a future browser-enabled shared fetcher can adopt it in ~1 hour.
+
+---
+
+## Why it's deferred (DTD 3/4 verdict B, 2026-07-29)
+
+`richmondamerican.com` is a **Blazor Server** SPA with **prerendering DISABLED**:
+
+- A plain HTTP `GET` of a per-home page returns a **242 KB empty app shell** —
+  no `<title>`, no JSON-LD, no price/beds/baths/sqft anywhere in the HTML.
+- Facts arrive only as **DOM diffs over the Blazor WebSocket** (`wss://www.richmondamerican.com/_blazor?id=…`)
+  after `/_blazor/negotiate`, rendered by `blazor.server.js` in a real browser.
+  (Verified: 27 WS frames / ~528 KB per home; 2 frames carry the fact payload.)
+
+The shared `@homesonspec/collectors-common` `LiveFetcher` is **HTTP-only** (`GET` /
+`postJson` via `fetch()`, no browser) and every one of the ~30 existing adapters uses
+it. A one-off Playwright render smuggled inside this adapter's `fetch()` would:
+break the invariant that a `RawPage` == an HTTP response, add a hidden non-workspace
+dependency no other adapter has, and fracture the shared pattern. The panel
+(Qwen + Grok + Kimi = B; Claude dissent = A; Codex abstained on a bad key) ruled the
+correct fix is a **shared browser-enabled fetcher**, which is out of the
+"touch only collectors/richmond-american/, no shared edits" scope of TK-10001.
+
+---
+
+## robots.txt — the data path IS allowed
+
+`https://www.richmondamerican.com/robots.txt` disallows (do NOT use):
+`/Community/LoadAll/`, `/Home/GenerateSiteMapXml/`, `/bin/`, `/Views/`,
+`/communityInfo*`, `/my-account/*`, `/account/*`, `*.config/*.dll` etc.
+
+The per-home detail URLs below are **ALLOWED** (no rule matches them). Verified with
+UA token `homesonspecbot` against the standard longest-match precedence.
+
+---
+
+## VERIFIED render path (sitemap → per-home URL → honest-UA Blazor render → DOM)
+
+1. **Sitemap index** (allowed): `https://www.richmondamerican.com/sitemap.xml`
+   → `sitemap1.xml … sitemap7.xml`.
+   - `sitemap4.xml` = **communities**  (`/{state}/{metro}/{city}/{community}/`)
+   - `sitemap5.xml` = **floor plans** (plan-level aggregate — SKIP, not per-home)
+   - **`sitemap6.xml` = the QMI per-home inventory** — **1,681** URLs of the form
+     `/{state}/{metro}/{city}/{community}/{plan}/{homeId}/`
+     where `homeId` is `NNNNNNNN-NNNN` (e.g. `24880000-0046`). This is the feed.
+
+2. **Fetch each per-home URL in a real browser with the honest UA**
+   `HomesOnSpecBot/0.1 (+homesonspec.com; site-owner-facts-only)` (or the
+   framework UA `HomesOnSpecBot/0.1 (+https://homesonspec.com/bot; contact: data@homesonspec.com)`).
+   `waitUntil: "networkidle"`, then `waitForFunction` until `document.body.innerText`
+   contains a `$NNN,NNN` price OR a "Model For Tour" / "no longer available" marker,
+   then `waitForTimeout(~1500ms)`. (~9 s per home total.)
+
+   **IMPORTANT — landing pages hit a CHEQ bot-wall, detail pages do NOT.**
+   `/quick-move-in-homes`, `/inventory`, `/homes` etc. are SPA entry points guarded by
+   CHEQ (`obs.starsbuildingweb.com/ct`) that redirects an automated client to `/404/`.
+   The **per-home detail URLs from sitemap6 render fine** under the honest UA — do NOT
+   drive collection through the QMI landing/search pages; go straight from sitemap6.
+
+3. **Extract from the rendered DOM** (`document.body.innerText`, regex — see below).
+   No JSON-LD, no `window.__` state, no embedded JSON in the DOM: text-scrape only.
+
+### Field extraction (regex over rendered innerText — verified 8/8 across FL/CO/AZ/CA/UT)
+
+| Field       | Source                                               |
+|-------------|------------------------------------------------------|
+| address     | `document.title` (`"{addr} \| Home For Sale in {community}"`) OR innerText `NN..., City, ST, ZIP` |
+| plan name   | `<h1>` (e.g. "Agate", "Dillon II")                   |
+| community   | tail of `<title>` after "in " / the community URL slug |
+| price       | `\$([0-9]{3},[0-9]{3})`  — **null for "Model For Tour" homes (they have no sale price — correct, keep null)** |
+| beds        | `([0-9]+)\s*Bed\b`                                    |
+| baths       | `([0-9](?:\.[0-9])?)\s*Bath\b`  (half-baths already folded, e.g. 2.5) |
+| sqft        | `([0-9],?[0-9]{3})\s*Approx\.\s*sq\.?\s*ft` (number sits ABOVE the "Approx. sq. ft." label) |
+| garage      | `([0-9]+)\s*Car Garage`                              |
+| move-in     | `Ready for move-in (now|<Month> <Year>) at <community>` → MOVE_IN_READY when "now", else UNDER_CONSTRUCTION + estCompletion |
+| homeId      | trailing `NNNNNNNN-NNNN` path segment (→ builderInventoryId) |
+| state/city  | parsed from the address tail (`, City, ST, ZIP`) → `normalizeStateCode` |
+| **lat/lon** | **NOT available** — Blazor renders an SVG map, no lat/lon in any request. Leave null; the city+state gate works off the parsed address. |
+
+### Emission (mirror meritage-homes exactly)
+- Emit **community FIRST** (canonicalHints `{builderSlug:"richmond-american", communityName}`),
+  then **inventory_home** (`{builderSlug, communityName, address, builderInventoryId:homeId, planName}`).
+- `homeType: "SINGLE_FAMILY"`, `images: []` (facts-only, no images).
+- Skip a home with no address or no community (log, don't stage — same as Meritage).
+- `RICHMOND_PAGE_LIMIT` (default 10) caps homes/run; optional `RICHMOND_STATE` filter
+  the sitemap6 URL list by `/{state}/` slug.
+
+---
+
+## Self-test proof (data quality — 8 homes, 6 states)
+
+Ran `/tmp/richmond-extract2.mjs` on 8 sitemap6 URLs spread across the catalog:
+
+- **8/8** homes returned distinct real street addresses + distinct sqft
+  (2070 / 2020 / 2510 / 2600 / 2170 / 2580 / 4518 / 2420) — genuinely per-home,
+  NOT floorplan- or community-aggregate.
+- **6/8** have a real price ($437,997 … $979,950); the 2 nulls are correctly
+  "Model For Tour" homes (no sale price by definition — a real distinction, not a miss).
+- **8/8** beds / baths / sqft / garage / move-in complete.
+- States seen: FL, CO, AZ, CA, UT.
+
+## What to do to ship it
+Add a browser-render mode to `collectors-common` (a `RenderFetcher` alongside
+`LiveFetcher`, same robots + rate-limit + honest-UA rails, but a Playwright page render
+returning rendered HTML as the RawPage body). Then this adapter is a thin sitemap6
+crawler + innerText-regex `extract()` — the fields above are already verified.
← 5c09c690 collectors: add Mattamy Homes — cracked the 'bucket-D' natio · back to Homesonspec · lennar refresh runner: source DB pw from gitignored .env (wa 9c5e6719 →