`LISTING-COVERAGE.md`

55 lines
# Listing Coverage — M-B3 firm↔listing pilot probe matrix

The heart of the product: tie each publicly-displayed listing to the brokerage **firm**
that holds it. This file records every candidate site we probed, the structured-data
method found, the robots.txt verdict, and whether we piloted it.

## Legal rails honored on every probe

- **Facts only** stored: address, price, beds, baths, sqft, lat/lng, status, listing URL.
  We NEVER copy or store listing photos, agent remarks, or marketing descriptions.
- Every stored row keeps the **source URL** and is presented as *the broker's own listing*
  (link out), never as ours.
- robots.txt honored: a disallowed listings path = **SKIP** (recorded here). Crawl-delay
  respected; 2–4s jitter between fetches.
- Structured-data priority: (1) schema.org JSON-LD `RealEstateListing`/`SingleFamilyResidence`/
  `Product`/`Offer`, (2) public listings sitemap.xml, (3) a public JSON/API the site itself calls.
  No aggressive DOM-scraping fights; a 403/challenge = SKIP + record.
- **Pilot scope**: proof-of-concept only. ≤50 listings/site, 2 sites piloted. Blanket ingest
  is GATED (see `~/.claude/yolo-queue/pending-approval/usrealestate-listing-blanket-ingest.md`).

## Probe matrix

| Site | firm_id(s) | robots verdict (UA `*`) | Structured method | Facts in JSON-LD | Piloted? |
|---|---|---|---|---|---|
| **coldwellbanker.com** | 27268, 64079, 82366 | ALLOW — `User-agent: *` empty Disallow; dedicated `sitemapindex-listings-*` published | **JSON-LD** `Product`+`SingleFamilyResidence`+`Offer`+`GeoCoordinates`+`PostalAddress`; listings sitemap (new-day/new-week) | addr, price, geo, numberOfRooms, seller org | **YES** |
| **realtytexas.com** | 1808, 1872, 1875… | ALLOW — only `/api/*`,`/open-houses/*`,`/clients/*` disallowed; `sitemap.xml` → per-county `sitemap_listings_onmarket_*` | **JSON-LD** `SingleFamilyResidence`+`Offer`+`GeoCoordinates`+`PostalAddress`+`PropertyValue` | addr, price, beds, baths, **sqft (floorSize)**, yearBuilt, geo, MLS# | **YES** |
| exprealty.com | 1830, 27261 | PARTIAL — sitemaps published, but per-listing PDPs are `Disallow: /*?listing-id=*` and `/search/*`; the address-page sitemap entries resolve to area/`AggregateOffer` pages, not single-listing PDPs | JSON-LD present but area-level `AggregateOffer`, no clean per-listing facts at an allowed URL | area aggregates only | NO — no clean allowed per-listing facts |
| elliman.com | 65704, 27445 | ALLOW — `/sales/*`,`/sold/*` explicitly Allowed (filter query-strings disallowed) | Candidate — JSON-LD likely on `/sales/*` PDPs | (not fully probed; deferred to blanket phase) | NO — pilot cap reached at 2 clean sites |
| compass.com | 27334, 1931, 70948… | PARTIAL — LDP/PDP sitemaps published, `/api/` disallowed; PDPs themselves allowed for `*` | JS-heavy, strong bot defenses (challenge likely on server fetch) | risky — deferred | NO — anti-bot risk; skip per rails |
| thekcorealty.net | 27269, 27545 | ALLOW — `*` only disallows /CVS,/ckeditor | IDX widget site (likely 3rd-party framed IDX); no first-party listings sitemap found | none first-party | NO — no first-party structured feed |
| urgfl.com | 30123 | ALLOW — only `/wp-admin/` disallowed, Crawl-Delay 2 | WordPress IDX; no first-party listings sitemap / clean JSON-LD found in probe | none first-party | NO — no clean first-party structured feed |

## Piloted sites — extraction detail

### coldwellbanker (`source = 'coldwellbanker'`)
- **Discovery**: `robots.txt` → `sitemapindex-listings-new-day.xml` → leaf `sitemap-listings-new-day-brand-*.xml` → listing PDP URLs (`/{state}/{city}/{street}/lid-{ID}`).
- **source_id**: the `lid-{ID}` token from the canonical URL (stable per listing).
- **Facts**: `Product.offers.price`, `SingleFamilyResidence.address.*`, `.geo.latitude/longitude`,
  `.numberOfRooms`. Beds/baths not consistently in JSON-LD → left null when absent.
- **Firm resolution**: `Offer.seller.name` = "Coldwell Banker Realty" (brand) → domain match
  `coldwellbanker.com` → firm rows. We pin the listing to the firm whose `firm_site.url`
  domain matches the source host.

### realtytexas (`source = 'realtytexas'`)
- **Discovery**: `sitemap.xml` → `sitemap_listings_onmarket_us_tx_{county}_01.xml` → listing PDP
  URLs (`/{city}/{slug}-id-{ID}-mls-{MLS}.html`).
- **source_id**: the `mls-{MLS}` number from the URL (falls back to `id-{ID}`).
- **Facts**: `SingleFamilyResidence.numberOfBedrooms`, `.numberOfBathroomsTotal`,
  `.floorSize` ("N sqft"), `.address.*`, `.geo.*`, `.offers.price`, `.yearBuilt`.
- **Firm resolution**: source host `realtytexas.com` → `firm_site.url` domain match → firm rows.

## Region crosswalk
Each listing's `(addressRegion state, lat/lng)` is matched to a county `region_id` via nearest
county centroid within the listing's state (cheap, no geocode API). No match → `region_id` null.