`docs/SOURCING.md`

82 lines
# Data Sourcing Doctrine — broker-of-record is the source of truth

**Standing rule (Steve, 2026-07-30):** Every deal must carry its **listing broker firm + agent
name**. From that, resolve the **original listing on the broker's own site** and treat *that* as
the source of truth. **Do not source deal facts (price, status, description, contact) from
aggregators — CREXi, Redfin, Zillow, CoStar, LoopNet — unless the aggregator genuinely IS the
listing broker** (they never are; they're marketplaces).

## Why
Aggregators re-key, stale, and de-brand listings. The listing brokerage is the party of record and
its own site is the freshest, most accurate, legally-attributable version of the deal. Trusting the
marketplace over the broker is how a viewer shows a wrong price or a dead listing and mislabels the
firm as "Crexi" instead of the real brokerage.

## Source-of-truth tiers (highest → lowest)
1. **firm-direct** — pulled from the listing brokerage's own site/API (`sources/firms.js`). Canonical.
2. **broker-of-record known** — still aggregator-origin, but we have the real listing broker firm +
   agent (from the CREXi `/assets/<id>/brokers` block). Eligible for promotion to firm-direct by
   resolving the deal on that firm's site.
3. **aggregator** — origin is a marketplace and we have no broker-of-record yet. Lowest trust;
   flag for broker resolution.

An aggregator record is a **breadcrumb to find the broker**, never the final truth.

## How this is implemented
- `scripts/enrich-provenance.js` — joins the real listing broker (firm + agent) from
  `data/raw/broker-blocks.json` onto every deal in `data/ranked.json`, and stamps each deal with
  `source_of_truth` (`firm-direct` | `broker-of-record` | `aggregator`), `broker_firm`,
  `broker_agent`, and `broker_agents[]`. Idempotent, $0, local.
- The viewer (`public/deals.html`) surfaces a **Source** facet + the real broker firm/agent, so
  aggregator-dependence is visible and filterable.
## Phase 2 findings (2026-07-30, ~$0.60 Browserbase spent, Steve-approved)
Discovery run on all 13 dormant firms (`run-firm-discovery.sh`) + raw-body capture on the 2 with
populated APIs (`capture-firm-body.js`). **Verdict: national-brokerage firm-direct is the WRONG tool
for LA-county coverage.**
- **Populated listings API:** only Colliers (`/coveo/rest/search/v2`) and Newmark
  (`/api/properties/search`). But both serve a **nationwide, un-geo-filtered** index — a "Van Nuys, CA"
  search returned NY/London/KY/WA properties, **0 LA-county** — and prices are mostly text
  ("Negotiable"). Colliers' Coveo `raw` block has no structured price/address at all.
- **Empty results (interaction/geo-bbox gated):** M&M, CBRE, JLL, Matthews, Cushman, Lee.
- **Zero JSON (SSR / bot-walled):** Kidder, NAI, Berkadia, Stepp.
- **Skip (aggregator):** LoopNet (CoStar-owned) — excluded by doctrine anyway.
→ **Do NOT wire these as primary sources.** CREXi stays the LA discovery layer (LA-dense, priced,
  carries the broker-of-record). Firms left at `needs-discovery` in `sources/firms.js`.

## Phase 2-lite (recommended next, still CREXi-based)
Resolve the broker-of-record for the **1,747 aggregator-only** deals not yet in
`broker-blocks.json` by re-hitting CREXi `/assets/<id>/brokers` (`enrich-tier1-crexi.js` path),
pushing broker-of-record coverage from 43% toward ~90%. Metered (~$0.04–0.10 batched) — Steve-gated.
Optionally deep-link each deal to the broker's own site for canonical verification.

## Residential (SFR + condo) — the Redfin frontier
Residential is 100% Redfin-sourced (aggregator). The gis-csv feed carries NO listing agent
(`fetch-sfr-redfin.js` — broker_name/firm_name stay NULL); the listing broker is captured separately
per-listing via Redfin's `mainHouseInfoPanelInfo` detail endpoint by `fetch-sfr-agents.js` (SFR) and
`fetch-redfin-agents.js` (condos). Writeback path is intact end-to-end: those scripts
`UPDATE sfr SET broker_name, firm_name` → `/api/residential` serves them → `mapSfr` → the viewer's
doctrine classifier → the Brokerage facet. Newly-resolved brokers surface with no extra wiring.
- **Baseline:** ~839/17,700 SFR (4.7%) + ~640/2,190 condos (29%) have a broker; the rest unresolved.
- **Cost:** ~40 listings per Browserbase session (~$0.04); full SFR resolution ≈ $18. Metered,
  Steve-gated; the script self-caps at CC_MAX_COST (default $1.50) per run. Some agents are Redfin-
  suppressed → honest NULL, never fabricated.

## Hard lines
- **THE RULE (Steve, 2026-07-31): CREXi / Redfin may be used to DISCOVER the broker + firm ONLY —
  no more data. Then get ALL listing info from the broker's or firm's own site.** Aggregators
  (Redfin, Zillow, CoStar, CREXi, LoopNet) are a phone book for *who the listing broker is*, never
  the record of *what the deal is*. Allowed from an aggregator: broker name + firm (the discovery
  breadcrumb, + an address only as a locator to find the listing on the firm site). NOT allowed from
  an aggregator: price, status, cap rate, units, description, specs, images — those come from the
  **listing brokerage's own site/feed**, or a **licensed MLS/IDX/RETS feed** we're entitled to.
- Two-layer pipeline: **discovery layer** (CREXi/Redfin → broker+firm) → **truth layer** (firm's own
  site → the deal's real data). The broker-of-record resolution (Phase 1/2-lite) IS the allowed
  discovery step; the CREXi deal fields (price/units/status) are aggregator data pending firm-site
  truth, not canonical.
- Kills the residential-broker-from-Redfin *data* scrape (page-scrape AND Browserbase); residential
  SFR/condo detail stays provisional aggregator data until a firm-site or licensed feed supplies it.
  If we can't source a field legitimately, leave it blank + label honestly — never scrape a portal.
- Never present an aggregator record as canonical when a broker-of-record is known.
- Business-contact fields only (CCPA); record `source_url` per enriched field.
- Firm-direct scraping spend is shown per-run and gated.