`BROKER-COVERAGE.md`

91 lines
# Broker registry — state coverage matrix (M-B1)

Probed 2026-07-21, curl-only (DTD verdict: OPEN-FILES-ONLY this pass — no headless
scraping of gated lookup portals, no paid subscriptions). Row estimates are from the
live probe (Socrata `$select=count(*)` or file-size/row sampling). Statuses:

- **OPEN-INGESTED** — free bulk download, adapter shipped + loaded this pass
- **OPEN-TODO** — free bulk download exists, adapter deferred to a later pass
- **GATED-portal** — per-name lookup portal / form-post / session only, no bulk file
- **GATED-paid** — bulk roster exists but costs money
- **NOT-FOUND** — no bulk mechanism located this pass

| State | Source | URL | Status | Rows (est) | Notes |
|---|---|---|---|---|---|
| TX | TREC via data.texas.gov Socrata `s7ft-44qi` | https://data.texas.gov/api/views/s7ft-44qi/rows.csv?accessType=DOWNLOAD | OPEN-INGESTED | 321,788 | Refreshed daily. 18,945 Broker Company / 42,417 Broker Individual / 260,426 Sales Agent. Firm linkage via `related_license_*`. No city column (county only). trec.texas.gov "High Value Data Sets" page itself no longer links raw files — the Socrata dataset IS the bulk feed. |
| FL | DBPR free extracts | https://www2.myfloridalicense.com/sto/file_download/extracts/RE_rgn1.csv (…rgn1–14) | OPEN-INGESTED | ~380k lines across 14 regional files | Refreshed daily (Last-Modified 07-20/07-21). Headerless DBPR standard layout, latin-1. Directory listing disabled — filenames recovered via Wayback CDX, all 14 live-verified. Employer name + numeric license in cols 21/22 → firm linkage. `RealEstateCorpLicense.csv` (9MB) + `re_salesperson.csv` (180MB, history) also live but redundant with the rgn files. |
| NY | DOS via data.ny.gov Socrata `yg7h-zjbf` | https://data.ny.gov/api/views/yg7h-zjbf/rows.csv?accessType=DOWNLOAD | OPEN-INGESTED | 147,941 | "Active Real Estate Salespersons and Brokers", refreshed daily. Active-only. Firm = `business_name` (no firm license # in-row → firms dedupe by normalized name). 77,632 salespersons, 16,108 assoc brokers, 9,234 individual brokers + entity types. |
| CO | DRE via data.colorado.gov Socrata `4zse-6bnw` | https://data.colorado.gov/api/views/4zse-6bnw/rows.csv?accessType=DOWNLOAD | OPEN-INGESTED | 108,811 total; ~49k RE brokers + 16.7k RE companies kept | "Licensed Real Estate Professionals" mixes in MLOs (21,341), appraisers, HOAs — adapter filters to Real Estate Broker/Company types, license_type preserved. No employer linkage columns. |
| CT | DCP via data.ct.gov Socrata `6tja-6vdt` | https://data.ct.gov/api/views/6tja-6vdt/rows.csv?accessType=DOWNLOAD | OPEN-INGESTED | 18,281 | "Real Estate Salespersons with Associated Broker" — active affiliation pairs (RES.# ↔ REB.#), broker credential feeds firm table. Sister datasets (`eqtn-rppv` status, team members `5vfu-8yxm`) available for later passes. |
| VA | DPOR | https://www.dpor.virginia.gov/LicenseLookup/ | GATED-portal | — | Lookup portal only; no bulk file linked anywhere on the rebuilt Drupal site (grepped nav + guessed download paths → 404). DPOR bulk data goes through a records-request/order process. |
| AZ | ADRE public database | https://services.azre.gov/PdbWeb/ | GATED-portal | — | "Public Database" exists but is a JS SPA (PdbWeb) — exports are generated client-side (jszip), no direct bulk CSV URL. Candidate for a later pass via its underlying JSON API (needs endpoint mapping, out of scope under open-files-only). |
| NC | NCREC | https://www.ncrec.gov/Home/Databases | GATED-portal | — | 403 bot-block on direct fetch; NCREC licensee data is by data-request form. |
| GA | GREC | https://grec.state.ga.us | GATED-paid | — | GREC sells licensee rosters (paid data request); no free bulk file. Probe of licensee page → 404/portal. |
| WA | DOL professions | https://professions.dol.wa.gov/s/license-lookup | GATED-portal | — | Lookup portal only. data.wa.gov has license COUNTS (`a2n7-rij5`) but no licensee-level RE roster. Bulk = public-records request. |
| NJ | Division of Consumer Affairs | https://newjersey.mylicense.com/verification/ | GATED-portal | — | mylicense.com verification portal (form-post + session), no bulk export. |
| PA | PALS | https://www.pals.pa.gov/ | GATED-portal | — | PALS lookup portal; bulk licensee lists are paid data requests through the Bureau. |
| OH | eLicense / data.ohio.gov | https://elicense.ohio.gov | NOT-FOUND | — | No RE licensee dataset on any Socrata domain (catalog search empty); eLicense is a Salesforce portal. Ohio DOC offers roster downloads somewhere behind eLicense login — recheck next pass. |
| GA/others via NMLS | — | — | — | NMLS covers mortgage (MLO), not real-estate brokers — not applicable. |

## M-B2 firm website discovery + front-page crawl (2026-07-21)

Top-200 firms by agent_count ran through free-search website discovery
(Brave Search HTML primary — DDG html endpoint anomaly-flagged this IP with
HTTP-202 challenge pages mid-run and stayed flagged; Brave answered 200/200)
then a Playwright chromium front-page crawl (concurrency 5, robots.txt honored).

- **Discovery**: 209 firm_site rows, 209 with url — **100% hit rate** on the
  top-200 batch (big firms are easy; expect lower on the long tail).
- **Crawl**: 190 crawled ok (http_status recorded), 14 robots-disallowed,
  5 errors (4 nav timeouts, 1 SSL: resf.com). **103 sites IDX-positive**.
- **Contacts**: 348 rows (mailto/tel/contact-page) across 145 firms.
- Provenance in `ingest_runs` (`firm_discovery` / `firm_crawl`).
- Franchise collapse is real: many licensed entities resolve to one brand site
  (exprealty.com, compass.com, elliman.com, realtytexas.com) — expect
  firm_site.url duplication across firm rows by design (firm_id stays UNIQUE).

### M-B3 pilot candidates

Top-20 crawled sites that look like clean IDX listing-search candidates
(has_idx_listings=true, HTTP 200, ordered by agent_count; duplicates by
domain collapsed):

| Firm | St | Agents | Site |
|---|---|---|---|
| EXP REALTY LLC | TX/FL | 6,916 | https://exprealty.com |
| COLDWELL BANKER RESIDENTIAL REAL ESTATE LLC | FL | 5,753 | https://coldwellbanker.com |
| THE K COMPANY REALTY LLC | FL | 3,683 | https://thekcorealty.net |
| UNITED REALTY GROUP INC | FL | 3,567 | https://urgfl.com |
| COMPASS FLORIDA LLC (+TX/NY entities) | FL | 3,552 | https://compass.com |
| DOUGLAS ELLIMAN REAL ESTATE | NY | 3,239 | https://elliman.com |
| REAL BROKER, LLC | TX | 3,234 | https://realtytexas.com |
| CORCORAN GROUP | NY | 2,054 | https://corcoran.com |
| FATHOM REALTY, LLC | TX | 1,859 | https://fathomrealty.com |
| HOMESMART TEXAS, LLC | TX | 1,794 | https://homesmart.com |
| BEACHFRONT REALTY INC | FL | 1,715 | https://beachfrontonline.com |
| CHARLES RUTENBERG REALTY INC | FL | 1,645 | https://charles-rutenberg.com |
| EBBY HALLIDAY REAL ESTATE, LLC | TX | 1,561 | https://ebby.com |
| WILLIAM RAVEIS REAL ESTATE INC | CT | 1,518 | https://raveis.com |
| SIGNATURE PREMIER PROPERTIES | NY | 1,518 | https://signaturepremier.com |
| PREMIERE PLUS REALTY CO | FL | 1,368 | https://premiereplusrealty.com |
| LONDON FOSTER LLC | FL | 1,279 | https://londonfoster.com |
| AVANTI WAY REALTY LLC | FL | 1,193 | https://avantiway.com |
| PARK PLACE REALTY NETWORK LLC | FL | 1,001 | https://parkplacenetwork.com |
| HOULIHAN LAWRENCE INC | NY | 997 | https://houlihanlawrence.com |

(LPT REALTY, FL, 5,423 agents — https://company.lptsearch.com — is top-5 by size
but **robots-disallowed**; needs a partner/API route in M-B3, not a crawl.)

Notable bot-blocked / robots-disallowed domains for M-B3 planning:
daltonwade.com, preferredshore.com, company.lptsearch.com, ct/ny.exprealty.com
(subdomain robots differ from apex — apex exprealty.com crawls fine),
texaspremierrealty.com, realestateempire.com. resf.com (Related ISG) breaks on
SSL to headless — retry via static fetch fallback in M-B3.

## Next-pass candidates

1. **AZ PdbWeb JSON API** — map the SPA's fetch endpoints (likely open JSON, just undocumented).
2. **OH eLicense "download data" export** — reported to exist behind a public form.
3. **CT broker-side roster** (`eqtn-rppv` + DCP full credential extract) to add CT firm HQ cities.
4. Remaining Socrata domains sweep (`api.us.socrata.com/api/catalog/v1?q=salesperson`, `q=licensee`) for smaller states.