`docs/crexi-loopnet-scraper-spec.md`

79 lines
# Spec — Robust Crexi/LoopNet Agent-Attributed Listing Scraper

**Owner:** vp-directories / re-props · **Ticket:** TK-10703 follow-on · **Status:** ready to run fresh
**Author:** crcp-ui · **Date:** 2026-08-19

## Goal
Make the **broker graph the master database for ALL listings**, populated by **our own openclaw
scrapes of Crexi + LoopNet**, with **correct per-agent attribution** (each listing tied to the
actual listing agent, resolved to a broker in our graph). CoStar is **parked** (licensed/paid —
separate decision). We **ingest** aggregator data into our DB; we **never link users out** to Crexi.

## Why this is a dedicated build (recon findings, 2026-08-19)
- openclaw real-Chrome reaches **crexi.com unblocked** (no captcha) — confirmed. Plain `fetch` is
  403-walled; openclaw is the tool.
- Crexi is a **React SPA**: the search grid's listing cards **lazy-load on scroll** and use
  **obfuscated class names** (generic `[class*=card]` selectors returned 0). Body text grows from
  ~420 → ~5,700 chars after a `window.scrollTo` — the data is there, but needs real DOM work.
- **The listing agent is NOT on the search grid** — it's on each property's **detail page**
  ("Broker of Record / Listed by"). So correct per-agent attribution requires a **two-level crawl**:
  search grid → each property detail page → agent name.
- This is exactly why naive homepage/grid scraping (option 1/3 pilots) capped out; the volume +
  attribution lives behind the detail-page crawl.

## Architecture
```
scripts/scrape-crexi-loopnet.js  (openclaw + qwen3:14b, $0 local; DB writes reversible)
  1. SEARCH SWEEP (per market, per property type):
     - openclaw navigate to the Crexi/LoopNet sales search for LA County CRE
       (Multifamily / Retail-NNN / Mixed-use / Office / Industrial).
     - Handle the SPA: scroll to bottom in a loop until no new cards; or use the results-count +
       paginate via the URL (?page=N) / "next" button. Extract per-card: detail-page URL + price +
       address (from the rendered DOM — inspect real selectors, or LLM-extract the scrolled innerText).
  2. DETAIL CRAWL (per listing URL, dedup first):
     - openclaw navigate to the detail page; extract {address, city, zip, price, type, units,
       cap_rate, listing_agent(s), firm}. Prefer structured DOM; fall back to qwen3:14b on the
       detail-page innerText with a strict JSON schema.
  3. ATTRIBUTE + INGEST (broker graph = master):
     - Match listing_agent name -> broker (lower(name); confirm with firm when available — reuse the
       name+FIRM guard from just-listed/licensed-agents so we never attach the WRONG person).
     - INSERT listing (id, address, city, zip, price, type, units, cap_rate, firm_name,
       source='crexi'|'loopnet', created_at) + broker_listing (broker_id, listing_id, role='agent').
     - DEDUP by normalized address across ALL sources (broker-site, broker-site-oc, crexi, loopnet,
       and the existing crexi-originated rows) so one property never double-counts.
  4. REVERSIBLE: everything tagged source in ('crexi','loopnet'). Undo = one DELETE per source.
```

## Hard rules (carry over from this session)
- **Per-agent attribution ONLY when name+firm confirm** — else attach at firm level (role='firm-listing')
  or skip. Never show one agent another's listing (the Cody-caught misattribution bug).
- **Never fabricate** an agent/price/address. Skip rows missing a real street address + price.
- **Never link users out to Crexi/LoopNet** on any CRCP page — ingest-and-display-as-ours only.
- **openclaw = our own scrape** (Steve: "only our own scrapes"); **no paid data feeds/APIs**; CoStar parked.
- **Rate-limit + be polite**: throttle detail-page crawls (e.g. 1 req / 1.5s), cap per-run volume,
  resume-able (persist seen URLs) so a run can stop/restart without re-crawling.
- **$0**: openclaw local + qwen3:14b local. Show the (local) cost line each run.

## Deliverables
1. `scripts/scrape-crexi-loopnet.js` — the two-level openclaw crawler + LLM extractor + ingester
   (dry-run default; `--apply`; `--market`, `--type`, `--limit`, `--source crexi|loopnet`).
2. Real DOM selectors for Crexi + LoopNet search cards + detail-page agent block (from a live recon
   pass — do NOT guess; inspect the rendered DOM via `openclaw browser evaluate`).
3. Dedup migration: a normalized-address unique guard so re-runs + cross-source don't duplicate.
4. Verification: for 5 sampled agents, their `/api/agent-profile` shows the newly-ingested,
   correctly-attributed listings; spot-check 10 listings against the live Crexi/LoopNet page.

## Acceptance criteria
- ≥ 500 LA-County CRE listings ingested with a **real, name+firm-confirmed listing agent**.
- 0 misattributed listings in a 20-row manual audit.
- 0 duplicate addresses across sources.
- Every ingested listing reversible via `DELETE FROM listing WHERE source IN ('crexi','loopnet')`.
- No CRCP page links out to Crexi/LoopNet.

## Starting points already in place (this session)
- openclaw proven on crexi.com (unblocked). `~/.npm-global/bin/openclaw browser navigate|evaluate`.
- LLM extractor pattern: `scripts/backfill-openclaw-llm.js` (qwen3:14b, strict-JSON, garbage filter).
- Attribution guard pattern: `public/licensed-agents.html` contactFor (name+firm).
- Master-DB shape: `listing` (has firm_name + source cols) + `broker_listing(role)` + `broker`/`firm`.
- Agent profile already renders own + closed + firm listings: `/api/agent-profile`, `public/agent.html`.