spec: robust Crexi/LoopNet agent-attributed scraper (openclaw two-level crawl -> broker graph master DB, per-agent attribution, reversible; CoStar parked) — ready to run fresh

dddefafdf3874e42004d3e5dd43639daa5e19884 · 2026-08-19 10:05:22 -0700 · Steve Abrams
Files touched

A docs/crexi-loopnet-scraper-spec.md
Diff

commit dddefafdf3874e42004d3e5dd43639daa5e19884
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Wed Aug 19 10:05:22 2026 -0700

    spec: robust Crexi/LoopNet agent-attributed scraper (openclaw two-level crawl -> broker graph master DB, per-agent attribution, reversible; CoStar parked) — ready to run fresh
---
 docs/crexi-loopnet-scraper-spec.md | 78 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)

diff --git a/docs/crexi-loopnet-scraper-spec.md b/docs/crexi-loopnet-scraper-spec.md
new file mode 100644
index 0000000..0de3d9c
--- /dev/null
+++ b/docs/crexi-loopnet-scraper-spec.md
@@ -0,0 +1,78 @@
+# Spec — Robust Crexi/LoopNet Agent-Attributed Listing Scraper
+
+**Owner:** vp-directories / re-props · **Ticket:** TK-10703 follow-on · **Status:** ready to run fresh
+**Author:** crcp-ui · **Date:** 2026-08-19
+
+## Goal
+Make the **broker graph the master database for ALL listings**, populated by **our own openclaw
+scrapes of Crexi + LoopNet**, with **correct per-agent attribution** (each listing tied to the
+actual listing agent, resolved to a broker in our graph). CoStar is **parked** (licensed/paid —
+separate decision). We **ingest** aggregator data into our DB; we **never link users out** to Crexi.
+
+## Why this is a dedicated build (recon findings, 2026-08-19)
+- openclaw real-Chrome reaches **crexi.com unblocked** (no captcha) — confirmed. Plain `fetch` is
+  403-walled; openclaw is the tool.
+- Crexi is a **React SPA**: the search grid's listing cards **lazy-load on scroll** and use
+  **obfuscated class names** (generic `[class*=card]` selectors returned 0). Body text grows from
+  ~420 → ~5,700 chars after a `window.scrollTo` — the data is there, but needs real DOM work.
+- **The listing agent is NOT on the search grid** — it's on each property's **detail page**
+  ("Broker of Record / Listed by"). So correct per-agent attribution requires a **two-level crawl**:
+  search grid → each property detail page → agent name.
+- This is exactly why naive homepage/grid scraping (option 1/3 pilots) capped out; the volume +
+  attribution lives behind the detail-page crawl.
+
+## Architecture
+```
+scripts/scrape-crexi-loopnet.js  (openclaw + qwen3:14b, $0 local; DB writes reversible)
+  1. SEARCH SWEEP (per market, per property type):
+     - openclaw navigate to the Crexi/LoopNet sales search for LA County CRE
+       (Multifamily / Retail-NNN / Mixed-use / Office / Industrial).
+     - Handle the SPA: scroll to bottom in a loop until no new cards; or use the results-count +
+       paginate via the URL (?page=N) / "next" button. Extract per-card: detail-page URL + price +
+       address (from the rendered DOM — inspect real selectors, or LLM-extract the scrolled innerText).
+  2. DETAIL CRAWL (per listing URL, dedup first):
+     - openclaw navigate to the detail page; extract {address, city, zip, price, type, units,
+       cap_rate, listing_agent(s), firm}. Prefer structured DOM; fall back to qwen3:14b on the
+       detail-page innerText with a strict JSON schema.
+  3. ATTRIBUTE + INGEST (broker graph = master):
+     - Match listing_agent name -> broker (lower(name); confirm with firm when available — reuse the
+       name+FIRM guard from just-listed/licensed-agents so we never attach the WRONG person).
+     - INSERT listing (id, address, city, zip, price, type, units, cap_rate, firm_name,
+       source='crexi'|'loopnet', created_at) + broker_listing (broker_id, listing_id, role='agent').
+     - DEDUP by normalized address across ALL sources (broker-site, broker-site-oc, crexi, loopnet,
+       and the existing crexi-originated rows) so one property never double-counts.
+  4. REVERSIBLE: everything tagged source in ('crexi','loopnet'). Undo = one DELETE per source.
+```
+
+## Hard rules (carry over from this session)
+- **Per-agent attribution ONLY when name+firm confirm** — else attach at firm level (role='firm-listing')
+  or skip. Never show one agent another's listing (the Cody-caught misattribution bug).
+- **Never fabricate** an agent/price/address. Skip rows missing a real street address + price.
+- **Never link users out to Crexi/LoopNet** on any CRCP page — ingest-and-display-as-ours only.
+- **openclaw = our own scrape** (Steve: "only our own scrapes"); **no paid data feeds/APIs**; CoStar parked.
+- **Rate-limit + be polite**: throttle detail-page crawls (e.g. 1 req / 1.5s), cap per-run volume,
+  resume-able (persist seen URLs) so a run can stop/restart without re-crawling.
+- **$0**: openclaw local + qwen3:14b local. Show the (local) cost line each run.
+
+## Deliverables
+1. `scripts/scrape-crexi-loopnet.js` — the two-level openclaw crawler + LLM extractor + ingester
+   (dry-run default; `--apply`; `--market`, `--type`, `--limit`, `--source crexi|loopnet`).
+2. Real DOM selectors for Crexi + LoopNet search cards + detail-page agent block (from a live recon
+   pass — do NOT guess; inspect the rendered DOM via `openclaw browser evaluate`).
+3. Dedup migration: a normalized-address unique guard so re-runs + cross-source don't duplicate.
+4. Verification: for 5 sampled agents, their `/api/agent-profile` shows the newly-ingested,
+   correctly-attributed listings; spot-check 10 listings against the live Crexi/LoopNet page.
+
+## Acceptance criteria
+- ≥ 500 LA-County CRE listings ingested with a **real, name+firm-confirmed listing agent**.
+- 0 misattributed listings in a 20-row manual audit.
+- 0 duplicate addresses across sources.
+- Every ingested listing reversible via `DELETE FROM listing WHERE source IN ('crexi','loopnet')`.
+- No CRCP page links out to Crexi/LoopNet.
+
+## Starting points already in place (this session)
+- openclaw proven on crexi.com (unblocked). `~/.npm-global/bin/openclaw browser navigate|evaluate`.
+- LLM extractor pattern: `scripts/backfill-openclaw-llm.js` (qwen3:14b, strict-JSON, garbage filter).
+- Attribution guard pattern: `public/licensed-agents.html` contactFor (name+firm).
+- Master-DB shape: `listing` (has firm_name + source cols) + `broker_listing(role)` + `broker`/`firm`.
+- Agent profile already renders own + closed + firm listings: `/api/agent-profile`, `public/agent.html`.
← 7adb0f7 CRCP agent precision: /api/agent-profile + agent.html accept · back to Commercialrealestate · TK-10709: Crexi two-level openclaw crawl (grid->detail) -> b af11701 →