Phase 2 findings: national firm-direct APIs are nationwide/price-sparse, wrong for LA — keep CREXi as discovery layer; docs + capture tooling

95b03a38fd025cd58bc89e5e51ff47eaac583ca9 · 2026-07-30 22:41:26 -0700 · steve

- 13-firm discovery: only Colliers+Newmark have populated APIs, both un-geo-filtered (0 LA-county on a Van Nuys search) + mostly Negotiable prices
- recommend Phase 2-lite: resolve broker-of-record for the 1747 aggregator-only deals via CREXi broker blocks (43% -> ~90%)

Files touched

M docs/SOURCING.md
A scripts/capture-firm-body.js

Diff

commit 95b03a38fd025cd58bc89e5e51ff47eaac583ca9
Author: steve <steve@designerwallcoverings.com>
Date:   Thu Jul 30 22:41:26 2026 -0700

    Phase 2 findings: national firm-direct APIs are nationwide/price-sparse, wrong for LA — keep CREXi as discovery layer; docs + capture tooling
    
    - 13-firm discovery: only Colliers+Newmark have populated APIs, both un-geo-filtered (0 LA-county on a Van Nuys search) + mostly Negotiable prices
    - recommend Phase 2-lite: resolve broker-of-record for the 1747 aggregator-only deals via CREXi broker blocks (43% -> ~90%)
---
 docs/SOURCING.md             | 24 +++++++++++++----
 scripts/capture-firm-body.js | 64 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+), 5 deletions(-)

diff --git a/docs/SOURCING.md b/docs/SOURCING.md
index 3e10aa9..d2a9ec8 100644
--- a/docs/SOURCING.md
+++ b/docs/SOURCING.md
@@ -29,11 +29,25 @@ An aggregator record is a **breadcrumb to find the broker**, never the final tru
   `broker_agent`, and `broker_agents[]`. Idempotent, $0, local.
 - The viewer (`public/deals.html`) surfaces a **Source** facet + the real broker firm/agent, so
   aggregator-dependence is visible and filterable.
-- **Phase 2 (metered):** activate the dormant firm-direct scrapers in `sources/firms.js` (Marcus &
-  Millichap, Lee Associates, Colliers, CBRE, JLL, Matthews, Kidder, NAI, Cushman & Wakefield,
-  Newmark, Berkadia, Stepp) via `discover-firm.js` so firm-direct becomes the PRIMARY acquisition
-  path and CREXi is demoted to discovery-only. Browserbase ~$0.03–0.05 per firm discovery run —
-  Steve-gated spend.
+## Phase 2 findings (2026-07-30, ~$0.60 Browserbase spent, Steve-approved)
+Discovery run on all 13 dormant firms (`run-firm-discovery.sh`) + raw-body capture on the 2 with
+populated APIs (`capture-firm-body.js`). **Verdict: national-brokerage firm-direct is the WRONG tool
+for LA-county coverage.**
+- **Populated listings API:** only Colliers (`/coveo/rest/search/v2`) and Newmark
+  (`/api/properties/search`). But both serve a **nationwide, un-geo-filtered** index — a "Van Nuys, CA"
+  search returned NY/London/KY/WA properties, **0 LA-county** — and prices are mostly text
+  ("Negotiable"). Colliers' Coveo `raw` block has no structured price/address at all.
+- **Empty results (interaction/geo-bbox gated):** M&M, CBRE, JLL, Matthews, Cushman, Lee.
+- **Zero JSON (SSR / bot-walled):** Kidder, NAI, Berkadia, Stepp.
+- **Skip (aggregator):** LoopNet (CoStar-owned) — excluded by doctrine anyway.
+→ **Do NOT wire these as primary sources.** CREXi stays the LA discovery layer (LA-dense, priced,
+  carries the broker-of-record). Firms left at `needs-discovery` in `sources/firms.js`.
+
+## Phase 2-lite (recommended next, still CREXi-based)
+Resolve the broker-of-record for the **1,747 aggregator-only** deals not yet in
+`broker-blocks.json` by re-hitting CREXi `/assets/<id>/brokers` (`enrich-tier1-crexi.js` path),
+pushing broker-of-record coverage from 43% toward ~90%. Metered (~$0.04–0.10 batched) — Steve-gated.
+Optionally deep-link each deal to the broker's own site for canonical verification.
 
 ## Hard lines
 - Never present an aggregator record as canonical when a broker-of-record is known.
diff --git a/scripts/capture-firm-body.js b/scripts/capture-firm-body.js
new file mode 100644
index 0000000..647cf4e
--- /dev/null
+++ b/scripts/capture-firm-body.js
@@ -0,0 +1,64 @@
+// capture-firm-body.js — dump the FULL raw JSON bodies of a firm's listing API so we can write a
+// precise extract() (docs/SOURCING.md Phase 2). Unlike discover-firm.js (metadata only), this saves
+// the actual response payloads to data/raw/body-<firm>-<i>.json for the biggest arrays seen.
+//
+//   NODE_PATH=$HOME/.claude/skills/browserbase/node_modules node scripts/capture-firm-body.js colliers
+// COST: ~$0.04 per firm (1 Browserbase session). Printed at end.
+'use strict';
+const fs = require('fs');
+const path = require('path');
+const { chromium } = require('playwright-core');
+const Browserbase = require('@browserbasehq/sdk').default;
+const { byKey, TYPES } = require('./sources/firms');
+const ROOT = path.join(__dirname, '..');
+const bbEnv = fs.readFileSync(process.env.HOME + '/.claude/skills/browserbase/.env', 'utf8');
+const get = (t, k) => (t.match(new RegExp('^' + k + '=(.*)$', 'm')) || [])[1]?.replace(/['"]/g, '').trim();
+
+const KEY = process.argv[2];
+const firm = byKey(KEY);
+if (!firm) { console.error('usage: node scripts/capture-firm-body.js <firmKey>'); process.exit(2); }
+const CITY = (process.env.CC_NAV || 'Van Nuys').split(',')[0].trim();
+
+(async () => {
+  const bodies = [];   // {path, count, body}
+  let browser;
+  try {
+    const bb = new Browserbase({ apiKey: get(bbEnv, 'BROWSERBASE_API_KEY') });
+    const session = await bb.sessions.create({ projectId: get(bbEnv, 'BROWSERBASE_PROJECT_ID'), browserSettings: { solveCaptchas: true, viewport: { width: 1440, height: 1000 } } });
+    console.log('bb session', session.id, '| firm', firm.key, '| city', CITY);
+    browser = await chromium.connectOverCDP(session.connectUrl);
+    const page = browser.contexts()[0].pages()[0] || await browser.contexts()[0].newPage();
+    page.setDefaultTimeout(60000);
+
+    page.on('response', async (resp) => {
+      const url = resp.url();
+      if (!firm.apiHosts.some(h => url.includes(h))) return;
+      if (!/json/i.test(resp.headers()['content-type'] || '')) return;
+      let j; try { j = await resp.json(); } catch { return; }
+      // Keep bodies that plausibly carry a listings array anywhere in them.
+      const s = JSON.stringify(j);
+      if (s.length < 400) return;                 // skip tiny lookups
+      let u; try { u = new URL(url); } catch { u = { pathname: url }; }
+      bodies.push({ path: u.pathname, bytes: s.length, body: j });
+    });
+
+    for (const [slug] of TYPES.slice(0, 2)) {     // 2 asset classes is enough to sample the shape
+      for (const url of firm.navUrls(CITY, slug)) {
+        console.log('→', url);
+        try { await page.goto(url, { waitUntil: 'domcontentloaded' }); await page.waitForTimeout(9000); await page.mouse.wheel(0, 3000).catch(() => {}); await page.waitForTimeout(4000); }
+        catch (e) { console.log('nav err', String(e.message).split('\n')[0]); }
+      }
+    }
+  } catch (e) { console.error('FATAL', e.message); }
+  finally { if (browser) await browser.close().catch(() => {}); }
+
+  // Save the biggest bodies (most likely the listings payloads).
+  bodies.sort((a, b) => b.bytes - a.bytes);
+  const saved = bodies.slice(0, 4);
+  saved.forEach((b, i) => {
+    const fp = path.join(ROOT, 'data', 'raw', `body-${firm.key}-${i}.json`);
+    fs.writeFileSync(fp, JSON.stringify(b.body, null, 2));
+    console.log(`saved ${fp}  (${b.path}, ${(b.bytes / 1024).toFixed(1)}kb)`);
+  });
+  console.log(`\n${firm.name}: ${bodies.length} JSON bodies seen, saved top ${saved.length}. Cost: ~$0.04 (1 session).`);
+})();

← 8030c9b auto-save: 2026-07-30T22:21:42 (1 files) — data/ranked.json. · back to Commercialrealestate · auto-save: 2026-07-30T22:51:51 (1 files) — .gitignore a5884dc →