CRCP: harden per-broker grind — auto-dismiss JS dialogs + allSettled (one bad Crexi page no longer crashes the run)

c189e73db06c69dad3435859ab73c47f3b27417b · 2026-08-19 10:56:44 -0700 · Steve Abrams

Files touched

M scripts/backfill-crexi-brokers.js

Diff

commit c189e73db06c69dad3435859ab73c47f3b27417b
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Wed Aug 19 10:56:44 2026 -0700

    CRCP: harden per-broker grind — auto-dismiss JS dialogs + allSettled (one bad Crexi page no longer crashes the run)
---
 scripts/backfill-crexi-brokers.js | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/scripts/backfill-crexi-brokers.js b/scripts/backfill-crexi-brokers.js
index 330e314..edc68b0 100644
--- a/scripts/backfill-crexi-brokers.js
+++ b/scripts/backfill-crexi-brokers.js
@@ -62,8 +62,14 @@ async function worker(page) {
   console.log(`Crexi per-broker backfill: ${work.length} rows, ${TABS} parallel openclaw tabs ($0). Background grind.`);
   const browser = await chromium.connectOverCDP(CDP);
   const ctx = browser.contexts()[0] || await browser.newContext();
-  const pages = []; for (let i = 0; i < TABS; i++) pages.push(await ctx.newPage());
-  await Promise.all(pages.map(p => worker(p)));
+  const pages = [];
+  for (let i = 0; i < TABS; i++) {
+    const p = await ctx.newPage();
+    p.on('dialog', d => d.dismiss().catch(() => {}));   // auto-dismiss beforeunload/alert popups (crashed the run before)
+    p.setDefaultTimeout(30000);
+    pages.push(p);
+  }
+  await Promise.allSettled(pages.map(p => worker(p)));   // one worker/page dying never kills the batch
   for (const p of pages) await p.close().catch(()=>{});
   flush();
   console.log(`\nDone: ${done} pages, ${hit} broker phones added. ${work.length - done} left (re-run to continue).`);

← 1768107 repoint all 3 direct scrapers (crexi/firm-site/personal-site · back to Commercialrealestate · firm-site scraper: scroll for lazy-loaded grids + try common eab941c →