[object Object]

← back to Commercialrealestate

TK-10709: dedup across ALL sources, not just crexi

b048ea216aa46f8c4bf448e72b0e3782f65b2dbf · 2026-08-25 16:46:19 -0700 · Steve Abrams

Seed the seen-address set from every source in broker_direct_listing so a
crexi crawl never double-counts a property already listed under broker-site /
broker-site-oc (spec criterion #3). Caught + fixed 5 cross-source collisions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

Diff

commit b048ea216aa46f8c4bf448e72b0e3782f65b2dbf
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Tue Aug 25 16:46:19 2026 -0700

    TK-10709: dedup across ALL sources, not just crexi
    
    Seed the seen-address set from every source in broker_direct_listing so a
    crexi crawl never double-counts a property already listed under broker-site /
    broker-site-oc (spec criterion #3). Caught + fixed 5 cross-source collisions.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 scripts/scrape-crexi-loopnet.js | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/scrape-crexi-loopnet.js b/scripts/scrape-crexi-loopnet.js
index 3f087b8..02e2ef6 100644
--- a/scripts/scrape-crexi-loopnet.js
+++ b/scripts/scrape-crexi-loopnet.js
@@ -205,9 +205,11 @@ async function resolveBroker(agent, firm, license, phone) {
   const prog = loadProg();
   const seenIds = new Set(prog.seenIds);
   const seenAddrs = new Set(prog.seenAddrs);
-  // seed seenAddrs from what's already in the DB so we never re-write across the whole history
+  // seed seenAddrs from EVERY source already in the DB (not just crexi) so one property never
+  // double-counts across sources (spec criterion #3: 0 duplicate addresses across broker-site /
+  // broker-site-oc / crexi / loopnet). First source to list an address wins.
   try {
-    const have = (await db.pool.query(`SELECT address FROM broker_direct_listing WHERE source='crexi'`)).rows;
+    const have = (await db.pool.query(`SELECT address FROM broker_direct_listing`)).rows;
     for (const r of have) seenAddrs.add(nAddr(r.address));
   } catch {}

← d7cdfed TK-10709: remove Crexi profile link-out from broker page + r  ·  back to Commercialrealestate  ·  auto-data-snapshot: 2026-08-25T17:03:18 (1 data files) — dat 66e131f →