fix(assessor): stop on exceededTransferLimit, not short page (was truncating at 294k/2.43M) + CC_START_OFFSET resume

bfc32f8dc0422c335005e13f9977baac10f4d59b · 2026-06-30 14:20:52 -0700 · Steve Abrams

The ArcGIS parcel layer returns short pages (<maxRecordCount) mid-dataset while exceededTransferLimit is still true, so breaking on feats.length<PAGE silently stopped the pull at ~12%. Now terminates only on the server's real end-of-data signal, and CC_START_OFFSET resumes/appends a partial pull.

Files touched

M scripts/fetch-assessor.js

Diff

commit bfc32f8dc0422c335005e13f9977baac10f4d59b
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Tue Jun 30 14:20:52 2026 -0700

    fix(assessor): stop on exceededTransferLimit, not short page (was truncating at 294k/2.43M) + CC_START_OFFSET resume
    
    The ArcGIS parcel layer returns short pages (<maxRecordCount) mid-dataset while exceededTransferLimit is still true, so breaking on feats.length<PAGE silently stopped the pull at ~12%. Now terminates only on the server's real end-of-data signal, and CC_START_OFFSET resumes/appends a partial pull.
---
 scripts/fetch-assessor.js | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/scripts/fetch-assessor.js b/scripts/fetch-assessor.js
index 4f4a904..a4c9f77 100644
--- a/scripts/fetch-assessor.js
+++ b/scripts/fetch-assessor.js
@@ -67,8 +67,11 @@ async function main() {
     (!ALL_YEARS ? `\n[assessor] SCOPE NOTE: pulling RollYear=${ROLL_YEAR} only (current snapshot, 1 row/parcel). Historical rolls 2021-2024 intentionally skipped (5× redundant). Set CC_ALL_YEARS=1 to pull all.` : ''));
   console.log(`[assessor] streaming NDJSON → ${OUT}  ($0 — public open data)`);
 
-  const ws = fs.createWriteStream(OUT, { flags: 'w' });
-  let offset = 0, written = 0, pages = 0;
+  // Resume support: CC_START_OFFSET continues a partial pull (append) instead of re-fetching from 0.
+  const START = +(process.env.CC_START_OFFSET || 0) || 0;
+  const ws = fs.createWriteStream(OUT, { flags: START ? 'a' : 'w' });
+  let offset = START, written = START, pages = 0;
+  if (START) console.log(`[assessor] RESUME from offset ${START.toLocaleString()} (appending to existing file)`);
   while (written < target) {
     const url = `${BASE}/query?where=${encodeURIComponent(WHERE)}&outFields=${FIELDS}` +
       `&returnGeometry=false&orderByFields=AIN&resultOffset=${offset}&resultRecordCount=${PAGE}&f=json`;
@@ -94,7 +97,10 @@ async function main() {
       const pct = total ? ((written / target) * 100).toFixed(1) : '?';
       process.stdout.write(`[assessor] ${written.toLocaleString()}/${target.toLocaleString()} (${pct}%) · ${fmtBytes(sz)} · ${pages} pages\n`);
     }
-    if (feats.length < PAGE) break;  // last page
+    // Terminate ONLY on the server's real end-of-data signal. This ArcGIS layer returns SHORT pages
+    // (< maxRecordCount) mid-dataset while exceededTransferLimit is still true, so a short page is NOT
+    // the end — breaking on feats.length<PAGE truncated the pull at ~294k/2.43M (the old bug).
+    if (data.exceededTransferLimit === false) break;  // last (partial) page; no more rows after this
   }
   await new Promise(r => ws.end(r));
   const sz = fs.statSync(OUT).size;

← 531f7e7 Add DW-Wins-Dashboard.png — CRE build wins dashboard (emaile · back to Commercialrealestate · fix(condos): render listings — warrant_signals is an object, 22644c3 →