geocode: 2k-address chunks (was 10k) so bulk backfill fits inside the timeout

13b17887638e000779ad0cbbf20244458fc11156 · 2026-07-28 23:22:07 -0700 · Steve Abrams

The full 28k backfill timed out: a 10k-address Census batch takes 5-15min, exceeding even a
generous timeout. Default CHUNK=2000 (~1-2min/batch) + CHUNK/CURL_TIMEOUT env overrides. Per-sweep
incrementals (few hundred new homes) still finish well inside the 120s fail-fast; the one-time
backfill runs CHUNK=2000 CURL_TIMEOUT=300. Resilient: a failed chunk leaves its rows unlogged for
the next run to retry, not stamped nomatch.

Files touched

M scripts/geocode-homes.mjs

Diff

commit 13b17887638e000779ad0cbbf20244458fc11156
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Tue Jul 28 23:22:07 2026 -0700

    geocode: 2k-address chunks (was 10k) so bulk backfill fits inside the timeout
    
    The full 28k backfill timed out: a 10k-address Census batch takes 5-15min, exceeding even a
    generous timeout. Default CHUNK=2000 (~1-2min/batch) + CHUNK/CURL_TIMEOUT env overrides. Per-sweep
    incrementals (few hundred new homes) still finish well inside the 120s fail-fast; the one-time
    backfill runs CHUNK=2000 CURL_TIMEOUT=300. Resilient: a failed chunk leaves its rows unlogged for
    the next run to retry, not stamped nomatch.
---
 scripts/geocode-homes.mjs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/geocode-homes.mjs b/scripts/geocode-homes.mjs
index 1a94c762..be88b4b4 100644
--- a/scripts/geocode-homes.mjs
+++ b/scripts/geocode-homes.mjs
@@ -19,7 +19,9 @@ import { writeFileSync, readFileSync } from "node:fs";
 
 const DB = process.env.DATABASE_URL || "postgresql://macstudio3@localhost/homesonspec?host=/tmp";
 const SAMPLE = process.env.SAMPLE ? Number(process.env.SAMPLE) : null;
-const CHUNK = SAMPLE ? Math.min(SAMPLE, 10000) : 10000;
+// Census bulk geocode of 10k addresses takes 5-15min; keep chunks small enough to finish
+// inside CURL_TIMEOUT. Default 2000 (~1-2min/chunk). Override with CHUNK for tuning.
+const CHUNK = SAMPLE ? Math.min(SAMPLE, 10000) : Number(process.env.CHUNK || 2000);
 const UA = "HomesOnSpecBot/0.1 (+homesonspec.com; site-owner-facts-only)";
 const CURL_TIMEOUT = process.env.CURL_TIMEOUT || "120"; // fail fast — never stall a build-loop sweep

← 0086b346 geocode: contrarian FIX-FIRST — city+state gate, full proven · back to Homesonspec · dr-horton: capture community coords from in-feed ld+json (yo d99677fd →