← back to Homesonspec
scripts: all-night autonomous build-loop (ingest x states + enrichment sweeps)
48110bc7782ec0bc472e9eb1fd128b056c2f1373 · 2026-07-27 20:23:38 -0700 · Steve
Deterministic overnight driver: activates the 4 working adapters + dr-horton, sweeps
CA/TX/FL/AZ/NV/CO/GA/NC/SC/TN/WA/UT (op-40 each), drains public-records enrichment,
repeats freshness sweeps (max 6, ~2.5h apart) until tmp/BUILD-LOOP-STOP. Facts-only,
robots-enforced, LOCAL DB only; public go-live stays gated. Idempotent re-sweeps.
Files touched
Diff
commit 48110bc7782ec0bc472e9eb1fd128b056c2f1373
Author: Steve <steve@designerwallcoverings.com>
Date: Mon Jul 27 20:23:38 2026 -0700
scripts: all-night autonomous build-loop (ingest x states + enrichment sweeps)
Deterministic overnight driver: activates the 4 working adapters + dr-horton, sweeps
CA/TX/FL/AZ/NV/CO/GA/NC/SC/TN/WA/UT (op-40 each), drains public-records enrichment,
repeats freshness sweeps (max 6, ~2.5h apart) until tmp/BUILD-LOOP-STOP. Facts-only,
robots-enforced, LOCAL DB only; public go-live stays gated. Idempotent re-sweeps.
---
scripts/build-loop.sh | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
diff --git a/scripts/build-loop.sh b/scripts/build-loop.sh
new file mode 100644
index 0000000..1d6ae4c
--- /dev/null
+++ b/scripts/build-loop.sh
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# ALL-NIGHT autonomous "keep building" loop for HomesOnSpec inventory.
+# Layer 1 (this script): deterministic ingest + public-records enrichment across
+# the working adapters × top builder states, op-40 each, repeated in freshness
+# sweeps until a STOP file appears or MAX_SWEEPS is hit. Facts-only, robots-
+# enforced (LiveFetcher), LOCAL DB only — the public Kamatera go-live stays gated.
+# Idempotent: re-sweeps of unchanged data are cheap no-ops (contentHash dedup).
+# Stop anytime: touch ~/Projects/homesonspec/tmp/BUILD-LOOP-STOP
+set -uo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+cd "$ROOT/apps/workers" || exit 1
+export DATABASE_URL='postgresql://macstudio3@localhost/homesonspec?host=/tmp'
+mkdir -p "$ROOT/tmp"
+STOP="$ROOT/tmp/BUILD-LOOP-STOP"
+MAX_SWEEPS="${MAX_SWEEPS:-6}"
+SWEEP_SLEEP="${SWEEP_SLEEP:-9000}" # ~2.5h between freshness sweeps (polite)
+LOG() { echo "[$(date '+%m-%d %H:%M:%S')] $*"; }
+PSQL() { psql "$DATABASE_URL" -tAc "$1" 2>/dev/null; }
+CLI() { npx tsx src/cli.ts --adapter="$1" 2>&1 | grep -E '"published"|"pages"|Error|error|inactive|blocked|disallow' | head -3; }
+
+# state formats per adapter
+declare -A KB=( [CA]=CA [TX]=TX [FL]=FL [AZ]=AZ [NV]=NV [CO]=CO [GA]=GA [NC]=NC [SC]=SC [TN]=TN [WA]=WA [UT]=UT )
+declare -A DRH=( [CA]=california [TX]=texas [FL]=florida [AZ]=arizona [NV]=nevada [CO]=colorado [GA]=georgia [NC]="north-carolina" [SC]="south-carolina" [TN]=tennessee [WA]=washington [UT]=utah )
+declare -A PLT=( [CA]=California [TX]=Texas [FL]=Florida [AZ]=Arizona [NV]=Nevada [CO]=Colorado [GA]=Georgia [NC]="North Carolina" [SC]="South Carolina" [TN]=Tennessee [WA]=Washington [UT]=Utah )
+declare -A TRI=( [CA]=ca [TX]=tx [FL]=fl [AZ]=az [NV]=nv [CO]=co [GA]=ga [NC]=nc [SC]=sc [TN]=tn [WA]=wa [UT]=ut )
+STATES=(CA TX FL AZ NV CO GA NC SC TN WA UT)
+
+# wait until no other adapter run is in flight (avoid same-source races)
+while pgrep -f "cli.ts --adapter" | grep -qv $$ 2>/dev/null; do sleep 20; done
+
+for k in dr-horton-site kb-home-site pultegroup-site tri-pointe-site; do
+ PSQL "update \"SourceRegistry\" set active=true, health='HEALTHY', \"consecutiveFailures\"=0 where key='$k';" >/dev/null
+done
+
+sweep=0
+while [ ! -f "$STOP" ] && [ "$sweep" -lt "$MAX_SWEEPS" ]; do
+ sweep=$((sweep+1))
+ s0=$(PSQL "select count(*) from \"InventoryHome\" where status='PUBLISHED';")
+ b0=$(PSQL "select count(distinct \"builderId\") from \"InventoryHome\" where status='PUBLISHED';")
+ LOG "===== SWEEP $sweep/$MAX_SWEEPS start: $s0 homes / $b0 builders ====="
+ for S in "${STATES[@]}"; do
+ [ -f "$STOP" ] && break
+ LOG "-- $S --"
+ KBHOME_STATE="${KB[$S]}" CLI kb-home-site 2>&1 | sed "s/^/ kb $S /"
+ DRHORTON_STATE="${DRH[$S]}" DRHORTON_PAGE_LIMIT=40 CLI dr-horton-site 2>&1 | sed "s/^/ drh $S /"
+ PULTE_STATE="${PLT[$S]}" PULTE_PAGE_LIMIT=40 CLI pultegroup-site 2>&1 | sed "s/^/ plt $S /"
+ TRIPOINTE_STATE="${TRI[$S]}" TRIPOINTE_PAGE_LIMIT=40 CLI tri-pointe-site 2>&1 | sed "s/^/ tri $S /"
+ done
+ # drain enrichment queue
+ for i in $(seq 1 40); do
+ rem=$(PSQL "select count(*) from \"Community\" where lat is not null and amenities is null;")
+ [ "${rem:-0}" -eq 0 ] && { LOG "enrich: queue drained"; break; }
+ LOG "enrich pass $i: $rem left"
+ NEARBY_LIMIT=40 npx tsx scripts/enrich-nearby.ts >/dev/null 2>&1
+ done
+ s1=$(PSQL "select count(*) from \"InventoryHome\" where status='PUBLISHED';")
+ b1=$(PSQL "select count(distinct \"builderId\") from \"InventoryHome\" where status='PUBLISHED';")
+ enr=$(PSQL "select count(*) from \"Community\" where amenities is not null;")
+ LOG "===== SWEEP $sweep done: $s1 homes / $b1 builders (+$((s1-s0)) homes) · $enr communities enriched ====="
+ [ -f "$STOP" ] && break
+ [ "$sweep" -lt "$MAX_SWEEPS" ] && { LOG "sleeping ${SWEEP_SLEEP}s before next freshness sweep"; sleep "$SWEEP_SLEEP"; }
+done
+LOG "BUILD LOOP EXIT (sweeps=$sweep, stop=$( [ -f "$STOP" ] && echo yes || echo no ))"
← 8dda332 workers: public-records nearby-enrichment stage (NCES school
·
back to Homesonspec
·
auto-save: 2026-07-27T20:24:23 (1 files) — scripts/build-loo 76c1d27 →