[object Object]

← back to Homesonspec

lennar refresh runner: source DB pw from gitignored .env (was hardcoded) + re-establish tunnel per-state

9c5e6719f344c0dbb7849592b4e017edd5e56cba · 2026-07-29 09:54:55 -0700 · Steve Abrams

- Secret hygiene: DATABASE_URL password now read from .env.kamatera at runtime
  instead of hardcoded in the tracked script.
- Robustness: ensure_tunnel() re-checks/re-opens the keepalive tunnel before each
  state, so a mid-run network blip no longer silently fails the remaining states
  (the exact failure mode seen earlier today). Aborts cleanly if it can't recover.

Note: the old hardcoded pw is in git history (commit e70a69b); repo is local-only
(no remote), and the pw is inert without separate SSH access to Kamatera — flagging
for optional history-scrub/rotation, not urgent.

Files touched

Diff

commit 9c5e6719f344c0dbb7849592b4e017edd5e56cba
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Wed Jul 29 09:54:55 2026 -0700

    lennar refresh runner: source DB pw from gitignored .env (was hardcoded) + re-establish tunnel per-state
    
    - Secret hygiene: DATABASE_URL password now read from .env.kamatera at runtime
      instead of hardcoded in the tracked script.
    - Robustness: ensure_tunnel() re-checks/re-opens the keepalive tunnel before each
      state, so a mid-run network blip no longer silently fails the remaining states
      (the exact failure mode seen earlier today). Aborts cleanly if it can't recover.
    
    Note: the old hardcoded pw is in git history (commit e70a69b); repo is local-only
    (no remote), and the pw is inert without separate SSH access to Kamatera — flagging
    for optional history-scrub/rotation, not urgent.
---
 ops/lennar-mac-refresh.sh | 39 ++++++++++++++++++++++++++++-----------
 1 file changed, 28 insertions(+), 11 deletions(-)

diff --git a/ops/lennar-mac-refresh.sh b/ops/lennar-mac-refresh.sh
index 36f42afa..56bbc595 100755
--- a/ops/lennar-mac-refresh.sh
+++ b/ops/lennar-mac-refresh.sh
@@ -4,21 +4,38 @@
 # (throttled, polite), then tears the tunnel down. Intended to run on a Mac launchd schedule.
 # Live homes get re-verified + imaged; delisted homes age out via the normal freshness cadence.
 set -uo pipefail
-cd /Users/macstudio3/Projects/homesonspec/apps/workers
+ROOT=/Users/macstudio3/Projects/homesonspec
+cd "$ROOT/apps/workers"
 export PATH=$PATH:/usr/local/bin
 LOG=/tmp/lennar-mac-refresh.log; : > "$LOG"
 echo "=== Lennar Mac refresh $(date) ===" >> "$LOG"
-# 1. keepalive tunnel
-pkill -f "5433:localhost:5432" 2>/dev/null; sleep 1
-ssh -f -N -o ServerAliveInterval=30 -o ServerAliveCountMax=1000000 -o ExitOnForwardFailure=yes -L 5433:localhost:5432 root@45.61.58.125
-sleep 2
-pgrep -f "5433:localhost:5432" >/dev/null || { echo "TUNNEL FAILED" >> "$LOG"; exit 1; }
-export DATABASE_URL="postgresql://homesonspec:40b5436179ee30580aec75a1da1bffc2@localhost:5433/homesonspec"
-# 2. refresh each Lennar state (bounded, sequential = polite). Full national list.
-for S in california texas florida arizona colorado nevada north-carolina south-carolina georgia          alabama arkansas delaware tennessee oregon washington oklahoma new-jersey pennsylvania          virginia maryland new-york minnesota indiana; do
+
+# Secret sourced at runtime from the gitignored .env.kamatera — never hardcode it in a tracked file.
+DBPW=$(grep '^HOMESONSPEC_DB_PW=' "$ROOT/.env.kamatera" | tail -1 | cut -d= -f2)
+[ -n "$DBPW" ] || { echo "NO DB PW (check .env.kamatera)" >> "$LOG"; exit 1; }
+export DATABASE_URL="postgresql://homesonspec:$DBPW@localhost:5433/homesonspec"
+
+# Keepalive tunnel, re-established on demand — a mid-run network blip must NOT silently
+# fail the remaining states (the failure mode observed 2026-07-29).
+ensure_tunnel() {
+  pgrep -f "5433:localhost:5432" >/dev/null && return 0
+  ssh -f -N -o ServerAliveInterval=30 -o ServerAliveCountMax=1000000 -o ExitOnForwardFailure=yes \
+      -L 5433:localhost:5432 root@45.61.58.125
+  sleep 2
+  pgrep -f "5433:localhost:5432" >/dev/null
+}
+ensure_tunnel || { echo "TUNNEL FAILED" >> "$LOG"; exit 1; }
+
+# Refresh each Lennar state (bounded, sequential = polite). Full national list.
+STATES="california texas florida arizona colorado nevada north-carolina south-carolina georgia
+        alabama arkansas delaware tennessee oregon washington oklahoma new-jersey pennsylvania
+        virginia maryland new-york minnesota indiana"
+for S in $STATES; do
+  ensure_tunnel || { echo "--- $S: tunnel down, re-establish failed, aborting ---" >> "$LOG"; break; }
   echo "--- $S ---" >> "$LOG"
-  LENNAR_STATE_FILTER=$S LENNAR_PAGE_LIMIT=100000 FORCE_REEXTRACT=1     npx tsx src/cli.ts --adapter=lennar-site 2>&1 | grep -E '"published"|"pages"|Error' >> "$LOG"
+  LENNAR_STATE_FILTER="$S" LENNAR_PAGE_LIMIT=100000 FORCE_REEXTRACT=1 \
+    npx tsx src/cli.ts --adapter=lennar-site 2>&1 | grep -E '"published"|"pages"|Error' >> "$LOG"
 done
-# 3. teardown
+
 pkill -f "5433:localhost:5432" 2>/dev/null
 echo "=== done $(date) ===" >> "$LOG"

← 63d56d74 auto-save: 2026-07-29T09:37:47 (1 files) — collectors/richmo  ·  back to Homesonspec  ·  collectors: add K. Hovnanian adapter (khov.com RSC, 30th bui 04b55704 →