schu-reimport: launchd-managed, restartable, loop-until-clear + self-expire

b6ac11303400bc7b014dbd14ad4338c1a1636f34 · 2026-06-11 15:17:17 -0700 · SteveStudio2

com.steve.schu-reimport runs every 5 min (RunAtLoad+StartInterval 300). Each tick grinds
200-batches until net-new clears or 4x no-progress (rate bucket drained), then exits so the
next tick resumes after refill. Single-flight lock; survives close/reboot; self-boots-out +
renames plist when net-new<5. Clears all ~2,505 across ticks without a fixed cap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Files touched

M shopify/scripts/cadence/schu-reimport.sh

Diff

commit b6ac11303400bc7b014dbd14ad4338c1a1636f34
Author: SteveStudio2 <stevestudio2@SteveStudio2s-Mac-Studio.local>
Date:   Thu Jun 11 15:17:17 2026 -0700

    schu-reimport: launchd-managed, restartable, loop-until-clear + self-expire
    
    com.steve.schu-reimport runs every 5 min (RunAtLoad+StartInterval 300). Each tick grinds
    200-batches until net-new clears or 4x no-progress (rate bucket drained), then exits so the
    next tick resumes after refill. Single-flight lock; survives close/reboot; self-boots-out +
    renames plist when net-new<5. Clears all ~2,505 across ticks without a fixed cap.
    
    Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 shopify/scripts/cadence/schu-reimport.sh | 62 +++++++++++++++++++++++---------
 1 file changed, 46 insertions(+), 16 deletions(-)

diff --git a/shopify/scripts/cadence/schu-reimport.sh b/shopify/scripts/cadence/schu-reimport.sh
index 0c40ea42..ef9d0f72 100755
--- a/shopify/scripts/cadence/schu-reimport.sh
+++ b/shopify/scripts/cadence/schu-reimport.sh
@@ -1,25 +1,55 @@
 #!/usr/bin/env bash
-# Re-create the genuinely-uncreated Schumacher SKUs in SMALL batches with throttle-safe
-# pacing. The cadence importer now (a) only counts a create when a real product id comes
-# back, (b) backs off proactively on a low leaky-bucket. So each batch creates what it can;
-# throttle-failed rows stay net-new and the next batch picks them up. Idempotent + safe.
+# Re-create the genuinely-uncreated Schumacher SKUs — LAUNCHD-MANAGED + RESTARTABLE.
+#
+# com.steve.schu-reimport runs this every 5 min (RunAtLoad + StartInterval 300). Each tick
+# grinds 200-SKU batches until net-new clears OR it hits 4 consecutive no-progress batches
+# (Shopify rate-bucket empty), then exits — releasing the lock so the NEXT tick resumes after
+# the bucket refills. Single-flight lock = no overlap. SELF-EXPIRES (boots itself out of
+# launchd + renames its plist) once net-new is cleared, so it never runs forever.
+# The importer now only counts a create when a real product id returns (throttle-failed rows
+# stay net-new and get retried) — so this is fully idempotent and not a dup risk.
 set -u
 export PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin
 DIR="$HOME/Projects/Designer-Wallcoverings/shopify/scripts/cadence"
 LOG="$DIR/data/schu-reimport.log"
+LOCK="/tmp/schu-reimport.lock"
+LABEL="com.steve.schu-reimport"
+PLIST="$HOME/Library/LaunchAgents/$LABEL.plist"
+mkdir -p "$DIR/data"
 cd "$DIR" || exit 1
-NETNEW_SQL="SELECT count(*) FROM schumacher_catalog WHERE coalesce(shopify_product_id::text,'')='' AND cost>0 AND NOT coalesce(excluded,false) AND dw_sku IS NOT NULL AND dw_sku<>'';"
+NN_SQL="SELECT count(*) FROM schumacher_catalog WHERE coalesce(shopify_product_id::text,'')='' AND cost>0 AND NOT coalesce(excluded,false) AND dw_sku IS NOT NULL AND dw_sku<>'';"
+netnew() { psql -At -d dw_unified -c "$NN_SQL" 2>/dev/null; }
 
-echo "$(date '+%F %T') ===== schumacher re-import start =====" >> "$LOG"
-prev=999999
-for i in $(seq 1 25); do
-  nn=$(psql -At -d dw_unified -c "$NETNEW_SQL" 2>/dev/null)
-  echo "$(date '+%F %T') batch $i — net-new=$nn" >> "$LOG"
-  [ "${nn:-0}" -lt 5 ] && { echo "$(date '+%F %T') net-new cleared ($nn) — done." >> "$LOG"; break; }
-  # no-progress guard: if a full batch created nothing (all genuinely un-creatable), stop
-  if [ "${nn:-0}" -ge "$prev" ] && [ "$i" -gt 1 ]; then echo "$(date '+%F %T') no progress (still $nn) — stopping." >> "$LOG"; break; fi
-  prev=$nn
+# single-flight
+if ! mkdir "$LOCK" 2>/dev/null; then echo "$(date '+%F %T') still running — skip" >> "$LOG"; exit 0; fi
+trap 'rmdir "$LOCK" 2>/dev/null' EXIT
+
+# self-expire when the backfill is complete
+nn=$(netnew)
+if [ "${nn:-1}" -lt 5 ]; then
+  echo "$(date '+%F %T') net-new cleared ($nn) — self-expiring $LABEL" >> "$LOG"
+  launchctl bootout "gui/$(id -u)/$LABEL" 2>/dev/null
+  [ -f "$PLIST" ] && mv "$PLIST" "$PLIST.DONE"
+  exit 0
+fi
+
+echo "$(date '+%F %T') ===== reimport tick (net-new=$nn) =====" >> "$LOG"
+zero=0
+for i in $(seq 1 500); do
+  before=$(netnew)
+  [ "${before:-0}" -lt 5 ] && { echo "$(date '+%F %T') cleared mid-tick ($before)" >> "$LOG"; break; }
   node cadence-import.js --only Schumacher --skus 200 --commit >> "$LOG" 2>&1
-  sleep 20   # let the rate-limit bucket refill between batches
+  after=$(netnew)
+  if [ "${after:-0}" -ge "${before:-0}" ]; then zero=$((zero+1)); else zero=0; fi
+  echo "$(date '+%F %T') batch $i: $before -> $after (no-progress streak=$zero)" >> "$LOG"
+  if [ "$zero" -ge 4 ]; then echo "$(date '+%F %T') 4x no-progress (rate bucket drained) — exiting tick; next launchd tick resumes" >> "$LOG"; break; fi
+  sleep 15
 done
-echo "$(date '+%F %T') ===== schumacher re-import done — final net-new=$(psql -At -d dw_unified -c "$NETNEW_SQL" 2>/dev/null) =====" >> "$LOG"
+final=$(netnew)
+echo "$(date '+%F %T') ===== tick done — net-new=$final =====" >> "$LOG"
+# if cleared, self-expire immediately rather than waiting for next tick
+if [ "${final:-1}" -lt 5 ]; then
+  echo "$(date '+%F %T') backfill COMPLETE — self-expiring $LABEL" >> "$LOG"
+  launchctl bootout "gui/$(id -u)/$LABEL" 2>/dev/null
+  [ -f "$PLIST" ] && mv "$PLIST" "$PLIST.DONE"
+fi

← d05c6b58 Romo 339 cost backfill: pulled current trade prices from the · back to Designer Wallcoverings · Handle Shopify DAILY variant-creation cap (the real blocker) 8ed26702 →