add refiner_compare watchdog — auto-restart on crash, exit on 25/25

838379d81e32bdf7961ea60974174f466d72dca2 · 2026-05-19 12:08:44 -0700 · SteveStudio2

Files touched

A scripts/refiner_compare_watchdog.sh

Diff

commit 838379d81e32bdf7961ea60974174f466d72dca2
Author: SteveStudio2 <steve@designerwallcoverings.com>
Date:   Tue May 19 12:08:44 2026 -0700

    add refiner_compare watchdog — auto-restart on crash, exit on 25/25
---
 scripts/refiner_compare_watchdog.sh | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/scripts/refiner_compare_watchdog.sh b/scripts/refiner_compare_watchdog.sh
new file mode 100755
index 0000000..fc96663
--- /dev/null
+++ b/scripts/refiner_compare_watchdog.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+# Watchdog for refiner_compare.js — restarts it if it dies, exits when the
+# 25-pair job completes. The script itself is resume-safe (skips rendered
+# images), so a restart picks up where the crash left off.
+cd "$HOME/Projects/wallco-ai" || exit 1
+PROG="public/refiner-compare/progress.json"
+LOG=/tmp/refiner_compare.log
+restarts=0
+while true; do
+  done=$(python3 -c "import json;print(json.load(open('$PROG'))['done'])" 2>/dev/null || echo 0)
+  if [ "${done:-0}" -ge 25 ]; then
+    echo "$(date '+%H:%M:%S') job complete ($done/25) — watchdog exiting after $restarts restart(s)"
+    break
+  fi
+  if ! pgrep -f 'scripts/refiner_compare.js' >/dev/null 2>&1; then
+    if [ "$restarts" -ge 30 ]; then
+      echo "$(date '+%H:%M:%S') hit 30-restart cap at $done/25 — giving up, check $LOG"
+      break
+    fi
+    restarts=$((restarts+1))
+    echo "$(date '+%H:%M:%S') refiner_compare.js not running (at $done/25) — restart #$restarts"
+    nohup node scripts/refiner_compare.js >> "$LOG" 2>&1 &
+    sleep 5
+  fi
+  sleep 30
+done

← 252f179 refiner_compare: skip already-rendered images so a crash+res · back to Wallco Ai · refiner_compare: reorder into base pass then refiner pass — f042d90 →