watchdog: opt-in launchd plist + zsh script that bounces dashboard on /healthz 503

a353d859c2ddec273e1ed60f2a63987e6f419214 · 2026-05-12 18:17:16 -0700 · steve

launchd/sniper-watchdog.sh
  - curls /healthz with 5s timeout
  - 200 -> silent exit
  - anything else -> log UNHEALTHY + try pm2 restart, fall back to pgrep+nohup
  - all writes append-only to logs/watchdog.log

launchd/com.steve.sniper-watchdog.plist
  - StartInterval 60s, RunAtLoad true
  - PATH includes /opt/homebrew/bin for pm2 lookup
  - opt-in: must launchctl load to activate

plutil -lint passes; healthy-path smoke run exits 0 silently with no log
written. README updated with enable/disable instructions.

Also recorded 4.65h slice 6-7 RECHECK#2: 0/10. Cumulative 0/150 across
16 datapoints.

Files touched

M README.md
A launchd/com.steve.sniper-watchdog.plist
A launchd/sniper-watchdog.sh

Diff

commit a353d859c2ddec273e1ed60f2a63987e6f419214
Author: steve <steve@designerwallcoverings.com>
Date:   Tue May 12 18:17:16 2026 -0700

    watchdog: opt-in launchd plist + zsh script that bounces dashboard on /healthz 503
    
    launchd/sniper-watchdog.sh
      - curls /healthz with 5s timeout
      - 200 -> silent exit
      - anything else -> log UNHEALTHY + try pm2 restart, fall back to pgrep+nohup
      - all writes append-only to logs/watchdog.log
    
    launchd/com.steve.sniper-watchdog.plist
      - StartInterval 60s, RunAtLoad true
      - PATH includes /opt/homebrew/bin for pm2 lookup
      - opt-in: must launchctl load to activate
    
    plutil -lint passes; healthy-path smoke run exits 0 silently with no log
    written. README updated with enable/disable instructions.
    
    Also recorded 4.65h slice 6-7 RECHECK#2: 0/10. Cumulative 0/150 across
    16 datapoints.
---
 README.md                               | 13 ++++++++
 launchd/com.steve.sniper-watchdog.plist | 48 +++++++++++++++++++++++++++++
 launchd/sniper-watchdog.sh              | 53 +++++++++++++++++++++++++++++++++
 3 files changed, 114 insertions(+)

diff --git a/README.md b/README.md
index 56d9290..f345226 100644
--- a/README.md
+++ b/README.md
@@ -111,6 +111,19 @@ State of the world for **free** real-time CT-log access, as of 2026-05-12 ~23:05
 
 Meanwhile the **honeypot + DoH spot-check** path (the actual aggregator-attribution experiment) is unaffected — that runs on Cloudflare 1.1.1.1 DoH and Node `child_process whois`, neither of which depends on CT logs.
 
+## Liveness watchdog (opt-in)
+
+`launchd/sniper-watchdog.sh` polls `/healthz` once a minute, restarts the
+dashboard on HTTP 503 (or unreachable). Defaults: present in repo, not loaded.
+PM2-first (looks for `sniper-dashboard`), falls back to pgrep + nohup respawn.
+
+To enable on Mac2:
+```
+cp launchd/com.steve.sniper-watchdog.plist ~/Library/LaunchAgents/
+launchctl load -w ~/Library/LaunchAgents/com.steve.sniper-watchdog.plist
+```
+Logs at `logs/watchdog.log` (silent on healthy; one line per UNHEALTHY+ACTION).
+
 ## YOLO loop notes
 
 When running in autonomous /loop mode, each tick must:
diff --git a/launchd/com.steve.sniper-watchdog.plist b/launchd/com.steve.sniper-watchdog.plist
new file mode 100644
index 0000000..7bb349d
--- /dev/null
+++ b/launchd/com.steve.sniper-watchdog.plist
@@ -0,0 +1,48 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<!--
+  com.steve.sniper-watchdog — minute-cadence /healthz probe + auto-restart.
+
+  OPT-IN. To enable on Mac2:
+    cp launchd/com.steve.sniper-watchdog.plist ~/Library/LaunchAgents/
+    launchctl load -w ~/Library/LaunchAgents/com.steve.sniper-watchdog.plist
+
+  To disable:
+    launchctl unload -w ~/Library/LaunchAgents/com.steve.sniper-watchdog.plist
+    rm ~/Library/LaunchAgents/com.steve.sniper-watchdog.plist
+
+  Logs:    ~/Projects/domain-sniper/logs/watchdog.log
+  Errors:  ~/Projects/domain-sniper/logs/watchdog.err
+-->
+<plist version="1.0">
+<dict>
+  <key>Label</key>
+  <string>com.steve.sniper-watchdog</string>
+
+  <key>ProgramArguments</key>
+  <array>
+    <string>/bin/zsh</string>
+    <string>-c</string>
+    <string>$HOME/Projects/domain-sniper/launchd/sniper-watchdog.sh</string>
+  </array>
+
+  <key>StartInterval</key>
+  <integer>60</integer>
+
+  <key>RunAtLoad</key>
+  <true/>
+
+  <key>StandardOutPath</key>
+  <string>/tmp/sniper-watchdog.stdout</string>
+  <key>StandardErrorPath</key>
+  <string>/Users/stevestudio2/Projects/domain-sniper/logs/watchdog.err</string>
+
+  <key>EnvironmentVariables</key>
+  <dict>
+    <key>PATH</key>
+    <string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
+    <key>HOME</key>
+    <string>/Users/stevestudio2</string>
+  </dict>
+</dict>
+</plist>
diff --git a/launchd/sniper-watchdog.sh b/launchd/sniper-watchdog.sh
new file mode 100755
index 0000000..f00608b
--- /dev/null
+++ b/launchd/sniper-watchdog.sh
@@ -0,0 +1,53 @@
+#!/bin/zsh
+# sniper-watchdog.sh — minute-cadence /healthz probe for the dashboard.
+# On HTTP 503 (or unreachable), bounces the dashboard process.
+#
+# Discovers the dashboard via either:
+#   1. PM2  — `pm2 jlist | jq` for a process named "sniper-dashboard"
+#   2. PID  — falls back to pgrep on `node dashboard.js`
+#
+# All logs go to ~/Projects/domain-sniper/logs/watchdog.log (append-only).
+#
+# OPT-IN — Steve must `launchctl load` the plist for this to run.
+# Default repo state: present, not loaded.
+
+set -u
+PROJ="${HOME}/Projects/domain-sniper"
+LOG="${PROJ}/logs/watchdog.log"
+mkdir -p "${PROJ}/logs"
+
+stamp() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }
+say()   { echo "$(stamp) $*" >> "$LOG"; }
+
+# Probe /healthz; -m caps the request at 5s.
+RESP=$(curl -sS -o /tmp/sniper-healthz.json -w "%{http_code}" -m 5 http://127.0.0.1:9895/healthz 2>/dev/null)
+CODE="${RESP:-000}"
+
+if [ "$CODE" = "200" ]; then
+  # Quiet on healthy unless we want noise.
+  exit 0
+fi
+
+say "UNHEALTHY  http=${CODE}  body=$(head -c 200 /tmp/sniper-healthz.json 2>/dev/null | tr -d '\n')"
+
+# Try pm2 first
+if command -v pm2 >/dev/null 2>&1; then
+  if pm2 jlist 2>/dev/null | grep -q '"name":"sniper-dashboard"'; then
+    say "ACTION pm2 restart sniper-dashboard"
+    pm2 restart sniper-dashboard >> "$LOG" 2>&1
+    exit 0
+  fi
+fi
+
+# Fallback: pgrep + spawn
+OLD_PIDS=$(pgrep -f "node dashboard.js" 2>/dev/null || true)
+if [ -n "$OLD_PIDS" ]; then
+  say "ACTION kill PIDs ${OLD_PIDS}"
+  kill $OLD_PIDS 2>/dev/null
+  sleep 2
+fi
+cd "$PROJ" || { say "FATAL cannot cd ${PROJ}"; exit 1; }
+nohup node dashboard.js >> "${PROJ}/logs/dashboard.log" 2>&1 &
+disown
+say "ACTION respawned dashboard via nohup pid=$!"
+exit 0

← dfb3b86 dashboard: /healthz liveness probe for pm2/launchd · back to Domain Sniper · watch-ct: migrate to ct-source-ctlog (6-log fan-out, default 02d8f7f →