backup-daemon-health: credit the user belt via its verified success log (fix false FAIL)

1256a2dd40be0d1a588aea98c54d3dab9e8c16d5 · 2026-08-17 22:58:57 -0700 · Steve Abrams

launchd runs/exit-code resets to 0 on reload/reboot and shows '(never exited)',
so the canary false-FAILed after the Aug pm2-fracture reboots even though the
user LaunchAgent wrote a verified Henry copy nightly. Now credit a belt when its
own log shows a recent '[Henry] PASS: <prefix>_' (stricter than exit code —
validates the outcome; anti-stale guard preserved via log-mtime recency).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

M scripts/daemon-health.sh

Diff

commit 1256a2dd40be0d1a588aea98c54d3dab9e8c16d5
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Mon Aug 17 22:58:57 2026 -0700

    backup-daemon-health: credit the user belt via its verified success log (fix false FAIL)
    
    launchd runs/exit-code resets to 0 on reload/reboot and shows '(never exited)',
    so the canary false-FAILed after the Aug pm2-fracture reboots even though the
    user LaunchAgent wrote a verified Henry copy nightly. Now credit a belt when its
    own log shows a recent '[Henry] PASS: <prefix>_' (stricter than exit code —
    validates the outcome; anti-stale guard preserved via log-mtime recency).
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 scripts/daemon-health.sh | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/scripts/daemon-health.sh b/scripts/daemon-health.sh
index 5aefb32..f7e347e 100755
--- a/scripts/daemon-health.sh
+++ b/scripts/daemon-health.sh
@@ -38,7 +38,7 @@ rows=()
 
 # check <db-name> <henry-dir> <dump-prefix> <base-label>
 check(){
-  local name="$1" dir="$2" prefix="$3" base="$4"
+  local name="$1" dir="$2" prefix="$3" base="$4" logf="${5:-}"
   local rootp userp root_ec user_ec newest age_h fresh verdict reason
   # BELT-AND-SUSPENDERS (Steve ruled 8/14 "Both", TK-10547): the Henry mirror is written by TWO
   # redundant schedulers — the -root SYSTEM LaunchDaemon (Option B: durable, survives logout, needs
@@ -65,6 +65,23 @@ check(){
   local root_ok=0 user_ok=0
   [ -n "$rootp" ] && [ "$root_json" = "0" ] && root_ok=1
   [ -n "$userp" ] && [ "$user_json" = "0" ] && user_ok=1
+  # launchd's runs/exit-code RESETS to 0 on every reload/reboot and shows "(never exited)" for a
+  # loaded-but-not-yet-refired job, so it cannot prove the USER belt landed a scheduled clean run
+  # (TK-10547 follow-up: after the Aug pm2-fracture reboots it false-FAILed while the belt wrote a
+  # VERIFIED Henry copy nightly). Ground truth = the belt's OWN success log: a recent
+  # "[Henry] PASS: <prefix>_..." line means the most recent run landed a verified copy. Recency
+  # (log mtime within the freshness window) preserves the anti-stale guard — a dead belt stops
+  # appending PASS and its dump goes stale -> still FAIL. This is STRICTER than the exit code: it
+  # validates the actual outcome, not just that launchd thinks a process exited 0.
+  local user_landed=0
+  if [ -n "$logf" ] && [ -f "$logf" ]; then
+    local lmtime lage_h
+    lmtime=$(stat -f %m "$logf" 2>/dev/null || echo 0)
+    lage_h=$(( (now - lmtime) / 3600 ))
+    if [ "$lage_h" -le "$STALE_WARN_H" ] && tail -n 25 "$logf" | grep -q "\[Henry\] PASS: ${prefix}_"; then
+      user_landed=1; user_ok=1
+    fi
+  fi
   local up=$(( root_ok + user_ok ))
 
   newest=$(ls -t "$dir/${prefix}"_*.dump 2>/dev/null | head -1)
@@ -75,6 +92,7 @@ check(){
   else age_h=-1; fresh="missing"; fi
 
   local writers="root(sys)=${root_word} user(gui)=${user_word}"
+  [ "$user_landed" = "1" ] && writers="$writers user-log=verified-PASS@${lage_h}h"
   # Henry-freshness is authoritative. Fresh via at least one healthy writer = data safe; both writers
   # healthy = PASS; exactly one down = WARN (degraded redundancy, fix the down belt); both down or
   # Henry missing/stale = FAIL (no working writer → the silent-death setup).
@@ -82,7 +100,7 @@ check(){
   elif [ "$fresh" = "stale" ]; then verdict="FAIL"; reason="Henry dump ${age_h}h old (> ${STALE_WARN_H}h) — neither writer is landing [$writers]"
   elif [ "$up" -eq 0 ]; then verdict="FAIL"; reason="Henry fresh (${age_h}h) but NO scheduled writer has landed a clean run — the mirror is only being kept fresh by manual rescue; the automated belts are not proven [$writers]"
   elif [ "$up" -eq 1 ] && [ "$REQUIRE_BOTH_BELTS" = "1" ]; then verdict="WARN"; reason="Henry fresh (${age_h}h) via one writer — redundancy DEGRADED, other belt down [$writers]"
-  elif [ "$up" -eq 1 ]; then verdict="PASS"; reason="Henry fresh (${age_h}h) via the accepted primary belt (root belt accepted-down pending FDA, Steve 8/15) [$writers]"
+  elif [ "$up" -eq 1 ]; then verdict="PASS"; reason="Henry fresh (${age_h}h) via a proven writer (verified PASS in its own log); root belt accepted-down pending FDA (TK-10547) [$writers]"
   else verdict="PASS"; reason="Henry fresh (${age_h}h), both writers healthy [$writers]"; fi
   [ "$(rank "$verdict")" -gt "$(rank "$worst")" ] && worst="$verdict"
 
@@ -94,8 +112,8 @@ check(){
 }
 
 echo "== on-prem backup daemon health ($(date -Iseconds)) =="
-check dw_unified  /Volumes/Henry/dw-backups/dw_unified  dw_unified  com.steve.nas-dwdump-mirror
-check realestate  /Volumes/Henry/dw-backups/realestate  realestate  com.steve.nas-realestate-dump-mirror
+check dw_unified  /Volumes/Henry/dw-backups/dw_unified  dw_unified  com.steve.nas-dwdump-mirror          "$HERE/../data/launchd.out.log"
+check realestate  /Volumes/Henry/dw-backups/realestate  realestate  com.steve.nas-realestate-dump-mirror  "$HERE/../data/launchd-realestate.out.log"
 echo "== overall: $worst =="
 
 # JSON heartbeat (PASS/WARN/FAIL vocab so fleet-health-rollup + meta-watchdog read it right)

← 9fc2571 daemon-health: fix jq --argjson crash on '(never exited)' — · back to Nas Setup · backup-repos-to-henry: cover ~/.claude and its 73 nested ski 7d6456f →