daemon-health: belt-and-suspenders model (Steve ruled 8/14 'Both', TK-10547)

4a447c8255c592fb82a5f6f62a340920388ce680 · 2026-08-14 10:18:30 -0700 · steve

Check BOTH redundant writers per DB — the -root SYSTEM daemon (Option B,
durable, needs FDA on /usr/local/bin/nas-backup-sh) AND the gui/ USER agent
(Option C, homebrew bash, proven writing). Verdict is outcome-based on Henry
freshness: both writers healthy + fresh = PASS; exactly one belt down = WARN
(degraded redundancy, data safe); both down or Henry missing/stale = FAIL.
Current reality = WARN until the FDA grant brings the root belt up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

M scripts/daemon-health.sh

Diff

commit 4a447c8255c592fb82a5f6f62a340920388ce680
Author: steve <steve@designerwallcoverings.com>
Date:   Fri Aug 14 10:18:30 2026 -0700

    daemon-health: belt-and-suspenders model (Steve ruled 8/14 'Both', TK-10547)
    
    Check BOTH redundant writers per DB — the -root SYSTEM daemon (Option B,
    durable, needs FDA on /usr/local/bin/nas-backup-sh) AND the gui/ USER agent
    (Option C, homebrew bash, proven writing). Verdict is outcome-based on Henry
    freshness: both writers healthy + fresh = PASS; exactly one belt down = WARN
    (degraded redundancy, data safe); both down or Henry missing/stale = FAIL.
    Current reality = WARN until the FDA grant brings the root belt up.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 scripts/daemon-health.sh | 65 +++++++++++++++++++++++++++---------------------
 1 file changed, 36 insertions(+), 29 deletions(-)

diff --git a/scripts/daemon-health.sh b/scripts/daemon-health.sh
index ace052b..6d7258c 100755
--- a/scripts/daemon-health.sh
+++ b/scripts/daemon-health.sh
@@ -1,5 +1,6 @@
 #!/bin/bash
-# daemon-health.sh — HONEST read-only health of the on-prem backup ROOT DAEMONS.
+# daemon-health.sh — HONEST read-only health of the on-prem backup MIRROR (belt-and-suspenders:
+# the -root SYSTEM daemon + the gui/ USER agent, both redundant writers to /Volumes/Henry).
 #
 # Why (TK-10547): pull.log + latest.json reflect whoever ran the script LAST (incl. a MANUAL
 # user-context rescue run), so a root daemon exiting 1 every night (TCC can't write /Volumes/
@@ -25,17 +26,24 @@ worst="PASS"   # PASS < WARN < FAIL
 rank(){ case "$1" in FAIL) echo 2;; WARN) echo 1;; *) echo 0;; esac; }
 rows=()
 
-# check <label> <henry-dir> <dump-prefix>
+# check <db-name> <henry-dir> <dump-prefix> <base-label>
 check(){
-  local label="$1" dir="$2" prefix="$3"
-  local print exit_code runs state verdict reason newest age_h fresh
-  # Option B (Steve-approved, TK-10547): the canonical scheduled job is the -root SYSTEM LaunchDaemon
-  # (runs nightly 03:45 as root). Option C (a user LaunchAgent) was REJECTED — it hits the same SIP-bash
-  # TCC wall and only runs while logged in. So look up the actually-scheduled job in system/, not gui/.
-  print=$(launchctl print "system/$label" 2>/dev/null)
-  exit_code=$(printf '%s\n' "$print" | awk -F'= ' '/last exit code/{print $2; exit}')
-  runs=$(printf '%s\n' "$print" | awk -F'= ' '/^\truns/{print $2; exit}')
-  state=$(printf '%s\n' "$print" | awk -F'= ' '/^\tstate/{print $2; exit}')
+  local name="$1" dir="$2" prefix="$3" base="$4"
+  local rootp userp root_ec user_ec newest age_h fresh verdict reason
+  # BELT-AND-SUSPENDERS (Steve ruled 8/14 "Both", TK-10547): the Henry mirror is written by TWO
+  # redundant schedulers — the -root SYSTEM LaunchDaemon (Option B: durable, survives logout, needs
+  # FDA on /usr/local/bin/nas-backup-sh) AND the gui/ USER LaunchAgent via /opt/homebrew/bin/bash
+  # (Option C: runs while logged in, proven writing). So check BOTH; the verdict is OUTCOME-based on
+  # Henry freshness (fresh via EITHER writer = data safe) but we still WARN if one belt is down so a
+  # broken writer is never hidden — the ticket's whole point.
+  rootp=$(launchctl print "system/${base}-root" 2>/dev/null)
+  userp=$(launchctl print "gui/$MYUID/${base}" 2>/dev/null)
+  root_ec=$(printf '%s\n' "$rootp" | awk -F'= ' '/last exit code/{print $2; exit}')
+  user_ec=$(printf '%s\n' "$userp" | awk -F'= ' '/last exit code/{print $2; exit}')
+  local root_ok=0 user_ok=0
+  [ -n "$rootp" ] && [ "${root_ec:-1}" = "0" ] && root_ok=1
+  [ -n "$userp" ] && [ "${user_ec:-1}" = "0" ] && user_ok=1
+  local up=$(( root_ok + user_ok ))
 
   newest=$(ls -t "$dir/${prefix}"_*.dump 2>/dev/null | head -1)
   if [ -n "$newest" ]; then
@@ -44,28 +52,27 @@ check(){
     [ "$age_h" -le "$STALE_WARN_H" ] && fresh="fresh" || fresh="stale"
   else age_h=-1; fresh="missing"; fi
 
-  # MECHANISM-LOUD verdict (Steve ruled 8/14, TK-10547): a broken SCHEDULED daemon = FAIL even when a
-  # manual rescue left Henry fresh — the entire point of this ticket is that monitoring must NOT hide a
-  # broken daemon behind a rosy manual PARTIAL. Henry-freshness enriches the REASON (data-safe vs
-  # data-lost), it does NOT soften the verdict. Only a clean daemon (exit 0) + fresh Henry = PASS.
-  if [ -z "$print" ]; then verdict="FAIL"; reason="launchd state unreadable for $label — scheduler gone"
-  elif [ "${exit_code:-1}" != "0" ] && [ "$fresh" = "fresh" ]; then verdict="FAIL"; reason="daemon last_exit=${exit_code:-?} but Henry dump fresh (${age_h}h) — mechanism broken, data safe (launcher swap pending)"
-  elif [ "${exit_code:-1}" != "0" ]; then verdict="FAIL"; reason="daemon last_exit=${exit_code:-?} — mechanism broken AND Henry ${fresh} (${age_h}h)"
-  elif [ "$fresh" = "missing" ]; then verdict="FAIL"; reason="no Henry dump for $prefix — mirror never landed"
-  elif [ "$fresh" = "stale" ]; then verdict="WARN"; reason="Henry dump ${age_h}h old (> ${STALE_WARN_H}h) — daemon exit 0 but mirror not landing"
-  else verdict="PASS"; reason="daemon exit 0, Henry dump ${age_h}h old"; fi
+  local writers="root(sys)=exit:${root_ec:-NA}$([ -n "$rootp" ]||echo /unloaded) user(gui)=exit:${user_ec:-NA}$([ -n "$userp" ]||echo /unloaded)"
+  # Henry-freshness is authoritative. Fresh via at least one healthy writer = data safe; both writers
+  # healthy = PASS; exactly one down = WARN (degraded redundancy, fix the down belt); both down or
+  # Henry missing/stale = FAIL (no working writer → the silent-death setup).
+  if [ "$fresh" = "missing" ]; then verdict="FAIL"; reason="no Henry dump for $prefix — neither writer landed [$writers]"
+  elif [ "$fresh" = "stale" ]; then verdict="FAIL"; reason="Henry dump ${age_h}h old (> ${STALE_WARN_H}h) — neither writer is landing [$writers]"
+  elif [ "$up" -eq 0 ]; then verdict="FAIL"; reason="Henry fresh (${age_h}h) but BOTH writers broken — nothing will refresh it [$writers]"
+  elif [ "$up" -eq 1 ]; then verdict="WARN"; reason="Henry fresh (${age_h}h) via one writer — redundancy DEGRADED, other belt down [$writers]"
+  else verdict="PASS"; reason="Henry fresh (${age_h}h), both writers healthy [$writers]"; fi
   [ "$(rank "$verdict")" -gt "$(rank "$worst")" ] && worst="$verdict"
 
-  echo "  $label -> $verdict ($reason)"
-  echo "     daemon: state=${state:-?} runs=${runs:-?} last_exit=${exit_code:-?} | henry: $fresh ${newest:+$(basename "$newest")} (${age_h}h)"
-  rows+=("$(jq -n --arg l "$label" --arg v "$verdict" --arg r "$reason" \
-     --argjson ec "${exit_code:-null}" --argjson runs "${runs:-null}" --argjson age "${age_h:-null}" \
-     '{daemon:$l,verdict:$v,reason:$r,last_exit:$ec,runs:$runs,henry_dump_age_h:$age}')")
+  echo "  $name -> $verdict ($reason)"
+  echo "     henry: $fresh ${newest:+$(basename "$newest")} (${age_h}h) | writers: $writers"
+  rows+=("$(jq -n --arg l "$name" --arg v "$verdict" --arg r "$reason" \
+     --argjson rec "${root_ec:-null}" --argjson uec "${user_ec:-null}" --argjson age "${age_h:-null}" \
+     '{db:$l,verdict:$v,reason:$r,root_last_exit:$rec,user_last_exit:$uec,henry_dump_age_h:$age}')")
 }
 
 echo "== on-prem backup daemon health ($(date -Iseconds)) =="
-check com.steve.nas-dwdump-mirror-root          /Volumes/Henry/dw-backups/dw_unified  dw_unified
-check com.steve.nas-realestate-dump-mirror-root /Volumes/Henry/dw-backups/realestate  realestate
+check dw_unified  /Volumes/Henry/dw-backups/dw_unified  dw_unified  com.steve.nas-dwdump-mirror
+check realestate  /Volumes/Henry/dw-backups/realestate  realestate  com.steve.nas-realestate-dump-mirror
 echo "== overall: $worst =="
 
 # JSON heartbeat (PASS/WARN/FAIL vocab so fleet-health-rollup + meta-watchdog read it right)
@@ -91,7 +98,7 @@ if [ "$ALERT" = "1" ]; then
   fi
   if [ "$fire" = "1" ]; then
     CNCP="${CNCP_URL:-http://localhost:3333}"
-    note="[BACKUP DAEMON HEALTH $(date +%F)] Henry mirror $worst — $(printf '%s\n' "${rows[@]}" | jq -r 'select(.verdict!="PASS")|"\(.daemon): \(.reason)"' | paste -sd'; ' -). The nightly root daemon can't write /Volumes/Henry (TCC/FDA). Fix: grant FDA to the daemon's launcher (TK-10547)."
+    note="[BACKUP DAEMON HEALTH $(date +%F)] Henry mirror $worst — $(printf '%s\n' "${rows[@]}" | jq -r 'select(.verdict!="PASS")|"\(.db): \(.reason)"' | paste -sd'; ' -). Belt-and-suspenders: WARN = one of the two writers is down (data still safe via the other); FAIL = both down or Henry stale. To restore the root belt, grant FDA to /usr/local/bin/nas-backup-sh (TK-10547)."
     curl -sS --max-time 10 "$CNCP/api/parking-lot" -H 'Content-Type: application/json' \
       -d "$(jq -n --arg u "onprem://henry-mirror-daemon" --arg note "$note" '{url:$u,note:$note}')" >/dev/null 2>&1 || true
     if [ -f "$HOME/.claude/skills/_shared/george-send.sh" ]; then

← 272bd47 daemon-health: fix false-PASS regression — restore mechanism · back to Nas Setup · belt-and-suspenders: stagger user-agent belts past their roo d41f040 →