Add read-only daemon-health probe: surfaces root-daemon last_exit + Henry dump freshness so a nightly-failing daemon can't hide behind a manual PARTIAL (TK-10547)

b7fbe234b65e892e5ca39c4f939eba7830430437 · 2026-08-13 19:11:18 -0700 · Steve

Files touched

A scripts/daemon-health.sh

Diff

commit b7fbe234b65e892e5ca39c4f939eba7830430437
Author: Steve <steve@designerwallcoverings.com>
Date:   Thu Aug 13 19:11:18 2026 -0700

    Add read-only daemon-health probe: surfaces root-daemon last_exit + Henry dump freshness so a nightly-failing daemon can't hide behind a manual PARTIAL (TK-10547)
---
 scripts/daemon-health.sh | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/scripts/daemon-health.sh b/scripts/daemon-health.sh
new file mode 100755
index 0000000..2314872
--- /dev/null
+++ b/scripts/daemon-health.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+# daemon-health.sh — HONEST read-only health of the on-prem backup ROOT DAEMONS.
+#
+# Why this exists (TK-10547, yoloforever cycle 2): pull.log + latest.json reflect whoever
+# ran the script LAST — including a MANUAL user-context run — so a root daemon that has
+# been exiting 1 every night (TCC can't write /Volumes/Henry) hides behind a rosy manual
+# PARTIAL. This probe reads the DAEMON's own launchd state (readable WITHOUT sudo) plus the
+# freshness of the newest Henry dump, and prints a truthful per-daemon verdict.
+#
+# READ-ONLY: no sudo, no writes to latest.json, no launchd mutation, no backup run. $0 local.
+set -uo pipefail
+
+STALE_HOURS="${STALE_HOURS:-30}"   # a nightly 03:45 job older than this = missed a run
+now=$(date +%s)
+rc=0
+
+# daemon <label> <henry-dir> <dump-prefix>
+check(){
+  local label="$1" dir="$2" prefix="$3"
+  local print exit_code runs state
+  print=$(launchctl print "system/$label" 2>/dev/null)
+  if [ -z "$print" ]; then echo "  $label: launchd state UNREADABLE"; rc=1; return; fi
+  exit_code=$(printf '%s\n' "$print" | awk -F'= ' '/last exit code/{print $2; exit}')
+  runs=$(printf '%s\n' "$print" | awk -F'= ' '/^\truns/{print $2; exit}')
+  state=$(printf '%s\n' "$print" | awk -F'= ' '/^\tstate/{print $2; exit}')
+
+  # freshness of the newest Henry dump for this DB
+  local newest age_h="?" fresh="NO DUMP"
+  newest=$(ls -t "$dir/${prefix}"_*.dump 2>/dev/null | head -1)
+  if [ -n "$newest" ]; then
+    local mtime; mtime=$(stat -f %m "$newest" 2>/dev/null || echo 0)
+    age_h=$(( (now - mtime) / 3600 ))
+    if [ "$age_h" -le "$STALE_HOURS" ]; then fresh="FRESH (${age_h}h)"; else fresh="STALE (${age_h}h)"; fi
+  fi
+
+  local verdict="OK"
+  { [ "${exit_code:-1}" != "0" ] || [ "$fresh" = "NO DUMP" ] || [[ "$fresh" == STALE* ]]; } && { verdict="DEGRADED"; rc=1; }
+  echo "  $label"
+  echo "     daemon: state=${state:-?} runs=${runs:-?} last_exit=${exit_code:-?}   (last_exit!=0 = the nightly write is failing)"
+  echo "     henry : $fresh   ${newest:+($(basename "$newest"))}"
+  echo "     >>> $verdict"
+}
+
+echo "== on-prem backup daemon health ($(date -Iseconds)) =="
+check com.steve.nas-dwdump-mirror-root        /Volumes/Henry/dw-backups/dw_unified  dw_unified
+check com.steve.nas-realestate-dump-mirror-root /Volumes/Henry/dw-backups/realestate realestate
+echo "== overall: $([ $rc -eq 0 ] && echo HEALTHY || echo DEGRADED — a daemon is exit!=0 or its Henry dump is stale) =="
+exit $rc

← 20be61d pull-dw-dump: derive retention glob from $BASE so realestate · back to Nas Setup · daemon-health: emit PASS/WARN/FAIL JSON heartbeat + guarded e1954e8 →