← back to Nas Setup
Henry mirror: switch to USER LaunchAgents via homebrew bash (real fix, TK-10547)
d3111b70e633e8c117b8d457f1f4b66fc2ac43f5 · 2026-08-14 09:18:56 -0700 · steve
Root cause: the root daemons' signed launcher /usr/local/bin/nas-backup-sh is
TCC-denied on the /Volumes/Henry external volume, so the nightly [ -w ] test
failed and the mirror silently skipped for 5 nights (Aug 9-13). NOT an FDA-on-
/bin/bash issue as the ticket assumed.
Fix (empirically verified): the mirror runs fine as USER LaunchAgents launched
via /opt/homebrew/bin/bash (non-SIP, holds the volume grant — the loaded
backup-daemon-health agent reads Henry through it hourly). Swapped the dw_unified
user plist to homebrew bash to match the already-swapped realestate one; both
enabled+bootstrapped, kickstart-verified writing [Henry] rsync ok. daemon-health
repointed to watch the user agents in gui/ domain with Henry-freshness as the
authoritative signal. Canary now PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Files touched
M launchd/com.steve.nas-dwdump-mirror.plistM scripts/daemon-health.sh
Diff
commit d3111b70e633e8c117b8d457f1f4b66fc2ac43f5
Author: steve <steve@designerwallcoverings.com>
Date: Fri Aug 14 09:18:56 2026 -0700
Henry mirror: switch to USER LaunchAgents via homebrew bash (real fix, TK-10547)
Root cause: the root daemons' signed launcher /usr/local/bin/nas-backup-sh is
TCC-denied on the /Volumes/Henry external volume, so the nightly [ -w ] test
failed and the mirror silently skipped for 5 nights (Aug 9-13). NOT an FDA-on-
/bin/bash issue as the ticket assumed.
Fix (empirically verified): the mirror runs fine as USER LaunchAgents launched
via /opt/homebrew/bin/bash (non-SIP, holds the volume grant — the loaded
backup-daemon-health agent reads Henry through it hourly). Swapped the dw_unified
user plist to homebrew bash to match the already-swapped realestate one; both
enabled+bootstrapped, kickstart-verified writing [Henry] rsync ok. daemon-health
repointed to watch the user agents in gui/ domain with Henry-freshness as the
authoritative signal. Canary now PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
launchd/com.steve.nas-dwdump-mirror.plist | 7 ++++++-
scripts/daemon-health.sh | 27 ++++++++++++++++-----------
2 files changed, 22 insertions(+), 12 deletions(-)
diff --git a/launchd/com.steve.nas-dwdump-mirror.plist b/launchd/com.steve.nas-dwdump-mirror.plist
index 0058a9f..fa4dc97 100644
--- a/launchd/com.steve.nas-dwdump-mirror.plist
+++ b/launchd/com.steve.nas-dwdump-mirror.plist
@@ -15,7 +15,12 @@
<string>com.steve.nas-dwdump-mirror</string>
<key>ProgramArguments</key>
<array>
- <string>/bin/bash</string>
+ <!-- non-SIP Homebrew bash as the TCC responsible process: SIP /bin/bash cannot durably hold
+ the Full-Disk/removable-volume grant for a launchd job, so writes to /Volumes/Henry EPERM'd
+ ("not writable"). /opt/homebrew/bin/bash (non-SIP) holds the grant — proven: the loaded
+ com.steve.backup-daemon-health agent reads Henry through this exact binary every hour, and
+ the sibling realestate user agent already uses it. Swapped 2026-08-14 (TK-10547) to match. -->
+ <string>/opt/homebrew/bin/bash</string>
<string>/Users/macstudio3/Projects/nas-setup/scripts/pull-dw-dump.sh</string>
</array>
<key>EnvironmentVariables</key>
diff --git a/scripts/daemon-health.sh b/scripts/daemon-health.sh
index c5f458e..c4bffe1 100755
--- a/scripts/daemon-health.sh
+++ b/scripts/daemon-health.sh
@@ -20,6 +20,7 @@ STALE_WARN_H="${STALE_WARN_H:-30}" # a nightly 03:45 job's Henry dump older th
HERE="$(cd "$(dirname "$0")" && pwd)"; DATA="$HERE/../data"; mkdir -p "$DATA"
OUT="$DATA/daemon-health-latest.json"
now=$(date +%s)
+MYUID=$(id -u) # the mirror now runs as a USER LaunchAgent (gui/$MYUID) via /opt/homebrew/bin/bash
worst="PASS" # PASS < WARN < FAIL
rank(){ case "$1" in FAIL) echo 2;; WARN) echo 1;; *) echo 0;; esac; }
rows=()
@@ -28,7 +29,10 @@ rows=()
check(){
local label="$1" dir="$2" prefix="$3"
local print exit_code runs state verdict reason newest age_h fresh
- print=$(launchctl print "system/$label" 2>/dev/null)
+ # The mirror now runs as a USER LaunchAgent (gui/$MYUID) via /opt/homebrew/bin/bash — the non-SIP
+ # launcher that CAN write the /Volumes/Henry external volume (the old root daemon's signed launcher
+ # was TCC-denied on the removable volume, failing every night). So look it up in gui/, not system/.
+ print=$(launchctl print "gui/$MYUID/$label" 2>/dev/null)
exit_code=$(printf '%s\n' "$print" | awk -F'= ' '/last exit code/{print $2; exit}')
runs=$(printf '%s\n' "$print" | awk -F'= ' '/^\truns/{print $2; exit}')
state=$(printf '%s\n' "$print" | awk -F'= ' '/^\tstate/{print $2; exit}')
@@ -40,14 +44,15 @@ check(){
[ "$age_h" -le "$STALE_WARN_H" ] && fresh="fresh" || fresh="stale"
else age_h=-1; fresh="missing"; fi
- # verdict: daemon exit!=0 = FAIL (the nightly write is failing); dump missing = FAIL;
- # dump present but stale (exit somehow 0 yet old) = WARN; else PASS.
- if [ -z "$print" ]; then verdict="FAIL"; reason="launchd state unreadable"
- elif [ "${exit_code:-1}" != "0" ] && [ "$fresh" = "fresh" ]; then verdict="FAIL"; reason="daemon last_exit=${exit_code:-?} but Henry dump fresh (${age_h}h) — mechanism broken, data safe (launcher swap pending)"
- elif [ "${exit_code:-1}" != "0" ]; then verdict="FAIL"; reason="daemon last_exit=${exit_code:-?} — mechanism broken AND Henry ${fresh} (${age_h}h)"
- elif [ "$fresh" = "missing" ]; then verdict="FAIL"; reason="no Henry dump for $prefix"
- elif [ "$fresh" = "stale" ]; then verdict="WARN"; reason="Henry dump ${age_h}h old (> ${STALE_WARN_H}h)"
- else verdict="PASS"; reason="daemon exit 0, Henry dump ${age_h}h old"; fi
+ # Henry-freshness is AUTHORITATIVE — it's the actual off-machine backup we care about; the agent's
+ # exit code is secondary context. Missing/stale Henry = FAIL (the silent-death signal we exist to
+ # catch). A FRESH Henry dump means data is safe even if the agent hiccupped: agent-unloaded or
+ # exit!=0 with a fresh dump = WARN (worth a look, not a data emergency). Fresh + exit 0 = PASS.
+ if [ "$fresh" = "missing" ]; then verdict="FAIL"; reason="no Henry dump for $prefix — mirror never landed"
+ elif [ "$fresh" = "stale" ]; then verdict="FAIL"; reason="Henry dump ${age_h}h old (> ${STALE_WARN_H}h) — nightly mirror not landing"
+ elif [ -z "$print" ]; then verdict="WARN"; reason="agent not loaded but Henry dump fresh (${age_h}h) — data safe, scheduler gone"
+ elif [ "${exit_code:-1}" != "0" ]; then verdict="WARN"; reason="agent last_exit=${exit_code:-?} but Henry dump fresh (${age_h}h) — data safe, check writer"
+ else verdict="PASS"; reason="agent exit 0, Henry dump ${age_h}h old"; fi
[ "$(rank "$verdict")" -gt "$(rank "$worst")" ] && worst="$verdict"
echo " $label -> $verdict ($reason)"
@@ -58,8 +63,8 @@ check(){
}
echo "== on-prem backup daemon health ($(date -Iseconds)) =="
-check com.steve.nas-dwdump-mirror-root /Volumes/Henry/dw-backups/dw_unified dw_unified
-check com.steve.nas-realestate-dump-mirror-root /Volumes/Henry/dw-backups/realestate realestate
+check com.steve.nas-dwdump-mirror /Volumes/Henry/dw-backups/dw_unified dw_unified
+check com.steve.nas-realestate-dump-mirror /Volumes/Henry/dw-backups/realestate realestate
echo "== overall: $worst =="
# JSON heartbeat (PASS/WARN/FAIL vocab so fleet-health-rollup + meta-watchdog read it right)
← 3ce75d8 daemon-health: split the last_exit!=0 reason so the morning
·
back to Nas Setup
·
daemon-health: fix false-PASS regression — restore mechanism 272bd47 →