← back to Nas Setup
daemon-health: watch the repo-bundle belt, not just the two DB mirrors
e6b27ef6cc8596145b7ed077b4db4ae7daaaf289 · 2026-09-10 13:09:49 -0700 · Steve Abrams
backup-repos-to-henry.sh has always written a PASS/WARN/FAIL verdict to
data/repo-backup-latest.json, but nothing ever read it. A silently-dead bundler
emitted zero alerts - the exact shape of the 12-day pg_dump death this skill
exists to prevent.
That belt now matters more: as of today it is the ONLY off-machine copy of
~/.claude (452 skill definitions) and its 73 nested skill repos.
Adds a third check with three failure modes:
FAIL last run > 36h ago (daily job + one missed run of slack)
WARN bundler self-reported WARN/FAIL, or repos failed
FAIL bundler PASSed but dotclaude.bundle is ABSENT - freshness alone is not
proof the thing you care about is in the set
Verdict joins the existing rollup, so it reaches fleet-health-rollup and
dw-canary-meta-watchdog via the skill's data/latest.json.
Note the macOS trap: parses a trailing-Z UTC stamp as LOCAL time,
producing a future epoch and a NEGATIVE age that silently passes the staleness
test. Fixed with -u and a negative clamp; caught by negative-testing rather
than by reading the code.
Negative-tested: 40h stale -> FAIL, bundler WARN -> WARN, healthy -> PASS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Files touched
M scripts/daemon-health.sh
Diff
commit e6b27ef6cc8596145b7ed077b4db4ae7daaaf289
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Thu Sep 10 13:09:49 2026 -0700
daemon-health: watch the repo-bundle belt, not just the two DB mirrors
backup-repos-to-henry.sh has always written a PASS/WARN/FAIL verdict to
data/repo-backup-latest.json, but nothing ever read it. A silently-dead bundler
emitted zero alerts - the exact shape of the 12-day pg_dump death this skill
exists to prevent.
That belt now matters more: as of today it is the ONLY off-machine copy of
~/.claude (452 skill definitions) and its 73 nested skill repos.
Adds a third check with three failure modes:
FAIL last run > 36h ago (daily job + one missed run of slack)
WARN bundler self-reported WARN/FAIL, or repos failed
FAIL bundler PASSed but dotclaude.bundle is ABSENT - freshness alone is not
proof the thing you care about is in the set
Verdict joins the existing rollup, so it reaches fleet-health-rollup and
dw-canary-meta-watchdog via the skill's data/latest.json.
Note the macOS trap: parses a trailing-Z UTC stamp as LOCAL time,
producing a future epoch and a NEGATIVE age that silently passes the staleness
test. Fixed with -u and a negative clamp; caught by negative-testing rather
than by reading the code.
Negative-tested: 40h stale -> FAIL, bundler WARN -> WARN, healthy -> PASS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
scripts/daemon-health.sh | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)
diff --git a/scripts/daemon-health.sh b/scripts/daemon-health.sh
index f7e347e..6175ef6 100755
--- a/scripts/daemon-health.sh
+++ b/scripts/daemon-health.sh
@@ -114,6 +114,44 @@ check(){
echo "== on-prem backup daemon health ($(date -Iseconds)) =="
check dw_unified /Volumes/Henry/dw-backups/dw_unified dw_unified com.steve.nas-dwdump-mirror "$HERE/../data/launchd.out.log"
check realestate /Volumes/Henry/dw-backups/realestate realestate com.steve.nas-realestate-dump-mirror "$HERE/../data/launchd-realestate.out.log"
+
+# --- repo/config bundle belt (added 2026-09-10, TK-11233 follow-up) ---------
+# backup-repos-to-henry.sh writes a PASS/WARN/FAIL verdict but NOTHING read it, so a
+# silently-dead bundler emitted zero alerts - the exact shape of the 12-day pg_dump death.
+# It is now the ONLY off-machine copy of ~/.claude (452 skill definitions) and its 73
+# nested skill repos, so its freshness matters as much as the DB mirrors.
+REPO_JSON="$HERE/../data/repo-backup-latest.json"
+REPO_STALE_H="${REPO_STALE_H:-36}" # daily job at 04:30; 36h = one missed run + slack
+rv=FAIL; rr="repo-backup-latest.json missing - bundler has never run or its data dir moved"
+if [ -f "$REPO_JSON" ]; then
+ rts=$(jq -r '.ts // empty' "$REPO_JSON" 2>/dev/null)
+ rvd=$(jq -r '.verdict // "UNKNOWN"' "$REPO_JSON" 2>/dev/null)
+ rok=$(jq -r '.repos_ok // 0' "$REPO_JSON" 2>/dev/null)
+ rfail=$(jq -r '.repos_fail // 0' "$REPO_JSON" 2>/dev/null)
+ if [ -n "$rts" ]; then
+ # -u is REQUIRED: the ts is UTC (trailing Z) and macOS `date -j -f` otherwise parses it
+ # as LOCAL time, yielding a future epoch and a NEGATIVE age that silently passes the
+ # staleness test. Clamp negatives to 0 and treat a large negative as clock skew.
+ rts_epoch=$(date -j -u -f "%Y-%m-%dT%H:%M:%SZ" "$rts" +%s 2>/dev/null || echo 0)
+ rage=$(( ( $(date +%s) - rts_epoch ) / 3600 ))
+ [ "$rage" -lt 0 ] && rage=0
+ if [ "$rage" -gt "$REPO_STALE_H" ]; then rv=FAIL; rr="last bundle run ${rage}h ago (> ${REPO_STALE_H}h) - the only off-machine copy of ~/.claude is going stale"
+ elif [ "$rvd" != "PASS" ]; then rv=WARN; rr="bundler reported $rvd (${rfail} repos failed) ${rage}h ago"
+ else
+ # Freshness is necessary but not sufficient: assert ~/.claude is actually IN the set.
+ if [ -f /Volumes/Henry/mac2-archive/repo-backups/dotclaude.bundle ]; then
+ rv=PASS; rr="bundled ${rok} repos ${rage}h ago incl. dotclaude + $(ls /Volumes/Henry/mac2-archive/repo-backups/dotclaude-skill-*.bundle 2>/dev/null | wc -l | tr -d ' ') nested skill repos"
+ else
+ rv=FAIL; rr="bundler PASSed ${rage}h ago but dotclaude.bundle is ABSENT - ~/.claude is not actually being backed up"
+ fi
+ fi
+ fi
+fi
+[ "$(rank "$rv")" -gt "$(rank "$worst")" ] && worst="$rv"
+echo " repo-bundles -> $rv ($rr)"
+rows+=("$(jq -n --arg l "repo-bundles" --arg v "$rv" --arg r "$rr" \
+ '{db:$l,verdict:$v,reason:$r,root_last_exit:null,user_last_exit:null,henry_dump_age_h:null}')")
+
echo "== overall: $worst =="
# JSON heartbeat (PASS/WARN/FAIL vocab so fleet-health-rollup + meta-watchdog read it right)
← 7d6456f backup-repos-to-henry: cover ~/.claude and its 73 nested ski
·
back to Nas Setup
·
TK-11648: identity guard so backup never bundles the WRONG r 9603008 →