watchdog cycle2 (Cody-gated): atomic heartbeat write (tempfile+os.replace) + non-zero exit on write failure

95e9e044aecf8b89adab77c5d11421a91cf42d66 · 2026-08-17 22:56:06 -0700 · steve

Files touched

M data/latest.json
M watchdog.sh

Diff

commit 95e9e044aecf8b89adab77c5d11421a91cf42d66
Author: steve <steve@designerwallcoverings.com>
Date:   Mon Aug 17 22:56:06 2026 -0700

    watchdog cycle2 (Cody-gated): atomic heartbeat write (tempfile+os.replace) + non-zero exit on write failure
---
 data/latest.json |  2 +-
 watchdog.sh      | 22 ++++++++++++++++++++--
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/data/latest.json b/data/latest.json
index 4b4a456..97346a8 100644
--- a/data/latest.json
+++ b/data/latest.json
@@ -1 +1 @@
-{"generated_at": "2026-08-18T05:50:41Z", "verdict": "PASS", "status": "PASS", "serving": 3, "expected": 3, "fabric": "LAN", "self_up": 1, "down": "", "healed": "", "notes": [], "last_incident": "2026-08-17T21:35:31Z", "headline": "PASS: exo cluster 3/3 serving | fabric: LAN"}
\ No newline at end of file
+{"generated_at": "2026-08-18T05:56:06Z", "verdict": "PASS", "status": "PASS", "serving": 3, "expected": 3, "fabric": "LAN", "self_up": 1, "down": "", "healed": "", "notes": [], "last_incident": "2026-08-17T21:35:31Z", "headline": "PASS: exo cluster 3/3 serving | fabric: LAN"}
\ No newline at end of file
diff --git a/watchdog.sh b/watchdog.sh
index eb03d57..de2b5cf 100755
--- a/watchdog.sh
+++ b/watchdog.sh
@@ -137,7 +137,7 @@ VERDICT="$verdict" SERVING="$serving" EXPECTED="$EXPECTED" FABRIC="$fabric" \
 SELF_UP="$self_up" DOWNSTR="$downstr" HEALSTR="$healstr" HEAD="$verdict: $head" \
 GEN="$(now)" LAST_INCIDENT="${last_incident:-}" NOTES_FILE="$NOTES_TMP" \
 python3 - "$LATEST" <<'PY'
-import json, os, sys
+import json, os, sys, tempfile
 try:
     with open(os.environ['NOTES_FILE']) as f:
         notes = [l.rstrip('\n') for l in f if l.strip()]
@@ -152,8 +152,23 @@ d = {
     "notes": notes, "last_incident": os.environ.get('LAST_INCIDENT', ''),
     "headline": os.environ['HEAD'],
 }
-open(sys.argv[1], 'w').write(json.dumps(d))
+# ATOMIC write (Cody cycle 2): temp in same dir -> os.replace, so consumers
+# (fleet-health-rollup, dw-canary-meta-watchdog) never read a half-written file.
+target = sys.argv[1]
+tmp = None
+try:
+    fd, tmp = tempfile.mkstemp(dir=os.path.dirname(target) or '.', prefix='.hb.')
+    with os.fdopen(fd, 'w') as f:
+        f.write(json.dumps(d))
+    os.replace(tmp, target)
+except Exception as e:
+    sys.stderr.write("heartbeat write failed: %s\n" % e)
+    if tmp:
+        try: os.unlink(tmp)
+        except Exception: pass
+    sys.exit(1)   # surface the failure instead of silently shipping garbage
 PY
+hb_rc=$?
 rm -f "$NOTES_TMP"
 
 # Alert on any non-PASS (append + macOS notification). De-nag: only if state changed.
@@ -167,3 +182,6 @@ if [ "$verdict" != "PASS" ]; then
   fi
 fi
 echo "$verdict :: $head"
+# If the heartbeat couldn't be written, exit non-zero so launchd-job-canary flags it
+# (a silent write failure would otherwise look healthy while the heartbeat goes stale).
+[ "${hb_rc:-0}" = "0" ] || { echo "WARN: heartbeat write failed (rc=$hb_rc)"; exit 1; }

← d1fc0d7 watchdog cycle2: heartbeat carries notes[] + last_incident ( · back to Exo Cluster Watchdog · auto-data-snapshot: 2026-08-17T23:04:06 (1 data files) — dat bc46137 →