← back to Exo Cluster Watchdog
watchdog cycle2: heartbeat carries notes[] + last_incident (flap observability; Cody Hole 3) via python JSON
d1fc0d7d2e48ee406980ae86e273060dfa145f4e · 2026-08-17 22:50:50 -0700 · steve
Files touched
M data/latest.jsonM watchdog.sh
Diff
commit d1fc0d7d2e48ee406980ae86e273060dfa145f4e
Author: steve <steve@designerwallcoverings.com>
Date: Mon Aug 17 22:50:50 2026 -0700
watchdog cycle2: heartbeat carries notes[] + last_incident (flap observability; Cody Hole 3) via python JSON
---
data/latest.json | 2 +-
watchdog.sh | 50 +++++++++++++++++++++++++++++++++++++-------------
2 files changed, 38 insertions(+), 14 deletions(-)
diff --git a/data/latest.json b/data/latest.json
index 99119b8..4b4a456 100644
--- a/data/latest.json
+++ b/data/latest.json
@@ -1 +1 @@
-{"generated_at":"2026-08-18T05:39:17Z","verdict":"PASS","status":"PASS","serving":3,"expected":3,"fabric":"LAN","self_up":1,"down":"","healed":"","headline":"PASS: exo cluster 3/3 serving | fabric: LAN"}
\ No newline at end of file
+{"generated_at": "2026-08-18T05:50:41Z", "verdict": "PASS", "status": "PASS", "serving": 3, "expected": 3, "fabric": "LAN", "self_up": 1, "down": "", "healed": "", "notes": [], "last_incident": "2026-08-17T21:35:31Z", "headline": "PASS: exo cluster 3/3 serving | fabric: LAN"}
\ No newline at end of file
diff --git a/watchdog.sh b/watchdog.sh
index 290902c..eb03d57 100755
--- a/watchdog.sh
+++ b/watchdog.sh
@@ -118,19 +118,43 @@ head="exo cluster $serving/$EXPECTED serving | fabric: $fabric"
[ -n "$downstr" ] && head="$head | down: $downstr"
[ -n "$healstr" ] && head="$head | healed: $healstr"
-# Write heartbeat
-{
- printf '{'
- printf '"generated_at":"%s",' "$(now)"
- printf '"verdict":"%s",' "$verdict"
- printf '"status":"%s",' "$verdict"
- printf '"serving":%d,"expected":%d,' "$serving" "$EXPECTED"
- printf '"fabric":"%s",' "$fabric"
- printf '"self_up":%s,' "$self_up"
- printf '"down":"%s","healed":"%s",' "$downstr" "$healstr"
- printf '"headline":"%s: %s"' "$verdict" "$head"
- printf '}'
-} > "$LATEST"
+# last_incident: keep flap history visible even when currently PASS (Cody Hole 3, cycle 2).
+# non-PASS now -> stamp now; else carry the last recorded incident ts from alerts.log.
+if [ "$verdict" != "PASS" ]; then
+ last_incident="$(now)"
+else
+ last_incident=$(tail -1 "$ALERTS" 2>/dev/null | awk '{print $1}')
+fi
+
+# notes -> temp file (one per line) so python encodes them as a proper JSON array (no
+# hand-rolled quote escaping of arbitrary note text).
+NOTES_TMP="$DATA/.notes.$$"; : > "$NOTES_TMP"
+for nz in "${notes[@]:-}"; do [ -n "$nz" ] && printf '%s\n' "$nz" >> "$NOTES_TMP"; done
+
+# Write heartbeat (python3 = correct JSON escaping). verdict/status stay PASS/WARN/FAIL
+# (fleet-health-rollup vocabulary); adds notes[] + last_incident for flap observability.
+VERDICT="$verdict" SERVING="$serving" EXPECTED="$EXPECTED" FABRIC="$fabric" \
+SELF_UP="$self_up" DOWNSTR="$downstr" HEALSTR="$healstr" HEAD="$verdict: $head" \
+GEN="$(now)" LAST_INCIDENT="${last_incident:-}" NOTES_FILE="$NOTES_TMP" \
+python3 - "$LATEST" <<'PY'
+import json, os, sys
+try:
+ with open(os.environ['NOTES_FILE']) as f:
+ notes = [l.rstrip('\n') for l in f if l.strip()]
+except Exception:
+ notes = []
+d = {
+ "generated_at": os.environ['GEN'],
+ "verdict": os.environ['VERDICT'], "status": os.environ['VERDICT'],
+ "serving": int(os.environ['SERVING']), "expected": int(os.environ['EXPECTED']),
+ "fabric": os.environ['FABRIC'], "self_up": int(os.environ['SELF_UP']),
+ "down": os.environ['DOWNSTR'], "healed": os.environ['HEALSTR'],
+ "notes": notes, "last_incident": os.environ.get('LAST_INCIDENT', ''),
+ "headline": os.environ['HEAD'],
+}
+open(sys.argv[1], 'w').write(json.dumps(d))
+PY
+rm -f "$NOTES_TMP"
# Alert on any non-PASS (append + macOS notification). De-nag: only if state changed.
PREVF="$DATA/.prev_verdict"
← 3bcedd0 chore: lint (syntax ✓), refactor (Cody-gated), v0.2.0 (sessi
·
back to Exo Cluster Watchdog
·
watchdog cycle2 (Cody-gated): atomic heartbeat write (tempfi 95e9e04 →