← back to Ticket System
Prepare Exo watchdog empty-notes alert fix with isolated regression proof
fef7ba46dd5c9071b05ee79b6ca86fed8aa57679 · 2026-09-08 18:37:17 -0700 · Steve Abrams
Files touched
A verification/tk11129-alert-empty-notes-20260909/README.mdA verification/tk11129-alert-empty-notes-20260909/approval-draft.mdA verification/tk11129-alert-empty-notes-20260909/verify.pyA verification/tk11129-alert-empty-notes-20260909/watchdog.candidate.shA verification/tk11129-alert-empty-notes-20260909/watchdog.original.sh
Diff
commit fef7ba46dd5c9071b05ee79b6ca86fed8aa57679
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Tue Sep 8 18:37:17 2026 -0700
Prepare Exo watchdog empty-notes alert fix with isolated regression proof
---
.../tk11129-alert-empty-notes-20260909/README.md | 15 ++
.../approval-draft.md | 51 ++++++
.../tk11129-alert-empty-notes-20260909/verify.py | 121 +++++++++++++
.../watchdog.candidate.sh | 187 +++++++++++++++++++++
.../watchdog.original.sh | 187 +++++++++++++++++++++
5 files changed, 561 insertions(+)
diff --git a/verification/tk11129-alert-empty-notes-20260909/README.md b/verification/tk11129-alert-empty-notes-20260909/README.md
new file mode 100644
index 00000000..297c9338
--- /dev/null
+++ b/verification/tk11129-alert-empty-notes-20260909/README.md
@@ -0,0 +1,15 @@
+# TK-11129 isolated alert fix
+
+The watchdog alert branch crashes under macOS Bash 3.2 when `notes` is empty. The retained candidate changes one expansion to accept an empty array. It has not been installed.
+
+Run the exact alert-block regression from this directory, supplying a fresh evidence filename:
+
+```sh
+python3 verify.py --evidence /private/tmp/tk11129-proof-UNIQUE.json
+```
+
+The runner retains a fresh `mktemp -d` directory, reproduces the baseline failure, verifies candidate append and state behavior in eight cases, and checks both complete scripts for syntax without executing them. Notifications are stubbed. It refuses changed source structure and existing evidence filenames. Current-source and retained-candidate hashes are recorded in `e2e-proof-mktemp.json`. The first proof is retained as historical evidence; the mktemp proof supersedes its scratch method.
+
+Full watchdog execution can heal or restart services and is outside this preparation. Live installation and operational recovery remain blocked by the exact approval in `approval-draft.md`. Current healthy scheduler runs do not prove recovery of the alert branch.
+
+Parent independently reran eight cases and verified the exact candidate bytes and unchanged live source. Cycle evidence is retained at `../../data/codex-yoloforever/cycle-20260909T0123Z.sqpb5o/`.
diff --git a/verification/tk11129-alert-empty-notes-20260909/approval-draft.md b/verification/tk11129-alert-empty-notes-20260909/approval-draft.md
new file mode 100644
index 00000000..a0eb463e
--- /dev/null
+++ b/verification/tk11129-alert-empty-notes-20260909/approval-draft.md
@@ -0,0 +1,51 @@
+# TK-11129 — approve one-line Exo watchdog alert fix
+
+Status: DRAFT; installation and runtime action have not occurred.
+Owner: VP Operations. Finalizer: codex-yoloforever. Source ticket: TK-11129-cron-issue-com-steve-exo-cluster-watchdo.
+A2A packet: M-02349; correlation dm-mttf6rvi-53494-fv8w4y.
+
+## Decision
+
+- **APPROVE** — VP Operations recommendation: install the scoped one-line alert expansion change only, following the source-hash check and verification plan below.
+- **REVISE** — return the candidate or verification plan for a specified revision.
+- **BLOCK** — retain the reproduced defect and halt installation.
+
+Default: **HOLD**. Steve decision: **PENDING**. This recommendation does not authorize installation.
+
+## Exact requested approval
+
+Approve changing only the non-PASS alert echo in `/Users/macstudio3/Projects/exo-cluster-watchdog/watchdog.sh`, replacing `${notes[*]}` with `${notes[*]:-}`, using the retained candidate below after a matching-source hash check.
+No process restart, schedule change, node healing, SSH action, forced watchdog invocation, or unrelated file update is part of this approval.
+
+## Why this is needed
+
+macOS `/bin/bash` 3.2 with `set -u` fails on an empty-array `${notes[*]}` expansion. The exact installed alert block fails with empty notes and WARN, while nonempty notes and PASS work. The candidate preserves existing alert content and note text/order, and allows empty notes to append a valid alert. Current natural PASS runs do not repair this latent alert-path failure.
+A peer dropping between the first and second probe can plausibly yield WARN with empty notes; this production trigger is inferred from source, not demonstrated by incident correlation.
+
+## Candidate and evidence
+
+- Installed source SHA-256: `96aba9b28c3bab39eb86f256f571d2cdd3b78f4a92f829f0e8d1c5c21d6d8a60`.
+- Candidate SHA-256: `f661e30b22281685358afc886360db949e147402882e81026853945cb7e5318b`.
+- Candidate: `/Users/macstudio3/Projects/ticket-system/verification/tk11129-alert-empty-notes-20260909/watchdog.candidate.sh`.
+- Original retained for rollback: `/Users/macstudio3/Projects/ticket-system/verification/tk11129-alert-empty-notes-20260909/watchdog.original.sh`.
+- Reviewable patch: `/Users/macstudio3/Projects/ticket-system/verification/tk11129-alert-empty-notes-20260909/watchdog.patch`.
+- Reusable exact-block runner: `/Users/macstudio3/Projects/ticket-system/verification/tk11129-alert-empty-notes-20260909/verify.py`.
+- Evidence: `/Users/macstudio3/Projects/ticket-system/verification/tk11129-alert-empty-notes-20260909/e2e-proof-mktemp.json` (8 cases PASS; baseline empty/WARN failure required and observed; both full files pass syntax-only `/bin/bash -n`).
+- Retained fresh scratch: `/private/tmp/tk11129-alert-regression.4ZGZp4`.
+
+## Install plan after Steve approval
+
+1. Re-read the canonical ticket and confirm no new owner/source edits. Require the installed watchdog SHA-256 to match the exact source hash above; stop and rebase the candidate if it differs. Inspect the live repository diff and preserve all unrelated runtime data; the executor must follow standing snapshot/commit rules without mixing unrelated changes into this patch.
+2. Create a fresh retained backup directory and copy the installed watchdog into it. Confirm backup hash matches the source hash above. No cleanup or deletion is needed.
+3. Run `/bin/bash -n` on the retained candidate, review the single-line patch, then copy the candidate over the installed watchdog while preserving existing ownership and permissions. Verify the installed hash equals the candidate hash above and the source diff is exactly one line.
+4. Exercise the installed alert block in a fresh isolated scratch directory with notification stubbed; cover empty/nonempty WARN, multiple ordered notes, append preservation, repeated-state deduplication, and PASS. Never invoke the full watchdog manually because it can restart or heal nodes.
+5. Observe at least two naturally scheduled launchd runs: run counters increase, last exit is zero, heartbeat generation advances within cadence, and direct GET `/state` results agree with the current state. Compare newly appended stderr with the pre-install offset. Preserve evidence and commit only the approved source fix.
+6. Do not label the incident recovered merely because PASS heartbeats succeed. Record the isolated installed alert-branch proof separately from natural scheduler proof, and retain any unresolved incident-correlation gap.
+
+## Rollback after approved installation
+
+If the approved change causes a regression, require the installed hash still equals the candidate hash (stop if another writer changed it), copy the retained pre-install original over the installed watchdog with permissions preserved, and verify the original SHA-256. Observe the next natural launchd run and heartbeat; no restart, deletion, or forced watchdog invocation is needed. Log the regression and rollback on TK-11129 and retain the ticket as blocked.
+
+## Remaining gate
+
+Steve approval of this exact installed-file change, then the installation and natural scheduler/installed-alert-boundary evidence. All present results are isolated preparation plus read-only observation; the live source remains unchanged.
diff --git a/verification/tk11129-alert-empty-notes-20260909/verify.py b/verification/tk11129-alert-empty-notes-20260909/verify.py
new file mode 100644
index 00000000..fd7102b7
--- /dev/null
+++ b/verification/tk11129-alert-empty-notes-20260909/verify.py
@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+"""Retained, isolated regression of the watchdog alert block; never run watchdog."""
+import argparse
+import datetime
+import difflib
+import hashlib
+import json
+import os
+from pathlib import Path
+import shlex
+import subprocess
+
+SOURCE = Path('/Users/macstudio3/Projects/exo-cluster-watchdog/watchdog.sh')
+OLD = '${notes[*]}'
+NEW = '${notes[*]:-}'
+
+def sha(data):
+ return hashlib.sha256(data).hexdigest()
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--evidence', required=True, type=Path)
+ args = parser.parse_args()
+ evidence = args.evidence
+ if evidence.exists():
+ raise SystemExit('Refusing to overwrite existing evidence: ' + str(evidence))
+ original_bytes = SOURCE.read_bytes()
+ original = original_bytes.decode()
+ assert original.count(OLD) == 1, 'Source drift: expected exactly one unsafe notes expansion'
+ candidate = original.replace(OLD, NEW)
+ scratch = Path(subprocess.check_output(
+ ['mktemp', '-d', '/private/tmp/tk11129-alert-regression.XXXXXX'], text=True).strip())
+ original_path = scratch / 'watchdog.original.sh'
+ candidate_path = scratch / 'watchdog.candidate.sh'
+ original_path.write_bytes(original_bytes)
+ candidate_path.write_text(candidate)
+ patch = ''.join(difflib.unified_diff(original.splitlines(True), candidate.splitlines(True),
+ fromfile='a/watchdog.sh', tofile='b/watchdog.sh'))
+ (scratch / 'watchdog.patch').write_text(patch)
+ syntax = []
+ for path in (original_path, candidate_path):
+ result = subprocess.run(['/bin/bash', '-n', str(path)], capture_output=True, text=True)
+ syntax.append({'path': str(path), 'rc': result.returncode,
+ 'stdout': result.stdout, 'stderr': result.stderr})
+ assert result.returncode == 0
+ rows = []
+ cases = [
+ ('empty-warn-changed', [], 'WARN', 'PASS'),
+ ('nonempty-warn-unchanged', ['peer offline'], 'WARN', 'WARN'),
+ ('multi-fail-changed', ['peer one offline', 'quote " and $ literal'], 'FAIL', 'WARN'),
+ ('empty-pass', [], 'PASS', 'WARN'),
+ ]
+ for version, full in [('original', original), ('candidate', candidate)]:
+ start = full.index('# Alert on any non-PASS')
+ end = full.index('# If the heartbeat couldn', start)
+ block = full[start:end]
+ assert 'launchctl' not in block and 'ssh ' not in block and 'rm ' not in block
+ for label, notes, verdict, prev in cases:
+ case = scratch / (version + '-' + label)
+ case.mkdir()
+ sentinel = 'sentinel-existing-alert\n'
+ (case / 'alerts.log').write_text(sentinel)
+ (case / '.prev_verdict').write_text(prev + '\n')
+ setup = '\n'.join([
+ 'set -uo pipefail',
+ 'DATA=' + shlex.quote(str(case)),
+ 'ALERTS="$DATA/alerts.log"',
+ 'NOTIFICATION_TRACE="$DATA/notification.calls"',
+ 'now() { printf "%s\\n" "2026-09-09T00:00:00Z"; }',
+ 'osascript() { printf "%s\\n" "$*" >> "$NOTIFICATION_TRACE"; }',
+ 'head="exo cluster test"',
+ 'verdict=' + shlex.quote(verdict),
+ 'notes=(' + ' '.join(shlex.quote(n) for n in notes) + ')',
+ '',
+ ])
+ script = case / 'alert-block.sh'
+ script.write_text(setup + block)
+ result = subprocess.run(['/bin/bash', str(script)], capture_output=True, text=True)
+ alerts = (case / 'alerts.log').read_text()
+ notifications = (case / 'notification.calls').read_text() if (case / 'notification.calls').exists() else ''
+ failing_original = version == 'original' and not notes and verdict != 'PASS'
+ expected_alerts = sentinel if failing_original or verdict == 'PASS' else (
+ sentinel + '2026-09-09T00:00:00Z ' + verdict + ' :: exo cluster test :: ' + ' '.join(notes) + '\n')
+ checks = {
+ 'exit': result.returncode != 0 if failing_original else result.returncode == 0,
+ 'original_failure_reproduced': 'notes[*]: unbound variable' in result.stderr if failing_original else not result.stderr,
+ 'alert_exact_append_and_existing_content_preserved': alerts == expected_alerts,
+ 'previous_verdict_persisted': (case / '.prev_verdict').read_text() == verdict + '\n',
+ 'notification_deduplication': bool(notifications) == (not failing_original and verdict != 'PASS' and verdict != prev),
+ 'completion_output': result.stdout == ('' if failing_original else verdict + ' :: exo cluster test\n'),
+ }
+ rows.append({'version': version, 'case': label, 'script': str(script),
+ 'rc': result.returncode, 'stdout': result.stdout, 'stderr': result.stderr,
+ 'alerts': alerts, 'notifications': notifications, 'checks': checks,
+ 'verdict': 'PASS' if all(checks.values()) else 'FAIL'})
+ result = {
+ 'intent': 'Prove one-line notes expansion fix at exact isolated Bash alert append boundary',
+ 'risk_tier': 'R1 isolated candidate; live installation remains R4 gated',
+ 'timestamp': datetime.datetime.now(datetime.timezone.utc).isoformat(),
+ 'source': str(SOURCE), 'source_sha256_before': sha(original_bytes),
+ 'source_sha256_after': sha(SOURCE.read_bytes()), 'candidate_sha256': sha(candidate.encode()),
+ 'original': str(original_path), 'candidate': str(candidate_path), 'patch': str(scratch / 'watchdog.patch'),
+ 'retained_scratch': str(scratch),
+ 'bash_version': subprocess.run(['/bin/bash', '--version'], capture_output=True, text=True).stdout.splitlines()[0],
+ 'syntax': syntax, 'cases': rows,
+ 'boundary': 'Exact original/candidate alert block, real filesystem append/prev-state; notification invocation stubbed',
+ 'cleanup': 'All scratch retained; no deletion or cleanup commands performed',
+ 'critical_live_journey': {'verdict': 'SKIP', 'reason': 'No live installation authorized; no full watchdog run or healing attempted'},
+ 'source_unchanged': original_bytes == SOURCE.read_bytes(),
+ 'isolated_verdict': 'PASS' if all(r['verdict'] == 'PASS' for r in rows) and original_bytes == SOURCE.read_bytes() else 'FAIL',
+ 'overall_status': 'partial',
+ }
+ evidence.write_text(json.dumps(result, indent=2) + '\n')
+ print(json.dumps({'evidence': str(evidence), 'scratch': str(scratch),
+ 'isolated_verdict': result['isolated_verdict'], 'cases': len(rows),
+ 'source_unchanged': result['source_unchanged']}))
+ if result['isolated_verdict'] != 'PASS':
+ raise SystemExit(1)
+
+if __name__ == '__main__':
+ main()
diff --git a/verification/tk11129-alert-empty-notes-20260909/watchdog.candidate.sh b/verification/tk11129-alert-empty-notes-20260909/watchdog.candidate.sh
new file mode 100644
index 00000000..80946b95
--- /dev/null
+++ b/verification/tk11129-alert-empty-notes-20260909/watchdog.candidate.sh
@@ -0,0 +1,187 @@
+#!/bin/bash
+# exo-cluster-watchdog — "NEVER allow exo to just be off."
+# Runs every 2 min (launchd com.steve.exo-cluster-watchdog).
+# For each node: probe exo :52415. If DOWN and reachable, RESTART it.
+# Local node -> kickstart its own exo-keepalive. Peers -> ssh in and heal.
+# Anything unreachable/unrecoverable -> loud alert + FAIL/WARN heartbeat.
+#
+# Heartbeat verdict uses the fleet-health vocabulary (PASS/WARN/FAIL) so
+# fleet-health-rollup reads it correctly (per TK-10546 rule).
+set -uo pipefail
+
+DIR="$HOME/Projects/exo-cluster-watchdog"
+DATA="$DIR/data"
+LATEST="$DATA/latest.json"
+ALERTS="$DATA/alerts.log"
+PORT=52415
+mkdir -p "$DATA"
+
+# Node table: name | host | ssh_target(LOCAL | user@host | NOAUTH)
+# NOAUTH = do NOT ssh this node (key not authorized / avoid touching it); probe+alert only.
+NODES=(
+ "macstudio3|127.0.0.1|LOCAL"
+ "mac1|192.168.1.133|steve@192.168.1.133"
+ "mac2|192.168.1.54|NOAUTH"
+)
+EXPECTED=${#NODES[@]}
+
+now() { date -u +%Y-%m-%dT%H:%M:%SZ; }
+exo_up() { curl -s --max-time 4 "http://$1:$PORT/state" >/dev/null 2>&1; }
+pings() { ping -c1 -t2 "$1" >/dev/null 2>&1; }
+
+# Fabric detection (LOCAL queries — reliable even if this shell's net egress is sandboxed).
+# TB only if a 10.0.0.x address lives on the Thunderbolt Bridge (bridge0) or a Thunderbolt
+# hardware en* port — NOT a VPN utun / Colima / Docker bridge that also uses 10.0.0.x.
+detect_fabric() {
+ local ifc
+ ifc=$(ifconfig 2>/dev/null | awk '/^[a-z0-9]+:/{i=$1; sub(":","",i)} /inet 10\.0\.0\./{print i; exit}')
+ [ -z "$ifc" ] && { echo LAN; return; }
+ case "$ifc" in
+ bridge0) echo TB ;; # macOS Thunderbolt Bridge
+ en*) networksetup -listallhardwareports 2>/dev/null \
+ | grep -B1 "Device: $ifc\$" | grep -qi thunderbolt && echo TB || echo LAN ;;
+ *) echo LAN ;; # utun/vmenet/bridge100/etc = not TB
+ esac
+}
+
+# Remote self-contained heal — HONEST exit codes (only claims success if it really acted):
+# 0 = exo already up locally on the peer 7 = kicked its exo-keepalive launchd label
+# 8 = started exo directly (keepalive absent, but ~/exo+uv exist for THIS ssh user)
+# 9 = CANNOT heal: no keepalive AND this ssh user has no ~/exo (exo likely runs under a
+# different account) -> needs harden-peer.sh under the exo-owning user; do NOT fake it.
+REMOTE_HEAL='curl -s --max-time 3 http://127.0.0.1:52415/state >/dev/null 2>&1 && exit 0; \
+ launchctl kickstart -k gui/$(id -u)/com.steve.exo-keepalive 2>/dev/null && exit 7; \
+ if [ -x "$HOME/.local/bin/uv" ] && [ -d "$HOME/exo" ]; then \
+ cd "$HOME/exo" && nohup "$HOME/.local/bin/uv" run --extra mlx exo >/tmp/exo-watchdog.out 2>&1 & exit 8; \
+ fi; exit 9'
+
+serving=0; down=(); healed=(); heal_candidates=(); unreachable=(); notes=()
+
+for row in "${NODES[@]}"; do
+ IFS='|' read -r name host tgt <<<"$row"
+ if exo_up "$host"; then
+ serving=$((serving+1)); continue
+ fi
+ # exo is DOWN on this node -> try to heal
+ if [ "$tgt" = "LOCAL" ]; then
+ launchctl kickstart -k "gui/$(id -u)/com.steve.exo-keepalive" 2>/dev/null
+ heal_candidates+=("$name|$host|local-kick"); notes+=("$name exo was down -> local exo-keepalive kicked (pending re-probe)")
+ elif [ "$tgt" = "NOAUTH" ]; then
+ down+=("$name"); notes+=("$name exo down — ssh heal DISABLED (key not authorized yet); probe+alert only, NOT touching this box")
+ elif pings "$host"; then
+ # host up; try ssh heal (BatchMode so it fails fast if key not authorized)
+ rc=$(ssh -o BatchMode=yes -o ConnectTimeout=6 -o StrictHostKeyChecking=accept-new "$tgt" "$REMOTE_HEAL" >/dev/null 2>&1; echo $?)
+ if [ "$rc" = "7" ] || [ "$rc" = "8" ] || [ "$rc" = "0" ]; then
+ heal_candidates+=("$name|$host|ssh-rc$rc"); notes+=("$name exo down -> ssh heal rc=$rc (pending re-probe)")
+ elif [ "$rc" = "9" ]; then
+ down+=("$name"); notes+=("$name exo down + reachable, but NO keepalive and this ssh-user has no ~/exo (exo runs under another account) — run harden-peer.sh as the exo-owning user; NOT faking a heal")
+ else
+ down+=("$name"); notes+=("$name UP on LAN but exo down AND ssh heal failed (rc=$rc, no key/authz) — CANNOT self-heal, needs Steve")
+ fi
+ else
+ unreachable+=("$name"); down+=("$name"); notes+=("$name UNREACHABLE (machine off/asleep) — cannot power on remotely")
+ fi
+done
+
+# Re-probe after heal attempts (give exo a moment to bind) to recount serving
+sleep 6
+serving=0
+for row in "${NODES[@]}"; do IFS='|' read -r name host tgt <<<"$row"; exo_up "$host" && serving=$((serving+1)); done
+
+# Resolve heal candidates by a FRESH probe: 'healed' means VERIFIED up, not merely attempted.
+# A candidate that still isn't serving becomes 'down' — so a WARN can never ship with a
+# populated 'healed' for a node that's actually still down (Cody gate, cycle 1).
+for cand in "${heal_candidates[@]:-}"; do
+ [ -z "$cand" ] && continue
+ IFS='|' read -r cname chost cvia <<<"$cand"
+ if exo_up "$chost"; then
+ healed+=("$cname($cvia)")
+ else
+ down+=("$cname"); notes+=("$cname heal attempted ($cvia) but still DOWN on re-probe — NOT counting as healed")
+ fi
+done
+
+# Verdict (fleet vocab)
+self_up=$(exo_up 127.0.0.1 && echo 1 || echo 0)
+if [ "$serving" -ge "$EXPECTED" ]; then
+ verdict="PASS"
+elif [ "$self_up" = "0" ]; then
+ verdict="FAIL" # this node's exo is down and didn't recover
+else
+ verdict="WARN" # serving < expected but self is up (peer(s) degraded)
+fi
+
+fabric=$(detect_fabric)
+downstr=$(IFS=,; echo "${down[*]:-}")
+healstr=$(IFS=,; echo "${healed[*]:-}")
+head="exo cluster $serving/$EXPECTED serving | fabric: $fabric"
+[ -n "$downstr" ] && head="$head | down: $downstr"
+[ -n "$healstr" ] && head="$head | healed: $healstr"
+
+# last_incident: keep flap history visible even when currently PASS (Cody Hole 3, cycle 2).
+# non-PASS now -> stamp now; else carry the last recorded incident ts from alerts.log.
+if [ "$verdict" != "PASS" ]; then
+ last_incident="$(now)"
+else
+ last_incident=$(tail -1 "$ALERTS" 2>/dev/null | awk '{print $1}')
+fi
+
+# notes -> temp file (one per line) so python encodes them as a proper JSON array (no
+# hand-rolled quote escaping of arbitrary note text).
+NOTES_TMP="$DATA/.notes.$$"; : > "$NOTES_TMP"
+for nz in "${notes[@]:-}"; do [ -n "$nz" ] && printf '%s\n' "$nz" >> "$NOTES_TMP"; done
+
+# Write heartbeat (python3 = correct JSON escaping). verdict/status stay PASS/WARN/FAIL
+# (fleet-health-rollup vocabulary); adds notes[] + last_incident for flap observability.
+VERDICT="$verdict" SERVING="$serving" EXPECTED="$EXPECTED" FABRIC="$fabric" \
+SELF_UP="$self_up" DOWNSTR="$downstr" HEALSTR="$healstr" HEAD="$verdict: $head" \
+GEN="$(now)" LAST_INCIDENT="${last_incident:-}" NOTES_FILE="$NOTES_TMP" \
+python3 - "$LATEST" <<'PY'
+import json, os, sys, tempfile
+try:
+ with open(os.environ['NOTES_FILE']) as f:
+ notes = [l.rstrip('\n') for l in f if l.strip()]
+except Exception:
+ notes = []
+d = {
+ "generated_at": os.environ['GEN'],
+ "verdict": os.environ['VERDICT'], "status": os.environ['VERDICT'],
+ "serving": int(os.environ['SERVING']), "expected": int(os.environ['EXPECTED']),
+ "fabric": os.environ['FABRIC'], "self_up": int(os.environ['SELF_UP']),
+ "down": os.environ['DOWNSTR'], "healed": os.environ['HEALSTR'],
+ "notes": notes, "last_incident": os.environ.get('LAST_INCIDENT', ''),
+ "headline": os.environ['HEAD'],
+}
+# ATOMIC write (Cody cycle 2): temp in same dir -> os.replace, so consumers
+# (fleet-health-rollup, dw-canary-meta-watchdog) never read a half-written file.
+target = sys.argv[1]
+tmp = None
+try:
+ fd, tmp = tempfile.mkstemp(dir=os.path.dirname(target) or '.', prefix='.hb.')
+ with os.fdopen(fd, 'w') as f:
+ f.write(json.dumps(d))
+ os.replace(tmp, target)
+except Exception as e:
+ sys.stderr.write("heartbeat write failed: %s\n" % e)
+ if tmp:
+ try: os.unlink(tmp)
+ except Exception: pass
+ sys.exit(1) # surface the failure instead of silently shipping garbage
+PY
+hb_rc=$?
+rm -f "$NOTES_TMP"
+
+# Alert on any non-PASS (append + macOS notification). De-nag: only if state changed.
+PREVF="$DATA/.prev_verdict"
+prev=$(cat "$PREVF" 2>/dev/null || echo "")
+echo "$verdict" > "$PREVF"
+if [ "$verdict" != "PASS" ]; then
+ echo "$(now) $verdict :: $head :: ${notes[*]:-}" >> "$ALERTS"
+ if [ "$verdict" != "$prev" ]; then
+ osascript -e "display notification \"$head\" with title \"exo watchdog: $verdict\"" >/dev/null 2>&1 || true
+ fi
+fi
+echo "$verdict :: $head"
+# If the heartbeat couldn't be written, exit non-zero so launchd-job-canary flags it
+# (a silent write failure would otherwise look healthy while the heartbeat goes stale).
+[ "${hb_rc:-0}" = "0" ] || { echo "WARN: heartbeat write failed (rc=$hb_rc)"; exit 1; }
diff --git a/verification/tk11129-alert-empty-notes-20260909/watchdog.original.sh b/verification/tk11129-alert-empty-notes-20260909/watchdog.original.sh
new file mode 100644
index 00000000..de2b5cf7
--- /dev/null
+++ b/verification/tk11129-alert-empty-notes-20260909/watchdog.original.sh
@@ -0,0 +1,187 @@
+#!/bin/bash
+# exo-cluster-watchdog — "NEVER allow exo to just be off."
+# Runs every 2 min (launchd com.steve.exo-cluster-watchdog).
+# For each node: probe exo :52415. If DOWN and reachable, RESTART it.
+# Local node -> kickstart its own exo-keepalive. Peers -> ssh in and heal.
+# Anything unreachable/unrecoverable -> loud alert + FAIL/WARN heartbeat.
+#
+# Heartbeat verdict uses the fleet-health vocabulary (PASS/WARN/FAIL) so
+# fleet-health-rollup reads it correctly (per TK-10546 rule).
+set -uo pipefail
+
+DIR="$HOME/Projects/exo-cluster-watchdog"
+DATA="$DIR/data"
+LATEST="$DATA/latest.json"
+ALERTS="$DATA/alerts.log"
+PORT=52415
+mkdir -p "$DATA"
+
+# Node table: name | host | ssh_target(LOCAL | user@host | NOAUTH)
+# NOAUTH = do NOT ssh this node (key not authorized / avoid touching it); probe+alert only.
+NODES=(
+ "macstudio3|127.0.0.1|LOCAL"
+ "mac1|192.168.1.133|steve@192.168.1.133"
+ "mac2|192.168.1.54|NOAUTH"
+)
+EXPECTED=${#NODES[@]}
+
+now() { date -u +%Y-%m-%dT%H:%M:%SZ; }
+exo_up() { curl -s --max-time 4 "http://$1:$PORT/state" >/dev/null 2>&1; }
+pings() { ping -c1 -t2 "$1" >/dev/null 2>&1; }
+
+# Fabric detection (LOCAL queries — reliable even if this shell's net egress is sandboxed).
+# TB only if a 10.0.0.x address lives on the Thunderbolt Bridge (bridge0) or a Thunderbolt
+# hardware en* port — NOT a VPN utun / Colima / Docker bridge that also uses 10.0.0.x.
+detect_fabric() {
+ local ifc
+ ifc=$(ifconfig 2>/dev/null | awk '/^[a-z0-9]+:/{i=$1; sub(":","",i)} /inet 10\.0\.0\./{print i; exit}')
+ [ -z "$ifc" ] && { echo LAN; return; }
+ case "$ifc" in
+ bridge0) echo TB ;; # macOS Thunderbolt Bridge
+ en*) networksetup -listallhardwareports 2>/dev/null \
+ | grep -B1 "Device: $ifc\$" | grep -qi thunderbolt && echo TB || echo LAN ;;
+ *) echo LAN ;; # utun/vmenet/bridge100/etc = not TB
+ esac
+}
+
+# Remote self-contained heal — HONEST exit codes (only claims success if it really acted):
+# 0 = exo already up locally on the peer 7 = kicked its exo-keepalive launchd label
+# 8 = started exo directly (keepalive absent, but ~/exo+uv exist for THIS ssh user)
+# 9 = CANNOT heal: no keepalive AND this ssh user has no ~/exo (exo likely runs under a
+# different account) -> needs harden-peer.sh under the exo-owning user; do NOT fake it.
+REMOTE_HEAL='curl -s --max-time 3 http://127.0.0.1:52415/state >/dev/null 2>&1 && exit 0; \
+ launchctl kickstart -k gui/$(id -u)/com.steve.exo-keepalive 2>/dev/null && exit 7; \
+ if [ -x "$HOME/.local/bin/uv" ] && [ -d "$HOME/exo" ]; then \
+ cd "$HOME/exo" && nohup "$HOME/.local/bin/uv" run --extra mlx exo >/tmp/exo-watchdog.out 2>&1 & exit 8; \
+ fi; exit 9'
+
+serving=0; down=(); healed=(); heal_candidates=(); unreachable=(); notes=()
+
+for row in "${NODES[@]}"; do
+ IFS='|' read -r name host tgt <<<"$row"
+ if exo_up "$host"; then
+ serving=$((serving+1)); continue
+ fi
+ # exo is DOWN on this node -> try to heal
+ if [ "$tgt" = "LOCAL" ]; then
+ launchctl kickstart -k "gui/$(id -u)/com.steve.exo-keepalive" 2>/dev/null
+ heal_candidates+=("$name|$host|local-kick"); notes+=("$name exo was down -> local exo-keepalive kicked (pending re-probe)")
+ elif [ "$tgt" = "NOAUTH" ]; then
+ down+=("$name"); notes+=("$name exo down — ssh heal DISABLED (key not authorized yet); probe+alert only, NOT touching this box")
+ elif pings "$host"; then
+ # host up; try ssh heal (BatchMode so it fails fast if key not authorized)
+ rc=$(ssh -o BatchMode=yes -o ConnectTimeout=6 -o StrictHostKeyChecking=accept-new "$tgt" "$REMOTE_HEAL" >/dev/null 2>&1; echo $?)
+ if [ "$rc" = "7" ] || [ "$rc" = "8" ] || [ "$rc" = "0" ]; then
+ heal_candidates+=("$name|$host|ssh-rc$rc"); notes+=("$name exo down -> ssh heal rc=$rc (pending re-probe)")
+ elif [ "$rc" = "9" ]; then
+ down+=("$name"); notes+=("$name exo down + reachable, but NO keepalive and this ssh-user has no ~/exo (exo runs under another account) — run harden-peer.sh as the exo-owning user; NOT faking a heal")
+ else
+ down+=("$name"); notes+=("$name UP on LAN but exo down AND ssh heal failed (rc=$rc, no key/authz) — CANNOT self-heal, needs Steve")
+ fi
+ else
+ unreachable+=("$name"); down+=("$name"); notes+=("$name UNREACHABLE (machine off/asleep) — cannot power on remotely")
+ fi
+done
+
+# Re-probe after heal attempts (give exo a moment to bind) to recount serving
+sleep 6
+serving=0
+for row in "${NODES[@]}"; do IFS='|' read -r name host tgt <<<"$row"; exo_up "$host" && serving=$((serving+1)); done
+
+# Resolve heal candidates by a FRESH probe: 'healed' means VERIFIED up, not merely attempted.
+# A candidate that still isn't serving becomes 'down' — so a WARN can never ship with a
+# populated 'healed' for a node that's actually still down (Cody gate, cycle 1).
+for cand in "${heal_candidates[@]:-}"; do
+ [ -z "$cand" ] && continue
+ IFS='|' read -r cname chost cvia <<<"$cand"
+ if exo_up "$chost"; then
+ healed+=("$cname($cvia)")
+ else
+ down+=("$cname"); notes+=("$cname heal attempted ($cvia) but still DOWN on re-probe — NOT counting as healed")
+ fi
+done
+
+# Verdict (fleet vocab)
+self_up=$(exo_up 127.0.0.1 && echo 1 || echo 0)
+if [ "$serving" -ge "$EXPECTED" ]; then
+ verdict="PASS"
+elif [ "$self_up" = "0" ]; then
+ verdict="FAIL" # this node's exo is down and didn't recover
+else
+ verdict="WARN" # serving < expected but self is up (peer(s) degraded)
+fi
+
+fabric=$(detect_fabric)
+downstr=$(IFS=,; echo "${down[*]:-}")
+healstr=$(IFS=,; echo "${healed[*]:-}")
+head="exo cluster $serving/$EXPECTED serving | fabric: $fabric"
+[ -n "$downstr" ] && head="$head | down: $downstr"
+[ -n "$healstr" ] && head="$head | healed: $healstr"
+
+# last_incident: keep flap history visible even when currently PASS (Cody Hole 3, cycle 2).
+# non-PASS now -> stamp now; else carry the last recorded incident ts from alerts.log.
+if [ "$verdict" != "PASS" ]; then
+ last_incident="$(now)"
+else
+ last_incident=$(tail -1 "$ALERTS" 2>/dev/null | awk '{print $1}')
+fi
+
+# notes -> temp file (one per line) so python encodes them as a proper JSON array (no
+# hand-rolled quote escaping of arbitrary note text).
+NOTES_TMP="$DATA/.notes.$$"; : > "$NOTES_TMP"
+for nz in "${notes[@]:-}"; do [ -n "$nz" ] && printf '%s\n' "$nz" >> "$NOTES_TMP"; done
+
+# Write heartbeat (python3 = correct JSON escaping). verdict/status stay PASS/WARN/FAIL
+# (fleet-health-rollup vocabulary); adds notes[] + last_incident for flap observability.
+VERDICT="$verdict" SERVING="$serving" EXPECTED="$EXPECTED" FABRIC="$fabric" \
+SELF_UP="$self_up" DOWNSTR="$downstr" HEALSTR="$healstr" HEAD="$verdict: $head" \
+GEN="$(now)" LAST_INCIDENT="${last_incident:-}" NOTES_FILE="$NOTES_TMP" \
+python3 - "$LATEST" <<'PY'
+import json, os, sys, tempfile
+try:
+ with open(os.environ['NOTES_FILE']) as f:
+ notes = [l.rstrip('\n') for l in f if l.strip()]
+except Exception:
+ notes = []
+d = {
+ "generated_at": os.environ['GEN'],
+ "verdict": os.environ['VERDICT'], "status": os.environ['VERDICT'],
+ "serving": int(os.environ['SERVING']), "expected": int(os.environ['EXPECTED']),
+ "fabric": os.environ['FABRIC'], "self_up": int(os.environ['SELF_UP']),
+ "down": os.environ['DOWNSTR'], "healed": os.environ['HEALSTR'],
+ "notes": notes, "last_incident": os.environ.get('LAST_INCIDENT', ''),
+ "headline": os.environ['HEAD'],
+}
+# ATOMIC write (Cody cycle 2): temp in same dir -> os.replace, so consumers
+# (fleet-health-rollup, dw-canary-meta-watchdog) never read a half-written file.
+target = sys.argv[1]
+tmp = None
+try:
+ fd, tmp = tempfile.mkstemp(dir=os.path.dirname(target) or '.', prefix='.hb.')
+ with os.fdopen(fd, 'w') as f:
+ f.write(json.dumps(d))
+ os.replace(tmp, target)
+except Exception as e:
+ sys.stderr.write("heartbeat write failed: %s\n" % e)
+ if tmp:
+ try: os.unlink(tmp)
+ except Exception: pass
+ sys.exit(1) # surface the failure instead of silently shipping garbage
+PY
+hb_rc=$?
+rm -f "$NOTES_TMP"
+
+# Alert on any non-PASS (append + macOS notification). De-nag: only if state changed.
+PREVF="$DATA/.prev_verdict"
+prev=$(cat "$PREVF" 2>/dev/null || echo "")
+echo "$verdict" > "$PREVF"
+if [ "$verdict" != "PASS" ]; then
+ echo "$(now) $verdict :: $head :: ${notes[*]}" >> "$ALERTS"
+ if [ "$verdict" != "$prev" ]; then
+ osascript -e "display notification \"$head\" with title \"exo watchdog: $verdict\"" >/dev/null 2>&1 || true
+ fi
+fi
+echo "$verdict :: $head"
+# If the heartbeat couldn't be written, exit non-zero so launchd-job-canary flags it
+# (a silent write failure would otherwise look healthy while the heartbeat goes stale).
+[ "${hb_rc:-0}" = "0" ] || { echo "WARN: heartbeat write failed (rc=$hb_rc)"; exit 1; }
← aad067f0 auto-data-snapshot: 2026-09-08T18:35:21 (3 data files) — ver
·
back to Ticket System
·
Verify pinned installed watchdog bytes and reject unsafe sou 0917d391 →