← back to Exo Cluster Watchdog
TK-11996: memory/boot-grace/thrash guard on exo instance re-POST (1207 re-POSTs of 17GB model drove swap to 53GB -> power-button resets)
07252001e38c4c7daa338c84930ec1b896c9fe4a · 2026-09-25 13:33:11 -0700 · Steve Abrams
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D7LB6rCwpA7ubJTqYzqXEX
Files touched
A instance-load-guard.shA test-instance-load-guard.shM watchdog.sh
Diff
commit 07252001e38c4c7daa338c84930ec1b896c9fe4a
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Fri Sep 25 13:33:11 2026 -0700
TK-11996: memory/boot-grace/thrash guard on exo instance re-POST (1207 re-POSTs of 17GB model drove swap to 53GB -> power-button resets)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D7LB6rCwpA7ubJTqYzqXEX
---
instance-load-guard.sh | 70 +++++++++++++++++++++++++++++++++++++++++++++
test-instance-load-guard.sh | 24 ++++++++++++++++
watchdog.sh | 7 +++++
3 files changed, 101 insertions(+)
diff --git a/instance-load-guard.sh b/instance-load-guard.sh
new file mode 100755
index 0000000..4dd7dd7
--- /dev/null
+++ b/instance-load-guard.sh
@@ -0,0 +1,70 @@
+#!/bin/bash
+# instance-load-guard.sh — decide whether watchdog.sh may re-POST the wanted exo model
+# instance right now. Prints exactly one line: "ALLOW" or "BLOCK: <reason>". Exit 0 always.
+#
+# Why (TK-11996, 2026-09-25): the TK-11903 instance keepalive re-POSTed the 17GB
+# Qwen3-30B instance ~200x/day (1,207 re-POSTs since 09-18) while the ring kept evicting it.
+# On 09-25 each re-POST landed on a box already holding the SAME model in com.dw.mlx-qwen
+# (~22GB) + dw-local-ai (~15GB): swap went 20GB -> 53GB within a minute of the 11:28
+# CreateInstance, the Mac went unresponsive and was power-button reset at 11:35; after that
+# boot the 11:41 + 11:47 re-POSTs drove mem-free to 17% and it was reset again at 11:52.
+#
+# Blocks the re-POST when ANY of:
+# 1. boot grace — uptime < GUARD_MIN_UPTIME_S (default 900s): post-boot RunAtLoad stampede
+# 2. memory — kern.memorystatus_level (system-wide free %) < GUARD_MIN_MEMFREE (40)
+# 3. thrash — >= GUARD_MAX_REPOSTS_PER_HOUR (4) re-POSTs already in the last 60 min
+# 4. NOT MEASURED — memfree or uptime unreadable (never assume it is safe to load 17GB)
+#
+# Test seam: ONLY with --test are GUARD_TEST_UPTIME / GUARD_TEST_MEMFREE / GUARD_TEST_ALERTS /
+# GUARD_TEST_NOW honoured. The launchd path never passes --test.
+set -uo pipefail
+MIN_UPTIME=${GUARD_MIN_UPTIME_S:-900}
+MIN_MEMFREE=${GUARD_MIN_MEMFREE:-40}
+MAX_PER_HOUR=${GUARD_MAX_REPOSTS_PER_HOUR:-4}
+ALERTS_DEFAULT="$HOME/Projects/exo-cluster-watchdog/data/alerts.log"
+
+if [ "${1:-}" = "--test" ]; then
+ uptime_s="${GUARD_TEST_UPTIME:-}"; memfree="${GUARD_TEST_MEMFREE:-}"
+ alerts="${GUARD_TEST_ALERTS:-/dev/null}"; nowe="${GUARD_TEST_NOW:-$(date +%s)}"
+else
+ nowe=$(date +%s)
+ boot=$(sysctl -n kern.boottime 2>/dev/null | sed -E 's/^\{ sec = ([0-9]+),.*/\1/')
+ if [[ "$boot" =~ ^[0-9]+$ ]]; then uptime_s=$((nowe - boot)); else uptime_s=""; fi
+ memfree=$(sysctl -n kern.memorystatus_level 2>/dev/null)
+ alerts="$ALERTS_DEFAULT"
+fi
+
+if ! [[ "$uptime_s" =~ ^[0-9]+$ ]] || ! [[ "$memfree" =~ ^[0-9]+$ ]]; then
+ echo "BLOCK: NOT MEASURED (uptime='${uptime_s}' memfree='${memfree}') — refusing to load a 17GB model blind"; exit 0
+fi
+if [ "$uptime_s" -lt "$MIN_UPTIME" ]; then
+ echo "BLOCK: boot grace (uptime ${uptime_s}s < ${MIN_UPTIME}s)"; exit 0
+fi
+if [ "$memfree" -lt "$MIN_MEMFREE" ]; then
+ echo "BLOCK: memory (system free ${memfree}% < ${MIN_MEMFREE}%)"; exit 0
+fi
+# count re-POSTs in the last hour (alerts.log lines start with an ISO-8601 UTC timestamp)
+recent=$(NOWE="$nowe" python3 - "$alerts" <<'PY' 2>/dev/null
+import os, sys, datetime
+now = int(os.environ["NOWE"]); n = 0
+try:
+ for line in open(sys.argv[1], errors="replace"):
+ if "re-POSTed" not in line: continue
+ ts = line.split(" ", 1)[0]
+ try:
+ t = datetime.datetime.strptime(ts, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=datetime.timezone.utc).timestamp()
+ except ValueError:
+ continue
+ if 0 <= now - t <= 3600: n += 1
+except FileNotFoundError:
+ pass
+print(n)
+PY
+)
+if ! [[ "$recent" =~ ^[0-9]+$ ]]; then
+ echo "BLOCK: NOT MEASURED (re-POST history unreadable)"; exit 0
+fi
+if [ "$recent" -ge "$MAX_PER_HOUR" ]; then
+ echo "BLOCK: thrash breaker (${recent} re-POSTs in last 60m >= ${MAX_PER_HOUR}; instance keeps getting evicted)"; exit 0
+fi
+echo "ALLOW (uptime ${uptime_s}s, free ${memfree}%, ${recent} re-POSTs/60m)"
diff --git a/test-instance-load-guard.sh b/test-instance-load-guard.sh
new file mode 100755
index 0000000..242aa7b
--- /dev/null
+++ b/test-instance-load-guard.sh
@@ -0,0 +1,24 @@
+#!/bin/bash
+# Negative test for instance-load-guard.sh (TK-11996). Proves the guard goes RED (BLOCK) on
+# each injected fault and GREEN (ALLOW) only on a healthy, measured input. Touches no live data.
+set -uo pipefail
+G="$(cd "$(dirname "$0")" && pwd)/instance-load-guard.sh"
+T=$(mktemp -d); trap 'rm -rf "$T"' EXIT
+NOW=1790370000
+iso(){ python3 -c "import datetime,sys;print(datetime.datetime.fromtimestamp(int(sys.argv[1]),datetime.timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ'))" "$1"; }
+: > "$T/empty.log"
+for i in 1 2 3 4 5; do echo "$(iso $((NOW - i*600))) WARN :: 0 instances on ring -> re-POSTed m (http 200)" >> "$T/thrash.log"; done
+echo "$(iso $((NOW - 7200))) WARN :: re-POSTed old (http 200)" > "$T/old.log"
+pass=0; fail=0
+check(){ # name expect(ALLOW|BLOCK) uptime memfree alerts
+ out=$(GUARD_TEST_NOW=$NOW GUARD_TEST_UPTIME="$3" GUARD_TEST_MEMFREE="$4" GUARD_TEST_ALERTS="$5" bash "$G" --test)
+ if [[ "$out" == "$2"* ]]; then pass=$((pass+1)); echo "ok $1 -> $out"; else fail=$((fail+1)); echo "FAIL $1 -> $out (expected $2)"; fi
+}
+check "healthy" ALLOW 7200 60 "$T/empty.log"
+check "old re-POST ignored" ALLOW 7200 60 "$T/old.log"
+check "boot grace" BLOCK 300 60 "$T/empty.log"
+check "memory 17% (11:45)" BLOCK 7200 17 "$T/empty.log"
+check "thrash 5/hr" BLOCK 7200 60 "$T/thrash.log"
+check "memfree unmeasured" BLOCK 7200 "" "$T/empty.log"
+check "uptime unmeasured" BLOCK "" 60 "$T/empty.log"
+echo "pass=$pass fail=$fail"; [ "$fail" = 0 ]
diff --git a/watchdog.sh b/watchdog.sh
index 86d0e8a..6b16c5a 100755
--- a/watchdog.sh
+++ b/watchdog.sh
@@ -141,9 +141,16 @@ print(walk(json.load(open(sys.argv[1]))) or "?")' "$WANTED" 2>/dev/null)
if [ "$TEST_MODE" = 1 ]; then
instance_note="0 instances on ring -> WOULD re-POST $wanted_model (test mode, not sent)"
else
+ # TK-11996: memory / boot-grace / thrash guard — a 17GB load onto a swapping box hung
+ # the Mac into power-button resets (09-25 11:35 + 11:52). Unmeasured => BLOCK.
+ guard=$(bash "$DIR/instance-load-guard.sh" 2>/dev/null || echo "BLOCK: guard failed to run")
+ if [[ "$guard" != ALLOW* ]]; then
+ instance_note="0 instances on ring -> re-POST SUPPRESSED ($guard); $wanted_model not loaded"
+ else
body=$(python3 -c 'import json,sys; print(json.dumps({"instance": json.load(open(sys.argv[1]))}))' "$WANTED")
rc=$(curl -s --max-time 10 -o /dev/null -w '%{http_code}' -X POST "http://127.0.0.1:$PORT/instance" -H 'Content-Type: application/json' -d "$body")
instance_note="0 instances on ring -> re-POSTed $wanted_model (http $rc); pending re-probe, NOT counted as healed"
+ fi
fi
else
instance_note="0 instances on ring; relaunch attempted $((nowe - last))s ago, waiting for load (no re-POST)"
← f31f413 auto-data-snapshot: 2026-09-25T13:25:27 (2 data files) — dat
·
back to Exo Cluster Watchdog
·
TK-11996: guard uses absolute /usr/sbin/sysctl (launchd PATH 450bc37 →