raise exo rotate-guard threshold 2.0->8.0GB so wedged follower converges (TK-10707)

f9a667cab929d41533b9a402bb70085f6f85e44f · 2026-08-26 10:25:22 -0700 · Steve Abrams

Files touched

A exo-eventlog-rotate-guard.sh

Diff

commit f9a667cab929d41533b9a402bb70085f6f85e44f
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Wed Aug 26 10:25:22 2026 -0700

    raise exo rotate-guard threshold 2.0->8.0GB so wedged follower converges (TK-10707)
---
 exo-eventlog-rotate-guard.sh | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/exo-eventlog-rotate-guard.sh b/exo-eventlog-rotate-guard.sh
new file mode 100755
index 0000000..c30e03e
--- /dev/null
+++ b/exo-eventlog-rotate-guard.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+# exo-eventlog-rotate-guard.sh — bound the unbounded exo active event log (TK-10236 ENOSPC class).
+#
+# WHY: exo's DiskEventLog._rotate() compresses events.bin -> events.<stamp>.bin.zst (keeps <=5
+#      archives) ONLY on startup/close — never in-session. So a long-running exo (kept alive for
+#      weeks) grows ~/.exo/event_log/api/events.bin without limit (3.2G over ~5 days, Aug 2026).
+#      A 3.2G network-event log compresses to ~100M .zst, so a restart reclaims ~97%.
+#
+# WHAT: if events.bin > THRESHOLD_GB, kickstart exo-keepalive. exo dies + restarts; on restart it
+#       sees the stale events.bin and rotates+compresses it, then starts a fresh empty log.
+#       NO manual file deletion — exo's own rotation does it safely. Reversible: it's a restart.
+#
+# COST of firing: one brief exo-cluster restart (in-flight inference drops, tunnel blips ~seconds).
+#       The exo-cluster-watchdog already restarts exo routinely when down, so this is well-tolerated.
+#
+# Tune THRESHOLD_GB up to restart less often (default 2.0G ≈ every ~3 days at current growth).
+set -uo pipefail
+LOG="$HOME/.exo/event_log/api/events.bin"
+THRESHOLD_GB="${THRESHOLD_GB:-8.0}"  # raised 2.0->8.0 2026-08-26 TK-10707: let the wedged follower finish converging (idx 873K->6M) without the hourly restart resetting it. REVERT to 2.0 once converged.
+STAMP="$(date -u +%FT%TZ)"
+[ -f "$LOG" ] || { echo "$STAMP no events.bin — nothing to do"; exit 0; }
+sz=$(stat -f '%z' "$LOG" 2>/dev/null || echo 0)
+human=$(awk -v b="$sz" 'BEGIN{printf "%.2fG", b/1073741824}')
+thr=$(awk -v g="$THRESHOLD_GB" 'BEGIN{printf "%d", g*1073741824}')
+if [ "$sz" -gt "$thr" ]; then
+  echo "$STAMP events.bin ${human} > ${THRESHOLD_GB}G — restarting exo to trigger native rotation"
+  launchctl kickstart -k "gui/$(id -u)/com.steve.exo-keepalive"
+  echo "$STAMP kickstarted com.steve.exo-keepalive (exo will rotate events.bin -> .zst on restart)"
+else
+  echo "$STAMP events.bin ${human} <= ${THRESHOLD_GB}G — ok"
+fi

← 9868833 auto-data-snapshot: 2026-08-26T10:15:42 (1 data files) — dat · back to Exo Cluster Watchdog · auto-data-snapshot: 2026-08-26T10:53:15 (1 data files) — dat 4912efc →