drunk-animals: daemon 5-min hard timeout on hung ticks

6fb2136be816148888a64c1c8029c86a01ad3717 · 2026-05-13 07:32:28 -0700 · SteveStudio2

Last night a Replicate call hung for 16 min mid-tick, blocking 3
subsequent 4-min cycles. Daemon now SIGKILLs any tick that exceeds
5 min so cadence recovers within one cycle even when Replicate is
sick.

clearTimeout on exit/error so the killer doesn't fire on healthy
ticks.

Files touched

M scripts/drunk_animals_daemon.js

Diff

commit 6fb2136be816148888a64c1c8029c86a01ad3717
Author: SteveStudio2 <stevestudio2@SteveStacStudio.lan>
Date:   Wed May 13 07:32:28 2026 -0700

    drunk-animals: daemon 5-min hard timeout on hung ticks
    
    Last night a Replicate call hung for 16 min mid-tick, blocking 3
    subsequent 4-min cycles. Daemon now SIGKILLs any tick that exceeds
    5 min so cadence recovers within one cycle even when Replicate is
    sick.
    
    clearTimeout on exit/error so the killer doesn't fire on healthy
    ticks.
---
 scripts/drunk_animals_daemon.js | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/scripts/drunk_animals_daemon.js b/scripts/drunk_animals_daemon.js
index d1a47d2..c31ad85 100644
--- a/scripts/drunk_animals_daemon.js
+++ b/scripts/drunk_animals_daemon.js
@@ -69,11 +69,22 @@ function fire() {
   preflight();
   const t0 = Date.now();
   const child = spawn('node', [TICK_SCRIPT], { stdio: 'inherit', cwd: ROOT });
+  // Hard timeout — last night a Replicate call hung for 16 min, blocking
+  // 3+ subsequent ticks. Cap at 5 min so the next 4-min tick can fire on
+  // time even if Replicate is sick.
+  const killTimer = setTimeout(() => {
+    if (!child.killed) {
+      console.error(`[${new Date().toISOString()}] tick exceeded 5min — sending SIGKILL`);
+      try { child.kill('SIGKILL'); } catch {}
+    }
+  }, 5 * 60_000);
   child.on('exit', code => {
+    clearTimeout(killTimer);
     running = false;
     console.log(`[${new Date().toISOString()}] tick done in ${((Date.now()-t0)/1000).toFixed(1)}s (exit ${code})`);
   });
   child.on('error', err => {
+    clearTimeout(killTimer);
     running = false;
     console.error(`[${new Date().toISOString()}] tick spawn error:`, err.message);
   });

← 5312be8 drunk-animals: daemon preflight self-heals id_seq desync · back to Wallco Ai · stoned-animals: new collection — Grateful Dead × Warhol, red 5f6728c →