5x/contrarian FIX FIRST: make exit code honest — transient source-down exits 0, SUSTAINED (>=10m) exits 1

04dd24f88726cb2f0e27032d94a9045323093e45 · 2026-09-14 00:35:45 -0700 · Steve Abrams

Cody caught that commit 3fad9e6 cited cron-fire-canary's artifact-freshness as
the safety net for its silent exit-0, but the job is manifest-flagged
artifact_weak (publish.log is StandardOutPath AND StandardErrorPath, refreshed
by a console line on EVERY tick regardless of outcome) — so the canary trusts
the launchd EXIT CODE for this job, and a blanket exit-0 made a SUSTAINED
source outage invisible (TK-11431 false-green).

Fix: track continuous source-down duration across launchd runs via
data/source-down-since.json. Exit 0 only while inside a 10-min self-heal grace
(transient pm2 restart), exit 1 once the outage is sustained so the canary
genuinely warns. Success/recovery clears the state. A real publish-path failure
(source up but rsync/JSON fails) still exits 1.

Verified: success->0+clear; first-down->0+record; sustained(20m)->1;
recovery->0+clear; source-up-but-garbage->1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Yk9ehSqj2L4frMztZNM2j

Files touched

M .gitignore
M publish-snapshot.js

Diff

commit 04dd24f88726cb2f0e27032d94a9045323093e45
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Mon Sep 14 00:35:45 2026 -0700

    5x/contrarian FIX FIRST: make exit code honest — transient source-down exits 0, SUSTAINED (>=10m) exits 1
    
    Cody caught that commit 3fad9e6 cited cron-fire-canary's artifact-freshness as
    the safety net for its silent exit-0, but the job is manifest-flagged
    artifact_weak (publish.log is StandardOutPath AND StandardErrorPath, refreshed
    by a console line on EVERY tick regardless of outcome) — so the canary trusts
    the launchd EXIT CODE for this job, and a blanket exit-0 made a SUSTAINED
    source outage invisible (TK-11431 false-green).
    
    Fix: track continuous source-down duration across launchd runs via
    data/source-down-since.json. Exit 0 only while inside a 10-min self-heal grace
    (transient pm2 restart), exit 1 once the outage is sustained so the canary
    genuinely warns. Success/recovery clears the state. A real publish-path failure
    (source up but rsync/JSON fails) still exits 1.
    
    Verified: success->0+clear; first-down->0+record; sustained(20m)->1;
    recovery->0+clear; source-up-but-garbage->1.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_017Yk9ehSqj2L4frMztZNM2j
---
 .gitignore          |  1 +
 publish-snapshot.js | 30 +++++++++++++++++++++++++++++-
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/.gitignore b/.gitignore
index 514a103..60f0411 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,4 @@ data/owner.key
 data/snapshot.json
 cta/
 cta-report.png
+data/source-down-since.json
diff --git a/publish-snapshot.js b/publish-snapshot.js
index d0d8409..009c5a9 100644
--- a/publish-snapshot.js
+++ b/publish-snapshot.js
@@ -45,6 +45,26 @@ function getLocal() {
 
 const sleep = ms => new Promise(r => setTimeout(r, ms));
 
+// Track how long the source has been continuously unreachable, ACROSS launchd
+// runs (this script runs once per 180s fire — no in-memory state survives).
+// A brief blip (e.g. a pm2 restart) self-heals and should NOT flap the canary,
+// but a SUSTAINED outage must stay visible: cron-fire-canary flags this job as
+// artifact_weak (its publish.log refreshes every tick regardless of outcome),
+// so the launchd EXIT CODE is the only signal it can trust here. We therefore
+// exit 0 only while the outage is inside the self-heal grace window, and exit 1
+// once it is sustained — so a real, prolonged source-down is genuinely reported
+// (TK-11431: a failed input must not be silently reported as success).
+const DOWN_STATE = path.join(__dirname, 'data', 'source-down-since.json');
+const SOURCE_DOWN_GRACE_MS = 10 * 60 * 1000; // ~3-4 ticks: tolerate a transient restart
+
+function clearDownState() { try { fs.unlinkSync(DOWN_STATE); } catch (_) {} }
+function recordDownAndAgeMs() {
+  let since = null;
+  try { since = JSON.parse(fs.readFileSync(DOWN_STATE, 'utf8')).since; } catch (_) {}
+  if (!since) { since = Date.now(); try { fs.writeFileSync(DOWN_STATE, JSON.stringify({ since })); } catch (_) {} }
+  return Date.now() - since;
+}
+
 async function publishOnce() {
   const data = await getLocal();
   const snap = {
@@ -78,14 +98,22 @@ async function publishOnce() {
 (async () => {
   try {
     await publishOnce();
+    clearDownState();               // source reachable + published -> outage (if any) is over
   } catch (e1) {
     console.error(`publish attempt 1 failed: ${e1.message} — retrying once in 5s`);
     await sleep(5000);
     try {
       await publishOnce();
+      clearDownState();
     } catch (e2) {
       if (e2 && e2.sourceDown) {
-        console.error(`source :9790 unreachable (${e2.code}) after retry — skipping this tick (self-healing; watched by keep-alive). Snapshot not updated.`);
+        const downMs = recordDownAndAgeMs();
+        const downMin = Math.round(downMs / 60000);
+        if (downMs >= SOURCE_DOWN_GRACE_MS) {
+          console.error(`source :9790 unreachable (${e2.code}) for ~${downMin}m (>= ${SOURCE_DOWN_GRACE_MS / 60000}m grace) — SUSTAINED outage, exit 1 so cron-fire-canary warns.`);
+          process.exit(1);
+        }
+        console.error(`source :9790 unreachable (${e2.code}) for ~${downMin}m — within self-heal grace, skipping this tick (exit 0). keep-alive should restore :9790.`);
         process.exit(0);
       }
       console.error(`publish failed (after retry): ${e2.message}`);

← 234c987 5x: REPORT.md — clean twice (favicon 404 fixed) · back to Doing Viewer · 5x: REPORT.md — fold in contrarian FIX FIRST round (false-gr 961d840 →