← back to Doing Viewer
TK-11446: exit 0 (not 1) when doing-viewer source :9790 is transiently down
3fad9e6098255c1a6eadccfa6c16432079cfa428 · 2026-09-14 00:16:19 -0700 · Steve Abrams
The canary reopened TK-11446 with a NEW failure mode the earlier timeout-retry
fix (9033cbe) didn't cover: connect ECONNREFUSED 127.0.0.1:9790. When the
doing-viewer source pm2 server is transiently down (e.g. a pm2 thundering-herd
restart, TK-11688), publish-snapshot.js exit-1'd every 180s tick, flapping
cron-fire-canary. That is a dependency outage, not a failure of this cron job's
own logic — and it is self-healing (keep-alive restarts the process; a LONG
outage makes the Kamatera snapshot go stale, which cron-fire-canary catches via
its primary artifact-freshness signal).
Classify connection-level source-down (ECONNREFUSED/ECONNRESET/ENOTFOUND/
EHOSTUNREACH) distinctly and exit 0 + WARN after a retry. A summarizer timeout,
malformed JSON (source up but broken), or a rsync-to-Kamatera failure still
exit 1 so a real publish-path problem genuinely warns — no false green.
Verified: source-up->exit0+published; source-down->exit0+WARN; source-up-but-
garbage->exit1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Yk9ehSqj2L4frMztZNM2j
Files touched
Diff
commit 3fad9e6098255c1a6eadccfa6c16432079cfa428
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Mon Sep 14 00:16:19 2026 -0700
TK-11446: exit 0 (not 1) when doing-viewer source :9790 is transiently down
The canary reopened TK-11446 with a NEW failure mode the earlier timeout-retry
fix (9033cbe) didn't cover: connect ECONNREFUSED 127.0.0.1:9790. When the
doing-viewer source pm2 server is transiently down (e.g. a pm2 thundering-herd
restart, TK-11688), publish-snapshot.js exit-1'd every 180s tick, flapping
cron-fire-canary. That is a dependency outage, not a failure of this cron job's
own logic — and it is self-healing (keep-alive restarts the process; a LONG
outage makes the Kamatera snapshot go stale, which cron-fire-canary catches via
its primary artifact-freshness signal).
Classify connection-level source-down (ECONNREFUSED/ECONNRESET/ENOTFOUND/
EHOSTUNREACH) distinctly and exit 0 + WARN after a retry. A summarizer timeout,
malformed JSON (source up but broken), or a rsync-to-Kamatera failure still
exit 1 so a real publish-path problem genuinely warns — no false green.
Verified: source-up->exit0+published; source-down->exit0+WARN; source-up-but-
garbage->exit1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Yk9ehSqj2L4frMztZNM2j
---
publish-snapshot.js | 32 +++++++++++++++++++++++++++++---
1 file changed, 29 insertions(+), 3 deletions(-)
diff --git a/publish-snapshot.js b/publish-snapshot.js
index ce382d0..d0d8409 100644
--- a/publish-snapshot.js
+++ b/publish-snapshot.js
@@ -16,6 +16,14 @@ const PASS = process.env.BASIC_PASS || 'DW2024!';
const SNAP = path.join(__dirname, 'data', 'snapshot.json');
const REMOTE = 'root@45.61.58.125:/root/Projects/doing-viewer/data/snapshot.json';
+// Connection-level errors that mean the SOURCE server (:9790) is simply not up
+// right now — a self-healing condition (keep-alive + dw-uptime-probe watch that
+// process; a LONG outage makes the Kamatera snapshot go stale, which
+// cron-fire-canary catches via artifact-freshness). These must NOT be conflated
+// with a real publish failure, so we tag them and exit 0 on this tick instead of
+// flapping the canary with exit=1 (TK-11446).
+const SOURCE_DOWN_CODES = new Set(['ECONNREFUSED', 'ECONNRESET', 'ENOTFOUND', 'EHOSTUNREACH']);
+
function getLocal() {
return new Promise((resolve, reject) => {
const auth = 'Basic ' + Buffer.from(`${USER}:${PASS}`).toString('base64');
@@ -25,7 +33,13 @@ function getLocal() {
http.get(LOCAL, { headers: { Authorization: auth }, timeout: 30000 }, (res) => {
let d = ''; res.on('data', c => d += c);
res.on('end', () => { try { resolve(JSON.parse(d)); } catch (e) { reject(e); } });
- }).on('error', reject).on('timeout', function () { this.destroy(); reject(new Error('timeout')); });
+ }).on('error', (err) => {
+ // Mark "source process is down" distinctly from "source up but publish broke".
+ if (err && SOURCE_DOWN_CODES.has(err.code)) err.sourceDown = true;
+ reject(err);
+ }).on('timeout', function () { this.destroy(); reject(new Error('timeout')); });
+ // NOTE: a 30s timeout is NOT treated as sourceDown — it means the server is
+ // reachable but the summarizer stalled, a genuine issue worth an exit=1 warn.
});
}
@@ -51,8 +65,16 @@ async function publishOnce() {
}
// Retry ONCE on a transient failure (slow local fetch or a slow Kamatera rsync)
-// before exiting non-zero. A single blip no longer crash-flags the job; a
-// SUSTAINED failure still exits 1 so cron-fire-canary genuinely warns (TK-11446).
+// before deciding the exit code. A single blip no longer crash-flags the job.
+// Exit-code semantics (TK-11446):
+// - SOURCE server (:9790) is simply down after a retry -> exit 0 + WARN.
+// This is self-healing and monitored elsewhere (keep-alive restarts the pm2
+// process; a LONG outage staleness is caught by cron-fire-canary's
+// artifact-freshness check). It is NOT a failure of THIS cron job's logic,
+// so we don't flap the canary with a false exit=1 for a dependency outage.
+// - Any OTHER sustained failure (summarizer timeout, malformed JSON = source
+// up but broken, rsync-to-Kamatera failure) -> exit 1, so the canary
+// genuinely warns on a real publish-path problem.
(async () => {
try {
await publishOnce();
@@ -62,6 +84,10 @@ async function publishOnce() {
try {
await publishOnce();
} catch (e2) {
+ if (e2 && e2.sourceDown) {
+ console.error(`source :9790 unreachable (${e2.code}) after retry — skipping this tick (self-healing; watched by keep-alive). Snapshot not updated.`);
+ process.exit(0);
+ }
console.error(`publish failed (after retry): ${e2.message}`);
process.exit(1);
}
← 9033cbe TK-11446: make doing-viewer publish resilient to transient t
·
back to Doing Viewer
·
5x sweep 1: /favicon.ico -> 204 (was 404, console error on e ea854b7 →