[object Object]

← back to Nas Setup

pull-dw-dump: retry remote-dump SSH discovery on transient failure (TK-10547)

42281539091a67f7550583f484972e11734060de · 2026-08-15 07:45:16 -0700 · steve

A single 03:55 ssh timeout to Kamatera lost a full day's dw_unified backup and
hard-FAILed the run (the canary correctly caught it 8/15). Add 4 attempts with
15/30/60s backoff so a momentary network/SSH blip auto-recovers instead of
costing a day of backup. Read-only, idempotent (rsync skip-if-complete).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

Diff

commit 42281539091a67f7550583f484972e11734060de
Author: steve <steve@designerwallcoverings.com>
Date:   Sat Aug 15 07:45:16 2026 -0700

    pull-dw-dump: retry remote-dump SSH discovery on transient failure (TK-10547)
    
    A single 03:55 ssh timeout to Kamatera lost a full day's dw_unified backup and
    hard-FAILed the run (the canary correctly caught it 8/15). Add 4 attempts with
    15/30/60s backoff so a momentary network/SSH blip auto-recovers instead of
    costing a day of backup. Read-only, idempotent (rsync skip-if-complete).
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 scripts/pull-dw-dump.sh | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/pull-dw-dump.sh b/scripts/pull-dw-dump.sh
index 201efe3..30e80c2 100755
--- a/scripts/pull-dw-dump.sh
+++ b/scripts/pull-dw-dump.sh
@@ -67,8 +67,16 @@ alert(){ # CNCP card + George email on FAIL. args: <reason>
 }
 
 # ── 0. newest remote dump (read-only) — resolved ONCE, reused per destination ──
-REMOTE=$(ssh -o ConnectTimeout=15 -o BatchMode=yes "$HOST" "ls -t $GLOB 2>/dev/null | head -1")
-if [ -z "$REMOTE" ]; then log "FAIL: no remote dump found ($GLOB)"; alert "no remote dump matching $GLOB"; exit 1; fi
+# RETRY on transient SSH failure (added 2026-08-15, TK-10547): a single 03:55 `ssh` timeout to
+# Kamatera lost a whole day's dw_unified backup + hard-FAILed the run. A blip shouldn't cost a day.
+# 4 attempts, 15s→30s→60s backoff (~2 min max) — cheap insurance against a momentary network/SSH hiccup.
+REMOTE=""
+for attempt in 1 2 3 4; do
+  REMOTE=$(ssh -o ConnectTimeout=15 -o BatchMode=yes "$HOST" "ls -t $GLOB 2>/dev/null | head -1")
+  [ -n "$REMOTE" ] && break
+  [ "$attempt" -lt 4 ] && { backoff=$((attempt*15)); log "WARN: remote-dump discovery attempt $attempt/4 failed (ssh unreachable?) — retrying in ${backoff}s"; sleep "$backoff"; }
+done
+if [ -z "$REMOTE" ]; then log "FAIL: no remote dump found ($GLOB) after 4 attempts — Kamatera unreachable or dump missing"; alert "no remote dump matching $GLOB after 4 ssh attempts"; exit 1; fi
 BASE="$(basename "$REMOTE")"
 log "newest remote dump: $REMOTE"

← 3c61bc3 daemon-health: accept single-belt operation as PASS (Steve 8  ·  back to Nas Setup  ·  daemon-health: auto-detect belt strictness from the loaded r 672d698 →