ops: fix preflight rollback check reporting a false "no backup" (TK-11364)

616032a64754337eeeaa8456887d6c391ee2d554 · 2026-09-10 07:47:16 -0700 · Steve

Section 7 globbed /root/backups/* which only matches the top level, so it
missed the real dumps in /root/backups/db/ and printed "no obvious dump found"
while a fresh 12.8GB homesonspec dump sat one directory down. On the rollback
check a false "none" is the worst possible answer — it either aborts a window
unnecessarily or, worse, normalizes proceeding without a verified rollback.

Now finds the actual dump, prints size + age, warns past a nightly cycle,
restates the pg_restore --list validation the checklist requires, and fails
loud with an explicit "there is no rollback" if none exists.

Also surfaces BACKUP-SKIPPED flags, which are themselves a rollback risk.
Verified live: one is present today — DW-Agents nightly skipped at 03:30 with
only 34G free, refusing to run to protect Postgres. That flag would otherwise
have sat silent while someone planned a conversion.

Verified read-only against prod; no mutations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JhnABoxWtpF7mbBowoBWhb

Files touched

M ops/phase2-drafts/preflight.sh

Diff

commit 616032a64754337eeeaa8456887d6c391ee2d554
Author: Steve <steve@designerwallcoverings.com>
Date:   Thu Sep 10 07:47:16 2026 -0700

    ops: fix preflight rollback check reporting a false "no backup" (TK-11364)
    
    Section 7 globbed /root/backups/* which only matches the top level, so it
    missed the real dumps in /root/backups/db/ and printed "no obvious dump found"
    while a fresh 12.8GB homesonspec dump sat one directory down. On the rollback
    check a false "none" is the worst possible answer — it either aborts a window
    unnecessarily or, worse, normalizes proceeding without a verified rollback.
    
    Now finds the actual dump, prints size + age, warns past a nightly cycle,
    restates the pg_restore --list validation the checklist requires, and fails
    loud with an explicit "there is no rollback" if none exists.
    
    Also surfaces BACKUP-SKIPPED flags, which are themselves a rollback risk.
    Verified live: one is present today — DW-Agents nightly skipped at 03:30 with
    only 34G free, refusing to run to protect Postgres. That flag would otherwise
    have sat silent while someone planned a conversion.
    
    Verified read-only against prod; no mutations.
    
    Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01JhnABoxWtpF7mbBowoBWhb
---
 ops/phase2-drafts/preflight.sh | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/ops/phase2-drafts/preflight.sh b/ops/phase2-drafts/preflight.sh
index 9dafb3e3..c4fec0a1 100644
--- a/ops/phase2-drafts/preflight.sh
+++ b/ops/phase2-drafts/preflight.sh
@@ -59,8 +59,29 @@ echo "   inserts over a 10s sample: $(( ${INS2:-0} - ${INS1:-0} ))   (want 0 bef
 echo "   (n_tup_ins resets on a stats reset; the DELTA is what matters, not the absolute)"
 echo
 echo "==================== 7. newest backup (this IS the rollback) ===================="
-ls -lt /var/backups/*homesonspec* /root/backups/* /var/lib/postgresql/backups/* 2>/dev/null | head -3 \
-  || echo "   no obvious dump found in the usual dirs — confirm where the pg_dump lands + that one is fresh since the last import"
+# The old glob was /root/backups/* which only matches the TOP level — it missed the real
+# dumps in /root/backups/db/ and printed "no obvious dump found" while a fresh 12.8 GB
+# homesonspec dump was sitting one directory down. On the rollback check, a false "none"
+# is the worst possible answer. Search recursively for the actual DB dump instead.
+DUMP=$(ls -t /root/backups/db/*homesonspec*.dump /var/backups/*homesonspec* \
+        /var/lib/postgresql/backups/*homesonspec* 2>/dev/null | head -1)
+if [ -n "${DUMP:-}" ]; then
+  echo "   newest homesonspec dump:"
+  ls -lh "$DUMP" | sed 's/^/     /'
+  AGE_H=$(( ( $(date +%s) - $(stat -c %Y "$DUMP") ) / 3600 ))
+  echo "     age: ${AGE_H}h"
+  [ "$AGE_H" -gt 26 ] && echo "     >> WARNING: older than a nightly cycle — is the dump job still running?"
+  echo "     >> checklist item 2 also requires VALIDATING it: pg_restore --list \"$DUMP\" >/dev/null"
+else
+  echo "   !! NO homesonspec dump found in /root/backups/db, /var/backups, /var/lib/postgresql/backups."
+  echo "   !! There is no rollback. Do NOT convert until a fresh verified dump exists."
+fi
+# A skipped backup is itself a rollback risk — surface it rather than letting it sit silent.
+for f in /root/backups/BACKUP-SKIPPED-*.flag; do
+  [ -e "$f" ] || continue
+  echo "   >> BACKUP-SKIPPED FLAG PRESENT: $f"
+  sed 's/^/        /' "$f"
+done
 echo
 echo "==================== 8. indexes on SourceEvidence (partition key = createdAt) ===================="
 echo "   MATERIAL: as of 2026-09-10 the only indexes are pkey(id), (entityType,entityId), (stagedRecordId)."

← 14ed1904 ops: make Phase-2 preflight actually enforce the disk gate ( · back to Homesonspec · ops: preflight now VALIDATES the rollback dump instead of su 82fadf19 →