← back to Dw Signup Fulfillment

TK-11285: stop the deploy itself from destroying the live applications

a95eaa2c8b1394b518bb3c18f884d0d9ad7e0f84 · 2026-09-10 08:18:41 -0700 · Steve Abrams

Found while preparing the approved deploy. deploy.sh runs `rsync -az --delete`
and .deploy.conf carried NO RSYNC_EXTRA_EXCLUDES, so a deploy would have
overwritten prod's trade-applications.jsonl (19064 bytes, 23 rows, 16 pending)
with this machine's fixture copy (727 bytes, 2 test rows), and deleted six
prod-only files outright including both recover-stuck-apps rollback maps and
the live trade-notify-failures.jsonl.

deploy.sh's delete-guard would have caught it and aborted rather than shipped
the damage - but the deploy would simply have been blocked until someone added
these, and a DEPLOY_ALLOW_DELETE=1 override would have destroyed the data.

Adds the surgical excludes for state the live server and the operator CLIs
write. Verified by a real --dry-run --itemize-changes against prod: nothing
under data/ is touched, the only deletion is a stale lib/*.js.bak, and the
delete-guard now returns PASS.

Also excludes /verification - local test evidence prod has no use for, and its
paths match the guard's own risky-pattern regex ("verificat.*\.json"), which
would trip an abort on files that are merely being added.

DEPLOY.md gains the runbook rule Steve chose for the third writer: never run
npm test / npm run selftest against the prod checkout. selftest snapshots the
real store, writes its own rows into it, and restores the snapshot on exit, not
gated by DRY_RUN - a ~480ms window that silently destroys whatever arrived.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NEDp3MXiKVJ5GDKaQCHZon

Files touched

• M .deploy.conf
• M DEPLOY.md

Diff

commit a95eaa2c8b1394b518bb3c18f884d0d9ad7e0f84
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Thu Sep 10 08:18:41 2026 -0700

    TK-11285: stop the deploy itself from destroying the live applications
    
    Found while preparing the approved deploy. deploy.sh runs `rsync -az --delete`
    and .deploy.conf carried NO RSYNC_EXTRA_EXCLUDES, so a deploy would have
    overwritten prod's trade-applications.jsonl (19064 bytes, 23 rows, 16 pending)
    with this machine's fixture copy (727 bytes, 2 test rows), and deleted six
    prod-only files outright including both recover-stuck-apps rollback maps and
    the live trade-notify-failures.jsonl.
    
    deploy.sh's delete-guard would have caught it and aborted rather than shipped
    the damage - but the deploy would simply have been blocked until someone added
    these, and a DEPLOY_ALLOW_DELETE=1 override would have destroyed the data.
    
    Adds the surgical excludes for state the live server and the operator CLIs
    write. Verified by a real --dry-run --itemize-changes against prod: nothing
    under data/ is touched, the only deletion is a stale lib/*.js.bak, and the
    delete-guard now returns PASS.
    
    Also excludes /verification - local test evidence prod has no use for, and its
    paths match the guard's own risky-pattern regex ("verificat.*\.json"), which
    would trip an abort on files that are merely being added.
    
    DEPLOY.md gains the runbook rule Steve chose for the third writer: never run
    npm test / npm run selftest against the prod checkout. selftest snapshots the
    real store, writes its own rows into it, and restores the snapshot on exit, not
    gated by DRY_RUN - a ~480ms window that silently destroys whatever arrived.
    
    Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01NEDp3MXiKVJ5GDKaQCHZon
---
 .deploy.conf | 10 ++++++++++
 DEPLOY.md    |  9 +++++++++
 2 files changed, 19 insertions(+)

diff --git a/.deploy.conf b/.deploy.conf
index a97c48e..d62490d 100644
--- a/.deploy.conf
+++ b/.deploy.conf
@@ -3,3 +3,13 @@ DEPLOY_HOST=45.61.58.125
 DEPLOY_PATH=/root/Projects/dw-signup-fulfillment
 HEALTH_URL=http://127.0.0.1:9862/healthz
 INSTALL_CMD="npm ci --omit=dev"
+
+# Runtime state the LIVE server (and the operator CLIs) write on prod. deploy.sh
+# runs `rsync -az --delete`, so without these the deploy would overwrite prod's
+# trade applications with this machine's fixture copy and delete the rollback
+# maps outright. Measured 2026-09-10: prod trade-applications.jsonl was 19064
+# bytes / 23 rows / 16 pending; the local copy was 727 bytes / 2 test rows.
+# Keep this as ONE assignment — a second RSYNC_EXTRA_EXCLUDES silently shadows it.
+# /verification is local test evidence — prod has no use for it, and shipping it
+# trips deploy.sh's delete-guard (its risky-pattern regex matches "verificat.*\.json").
+RSYNC_EXTRA_EXCLUDES="/verification /data/trade-applications.jsonl /data/trade-applications.jsonl.* /data/recovery-*.json /data/trade-notify-failures.jsonl /data/latest.json /data/honor-reissue-ledger.jsonl /data/honor-reissue-ledger.jsonl.*"
diff --git a/DEPLOY.md b/DEPLOY.md
index 1272420..d081ff6 100644
--- a/DEPLOY.md
+++ b/DEPLOY.md
@@ -22,6 +22,15 @@ flip `DRY_RUN=0` until every prior step is green.
 
 ## 0. Pre-flight (still DRY_RUN)
 
+> ⛔ **NEVER run `npm test` / `npm run selftest` against the PROD checkout.**
+> `scripts/selftest.js` snapshots `data/trade-applications.jsonl`, writes its own test rows
+> into the REAL file during the run, then restores the start snapshot on exit — on both the
+> success and error paths, and it is NOT gated by `DRY_RUN`. Anything that arrives in that
+> ~480 ms window (a submitted trade application, an approve/reject decision) is destroyed
+> silently. Reproduced 2026-09-10 under TK-11285. Run the suite on a non-prod checkout, or
+> against a throwaway data dir. Post-deploy verification on prod is `/healthz` + `pm2 logs`,
+> never the test suite.
+
 ```sh
 cd ~/Projects/dw-signup-fulfillment
 npm install

← 534d1f6 TK-11285: correct my own overclaim in the f9de58c write-up  ·  back to Dw Signup Fulfillment  ·  TK-11285: deploy runbook with pre-deploy backup, verify, and fea5a31 →