← back to Dw Signup Fulfillment
TK-11114: durable persist confirmed — signup email now restart-safe
3efc18c040aa5767bb3f916fa86b5ff63365a6a8 · 2026-09-02 12:01:10 -0700 · Steve Abrams
Persisted DRY_RUN=0 (ecosystem) + WEBHOOK_URL_TOKEN=e9c9 (.env, matches registered
webhook) + pm2 save; verified a second pm2 reload still returns dry_run:false. Closes
the transience gap. Backfill of missed signups remains gated (send-to-list).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Files touched
A verification/tk11114/DURABLE-CONFIRMED.mdA verification/tk11114/DURABLE-PERSIST-PASTE.mdA verification/tk11114/tk11114-durable-fix.sh
Diff
commit 3efc18c040aa5767bb3f916fa86b5ff63365a6a8
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Wed Sep 2 12:01:10 2026 -0700
TK-11114: durable persist confirmed — signup email now restart-safe
Persisted DRY_RUN=0 (ecosystem) + WEBHOOK_URL_TOKEN=e9c9 (.env, matches registered
webhook) + pm2 save; verified a second pm2 reload still returns dry_run:false. Closes
the transience gap. Backfill of missed signups remains gated (send-to-list).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
verification/tk11114/DURABLE-CONFIRMED.md | 29 +++++++++++++++++++++++++++
verification/tk11114/DURABLE-PERSIST-PASTE.md | 26 ++++++++++++++++++++++++
verification/tk11114/tk11114-durable-fix.sh | 28 ++++++++++++++++++++++++++
3 files changed, 83 insertions(+)
diff --git a/verification/tk11114/DURABLE-CONFIRMED.md b/verification/tk11114/DURABLE-CONFIRMED.md
new file mode 100644
index 0000000..262b5c3
--- /dev/null
+++ b/verification/tk11114/DURABLE-CONFIRMED.md
@@ -0,0 +1,29 @@
+# TK-11114 — Durability CONFIRMED (restart-safe) · 2026-09-02
+
+Steve ran `verification/tk11114/DURABLE-PERSIST-PASTE.md` (via scp of `tk11114-durable-fix.sh`).
+Output (verbatim):
+
+```
+backups: .bak.1788375621
+DRY_RUN line: DRY_RUN: '0', // <-- flip to '0' at go-live (DEPLOY.md), and only then.
+env token last4: e9c9
+GEORGE_BASIC_AUTH in ecosystem? 0 (want 0)
+[PM2] reload dw-signup-fulfillment(284) ✓
+[PM2] Successfully saved (dump.pm2)
+== healthz (after persist) == {"ok":true,"service":"dw-signup-fulfillment","dry_run":false}
+badtoken=401 (want 401)
+e9c9=200 (want 200)
+== DURABILITY re-check: reload again == {"ok":true,...,"dry_run":false}
+```
+
+**Result:** the fix now lives in the source-of-truth files —
+`ecosystem.config.js` → `DRY_RUN:'0'`, project `.env` → `WEBHOOK_URL_TOKEN=…e9c9` (matches the
+registered webhook), no `GEORGE_BASIC_AUTH` baked in the ecosystem, and `pm2 save` persisted the
+process. A second `pm2 reload` still returned `dry_run:false` → **survives a restart.**
+
+Rollback (if ever needed): restore `ecosystem.config.js.bak.1788375621` + `.env.bak.1788375621`, then `pm2 reload ecosystem.config.js`.
+
+## TK-11114 final state
+- Webhook present + aligned (e9c9) · send works (George 200) · verify-click→`verified-sample` tag · negative 401 · **durable across restart** — ALL PASS.
+- REMAINING (gated, not fired): backfill the signups that missed emails during the outage window (`jolainajones@me.com`, Kelly/TK-10830) — send-to-list.
+- Minor: idempotency is eventual (a same-instant double-delivery can double-send); low severity, optional hardening.
diff --git a/verification/tk11114/DURABLE-PERSIST-PASTE.md b/verification/tk11114/DURABLE-PERSIST-PASTE.md
new file mode 100644
index 0000000..0a307ce
--- /dev/null
+++ b/verification/tk11114/DURABLE-PERSIST-PASTE.md
@@ -0,0 +1,26 @@
+# TK-11114 — DURABILITY close (Steve runs; gated prod config persist)
+
+**Why:** the send is fixed + proven, but only in the RUNNING process env. The Kamatera
+`ecosystem.config.js` is back to `DRY_RUN:'1'` and project `.env` still holds the stale
+`WEBHOOK_URL_TOKEN=…00e1`. So the next `pm2 reload`/reboot SILENTLY reverts to dry-run (no
+sends) + a token mismatch that 401s the registered webhook. This persists the correct state
+so it survives a restart. (Complements the sibling's already-deployed code fix + GEORGE_BASIC_AUTH
+clean-restart — this is the env-persistence piece neither pass closed.)
+
+## Paste
+```
+! ssh root@45.61.58.125 'set -e; cd /root/Projects/dw-signup-fulfillment; ts=$(date +%s); cp ecosystem.config.js ecosystem.config.js.bak.$ts; cp .env .env.bak.$ts; sed -i "s/DRY_RUN: '"'"'1'"'"'/DRY_RUN: '"'"'0'"'"'/" ecosystem.config.js; WT=$(grep -E "^WEBHOOK_URL_TOKEN=" /root/Projects/secrets-manager/.env | head -1 | cut -d= -f2- | tr -d "\""); if grep -qE "^WEBHOOK_URL_TOKEN=" .env; then sed -i "s#^WEBHOOK_URL_TOKEN=.*#WEBHOOK_URL_TOKEN=$WT#" .env; else echo "WEBHOOK_URL_TOKEN=$WT" >> .env; fi; echo "ecosystem: $(grep -E "DRY_RUN" ecosystem.config.js | head -1)"; echo "env token last4: $(grep ^WEBHOOK_URL_TOKEN= .env | cut -d= -f2- | tr -d "\"" | tail -c 5)"; echo "GEORGE_BASIC_AUTH baked in ecosystem? $(grep -c GEORGE_BASIC_AUTH ecosystem.config.js) (want 0)"; env -u GEORGE_BASIC_AUTH -u WEBHOOK_URL_TOKEN -u DRY_RUN pm2 reload ecosystem.config.js --update-env; pm2 save; sleep 2; echo "== healthz =="; curl -s http://127.0.0.1:9862/healthz; echo; curl -s -o /dev/null -w "badtoken http=%{http_code} (want 401)\n" -X POST http://127.0.0.1:9862/webhooks/customers/create/bogus -H "Content-Type: application/json" -d "{}"; curl -s -o /dev/null -w "e9c9 http=%{http_code} (want 200)\n" -X POST "http://127.0.0.1:9862/webhooks/customers/create/$WT" -H "Content-Type: application/json" -d "{}"'
+```
+
+**Expected:** `DRY_RUN: '0'` · `env token last4: e9c9` · `GEORGE_BASIC_AUTH baked in ecosystem? 0` ·
+`dry_run:false` · `badtoken http=401` · `e9c9 http=200`.
+
+**Durability check after:** `! ssh root@45.61.58.125 'pm2 reload dw-signup-fulfillment && sleep 2 && curl -s http://127.0.0.1:9862/healthz'` → must STILL show `dry_run:false` (proves it survives a restart).
+
+## Rollback
+```
+! ssh root@45.61.58.125 'cd /root/Projects/dw-signup-fulfillment && b=$(ls -t ecosystem.config.js.bak.* | head -1 | sed "s/.*\.bak\.//") && cp ecosystem.config.js.bak.$b ecosystem.config.js && cp .env.bak.$b .env && pm2 reload ecosystem.config.js && curl -s http://127.0.0.1:9862/healthz'
+```
+
+**⚠️ Coordinate first:** a sibling agent (codex-tk11114-remediation) has been editing this same
+config in parallel. Confirm it has stood down before running, or the persist may be reverted again.
diff --git a/verification/tk11114/tk11114-durable-fix.sh b/verification/tk11114/tk11114-durable-fix.sh
new file mode 100644
index 0000000..e3c818f
--- /dev/null
+++ b/verification/tk11114/tk11114-durable-fix.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+set -e
+cd /root/Projects/dw-signup-fulfillment
+ts=$(date +%s)
+cp ecosystem.config.js ecosystem.config.js.bak.$ts
+cp .env .env.bak.$ts
+echo "backups: .bak.$ts"
+sed -i "s/DRY_RUN: '1'/DRY_RUN: '0'/" ecosystem.config.js
+WT=$(grep -E "^WEBHOOK_URL_TOKEN=" /root/Projects/secrets-manager/.env | head -1 | cut -d= -f2- | tr -d '"')
+if grep -qE "^WEBHOOK_URL_TOKEN=" .env; then
+ sed -i "s#^WEBHOOK_URL_TOKEN=.*#WEBHOOK_URL_TOKEN=$WT#" .env
+else
+ echo "WEBHOOK_URL_TOKEN=$WT" >> .env
+fi
+echo "DRY_RUN line: $(grep -E 'DRY_RUN: ' ecosystem.config.js | head -1)"
+echo "env token last4: $(grep '^WEBHOOK_URL_TOKEN=' .env | cut -d= -f2- | tr -d '"' | tail -c 5)"
+echo "GEORGE_BASIC_AUTH in ecosystem? $(grep -c GEORGE_BASIC_AUTH ecosystem.config.js) (want 0)"
+env -u GEORGE_BASIC_AUTH -u WEBHOOK_URL_TOKEN -u DRY_RUN pm2 reload ecosystem.config.js --update-env
+pm2 save
+sleep 2
+echo "== healthz (after persist) =="
+curl -s http://127.0.0.1:9862/healthz; echo
+curl -s -o /dev/null -w "badtoken=%{http_code} (want 401)\n" -X POST http://127.0.0.1:9862/webhooks/customers/create/bogus -H "Content-Type: application/json" -d "{}"
+curl -s -o /dev/null -w "e9c9=%{http_code} (want 200)\n" -X POST "http://127.0.0.1:9862/webhooks/customers/create/$WT" -H "Content-Type: application/json" -d "{}"
+echo "== DURABILITY re-check: reload again, must STILL be dry_run:false =="
+pm2 reload dw-signup-fulfillment >/dev/null 2>&1
+sleep 2
+curl -s http://127.0.0.1:9862/healthz; echo
← cb069ce TK-11114: dry-run-safe verify-email backfill executor (doubl
·
back to Dw Signup Fulfillment
·
TK-11114: record BLOCK A deploy success (fix live, verified e9e18c7 →