← back to Secrets Manager
rotate-dw-admin-full.sh: repair 4 reproduced defects + synthetic harness (TK-11480)
5673bb9ca1ce23a2ffc4a023df97e8bdef76d988 · 2026-09-26 08:53:47 -0700 · Steve Abrams
Fixes: password in python3 argv (now printf builtin); ssh pipe+heredoc stdin
collision (script now the ssh command, pw alone on stdin); CHG passed as argv
not env (zero restarts); no errexit (failed Mac2 ALTER continued to prod).
Hardening per contrarian review: pm2 jlist failure fails closed (was a silent
zero-restart via process substitution), exclusive timestamped backups, opt-in
split-brain start, broad-cwd restart guard, partial-rotation STATE report,
xtrace disabled. test/rotate-dw-admin-full.test.sh: 19/19 green; RED vs original.
No secret read, script never executed against a real host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J3gBZfBgR2c7TZugF5Ybz4
Files touched
M rotate-dw-admin-full.shA test/rotate-dw-admin-full.test.sh
Diff
commit 5673bb9ca1ce23a2ffc4a023df97e8bdef76d988
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Sat Sep 26 08:53:47 2026 -0700
rotate-dw-admin-full.sh: repair 4 reproduced defects + synthetic harness (TK-11480)
Fixes: password in python3 argv (now printf builtin); ssh pipe+heredoc stdin
collision (script now the ssh command, pw alone on stdin); CHG passed as argv
not env (zero restarts); no errexit (failed Mac2 ALTER continued to prod).
Hardening per contrarian review: pm2 jlist failure fails closed (was a silent
zero-restart via process substitution), exclusive timestamped backups, opt-in
split-brain start, broad-cwd restart guard, partial-rotation STATE report,
xtrace disabled. test/rotate-dw-admin-full.test.sh: 19/19 green; RED vs original.
No secret read, script never executed against a real host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J3gBZfBgR2c7TZugF5Ybz4
---
rotate-dw-admin-full.sh | 202 +++++++++++++++++++++++++-------------
test/rotate-dw-admin-full.test.sh | 159 ++++++++++++++++++++++++++++++
2 files changed, 291 insertions(+), 70 deletions(-)
diff --git a/rotate-dw-admin-full.sh b/rotate-dw-admin-full.sh
index 6ec6e86..4613f66 100755
--- a/rotate-dw-admin-full.sh
+++ b/rotate-dw-admin-full.sh
@@ -8,44 +8,64 @@
# 2. ALTER ROLE dw_admin on Mac2 local PG AND prod Kamatera (each gated by y/N).
# 3. Fans the new value to Mac2 consumers via the canonical secrets-manager CLI.
# 4. On Kamatera: rewrites the dw_admin DSN password in-place wherever it lives
-# (app .env / ecosystem / pm2 dump), backing each file up to *.pre-rot.bak,
+# (app .env / ecosystem / pm2 dump), backing each file up first,
# then restarts ONLY the apps actually affected, in batches of <=6
# (NEVER `pm2 restart all` — that OOM-kills the daemon: 186 apps).
-# 5. Starts the 2 split-brain workers (am-recrawl, vendor-review-worker-enhanced)
-# which now carry the new pw, converging the fleet on one password.
+# 5. (only with --start-split-brain) starts the 2 split-brain workers
+# (am-recrawl, vendor-review-worker-enhanced) so they carry the new pw.
# 6. Verifies: 0 dw_admin auth failures, pm2 health, secrets-cli check.
#
+# FAIL-CLOSED (TK-11480 repair, 2026-09-26): errexit + pipefail everywhere. Any
+# failed stage STOPS the run and prints exactly which sides were already changed
+# (e.g. "mac2_alter=yes prod_alter=no") so a half-rotation is never silent.
+# Synthetic test: test/rotate-dw-admin-full.test.sh (no real host/DB/secret).
+#
# SECRET HYGIENE (per Steve's standing rules)
# The new pw exists only in: this process memory, the ALTER SQL piped via stdin,
# the secrets registry, and the rewritten consumer envs. It is NEVER printed
# (unless --show-once), never an argv, never in shell history.
#
# USAGE
-# ./rotate-dw-admin-full.sh --dry-run # print the plan, change NOTHING
-# ./rotate-dw-admin-full.sh # execute (confirms before each write)
-# ./rotate-dw-admin-full.sh --show-once # also print the new pw ONCE at the end
+# ./rotate-dw-admin-full.sh --dry-run # print the plan, change NOTHING
+# ./rotate-dw-admin-full.sh # execute (confirms before each write)
+# ./rotate-dw-admin-full.sh --show-once # also print the new pw ONCE at the end
+# ./rotate-dw-admin-full.sh --start-split-brain # also start the 2 stopped workers
#
# ROLLBACK
-# Every rewritten file has a sibling *.pre-rot.bak on Kamatera. To revert prod
-# dw_admin to the prior pw, re-run the 2026-06-03 recovery (extract from any
-# *.pre-rot.bak DSN, ALTER via stdin). Mac2 PG + registry similarly.
+# Every rewritten file has a sibling *.pre-rot.<UTC-ts>.bak on Kamatera
+# (created exclusively, so a rerun never overwrites an earlier backup).
+# To revert prod dw_admin to the prior pw, re-run the 2026-06-03 recovery
+# (extract from any backup's DSN, ALTER via stdin). Mac2 PG + registry similarly.
-set -uo pipefail
+set -Eeuo pipefail
+# Never trace: xtrace would print the ALTER (and so the password) to scrollback.
+set +o xtrace; unset SHELLOPTS BASH_XTRACEFD 2>/dev/null || true
-DRY=0; SHOW=0
+DRY=0; SHOW=0; START_SB=0
for a in "$@"; do case "$a" in
- --dry-run) DRY=1 ;; --show-once) SHOW=1 ;;
+ --dry-run) DRY=1 ;; --show-once) SHOW=1 ;; --start-split-brain) START_SB=1 ;;
*) echo "unknown arg: $a" >&2; exit 2 ;;
esac; done
KAM="root@kamatera"
SECRETS_CLI="$HOME/Projects/secrets-manager/cli.js"
REG_KEY="PG_DW_ADMIN_PASSWORD"
-SPLIT_BRAIN=(am-recrawl vendor-review-worker-enhanced)
say(){ printf '\n\033[1m%s\033[0m\n' "$*"; }
confirm(){ [ "$DRY" = 1 ] && { echo "[dry-run] would prompt: $1"; return 0; }
- read -r -p "$1 [y/N] " r; [ "$r" = y ] || { echo "aborted."; exit 1; }; }
+ local r=""; read -r -p "$1 [y/N] " r || true
+ [ "$r" = y ] || { echo "aborted by operator at: $1"; report_state; exit 1; }; }
+
+# Which sides have actually been changed — printed on ANY failure or abort.
+DONE_MAC2=no; DONE_PROD=no; DONE_REG=no; DONE_FAN=no
+report_state(){
+ echo "STATE: mac2_alter=$DONE_MAC2 prod_alter=$DONE_PROD registry=$DONE_REG kamatera_fan=$DONE_FAN" >&2
+ if [ "$DONE_MAC2$DONE_PROD" != nono ] && [ "$DONE_FAN" != yes ]; then
+ echo "!! PARTIAL ROTATION: the role password changed but consumers were NOT all repointed." >&2
+ echo "!! Do not walk away — rerun the remaining stage, or roll back per ROLLBACK above." >&2
+ fi
+}
+trap 'rc=$?; echo "FAILED at line $LINENO (exit $rc)" >&2; report_state; exit $rc' ERR
command -v openssl >/dev/null || { echo "need openssl" >&2; exit 1; }
command -v node >/dev/null || { echo "need node" >&2; exit 1; }
@@ -55,34 +75,33 @@ say "dw_admin FULL rotation $([ "$DRY" = 1 ] && echo '(DRY RUN — no changes)')
# ---- 1. mint -------------------------------------------------------------
NEW_PG="$(openssl rand -base64 48 | tr -dc 'A-Za-z0-9' | head -c 32)"
-[ "${#NEW_PG}" -ge 20 ] || { echo "pw-gen failed" >&2; exit 1; }
+[[ "$NEW_PG" =~ ^[A-Za-z0-9]{32}$ ]] || { echo "pw-gen failed" >&2; exit 1; }
echo "[1] minted a new 32-char password (hidden)"
-# Emit the ALTER statement on stdout, dollar-quoted so any char is safe.
-alter_sql(){ python3 - "$NEW_PG" <<'PY'
-import sys
-pw=sys.argv[1]; tag="$dwrot$"
-assert tag not in pw, "tag collision — regenerate"
-print(f"ALTER ROLE dw_admin PASSWORD {tag}{pw}{tag};")
-PY
-}
+# Emit the ALTER statement on stdout. printf is a bash BUILTIN, so the password
+# never becomes an argv of any exec'd process (the old python3 helper put it in
+# argv, visible to `ps`). NEW_PG is validated [A-Za-z0-9]{32}, so plain quoting
+# is injection-safe.
+alter_sql(){ printf "ALTER ROLE dw_admin PASSWORD '%s';\n" "$NEW_PG"; }
# ---- 2. Mac2 local PG ----------------------------------------------------
say "[2] ALTER ROLE dw_admin on Mac2 local PG"
if [ "$DRY" = 1 ]; then
- echo "[dry-run] alter_sql | psql -d postgres -f - (superuser = \$USER; adjust -U if needed)"
+ echo "[dry-run] alter_sql | psql -X -q -d postgres -f - (superuser = \$USER; adjust -U if needed)"
else
confirm " apply to Mac2 local PG (psql -d postgres as \$USER)?"
- alter_sql | psql -d postgres -v ON_ERROR_STOP=1 -f - && echo " Mac2: ALTER ok"
+ alter_sql | psql -X -q -d postgres -v ON_ERROR_STOP=1 -f -
+ DONE_MAC2=yes; echo " Mac2: ALTER ok"
fi
# ---- 3. prod Kamatera PG -------------------------------------------------
say "[3] ALTER ROLE dw_admin on prod Kamatera"
if [ "$DRY" = 1 ]; then
- echo "[dry-run] alter_sql | ssh $KAM 'sudo -n -u postgres psql -d postgres -f -'"
+ echo "[dry-run] alter_sql | ssh $KAM 'sudo -n -u postgres psql -X -q -d postgres -f -'"
else
confirm " apply to PROD Kamatera?"
- alter_sql | ssh "$KAM" "sudo -n -u postgres psql -d postgres -v ON_ERROR_STOP=1 -f -" && echo " prod: ALTER ok"
+ alter_sql | ssh "$KAM" "sudo -n -u postgres psql -X -q -d postgres -v ON_ERROR_STOP=1 -f -"
+ DONE_PROD=yes; echo " prod: ALTER ok"
fi
# ---- 4. fan to Mac2 consumers via secrets-manager ------------------------
@@ -90,23 +109,28 @@ say "[4] route new pw to Mac2 consumers (secrets-manager import-paste, stdin)"
if [ "$DRY" = 1 ]; then
echo "[dry-run] printf '$REG_KEY=<new>' | node $SECRETS_CLI import-paste"
else
- printf '%s=%s\n' "$REG_KEY" "$NEW_PG" | node "$SECRETS_CLI" import-paste && echo " registry + Mac2 .envs updated"
+ printf '%s=%s\n' "$REG_KEY" "$NEW_PG" | node "$SECRETS_CLI" import-paste
+ DONE_REG=yes; echo " registry + Mac2 .envs updated"
fi
# ---- 5. fan to Kamatera (in-place DSN rewrite) + targeted batched restart -
-say "[5] fan to prod Kamatera (rewrite dw_admin DSNs in place, restart affected apps <=6/batch)"
-if [ "$DRY" = 1 ]; then
- echo "[dry-run] push pw via stdin to remote rewriter; restart only apps whose cwd contains a rewritten file"
-else
- confirm " rewrite prod app envs + restart affected dw_admin apps (batched)?"
- printf '%s' "$NEW_PG" | ssh "$KAM" 'bash -s' <<'REMOTE'
-set -uo pipefail
-NEW_PG="$(cat)" # pw arrives on stdin — never in argv
-export NEW_PG
+# Why not a heredoc: ssh has ONE stdin, and it must carry the password alone.
+# So the (secret-free) script moves into the ssh command, quoted with %q.
+# Assumes root's login shell on Kamatera is bash (it is; a change fails loudly).
+# The remote program travels as the ssh COMMAND (it contains no secret); the
+# password travels alone on ssh STDIN. (The old version fed both a pipe and a
+# heredoc to the same stdin — the heredoc won, so the remote read the script
+# text as the "password".)
+read -r -d '' REMOTE_FAN <<'REMOTE' || true
+set -Eeuo pipefail
+IFS= read -r NEW_PG # pw arrives on stdin — never in argv
+[[ "$NEW_PG" =~ ^[A-Za-z0-9]{32}$ ]] || { echo "remote: bad pw on stdin" >&2; exit 3; }
+export NEW_PG DWROT_TS="$(date -u +%Y%m%dT%H%M%SZ)"
# 5a. rewrite every file carrying a dw_admin DSN; record which dirs changed.
+# Any failed write exits non-zero -> errexit stops BEFORE any restart.
CHANGED_DIRS="$(python3 - <<'PY'
-import os,re,sys
-new=os.environ["NEW_PG"]
+import os,re,sys,shutil
+new=os.environ["NEW_PG"]; ts=os.environ["DWROT_TS"]
roots=["/root/DW-Agents","/root/public-projects","/root/Projects","/etc/environment","/root/.pm2/dump.pm2"]
pat=re.compile(r'(postgres(?:ql)?://dw_admin:)[^@]+(@)')
files=set()
@@ -114,68 +138,106 @@ for r in roots:
if os.path.isfile(r): files.add(r)
elif os.path.isdir(r):
for dp,_,fs in os.walk(r):
- if "node_modules" in dp or "/.git/" in dp: continue
+ if "node_modules" in dp or "/.git/" in dp or dp.endswith("/.git"): continue
for f in fs:
if f.endswith((".env",".local",".js",".cjs",".json")): files.add(os.path.join(dp,f))
-dirs=set()
-for f in files:
- try: s=open(f,encoding="utf-8",errors="replace").read()
+dirs=set(); errs=0
+for f in sorted(files):
+ try: s=open(f,encoding="utf-8",errors="surrogateescape").read()
except Exception: continue
if "dw_admin:" not in s: continue
- ns=pat.sub(r'\g<1>'+new+r'\g<2>', s)
- if ns!=s:
- try:
- open(f+".pre-rot.bak","w").write(s); open(f,"w").write(ns)
- dirs.add(os.path.dirname(f))
- except Exception as e:
- sys.stderr.write(f"skip {f}: {e}\n")
+ ns=pat.sub(lambda m: m.group(1)+new+m.group(2), s)
+ if ns==s: continue
+ try:
+ bak=f"{f}.pre-rot.{ts}.bak"
+ with open(bak,"x",encoding="utf-8",errors="surrogateescape") as b: b.write(s) # exclusive: never clobber
+ shutil.copymode(f,bak)
+ tmp=f"{f}.dwrot-tmp"
+ with open(tmp,"w",encoding="utf-8",errors="surrogateescape") as t: t.write(ns)
+ shutil.copymode(f,tmp); os.replace(tmp,f)
+ dirs.add(os.path.dirname(f))
+ except Exception as e:
+ errs+=1; sys.stderr.write(f"REWRITE FAILED {f}: {e}\n")
sys.stderr.write(f"rewrote files in {len(dirs)} dirs\n")
print("\n".join(sorted(dirs)))
+sys.exit(1 if errs else 0)
PY
)"
-# 5b. restart ONLY online apps whose cwd is under a changed dir, in batches of 6.
-mapfile -t TARGETS < <(pm2 jlist | python3 -c '
+# 5b. restart ONLY online apps whose cwd relates to a changed dir, <=6 per batch.
+# CHG is passed in the ENVIRONMENT (the old version appended it as an argv,
+# so python never saw it and zero apps restarted). Broad parent cwds never
+# match "downward", so an app running from /root can't sweep the whole box.
+# Captured via plain command substitution (NOT `done < <(...)`): errexit +
+# pipefail cannot see a failure inside process substitution, so a dead
+# `pm2 jlist` would silently mean "0 apps restarted" and a false-green DONE.
+PM2_JSON="$(pm2 jlist)" || { echo "pm2 jlist failed — refusing to guess restart targets" >&2; exit 6; }
+TARGET_LIST="$(printf '%s' "$PM2_JSON" | CHG="$CHANGED_DIRS" python3 -c '
import sys,json,os
+BROAD={"/","/root","/root/Projects","/root/public-projects","/root/DW-Agents","/etc","/root/.pm2"}
dirs=[d for d in os.environ.get("CHG","").splitlines() if d]
for p in json.load(sys.stdin):
e=p["pm2_env"]
if e.get("status")!="online": continue
- cwd=e.get("pm_cwd","") or ""
- if any(cwd==d or cwd.startswith(d+"/") or d.startswith(cwd+"/") for d in dirs):
- print(p["name"])
-' CHG="$CHANGED_DIRS")
+ cwd=(e.get("pm_cwd","") or "").rstrip("/")
+ if not cwd: continue
+ hit=any(cwd==d or cwd.startswith(d+"/") for d in dirs if d not in BROAD) or \
+ (cwd not in BROAD and any(d.startswith(cwd+"/") for d in dirs))
+ if hit: print(p["name"])
+')"
+TARGETS=()
+while IFS= read -r a; do [ -n "$a" ] && TARGETS+=("$a"); done <<< "$TARGET_LIST"
echo "affected apps: ${#TARGETS[@]}"
+FAILED_RESTARTS=()
+restart_batch(){ pm2 restart "$@" --update-env >/dev/null 2>&1 || FAILED_RESTARTS+=("$@"); }
batch=()
-for a in "${TARGETS[@]}"; do
+for a in ${TARGETS[@]+"${TARGETS[@]}"}; do
batch+=("$a")
- if [ "${#batch[@]}" -ge 6 ]; then pm2 restart "${batch[@]}" --update-env >/dev/null 2>&1; sleep 4; batch=(); fi
+ if [ "${#batch[@]}" -ge 6 ]; then restart_batch "${batch[@]}"; sleep 4; batch=(); fi
done
-[ "${#batch[@]}" -gt 0 ] && pm2 restart "${batch[@]}" --update-env >/dev/null 2>&1
-# 5c. converge the split-brain workers (now carry the new pw) + persist.
-pm2 start am-recrawl vendor-review-worker-enhanced >/dev/null 2>&1 || true
-pm2 save >/dev/null 2>&1
+if [ "${#batch[@]}" -gt 0 ]; then restart_batch "${batch[@]}"; fi
+# 5c. split-brain workers only when explicitly requested (--start-split-brain).
+if [ "${DWROT_START_SB:-0}" = 1 ]; then pm2 start am-recrawl vendor-review-worker-enhanced; fi
+pm2 save >/dev/null
+if [ "${#FAILED_RESTARTS[@]}" -gt 0 ]; then
+ echo "RESTART FAILED for: ${FAILED_RESTARTS[*]} (all other batches ran)" >&2; exit 4
+fi
echo "remote fan + targeted restart done"
REMOTE
-fi
-# ---- 6. verify -----------------------------------------------------------
-say "[6] verify"
+say "[5] fan to prod Kamatera (rewrite dw_admin DSNs in place, restart affected apps <=6/batch)"
if [ "$DRY" = 1 ]; then
- echo "[dry-run] would: check 0 dw_admin auth failures, pm2 health, secrets-cli check"
+ echo "[dry-run] pw via ssh stdin (script as ssh command); restart only apps whose cwd relates to a rewritten file"
+ if [ "$START_SB" = 1 ]; then echo "[dry-run] would also pm2 start am-recrawl vendor-review-worker-enhanced"; fi
else
- ssh "$KAM" 'bash -s' <<'REMOTE'
+ confirm " rewrite prod app envs + restart affected dw_admin apps (batched)?"
+ printf '%s\n' "$NEW_PG" | ssh "$KAM" "DWROT_START_SB=$START_SB bash -c $(printf '%q' "$REMOTE_FAN")"
+ DONE_FAN=yes
+fi
+
+# ---- 6. verify -----------------------------------------------------------
+read -r -d '' REMOTE_VERIFY <<'REMOTE' || true
+set -uo pipefail
LOG=/var/log/postgresql/postgresql-14-main.log
sleep 12
-f=$(awk -v c="$(date -u -d "12 seconds ago" "+%Y-%m-%d %H:%M:%S")" '/authentication failed for user "dw_admin"/ && $1" "$2 > c {n++} END{print n+0}' "$LOG" 2>/dev/null)
+f=$(awk -v c="$(date -u -d "12 seconds ago" "+%Y-%m-%d %H:%M:%S" 2>/dev/null)" '/authentication failed for user "dw_admin"/ && $1" "$2 > c {n++} END{print n+0}' "$LOG" 2>/dev/null || echo 0)
echo " dw_admin auth failures (last 12s): $f (want 0)"
pm2 jlist | python3 -c "import sys,json;from collections import Counter;print(' pm2:',dict(Counter(p['pm2_env'].get('status') for p in json.load(sys.stdin))))"
-sudo -n -u postgres psql -d dw_unified -tAc "SELECT ' dw_unified rows: '||count(*) FROM products;" 2>&1 | head -1
+sudo -n -u postgres psql -X -d dw_unified -tAc "SELECT ' dw_unified rows: '||count(*) FROM products;" 2>&1 | head -1
+[ "${f:-0}" = 0 ]
REMOTE
+
+say "[6] verify"
+if [ "$DRY" = 1 ]; then
+ echo "[dry-run] would: check 0 dw_admin auth failures, pm2 health, secrets-cli check"
+else
+ ssh "$KAM" "bash -c $(printf '%q' "$REMOTE_VERIFY")" </dev/null \
+ || { echo "VERIFY FAILED: dw_admin auth failures seen after rotation" >&2; report_state; exit 5; }
node "$SECRETS_CLI" check 2>&1 | grep -iE "dw_admin|PG_DW|FAIL|VALID" | head -5 || true
fi
+trap - ERR
say "DONE. After verifying 0 failures + apps online:"
echo " • flip CNCP prod flag (server.js:1718 status:'compromised' -> 'ok')"
echo " • the box is now on a FRESH password (compromised-era pw retired)"
-[ "$SHOW" = 1 ] && printf ' new dw_admin pw (record in your vault, then clear): %s\n' "$NEW_PG"
+if [ "$SHOW" = 1 ]; then printf ' new dw_admin pw (record in your vault, then clear): %s\n' "$NEW_PG"; fi
unset NEW_PG
diff --git a/test/rotate-dw-admin-full.test.sh b/test/rotate-dw-admin-full.test.sh
new file mode 100755
index 0000000..f7a3927
--- /dev/null
+++ b/test/rotate-dw-admin-full.test.sh
@@ -0,0 +1,159 @@
+#!/usr/bin/env bash
+# Synthetic harness for rotate-dw-admin-full.sh (TK-11480).
+#
+# Runs the rotation script inside a throwaway sandbox where EVERY external
+# effect is a stub: openssl (emits a fixed SYNTHETIC password), psql, ssh,
+# sudo, pm2, and a fake secrets-manager cli.js under a fake $HOME. The ssh stub
+# executes the remote command locally with /root and /etc paths rewritten into
+# the sandbox, so the Kamatera-side DSN rewrite + pm2 restart logic really runs.
+# A python3 wrapper records argv so a password passed as an argument is caught.
+#
+# No real secret is read, no real DB/host/pm2 is touched. Nothing leaves the box.
+#
+# Usage: test/rotate-dw-admin-full.test.sh [path-to-script]
+# default script = ../rotate-dw-admin-full.sh. Pass an older copy to prove
+# the harness goes RED on the known defects (negative test).
+set -uo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+SCRIPT="$(cd "$(dirname "${1:-$HERE/../rotate-dw-admin-full.sh}")" && pwd)/$(basename "${1:-rotate-dw-admin-full.sh}")"
+[ -f "$SCRIPT" ] || { echo "no script: $SCRIPT" >&2; exit 2; }
+
+SYNTH="SYNTHpw0000000000000000000000Zz9" # 32 alnum chars, obviously fake
+OLD="OLDsynth"
+REAL_PY="$(command -v python3)"
+PASS=0; FAIL=0
+ok(){ PASS=$((PASS+1)); printf ' PASS %s\n' "$1"; }
+bad(){ FAIL=$((FAIL+1)); printf ' FAIL %s\n' "$1"; }
+
+mk_sandbox(){
+ SB="$(mktemp -d "${TMPDIR:-/tmp}/dwrot-test.XXXXXX")"
+ mkdir -p "$SB/bin" "$SB/home/Projects/secrets-manager" "$SB/root/public-projects/appA" \
+ "$SB/root/public-projects/appB/config" "$SB/root/public-projects/appC" "$SB/root/.pm2" "$SB/etc" "$SB/log"
+ : > "$SB/log/argv.log"
+ printf 'DATABASE_URL=postgres://dw_admin:%s@127.0.0.1:5432/dw_unified\n' "$OLD" > "$SB/root/public-projects/appA/.env"
+ printf 'module.exports={env:{PG:"postgresql://dw_admin:%s@localhost/dw_unified"}}\n' "$OLD" > "$SB/root/public-projects/appB/config/db.js"
+ printf 'DATABASE_URL=postgres://other_user:x@127.0.0.1/other\n' > "$SB/root/public-projects/appC/.env"
+ printf '{}\n' > "$SB/root/.pm2/dump.pm2"; : > "$SB/etc/environment"
+
+ # every stub appends "<name> <argv...>" to argv.log
+ cat > "$SB/bin/_log" <<EOF
+#!/bin/bash
+printf '%s\n' "\$*" >> "$SB/log/argv.log"
+EOF
+ cat > "$SB/bin/openssl" <<EOF
+#!/bin/bash
+"$SB/bin/_log" openssl "\$@"; printf '%s\n' "$SYNTH$SYNTH"
+EOF
+ cat > "$SB/bin/psql" <<EOF
+#!/bin/bash
+"$SB/bin/_log" psql "\$@"
+cat >> "$SB/log/psql.stdin"
+[ -n "\${FAKE_PSQL_FAIL:-}" ] && { echo "psql: FATAL (synthetic)" >&2; exit 2; }
+exit 0
+EOF
+ cat > "$SB/bin/sudo" <<EOF
+#!/bin/bash
+"$SB/bin/_log" sudo "\$@"
+while [ \$# -gt 0 ]; do case "\$1" in -n) shift;; -u) shift 2;; *) break;; esac; done
+exec "\$@"
+EOF
+ # ssh: log, then run the remote command locally with /root and /etc sandboxed
+ cat > "$SB/bin/ssh" <<EOF
+#!/bin/bash
+"$SB/bin/_log" ssh "\$@"
+args=(); for a in "\$@"; do case "\$a" in -n|-T) ;; *) args+=("\$a");; esac; done
+cmd="\${args[*]:1}"
+cmd="\${cmd//\/root\//$SB/root/}"; cmd="\${cmd//\"\/root\"/\"$SB/root\"}"; cmd="\${cmd//\/etc\/environment/$SB/etc/environment}"
+echo ssh >> "$SB/log/ssh.count"
+exec bash -c "\$cmd"
+EOF
+ cat > "$SB/bin/pm2" <<EOF
+#!/bin/bash
+"$SB/bin/_log" pm2 "\$@"
+if [ "\$1" = jlist ] && [ -n "\${FAKE_PM2_JLIST_FAIL:-}" ]; then echo "pm2 daemon unreachable (synthetic)" >&2; exit 1; fi
+if [ "\$1" = jlist ]; then cat <<J
+[{"name":"appA","pm2_env":{"status":"online","pm_cwd":"$SB/root/public-projects/appA"}},
+ {"name":"appB","pm2_env":{"status":"online","pm_cwd":"$SB/root/public-projects/appB"}},
+ {"name":"appC","pm2_env":{"status":"online","pm_cwd":"$SB/root/public-projects/appC"}},
+ {"name":"appD","pm2_env":{"status":"stopped","pm_cwd":"$SB/root/public-projects/appA"}},
+ {"name":"rootcwd","pm2_env":{"status":"online","pm_cwd":"$SB/root"}}]
+J
+fi
+exit 0
+EOF
+ cat > "$SB/bin/python3" <<EOF
+#!/bin/bash
+"$SB/bin/_log" python3 "\$@"
+exec "$REAL_PY" "\$@"
+EOF
+ cat > "$SB/home/Projects/secrets-manager/cli.js" <<EOF
+let s='';process.stdin.on('data',d=>s+=d).on('end',()=>{require('fs').appendFileSync('$SB/log/cli.stdin',s)});
+EOF
+ chmod +x "$SB/bin/"*
+}
+
+run_script(){ # $1 = stdin answers, rest = script args
+ local answers="$1"; shift
+ printf '%s' "$answers" | env -i PATH="$SB/bin:/usr/bin:/bin:$(dirname "$(command -v node)")" HOME="$SB/home" \
+ TMPDIR="$SB" FAKE_PSQL_FAIL="${FAKE_PSQL_FAIL:-}" FAKE_PM2_JLIST_FAIL="${FAKE_PM2_JLIST_FAIL:-}" bash "$SCRIPT" "$@" > "$SB/log/out" 2>&1
+ echo $?
+}
+
+echo "harness: $SCRIPT"
+
+# ---- T1: full supervised run, all confirms = y ---------------------------------
+mk_sandbox
+rc=$(run_script $'y\ny\ny\n')
+grep -q "$SYNTH" "$SB/log/argv.log" && bad "T1 password appeared in a process argv (defect 1)" \
+ || ok "T1 password never in any process argv"
+grep -q "ALTER ROLE dw_admin" "$SB/log/psql.stdin" 2>/dev/null && [ "$(grep -c "$SYNTH" "$SB/log/psql.stdin")" -ge 2 ] \
+ && ok "T1 ALTER reached Mac2 + prod psql via stdin" || bad "T1 ALTER did not reach both psql stdins"
+grep -q "PG_DW_ADMIN_PASSWORD=$SYNTH" "$SB/log/cli.stdin" 2>/dev/null && ok "T1 registry fan got the new pw on stdin" \
+ || bad "T1 registry fan missing"
+grep -q "dw_admin:$SYNTH@" "$SB/root/public-projects/appA/.env" && grep -q "dw_admin:$SYNTH@" "$SB/root/public-projects/appB/config/db.js" \
+ && ok "T1 remote received the REAL new pw and rewrote both DSNs" \
+ || bad "T1 remote DSN rewrite wrong/missing (defect 2: heredoc overrides stdin pipe)"
+grep -q 'other_user:x@' "$SB/root/public-projects/appC/.env" && ok "T1 non-dw_admin DSN untouched" || bad "T1 touched a non-dw_admin DSN"
+[ -n "$(find "$SB/root/public-projects/appA" -name '*.pre-rot.*.bak')" ] && ok "T1 backup written" || bad "T1 no backup written"
+restarts="$(grep '^pm2 restart' "$SB/log/argv.log" || true)"
+if echo "$restarts" | grep -q appA && echo "$restarts" | grep -q appB; then ok "T1 affected apps appA+appB restarted"
+else bad "T1 affected apps NOT restarted (defect 3: CHG passed as argv, not env)"; fi
+echo "$restarts" | grep -qE 'appC|appD|rootcwd' && bad "T1 restarted an unaffected/stopped/root-cwd app" || ok "T1 no unaffected app restarted"
+grep -q '^pm2 start' "$SB/log/argv.log" && bad "T1 started split-brain workers without --start-split-brain" || ok "T1 no unrequested worker starts"
+[ "$rc" = 0 ] && ok "T1 exit 0" || bad "T1 exit $rc"
+
+# ---- T2: rerun must not clobber the first backup --------------------------------
+first_bak="$(find "$SB/root/public-projects/appA" -name '*.pre-rot.*.bak' | head -1)"
+printf 'DATABASE_URL=postgres://dw_admin:%s@127.0.0.1:5432/dw_unified\n' "SECONDold" > "$SB/root/public-projects/appA/.env"
+sleep 1; run_script $'y\ny\ny\n' >/dev/null
+if [ -n "$first_bak" ] && grep -q "dw_admin:$OLD@" "$first_bak"; then ok "T2 original backup preserved on rerun"
+else bad "T2 rerun clobbered the original backup"; fi
+rm -rf "$SB"
+
+# ---- T3: Mac2 ALTER fails -> must stop before touching prod ---------------------
+mk_sandbox
+rc=$(FAKE_PSQL_FAIL=1 run_script $'y\ny\ny\n')
+[ ! -s "$SB/log/ssh.count" ] && ok "T3 no ssh to prod after local ALTER failure" \
+ || bad "T3 continued to prod after local failure (defect 4: no errexit)"
+[ "$rc" != 0 ] && ok "T3 non-zero exit ($rc)" || bad "T3 exited 0 despite failure"
+grep -q "dw_admin:$OLD@" "$SB/root/public-projects/appA/.env" && ok "T3 prod DSNs untouched" || bad "T3 prod DSNs changed"
+rm -rf "$SB"
+
+# ---- T4: dry-run changes nothing -----------------------------------------------
+mk_sandbox
+rc=$(run_script '' --dry-run)
+grep -qE '^(psql|ssh|pm2) ' "$SB/log/argv.log" && bad "T4 dry-run invoked psql/ssh/pm2" || ok "T4 dry-run touched nothing"
+grep -q "$SYNTH" "$SB/log/out" && bad "T4 dry-run printed the password" || ok "T4 dry-run never printed the password"
+rm -rf "$SB"
+
+# ---- T5: pm2 jlist fails mid-fan -> must exit non-zero, never a false-green DONE
+mk_sandbox
+rc=$(FAKE_PM2_JLIST_FAIL=1 run_script $'y\ny\ny\n')
+[ "$rc" != 0 ] && ok "T5 non-zero exit when pm2 jlist fails ($rc)" || bad "T5 exit 0 with pm2 jlist dead (silent zero-restart)"
+grep -q '^DONE' "$SB/log/out" && bad "T5 printed DONE despite no restart targets" || ok "T5 no false-green DONE"
+grep -q 'PARTIAL ROTATION' "$SB/log/out" && ok "T5 partial-rotation state reported" || bad "T5 no partial-rotation warning"
+rm -rf "$SB"
+
+echo "result: $PASS pass, $FAIL fail"
+[ "$FAIL" = 0 ]
← 576e596 auto-data-snapshot: 2026-09-25T11:52:49 (1 data files) — reg
·
back to Secrets Manager
·
rotate-dw-admin-full.sh: client-side SCRAM-SHA-256 so plaint eb703f4 →