← back to Secrets Manager
fix secrets fan: top-level routes invisible (22) + dw_admin never rewrote DATABASE_URL
41aafea01a21865125c3a246ab3a7b64e86d20f2 · 2026-06-09 14:11:11 -0700 · SteveStudio2
Root cause of the 2026-06-03 dw_admin rotation half-fire — three stacked bugs,
all fixed in cli.js with ZERO routes.json churn:
1. cli.js read only ROUTES.services[key], but ~22 routes (PG_DW_ADMIN_PASSWORD,
SHOPIFY_ORDERS_TOKEN, TWILIO_*, GOOGLE_DRIVE_*, ANTHROPIC_*…) live at routes.json
top-level -> invisible -> fanned 0 destinations. Fix: routeFor() top-level fallback.
2. fanOut had no env_file branch (dw_admin dests' type) -> silently skipped. Added it.
3. fan set a bare *_PASSWORD var, but dw_admin consumers read a full DATABASE_URL DSN.
Fix: rewriteDsnPassword() + DSN_REWRITE map rewrites the pw inside DATABASE_URL
for dw_admin (URL-encoding + / =).
Verified: unit + full-branch temp-file tests; route visible (8 dests); routes.json
byte-identical to HEAD. Corrected ROTATION-PRESTAGE.md (real step 3 + prod 3b SSH
step + revert guard). Prod DATABASE_URL still manual (3b).
Files touched
M ROTATION-PRESTAGE.mdM cli.js
Diff
commit 41aafea01a21865125c3a246ab3a7b64e86d20f2
Author: SteveStudio2 <stevestudio2@SteveStacStudio.lan>
Date: Tue Jun 9 14:11:11 2026 -0700
fix secrets fan: top-level routes invisible (22) + dw_admin never rewrote DATABASE_URL
Root cause of the 2026-06-03 dw_admin rotation half-fire — three stacked bugs,
all fixed in cli.js with ZERO routes.json churn:
1. cli.js read only ROUTES.services[key], but ~22 routes (PG_DW_ADMIN_PASSWORD,
SHOPIFY_ORDERS_TOKEN, TWILIO_*, GOOGLE_DRIVE_*, ANTHROPIC_*…) live at routes.json
top-level -> invisible -> fanned 0 destinations. Fix: routeFor() top-level fallback.
2. fanOut had no env_file branch (dw_admin dests' type) -> silently skipped. Added it.
3. fan set a bare *_PASSWORD var, but dw_admin consumers read a full DATABASE_URL DSN.
Fix: rewriteDsnPassword() + DSN_REWRITE map rewrites the pw inside DATABASE_URL
for dw_admin (URL-encoding + / =).
Verified: unit + full-branch temp-file tests; route visible (8 dests); routes.json
byte-identical to HEAD. Corrected ROTATION-PRESTAGE.md (real step 3 + prod 3b SSH
step + revert guard). Prod DATABASE_URL still manual (3b).
---
ROTATION-PRESTAGE.md | 115 ++++++++++++++++++++++++++++++++++-----------------
cli.js | 64 ++++++++++++++++++++++++++--
2 files changed, 138 insertions(+), 41 deletions(-)
diff --git a/ROTATION-PRESTAGE.md b/ROTATION-PRESTAGE.md
index de3cbf5..ee40816 100644
--- a/ROTATION-PRESTAGE.md
+++ b/ROTATION-PRESTAGE.md
@@ -1,49 +1,88 @@
-# dw_admin Rotation — Pre-Stage (ready for Steve's console window)
+# dw_admin Rotation — Pre-Stage (corrected 2026-06-09)
-> Prepared 2026-06-01. Goal: shrink the hands-on window to ~10 min so Kamatera
-> can be cleared for the 65-site DW fleet rebuild deploy. **No secret VALUES here.**
-> Claude has done all the code prep; the steps below are the console-only actions
-> only Steve can run (mint pw, ALTER ROLE, restart prod).
+> **No secret VALUES here.** Console-only steps are Steve's; Claude runs the
+> no-secret parts (local fan + verify). Goal: a tight, uninterrupted window so
+> the 2026-06-03 half-fire does NOT repeat.
---
-## ✅ Code prep DONE (by Claude, on branches — not deployed)
+## ⚠️ Why the last attempt (2026-06-03) HALF-FIRED — now fixed
-- `archive-agent/ecosystem.config.js` — DSN already env-first (no pw literal). ✔
-- `Designer-Wallcoverings/DW-Agents/ecosystem.config.js:134` — **fixed → env-first**
- (`process.env.DATABASE_URL || '…dw_admin@127.0.0.1…'`), committed on `rotation-prestage`.
- → both inline-DSN gotchas from the runbook are resolved; a restart now picks up
- the rotated pw from `DATABASE_URL` env with **no further config edits**.
-- gitleaks pre-commit hooks + secret-strip already removed the cleartext pw from
- tracked source fleet-wide (origin incident).
+Pre-flight on 2026-06-09 found the route-back fan was a **silent no-op** for
+dw_admin — three stacked bugs:
-## Console steps (Steve — ~10 min)
+1. The `PG_DW_ADMIN_PASSWORD` route lived at routes.json **top-level**, but
+ `cli.js` only reads `ROUTES.services[key]` → the route was invisible → the
+ fan wrote **only** master+desktop and skipped all 8 destinations.
+2. Even when found, the destinations are `type:"env_file"` — and `fanOut` had
+ **no `env_file` branch** → silently skipped.
+3. Even if written, the fan set a **bare `*_PASSWORD` var**, but every consumer
+ connects via a full **`DATABASE_URL` DSN** (pw embedded inline). The bare var
+ is read by nothing. So `ALTER` changed the real pw, the fan "succeeded," and
+ every app restarted still holding the OLD pw in `DATABASE_URL` → FATAL auth.
-1. **Mint** a new ≥20-char password (1Password / `openssl rand -base64 24`).
+**Fixed (commit on this repo, 2026-06-09) — all in cli.js, routes.json UNCHANGED:**
+- `routeFor(key)` now falls back to top-level routes, not just `.services` — so
+ `PG_DW_ADMIN_PASSWORD` (and ~21 OTHER invisible top-level routes: SHOPIFY_ORDERS_TOKEN,
+ TWILIO_*, GOOGLE_DRIVE_*, the ANTHROPIC_* keys…) actually fan.
+- `fanOut` gained an `env_file` branch (the dw_admin dests' type — previously unhandled).
+- New `rewriteDsnPassword()` + a code-driven `DSN_REWRITE` map (`PG_DW_ADMIN_PASSWORD →
+ {dsn:DATABASE_URL, user:dw_admin}`): the fan now **surgically rewrites the password
+ inside `DATABASE_URL`** for `dw_admin` in each of the 8 dests (URL-encoding `+ / =`),
+ in addition to setting the bare var.
+- Proven on temp .env: DSN rewritten+encoded, comments+other keys preserved, route
+ visible (8 dests), routes.json byte-identical to HEAD.
+
+**Still NOT auto-covered: prod.** The 8 fan dests are all **local Mac2**. The
+~40 Kamatera apps read their OWN `DATABASE_URL` (pm2 env). The fan does not
+touch prod — see step 3b, which is manual + SSH (Steve).
+
+---
+
+## Console steps — keep 2 → 3 → 4 ONE uninterrupted block
+
+**Before you start:** open a second shell already `ssh root@45.61.58.125` and
+locate prod's DATABASE_URL source so 3b is instant:
+```
+ssh root@45.61.58.125 "grep -rl 'dw_admin:' /root/.pm2/dump.pm2 /root/.env /root/**/ecosystem.config.js 2>/dev/null | head"
+```
+Keep the OLD pw recovery line staged (revert target lives in `/root/.pm2/dump.pm2`).
+
+1. **Mint** a URL-safe ≥20-char pw (avoid raw `+ / =` for the manual prod edit):
+ `openssl rand -hex 24` (or `openssl rand -base64 24 | tr -d '+/='`)
2. **ALTER ROLE on BOTH** databases:
- - **Prod Kamatera:** `ssh root@45.61.58.125` → `sudo -u postgres psql -c "ALTER ROLE dw_admin WITH PASSWORD '<NEW>';"`
- - **Local Mac2:** `psql -U postgres -c "ALTER ROLE dw_admin WITH PASSWORD '<NEW>';"`
-3. **Route back** (fans the new pw to every registered .env + MCP env):
- ```
- printf 'DW_ADMIN_DB_PASSWORD=<NEW>' | node ~/Projects/secrets-manager/cli.js import-paste
- ```
-4. **Restart consumers** so they reconnect with the new pw:
- - **Local Mac2:** `pm2 restart wallco-ai-9905` (the local PG consumer); others (run-notes-chat / claude-control-center / morning-review) don't hit dw_admin.
- - **Kamatera:** `ssh root@45.61.58.125 'pm2 restart all --update-env'` — the ~40 dw_admin pm2 procs (DW-Agents ×14, Norma ×8, pd-* ×7, site-factory, directories, dw-commerce hub). `--update-env` is required so they re-read the rotated `DATABASE_URL`.
-5. **Verify:**
- - `node ~/Projects/secrets-manager/cli.js check` (validates the new pw connects)
+ - Prod: `ssh root@45.61.58.125 'sudo -u postgres psql -c "ALTER ROLE dw_admin WITH PASSWORD '\''<NEW>'\'';"'`
+ - Local Mac2: `psql -U postgres -c "ALTER ROLE dw_admin WITH PASSWORD '<NEW>';"`
+3. **Route back the new pw:**
+ - **3a — Local (Claude can run this):**
+ `printf 'PG_DW_ADMIN_PASSWORD=<NEW>' | node ~/Projects/secrets-manager/cli.js import-paste`
+ → rewrites `DATABASE_URL` in all 8 local dests + sets the bare var. Confirm
+ the output shows `[DSN DATABASE_URL×1]` for each (NOT `⚠ NOT FOUND`).
+ - **3b — Prod (Steve, SSH):** update the prod `DATABASE_URL` source found
+ above with the new pw inline, e.g. if it's an env/ecosystem file:
+ `ssh root@45.61.58.125` → edit so `DATABASE_URL=postgresql://dw_admin:<NEW>@127.0.0.1:5432/dw_unified...`
+ (URL-encode any `+ / =`). If the live value lives only in `dump.pm2`, the
+ reliable refresh is `pm2 delete all && pm2 start <ecosystem> && pm2 save`
+ after the ecosystem file carries `process.env.DATABASE_URL` (already
+ env-first on the rotation-prestage branch).
+4. **Restart consumers:**
+ - Local: `pm2 restart wallco-ai-9905`
+ - Prod: `ssh root@45.61.58.125 'pm2 restart all --update-env'` (or the
+ delete+start from 3b if env didn't refresh).
+5. **Verify (Claude can run):**
+ - `node ~/Projects/secrets-manager/cli.js check`
- `ssh root@45.61.58.125 'pm2 ls'` → all `online`, no restart loops
- - spot-check one prod app responds (e.g. `curl -s -o /dev/null -w "%{http_code}" https://designerwallcoverings.com`)
-
-## After rotation → Kamatera is clear
-- Flip the cncp dashboard prod flag off (`server.js:1718` `status:'compromised'` → `ok`).
-- Then the DW fleet rebuild can deploy (canary `1890swallpaper` → parallel the other 64).
+ - `curl -s -o /dev/null -w "%{http_code}" https://designerwallcoverings.com`
+ - **If any dw_admin app is FATAL → REVERT immediately:** re-ALTER prod to the
+ OLD pw, `pm2 restart all --update-env`, and stop to debug. Do NOT leave it
+ half-applied.
-## Still outstanding (separate from dw_admin, but same incident)
-- Gemini API key #1 (7 Patty routes + 3 DW docs) — mint new at aistudio.google.com, delete old, route back.
-- Gemini API key #2 (`PoppyPetitions/lib/gemini.ts`) — same.
-- Norma scoped-user rotation (DTD-cleared, local-only) — does NOT fix dw_admin; optional now.
+## Naming note
+The route key + fan key is **`PG_DW_ADMIN_PASSWORD`** (the one cli.js reads).
+secrets `.env` also carries a stray `DW_ADMIN_PG_PASSWORD` and the old doc
+referenced `DW_ADMIN_DB_PASSWORD` — both dead aliases; always fan
+`PG_DW_ADMIN_PASSWORD`.
-## What Claude can do post-rotation (no secrets)
-- Run step 3's route-back + step 5 verification.
-- Drive the canary + parallel deploy once you flip the prod flag.
+## After rotation → Kamatera clear
+- Flip cncp prod flag (`server.js` `status:'compromised'` → `ok`).
+- Outstanding (same incident, separate creds): Gemini key #1, key #2.
diff --git a/cli.js b/cli.js
index b277e7d..be1b6a1 100755
--- a/cli.js
+++ b/cli.js
@@ -22,6 +22,19 @@ const { spawnSync } = require('child_process');
const ROOT = path.dirname(__filename);
const HOME = os.homedir();
const ROUTES = JSON.parse(fs.readFileSync(path.join(ROOT, 'routes.json'), 'utf8'));
+// routes.json carries some routes under .services AND some at top-level (legacy).
+// cli.js historically read ONLY .services[key], so ~22 top-level routes
+// (PG_DW_ADMIN_PASSWORD, SHOPIFY_ORDERS_TOKEN, TWILIO_*, GOOGLE_DRIVE_*, the
+// ANTHROPIC_* keys…) were INVISIBLE to the fan — fanning them wrote only
+// master+desktop and silently skipped every destination. routeFor() falls back
+// to the top-level entry so all routes fan. (Root cause of the 2026-06-03
+// dw_admin rotation half-fire.) Prefer .services on name clash.
+function routeFor(key) { return (ROUTES.services && ROUTES.services[key]) || ROUTES[key] || null; }
+// DB-password keys whose consumers connect via a full DSN, not a bare var — the
+// fan must rewrite the password INSIDE that DSN in each env_file dest.
+const DSN_REWRITE = {
+ PG_DW_ADMIN_PASSWORD: { dsn: 'DATABASE_URL', user: 'dw_admin' },
+};
const MASTER_ENV = path.join(ROOT, '.env');
const DESKTOP_ENV = path.join(HOME, 'Desktop', 'site-factory.env');
const REGISTRY = path.join(ROOT, 'registry.json');
@@ -70,6 +83,27 @@ function writeEnvFile(p, kv, preserveComments = true) {
fs.writeFileSync(p, body);
try { fs.chmodSync(p, 0o600); } catch {}
}
+// Rewrite the password segment of a DSN var (e.g. DATABASE_URL=postgres://user:PW@host)
+// inside an env-file body, for the given DB user. The new pw is URL-encoded so base64
+// chars (+ / =) don't corrupt the URL userinfo. Returns {body, changed}.
+//
+// WHY this exists: dw_admin consumers connect via a full DATABASE_URL DSN, NOT a bare
+// *_PASSWORD var. A fan that only sets the bare var leaves DATABASE_URL holding the OLD
+// pw → restart → FATAL auth. That was the 2026-06-03 rotation half-fire. The pg-rotation
+// route now carries `dsn: "DATABASE_URL"` so the fan rewrites the live DSN too.
+function rewriteDsnPassword(text, dsnVar, dbUser, newPw) {
+ if (!text) return { body: text, changed: 0 };
+ const enc = encodeURIComponent(newPw);
+ let changed = 0;
+ const re = new RegExp('^(\\s*' + dsnVar + '\\s*=\\s*["\']?)(postgres(?:ql)?:\\/\\/' + dbUser + ':)([^@]*)(@)', 'i');
+ const body = text.split('\n').map((line) => {
+ const m = line.match(re);
+ if (m) { changed++; return m[1] + m[2] + enc + m[4] + line.slice(m[0].length); }
+ return line;
+ }).join('\n');
+ return { body, changed };
+}
+
// Remote env via SSH. Hosts must have SSH key auth set up; we never prompt
// for a password (BatchMode=yes). Connect timeout caps stalls at 8s.
function readRemoteEnvFile(user, host, p) {
@@ -141,7 +175,7 @@ function httpsReq(url, opts = {}) {
// ─── verify ─────────────────────────────────────────────────────────────
async function verifyToken(key, value) {
- const cfg = ROUTES.services[key]?.verify;
+ const cfg = routeFor(key)?.verify;
if (!cfg) return { ok: true, skipped: true };
const headers = {};
if (cfg.auth === 'bearer') headers['Authorization'] = `Bearer ${value}`;
@@ -173,7 +207,7 @@ async function verifyToken(key, value) {
// ─── fan-out ────────────────────────────────────────────────────────────
function fanOut(key, value) {
- const destinations = ROUTES.services[key]?.destinations || [];
+ const destinations = routeFor(key)?.destinations || [];
const written = [];
// Master always
@@ -205,6 +239,30 @@ function fanOut(key, value) {
const cur = loadEnvFile(p); cur[key] = value;
writeEnvFile(p, cur, true); written.push(p);
}
+ } else if (d.type === 'env_file') {
+ // Local env file. Sets the bare key, AND — when this key feeds a DSN
+ // (per-dest `d.dsn`, or the code-driven DSN_REWRITE map) — surgically
+ // rewrites the password inside that DSN var (e.g. DATABASE_URL) for the
+ // DB user. The dw_admin rotation path NEEDS this: consumers read
+ // DATABASE_URL, not the bare *_PASSWORD var. (Pre-2026-06-09 the fan had
+ // no env_file branch at all → these dests were silently skipped.)
+ const p = expand(d.path);
+ if (!fs.existsSync(p)) { console.error(` ⚠ env_file skip (missing): ${p}`); continue; }
+ let text = fs.readFileSync(p, 'utf8');
+ let note = '';
+ const dsnSpec = d.dsn ? { dsn: d.dsn, user: d.dsnUser || 'dw_admin' } : DSN_REWRITE[key];
+ if (dsnSpec) {
+ const { body, changed } = rewriteDsnPassword(text, dsnSpec.dsn, dsnSpec.user, value);
+ text = body;
+ note = changed ? ` [DSN ${dsnSpec.dsn}×${changed}]` : ` [⚠ DSN ${dsnSpec.dsn} NOT FOUND]`;
+ }
+ // serializeEnvBody only rewrites keys present in the kv map, so the
+ // already-rewritten DSN line is preserved verbatim (not re-parsed).
+ const finalBody = serializeEnvBody(text, d.dsnOnly ? {} : { [key]: value });
+ fs.mkdirSync(path.dirname(p), { recursive: true });
+ fs.writeFileSync(p, finalBody);
+ try { fs.chmodSync(p, 0o600); } catch {}
+ written.push(p + note);
} else if (d.type === 'skill') {
const p = path.join(HOME, '.claude/skills', d.name, '.env');
const cur = loadEnvFile(p); cur[key] = value;
@@ -246,7 +304,7 @@ async function cmdAdd(key, value) {
const r = loadRegistry();
r.secrets[key] = {
digest: digest(value),
- label: ROUTES.services[key]?.label || key,
+ label: routeFor(key)?.label || key,
last_updated: new Date().toISOString(),
validated: !v.skipped,
verify_status: v.status || null,
← 8b0e8f0 add rotate-dw-admin-full.sh — complete the forward dw_admin
·
back to Secrets Manager
·
fan-out: honor per-destination d.key remap in env_file branc 93d8391 →