docs(sessions): resume — outage recovery, pm2 reconcile (TK-10606), TK-10293 gate verified green + Playwright env-fix learnings

06c151b8ed77f4ee1c43e0a0c431f30209287aaf · 2026-08-17 06:53:28 -0700 · Steve Abrams
Files touched

M docs/sessions/learnings.md
M docs/sessions/sessions.md
M docs/sessions/tasks-completed.md
Diff

commit 06c151b8ed77f4ee1c43e0a0c431f30209287aaf
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Mon Aug 17 06:53:28 2026 -0700

    docs(sessions): resume — outage recovery, pm2 reconcile (TK-10606), TK-10293 gate verified green + Playwright env-fix learnings
---
 docs/sessions/learnings.md       | 6 ++++++
 docs/sessions/sessions.md        | 3 +++
 docs/sessions/tasks-completed.md | 6 ++++++
 3 files changed, 15 insertions(+)

diff --git a/docs/sessions/learnings.md b/docs/sessions/learnings.md
index 66d7404..dc2a8e3 100644
--- a/docs/sessions/learnings.md
+++ b/docs/sessions/learnings.md
@@ -197,3 +197,9 @@ Technical learnings, gotchas, and patterns discovered during development. Append
 - `NextRequest.ip` is NOT stable across Next 16 runtimes — read `x-forwarded-for` (first IP) → `x-real-ip` → 'unknown'. The route trusts the XFF header because nginx terminates TLS and sets it; if Norma ever moves off nginx-fronted, revisit.
 - nginx vhost mismatch caught during deploy: `sdcc.agentabrams.com` actually proxies 127.0.0.1:9720 (Express `norma-sdcc-pitch`), NOT norma-email :7400 (Next.js). The public hostname is currently NOT a path to the Next.js login route. Smoke test ran against `http://127.0.0.1:7400` directly on prod via ssh.
 - Verified 10/15min/30min behavior: fresh restart → 11 bad POSTs from one IP → 10×401, 11th=429 with `Retry-After: 1800` and body `{"error":"too many attempts","retryAfter":1800}`. The "10 attempts came back 429" rounding seen on a NOT-freshly-restarted process is because pm2 had ambient activity that pre-incremented the counter — restart-then-test gives the spec-correct boundary.
+
+### Pre-deploy gate: Playwright env failures are NOT code regressions (TK-10293 verify, 2026-08-17)
+- `npm run gate` correctly fails CLOSED on e2e even when the app is fine: the two most common causes are (1) the Playwright browser binary version the tests want isn't installed (error `Executable doesn't exist at .../chromium_headless_shell-<N>` → `npx playwright install chromium`), and (2) a **stale `~/Library/Caches/ms-playwright/__dirlock`** left by an interrupted/parallel `playwright install` that makes a *subsequent* install silently no-op (exit 0, nothing downloaded). Symptom of #2: install "succeeds" but `ls ~/Library/Caches/ms-playwright/ | grep chromium` still shows only the OLD version.
+- Fix order: `ps aux | grep "[p]laywright install"` to confirm no LIVE install holds the lock → `rm -rf ~/Library/Caches/ms-playwright/__dirlock` → `npx playwright install chromium`. Only remove the lock after confirming nothing is actively installing (another agent session on this Mac can legitimately hold it).
+- The 5 non-browser suites (api-smoke / security-regression / write-smoke / crud-lifecycle / authz-matrix) run first and independently, so a red e2e with green everything-else = environment, not app. Full green today: api 20, write 35, crud 35, authz 12, e2e 9.
+- `npm run reload` = fast emergency recovery (build→restart→:7400 health-gate, no suite). `npm run deploy:local` = ship NEW code (build→`npm run gate` full battery→reload). Gate instance is isolated on :7414 / sdcc_test with sends sinked — safe to run anytime, never touches live :7400 or the sdcc DB.
diff --git a/docs/sessions/sessions.md b/docs/sessions/sessions.md
index b979069..9f88442 100644
--- a/docs/sessions/sessions.md
+++ b/docs/sessions/sessions.md
@@ -381,3 +381,6 @@ Resumed the stalled `dw-instagram` ticket TK-10573 (DOING 24h, ticketmaster-nudg
 **Remaining = 2 Steve-gated steps** (drafted to `~/.claude/yolo-queue/pending-approval/TK-10573-ig-redo-finish-20260815.md`), neither auto-executed:
 1. Repost the 1 failed post: `node post-to.js metallicwallpaper --product dwqw-58467-handle --confirm`
 2. openclaw-delete the 84 safe originals (HOLD card excluded until step 1 runs).
+
+## 2026-08-16 → 08-17 — Resume: outage recovery, pm2 reconcile, TK-10293 gate verified green
+Resumed after compaction. Recovered a second Norma :7400 outage (norma-email gone from pm2 after a reboot fractured the daemon into 4 Gods). Handed pm2 fleet reconciliation to vp-operations (TK-10606, now done: 1 God daemon, 126 apps, recurrence fixed, fracture-canary loaded+PASS) and the GA4 store grant to analytics-agent (TK-10200, blocked on Steve's browser action). Then verified the TK-10293 deliverable: full `npm run gate` battery green end-to-end (api 20 / security / write 35 / crud 35 / authz 12 / e2e 9). Along the way fixed a missing Playwright chromium binary + a stale ms-playwright `__dirlock` that was silently no-op'ing reinstalls. Everything in the Norma lane green; nothing pending on our end except Steve's GA4 grant.
diff --git a/docs/sessions/tasks-completed.md b/docs/sessions/tasks-completed.md
index 441618c..cbc8ebe 100644
--- a/docs/sessions/tasks-completed.md
+++ b/docs/sessions/tasks-completed.md
@@ -133,3 +133,9 @@ Tasks fully implemented and deployed.
 
 ### Request #58: Commit + push everything
 - 11 fresh commits today, all local. No remote per `feedback_never_push_to_remotes.md`. Prod files bit-for-bit match HEAD across all spot-checked paths.
+
+### Resume session (2026-08-16 → 08-17): outage recovery + fleet reconcile + gate verify
+- **Norma :7400 outage (pm2 fracture)** — norma-email had vanished from pm2 after a reboot spawned 4 God daemons (dual launchd resurrect race). Recovered norma-email safely (port free, no dup), verified :7400/login→200.
+- **TK-10606 pm2 fracture** — handed to vp-operations; reconciled to 1 canonical God daemon. Verified independently: 126 apps (125 online + `usre-broker-rotate` correctly stopped 4:15am cron one-shot), dump backed up, recurrence fixed (redundant `com.steve.pm2-resurrect.plist`→`.disabled-TK10606`), `com.steve.pm2-fracture-canary` loaded + PASS. Closed by claude-mail under Steve's "All Approved".
+- **TK-10200 GA4 grant** — handed to analytics-agent; dashboard verified live (ETL 150/150 props 0 err), SA email confirmed (`analytics-agent@dw-ga4-attribution.iam.gserviceaccount.com`), copy-paste grant steps refreshed in pending-approval. Stays blocked on Steve's one browser action.
+- **TK-10293 durable gate — VERIFIED GREEN** — ran full `npm run gate`: all 6 suites pass end-to-end. Fixed a missing Playwright chromium binary + a stale `__dirlock` (see learnings). Deliverable confirmed live-green.
← acd4004 ig(TK-10573): GATE2 fan-out — split 85 originals into 10 bat · back to Norma Platform · IG auto-post viewer with inline delete (GET /posts on :9810) 8529d5a →