`TK-10970-kamatera-pm2-fracture-memo.md`

33 lines
# TK-10970 — Kamatera pm2-fracture fix (GATED: remote deploy + remote process sweep)
Agent: vp-operations  Date: 2026-08-31

## Kamatera DOES fracture. 3 God daemons on /root/.pm2:
- 1148923 CANONICAL (65 sock fds, 425 children) DO NOT KILL
- 1122414 (Aug11) ORPHAN (0 sock, 0 kids) sweepable
- 319572 (Jun22) SEMI-ORPHAN (2 sock fds, 0 kids) reconfirm before sweep

## Unguarded pm2 callers feeding it:
pm2-fleet-watchdog.sh (cron */5min), pm2-daily-fleet.sh (5am), service-monitor.service (systemd),
crawl-microsites.js (has pm2 jlist, git head 55ac9e4, NO TK-10970 guard)

## GATED actions (all reversible):
1. Remote kill orphan: ssh root@45.61.58.125 'kill 1122414' (after live reconfirm sockets=0 kids=0). Undo: none (respawns). Verify: God count -> 1.
2. Guard crawl-microsites.js jlist w/ pm2DaemonReachable() rpc.sock-precheck (mirror Mac2 89c7ae5). scp up. Undo: git revert on Kamatera.
3. Guard pm2-fleet-watchdog.sh: prepend `[ -S /root/.pm2/rpc.sock ] && lsof -nP /root/.pm2/rpc.sock >/dev/null 2>&1 || exit 0`. Undo: .bak restore.
No spend/DNS/identity/customer-facing. Gated only for remote deploy + remote kill.

## Executed 2026-08-31 — Codex verification

- Fresh state invalidated the stale kill plan: PID `1122414` was already absent.
- PID `319572` is a valid, separate daemon for `/home/ralph/.pm2`; it was preserved.
- PID `1148923` remains the sole `/root/.pm2` daemon and owns both root PM2 sockets.
- Installed `/usr/local/bin/pm2` as a global root-call shim using Linux `flock -w 15`
  and `timeout --signal=KILL 20 /usr/bin/pm2`. This covers watchdog, daily fleet,
  service monitor, and crawler without modifying the dirty crawler worktree.
- SHA-256: `a5a0be4e7f4aafc256e25a3943daac8d6fd30f734bf1f5935b7db40f989e73b6`.
- Proof: 12 concurrent real `jlist` calls completed; forced contention failed closed
  with exit `75`; no new daemon appeared; `pm2.pid=1148923` persisted.
- Actual watchdog run advanced its heartbeat and reported `425` managed / `424` online.

Rollback: remove `/usr/local/bin/pm2` to restore PATH resolution to `/usr/bin/pm2`.