← back to Db Saturation Fix

README.md

21 lines

# Kamatera Postgres saturation — fix kit

**Diagnosis (2026-07-20):** `dw_unified` Postgres on 45.61.58.125 is at **500/500 connections**
(max_connections=500). 486 held by `dw_admin`, 471 actively running SELECT, only 1
idle-in-transaction. Cause = **268 pm2 apps sharing one Postgres**, each with its own pool,
collectively exhausting the ceiling. Crash-loopers (`momentum-crawler` 4023 restarts/errored,
`koroseal-quote-api` 2225) churn connections on top.

Everything here is for **Steve to run** — read the safety notes, nothing auto-executes.

## Order of operations
1. `bash 1-immediate-relief.sh`   → stop the *errored* crash-looper(s) to free slots NOW (reversible).
2. `bash 2-audit-pools.sh`        → READ-ONLY: find apps with big/unbounded `pg` pools.
3. Install PgBouncer with `pgbouncer.ini` (transaction pooling) → the durable fix.
4. Repoint each app's `PGHOST`/port at PgBouncer (6432), lower per-app pool `max` to 2–3.

## Why PgBouncer
With 268 apps you cannot give each a raw pool and stay under 500. PgBouncer (transaction mode)
multiplexes hundreds of client connections onto a small server pool (e.g. 40), so the fleet
scales without touching max_connections (each raw slot ~10MB RAM).