`RUNBOOK-GOLIVE.md`

110 lines
# ca-donations — GATED go-live runbook (Steve-run)

Every command here is **hard-gated** (remote push · external publish · DNS · scheduled-job install).
The public-tier code is **built and verified locally** behind default-off flags (commits `b285cb4` + `80ce7b1`).
Nothing below has been executed. Run each block yourself (paste with the `!` prefix or in a terminal).

Facts: Kamatera `45.61.58.125` · deploy path `/root/public-projects/ca-donations` · app PORT `9926`
· local DB `ca_donations` on `/tmp` socket · DNS `donations.agentabrams.com` is Cloudflare-proxied
(172.67.211.32 / 104.21.93.147) with origin currently Mac2-only.

Row counts at migration time: political_contributions **15.7M** (calaccess 6.95M + fec 8.75M),
charitable_orgs 161K, charitable_grants 40.5K.

---

## 0. Political rollup — EXISTS, refreshes on ingest (TK-10782, done)
The materialized k-anon rollup `political_agg` now EXISTS (schema.sql) and is rebuilt
ATOMICALLY on every political ingest (`scripts/refresh-political-agg.mjs`, called at the
end of ingest-calaccess-political.mjs + ingest-fec-ca.mjs). It bakes the k-anon floor in
(`HAVING count(distinct donor_name) >= 5`), so `/api/political/agg` reads an indexed table
(sub-second, ~0.4s — the old 503 from the 15.7M-row live scan is gone). ~39.4K groups across
recipient/employer/city/jurisdiction; min(distinct_donors) = 5, zero groups < 5.

**Go-live posture (Steve's decision):** aggregate political is served behind the standard
un/pw (`admin` / `DW2024!`), NOT public. The public surface is **charitable-only**.
`/api/political/agg` is NOT in the public allowlist — it returns 401 without creds even
when `PUBLIC_TIER=1`, and 200 (fast) WITH creds, exactly like raw `/api/political`. The
`PUBLIC_POLITICAL_AGG` flag has been retired — the aggregate is simply an authenticated
feature now. There is no "enable political agg publicly" step; do NOT set any such flag.

## 1. DB migration (Mac2 → Kamatera)
```sh
# On Mac2: dump (custom format, compressed). ~ multi-GB given 15.7M rows.
pg_dump -h /tmp -d ca_donations -Fc -f /tmp/ca_donations.dump
ls -lh /tmp/ca_donations.dump
# Copy to Kamatera
scp /tmp/ca_donations.dump root@45.61.58.125:/root/ca_donations.dump
# On Kamatera: create DB + restore
ssh root@45.61.58.125 'createdb ca_donations 2>/dev/null; \
  pg_restore --no-owner --clean --if-exists -d ca_donations /root/ca_donations.dump; \
  psql -d ca_donations -c "SELECT count(*) FROM political_contributions;"'
```

## 2. Deploy the app (uses the standard deploy.sh; adds PUBLIC_TIER for the public origin)
```sh
# From Mac2 project dir — rsync + npm ci + pm2 reload + smoke (deploy.sh reads .deploy.conf)
cd ~/Projects/ca-donations && ~/Projects/_shared/scripts/deploy.sh
# On Kamatera: (re)start the process WITH the public flag. Political raw + agg stay gated.
ssh root@45.61.58.125 'cd /root/public-projects/ca-donations && \
  PUBLIC_TIER=1 PORT=9926 pm2 start server.js --name ca-donations --update-env || \
  (pm2 set ca-donations && PUBLIC_TIER=1 pm2 restart ca-donations --update-env); \
  pm2 save'
# NOTE: PUBLIC_POLITICAL_AGG is retired — political (raw AND aggregate) stays Basic-Auth gated regardless.
# Set BASIC_AUTH_PASS to a fresh value for the gated political tier.
```

## 3. nginx (IP-bound) + webroot cert — per standing rules
```sh
ssh root@45.61.58.125 'cat >/etc/nginx/sites-available/donations.agentabrams.com <<NGINX
server {
  listen 45.61.58.125:80;
  server_name donations.agentabrams.com;
  location /.well-known/acme-challenge/ { root /var/www/certbot; }
  location / { proxy_pass http://127.0.0.1:9926; proxy_set_header Host \$host; proxy_set_header X-Forwarded-For \$remote_addr; }
}
NGINX
ln -sf /etc/nginx/sites-available/donations.agentabrams.com /etc/nginx/sites-enabled/ && nginx -t && systemctl reload nginx'
# Cert (webroot):
ssh root@45.61.58.125 'certbot certonly --webroot -w /var/www/certbot -d donations.agentabrams.com --non-interactive --agree-tos -m info@agentabrams.com'
# then add the listen 443 ssl server block referencing the issued fullchain/privkey and reload nginx.
```

## 4. DNS — point origin at Kamatera (Cloudflare, proxy stays ON)
Cloudflare zone `agentabrams.com`: set/confirm the `donations` A record → `45.61.58.125` (proxied 🟠).
Do via the cloudflare-dns MCP or the dashboard. Verify: `dig +short donations.agentabrams.com` then
`curl -I https://donations.agentabrams.com/healthz` (expect 200) and
`curl -I https://donations.agentabrams.com/api/political` (expect **401** — political stays gated).

## 5. DURABLE freshness fix — daily CAL-ACCESS loader (scheduled-job install = gated)
The freshness canary is armed; the LOADER has no scheduler (that's why it went stale).
Install this launchd job on Mac2 (the box that owns ingest) to run it daily:
```sh
cat > ~/Library/LaunchAgents/com.steve.ca-donations-loader.plist <<'PLIST'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>Label</key><string>com.steve.ca-donations-loader</string>
  <key>ProgramArguments</key><array>
    <string>/bin/zsh</string><string>-lc</string>
    <string>cd ~/Projects/ca-donations && curl -fsSL --retry 2 -o data/raw/dbwebexport.zip https://campaignfinance.cdn.sos.ca.gov/dbwebexport.zip && /opt/homebrew/bin/node scripts/ingest-calaccess-political.mjs >> data/raw/ingest.log 2>&1</string>
  </array>
  <key>StartCalendarInterval</key><dict><key>Hour</key><integer>5</integer><key>Minute</key><integer>15</integer></dict>
  <key>StandardOutPath</key><string>/tmp/ca-donations-loader.out</string>
  <key>StandardErrorPath</key><string>/tmp/ca-donations-loader.err</string>
</dict></plist>
PLIST
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.steve.ca-donations-loader.plist
launchctl list | grep ca-donations-loader
```
(Runs the loader daily at 05:15; the existing `com.steve.ca-donations-freshness` canary already alerts if it silently dies.)

---

### Rollback (if a go-live step misbehaves)
- App code (public tier): `git -C ~/Projects/ca-donations revert 80ce7b1 b285cb4` (returns to fully-gated).
- Political rollup (TK-10782): `git -C ~/Projects/ca-donations revert <sha>` then `psql -h /tmp -d ca_donations -c 'DROP TABLE political_agg'` (the endpoint reverts to the live-scan/503 version).
- Public origin: `ssh root@45.61.58.125 'pm2 restart ca-donations' ` after unsetting PUBLIC_TIER, or `pm2 delete`.
- DNS: repoint the `donations` A record away from 45.61.58.125.
- Loader job: `launchctl bootout gui/$(id -u)/com.steve.ca-donations-loader`.