ca-donations: gated go-live runbook (Kamatera migration + nginx/cert + DNS + daily-loader launchd) — Steve-run

15aa12876cc86cfb2c45528f4668574fedb78672 · 2026-08-24 12:03:17 -0700 · Steve
Files touched

A RUNBOOK-GOLIVE.md
Diff

commit 15aa12876cc86cfb2c45528f4668574fedb78672
Author: Steve <steve@designerwallcoverings.com>
Date:   Mon Aug 24 12:03:17 2026 -0700

    ca-donations: gated go-live runbook (Kamatera migration + nginx/cert + DNS + daily-loader launchd) — Steve-run
---
 RUNBOOK-GOLIVE.md | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 98 insertions(+)

diff --git a/RUNBOOK-GOLIVE.md b/RUNBOOK-GOLIVE.md
new file mode 100644
index 0000000..b40dc6d
--- /dev/null
+++ b/RUNBOOK-GOLIVE.md
@@ -0,0 +1,98 @@
+# ca-donations — GATED go-live runbook (Steve-run)
+
+Every command here is **hard-gated** (remote push · external publish · DNS · scheduled-job install).
+The public-tier code is **built and verified locally** behind default-off flags (commits `b285cb4` + `80ce7b1`).
+Nothing below has been executed. Run each block yourself (paste with the `!` prefix or in a terminal).
+
+Facts: Kamatera `45.61.58.125` · deploy path `/root/public-projects/ca-donations` · app PORT `9926`
+· local DB `ca_donations` on `/tmp` socket · DNS `donations.agentabrams.com` is Cloudflare-proxied
+(172.67.211.32 / 104.21.93.147) with origin currently Mac2-only.
+
+Row counts at migration time: political_contributions **15.7M** (calaccess 6.95M + fec 8.75M),
+charitable_orgs 161K, charitable_grants 40.5K.
+
+---
+
+## 0. PRE-FLIGHT — materialize the political rollup BEFORE any public-political flag
+`PUBLIC_POLITICAL_AGG` stays OFF at go-live (charitable-only public per the conditioned plan).
+Do NOT enable it until this exists — the live k-anon agg is a 15.7M-row full scan (~174s, guarded to 503).
+When/if you want aggregate political public, build a materialized rollup refreshed on ingest first
+(TK-10782 follow-up). Charitable go-live does NOT need this.
+
+## 1. DB migration (Mac2 → Kamatera)
+```sh
+# On Mac2: dump (custom format, compressed). ~ multi-GB given 15.7M rows.
+pg_dump -h /tmp -d ca_donations -Fc -f /tmp/ca_donations.dump
+ls -lh /tmp/ca_donations.dump
+# Copy to Kamatera
+scp /tmp/ca_donations.dump root@45.61.58.125:/root/ca_donations.dump
+# On Kamatera: create DB + restore
+ssh root@45.61.58.125 'createdb ca_donations 2>/dev/null; \
+  pg_restore --no-owner --clean --if-exists -d ca_donations /root/ca_donations.dump; \
+  psql -d ca_donations -c "SELECT count(*) FROM political_contributions;"'
+```
+
+## 2. Deploy the app (uses the standard deploy.sh; adds PUBLIC_TIER for the public origin)
+```sh
+# From Mac2 project dir — rsync + npm ci + pm2 reload + smoke (deploy.sh reads .deploy.conf)
+cd ~/Projects/ca-donations && ~/Projects/_shared/scripts/deploy.sh
+# On Kamatera: (re)start the process WITH the public flag. Political raw + agg stay gated.
+ssh root@45.61.58.125 'cd /root/public-projects/ca-donations && \
+  PUBLIC_TIER=1 PORT=9926 pm2 start server.js --name ca-donations --update-env || \
+  (pm2 set ca-donations && PUBLIC_TIER=1 pm2 restart ca-donations --update-env); \
+  pm2 save'
+# NOTE: leave PUBLIC_POLITICAL_AGG UNSET (off). Set BASIC_AUTH_PASS to a fresh value for the still-gated political tier.
+```
+
+## 3. nginx (IP-bound) + webroot cert — per standing rules
+```sh
+ssh root@45.61.58.125 'cat >/etc/nginx/sites-available/donations.agentabrams.com <<NGINX
+server {
+  listen 45.61.58.125:80;
+  server_name donations.agentabrams.com;
+  location /.well-known/acme-challenge/ { root /var/www/certbot; }
+  location / { proxy_pass http://127.0.0.1:9926; proxy_set_header Host \$host; proxy_set_header X-Forwarded-For \$remote_addr; }
+}
+NGINX
+ln -sf /etc/nginx/sites-available/donations.agentabrams.com /etc/nginx/sites-enabled/ && nginx -t && systemctl reload nginx'
+# Cert (webroot):
+ssh root@45.61.58.125 'certbot certonly --webroot -w /var/www/certbot -d donations.agentabrams.com --non-interactive --agree-tos -m info@agentabrams.com'
+# then add the listen 443 ssl server block referencing the issued fullchain/privkey and reload nginx.
+```
+
+## 4. DNS — point origin at Kamatera (Cloudflare, proxy stays ON)
+Cloudflare zone `agentabrams.com`: set/confirm the `donations` A record → `45.61.58.125` (proxied 🟠).
+Do via the cloudflare-dns MCP or the dashboard. Verify: `dig +short donations.agentabrams.com` then
+`curl -I https://donations.agentabrams.com/healthz` (expect 200) and
+`curl -I https://donations.agentabrams.com/api/political` (expect **401** — political stays gated).
+
+## 5. DURABLE freshness fix — daily CAL-ACCESS loader (scheduled-job install = gated)
+The freshness canary is armed; the LOADER has no scheduler (that's why it went stale).
+Install this launchd job on Mac2 (the box that owns ingest) to run it daily:
+```sh
+cat > ~/Library/LaunchAgents/com.steve.ca-donations-loader.plist <<'PLIST'
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0"><dict>
+  <key>Label</key><string>com.steve.ca-donations-loader</string>
+  <key>ProgramArguments</key><array>
+    <string>/bin/zsh</string><string>-lc</string>
+    <string>cd ~/Projects/ca-donations && curl -fsSL --retry 2 -o data/raw/dbwebexport.zip https://campaignfinance.cdn.sos.ca.gov/dbwebexport.zip && /opt/homebrew/bin/node scripts/ingest-calaccess-political.mjs >> data/raw/ingest.log 2>&1</string>
+  </array>
+  <key>StartCalendarInterval</key><dict><key>Hour</key><integer>5</integer><key>Minute</key><integer>15</integer></dict>
+  <key>StandardOutPath</key><string>/tmp/ca-donations-loader.out</string>
+  <key>StandardErrorPath</key><string>/tmp/ca-donations-loader.err</string>
+</dict></plist>
+PLIST
+launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.steve.ca-donations-loader.plist
+launchctl list | grep ca-donations-loader
+```
+(Runs the loader daily at 05:15; the existing `com.steve.ca-donations-freshness` canary already alerts if it silently dies.)
+
+---
+
+### Rollback (if a go-live step misbehaves)
+- App code: `git -C ~/Projects/ca-donations revert 80ce7b1 b285cb4` (returns to fully-gated).
+- Public origin: `ssh root@45.61.58.125 'pm2 restart ca-donations' ` after unsetting PUBLIC_TIER, or `pm2 delete`.
+- DNS: repoint the `donations` A record away from 45.61.58.125.
+- Loader job: `launchctl bootout gui/$(id -u)/com.steve.ca-donations-loader`.
← 80ce7b1 ca-donations: harden public gate — strict positive allowlist · back to Ca Donations · ca-donations: materialized k-anon political rollup (politica e807f8e →