← back to NationalPaperHangers
DEPLOY_KAMATERA.md
778 lines
# National Paper Hangers — Kamatera Deployment Runbook
**Status: READY TO EXECUTE — awaiting Steve's green-light.**
**Hard gate on Step H:** DNS cutover requires a new Cloudflare token with Zone:Edit permission.
The active CF token is DNS-edit only and cannot create zones.
- Source: Mac2 `/Users/stevestudio2/Projects/NationalPaperHangers/` · pm2 `national-paper-hangers` · port 9765
- Target: Kamatera VPS `45.61.58.125` · `/root/Projects/NationalPaperHangers/` · same port 9765
- Domain: `nationalpaperhangers.com` (GoDaddy, currently ns63/ns64.domaincontrol.com)
- Database: standalone PG `national_paper_hangers` (NOT dw_unified)
---
## Step A — Pre-flight
Run these on Kamatera before anything else. SSH as root.
### A1. Disk space
```bash
ssh root@45.61.58.125 "df -h /"
```
Expected output: at least 2 GB free (app + dump + npm modules ≈ 800 MB worst case).
Failure: `Use%` is 95%+. Fix: `du -sh /root/Projects/* | sort -rh | head -20` to find what's large,
then compress old dumps (`find /tmp -name '*.dump' -mtime +7 -delete`) or expand the VPS disk.
### A2. Node version
```bash
ssh root@45.61.58.125 "node --version && npm --version"
```
Expected: `v20.x.x` or higher (package.json engines: `>=20`).
Failure: older Node. Fix: `nvm install 20 && nvm alias default 20` or install via NodeSource:
```bash
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
```
### A3. PostgreSQL version
```bash
ssh root@45.61.58.125 "psql --version && pg_lsclusters 2>/dev/null || pg_config --version"
```
Expected: PostgreSQL 14 or higher (schema uses `gen_random_uuid()`, GIN indexes, BRIN).
Failure: PG 12 or lower. Fix: add PG apt repo and upgrade, or use Kamatera's PG 15 cluster if
already present for another project.
### A4. Port 9765 free
```bash
ssh root@45.61.58.125 "ss -tlnp | grep ':9765' || echo 'port 9765 is free'"
```
Expected: `port 9765 is free`
Failure: another process is on 9765. Fix: `ss -tlnp | grep 9765` to identify the owner, then
either kill it or assign NPH a different port (update ecosystem.kamatera.config.js PORT and nginx
proxy_pass accordingly).
### A5. Confirm national_paper_hangers DB does not already exist (first deploy only)
```bash
ssh root@45.61.58.125 "psql -U postgres -lqt | grep national_paper_hangers || echo 'DB absent'"
```
If absent, create it:
```bash
ssh root@45.61.58.125 "createdb -U postgres national_paper_hangers"
```
If present and this is a re-deploy, the pg_restore in Step B will drop+recreate objects idempotently.
---
## Step B — PG Migration (Mac2 → Kamatera)
These commands run on **Mac2** unless noted.
### B1. Dump the database (Mac2)
```bash
DUMP=/tmp/nph_$(date +%Y%m%d_%H%M%S).dump
pg_dump -Fc -d national_paper_hangers -f "$DUMP"
echo "Dump: $DUMP ($(du -sh $DUMP | cut -f1))"
```
Expected output: `Dump: /tmp/nph_20260505_143200.dump (2.1M)` — size varies with data volume.
Failure: `pg_dump: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed`.
Fix: ensure local PG is running (`brew services start postgresql@15` on Mac2) and the DB exists
(`psql -l | grep national_paper_hangers`).
**Schema-only option for test deploys** (no data, just structure):
```bash
pg_dump -Fc --schema-only -d national_paper_hangers -f "$DUMP"
# Then seed with: ssh root@45.61.58.125 "psql -d national_paper_hangers -f /root/Projects/NationalPaperHangers/db/seed.sql"
```
### B2. scp dump to Kamatera
```bash
scp "$DUMP" root@45.61.58.125:/tmp/
REMOTE_DUMP="/tmp/$(basename $DUMP)"
echo "Remote dump: $REMOTE_DUMP"
```
Expected: scp progress bar completes, exit 0.
Failure: `Permission denied (publickey)` — fix: ensure `~/.ssh/config` has the kamatera key entry,
or add `-i ~/.ssh/kamatera_id_rsa` to the scp command.
### B3. Create the pg role (Kamatera, first deploy only)
The app connects to PG with a dedicated role. On Kamatera, create it if absent:
```bash
ssh root@45.61.58.125 "psql -U postgres -c \"
DO \\\$\\\$ BEGIN
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname='nph_app') THEN
CREATE ROLE nph_app LOGIN PASSWORD 'STRONG_PASSWORD_HERE';
END IF;
END \\\$\\\$;
GRANT ALL PRIVILEGES ON DATABASE national_paper_hangers TO nph_app;
\""
```
Replace `STRONG_PASSWORD_HERE` with the actual PGPASSWORD value you'll use in .env.
The role name `nph_app` matches the PGUSER env var in the production .env example below.
### B4. pg_restore on Kamatera
```bash
ssh root@45.61.58.125 "pg_restore --no-owner -c --if-exists \
-U postgres \
-d national_paper_hangers \
$REMOTE_DUMP 2>&1 | tail -30"
```
Expected: a few lines of SET commands, exit 0. Errors like `ERROR: role "stevestudio2" does not exist`
from `--no-owner` suppression are normal and harmless.
Failure: `FATAL: database "national_paper_hangers" does not exist` — run Step A5 first.
Failure: `pg_restore: error: could not execute query` on a specific table — check if the migration
files in `db/migrations/` have been applied. Run migrations manually:
```bash
ssh root@45.61.58.125 "psql -d national_paper_hangers \
-f /root/Projects/NationalPaperHangers/db/migrations/001_claim_columns.sql \
-f /root/Projects/NationalPaperHangers/db/migrations/002_ad_signals.sql"
# Migration 003 uses CONCURRENTLY — cannot run inside a transaction block:
ssh root@45.61.58.125 "psql -d national_paper_hangers \
-f /root/Projects/NationalPaperHangers/db/migrations/003_perf.sql"
```
### B5. Grant table permissions (Kamatera)
After pg_restore with `--no-owner`, tables are owned by `postgres`. Grant access to the app role:
```bash
ssh root@45.61.58.125 "psql -U postgres -d national_paper_hangers -c \"
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO nph_app;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO nph_app;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO nph_app;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT USAGE, SELECT ON SEQUENCES TO nph_app;
\""
```
### B6. Clean up dump files
```bash
rm "$DUMP" # Mac2 local copy
ssh root@45.61.58.125 "rm $REMOTE_DUMP" # Kamatera /tmp copy
```
---
## Step C — Code Sync
### C1. rsync (Mac2 → Kamatera)
The deploy script handles this automatically. To run manually:
```bash
rsync -avz --delete \
--exclude='node_modules/' \
--exclude='.env' \
--exclude='.git/' \
--exclude='*.dump' \
--exclude='*.log' \
/Users/stevestudio2/Projects/NationalPaperHangers/ \
root@45.61.58.125:/root/Projects/NationalPaperHangers/
```
Expected: file list scrolls by, `sent N bytes received M bytes` at end, exit 0.
Failure: `ssh: connect to host 45.61.58.125 port 22: Connection refused` — check Tailscale is up
on Mac2 and Kamatera, or use the direct Kamatera IP with SSH key.
### C2. npm ci on Kamatera
```bash
ssh root@45.61.58.125 "cd /root/Projects/NationalPaperHangers && npm ci --omit=dev"
```
Expected: `added N packages` (around 80), `npm warn` lines for optional peers are fine. Exit 0.
Failure: `npm ci can only install packages when package-lock.json is present` — ensure
`package-lock.json` was NOT in `.gitignore` or rsync excludes. It should be synced.
Failure: bcrypt native compile errors — install build tools:
```bash
ssh root@45.61.58.125 "apt-get install -y build-essential python3"
```
Then re-run npm ci.
---
## Step D — Environment (.env on Kamatera)
Create `/root/Projects/NationalPaperHangers/.env` on Kamatera. This file is **never** rsync'd
(intentionally excluded). Write it directly on Kamatera:
```bash
ssh root@45.61.58.125 "cat > /root/Projects/NationalPaperHangers/.env << 'ENVEOF'
# ── Runtime ────────────────────────────────────────────────────────────────
PORT=9765
NODE_ENV=production
# ── Sessions ────────────────────────────────────────────────────────────────
# REQUIRED. Generate: openssl rand -base64 48
SESSION_SECRET=REPLACE_WITH_STRONG_SECRET_MIN_32_CHARS
# ── Booking token signing ────────────────────────────────────────────────────
# REQUIRED. Generate: openssl rand -base64 48
BOOKING_SIGNING_SECRET=REPLACE_WITH_STRONG_SECRET_MIN_32_CHARS
# ── Public origin ────────────────────────────────────────────────────────────
# REQUIRED. Used in email links and Stripe redirect URLs.
PUBLIC_URL=https://nationalpaperhangers.com
# ── Mailing address (CAN-SPAM §7) ────────────────────────────────────────────
# REQUIRED. Appears in the footer of every transactional email.
MAILING_ADDRESS=National Paper Hangers, [Street Address], [City, State ZIP]
# ── Postgres (Kamatera PG) ────────────────────────────────────────────────────
# REQUIRED.
PGHOST=localhost
PGPORT=5432
PGDATABASE=national_paper_hangers
PGUSER=nph_app
PGPASSWORD=REPLACE_WITH_PG_PASSWORD
# ── Stripe ────────────────────────────────────────────────────────────────────
# REQUIRED for subscription + Stripe webhook flows.
STRIPE_SECRET_KEY=sk_live_...
STRIPE_WEBHOOK_SECRET=whsec_...
# Price IDs from Stripe dashboard (Products → Prices):
STRIPE_PRICE_PRO_MONTH=price_...
STRIPE_PRICE_PRO_YEAR=price_...
STRIPE_PRICE_SIGNATURE_MONTH=price_...
STRIPE_PRICE_SIGNATURE_YEAR=price_...
STRIPE_PRICE_ENTERPRISE_MONTH=price_...
# ── George Gmail agent (via Tailscale) ───────────────────────────────────────
# REQUIRED for all transactional email (booking confirmations, claim flows, etc.)
# George is on Mac2 Tailscale IP — reachable from Kamatera via Tailnet.
GEORGE_URL=http://100.107.67.67:9850
GEORGE_ACCOUNT=info
GEORGE_USER=admin
GEORGE_PASS=REPLACE_WITH_GEORGE_PASSWORD
EMAIL_FROM=info@nationalpaperhangers.com
EMAIL_FROM_NAME=National Paper Hangers
# ── Optional ──────────────────────────────────────────────────────────────────
# GA4 measurement ID (injected client-side in views/layout.ejs if set).
# GA4_MEASUREMENT_ID=G-XXXXXXXXXX
ENVEOF
chmod 600 /root/Projects/NationalPaperHangers/.env"
```
### D — Variable reference
| Variable | Required | Description |
|---|---|---|
| `PORT` | yes | Must be 9765; matches nginx proxy_pass |
| `NODE_ENV` | yes | Must be `production` or server refuses to start |
| `SESSION_SECRET` | yes | ≥32 random chars; server throws on boot if missing/default |
| `BOOKING_SIGNING_SECRET` | yes | Signs booking confirmation tokens in lib/booking-token.js |
| `PUBLIC_URL` | yes | `https://nationalpaperhangers.com` — used in email links and Stripe redirects |
| `MAILING_ADDRESS` | yes | CAN-SPAM footer address; scrubbed from email.js templates |
| `PGHOST` | yes | `localhost` for Kamatera local PG |
| `PGPORT` | yes | `5432` |
| `PGDATABASE` | yes | `national_paper_hangers` |
| `PGUSER` | yes | `nph_app` (the role created in Step B3) |
| `PGPASSWORD` | yes | The password set in Step B3 |
| `STRIPE_SECRET_KEY` | yes | Live secret key from Stripe dashboard |
| `STRIPE_WEBHOOK_SECRET` | yes | From Stripe → Webhooks → endpoint secret |
| `STRIPE_PRICE_PRO_MONTH` | yes | Stripe price ID for Pro monthly tier |
| `STRIPE_PRICE_PRO_YEAR` | yes | Stripe price ID for Pro annual tier |
| `STRIPE_PRICE_SIGNATURE_MONTH` | yes | Stripe price ID for Signature monthly |
| `STRIPE_PRICE_SIGNATURE_YEAR` | yes | Stripe price ID for Signature annual |
| `STRIPE_PRICE_ENTERPRISE_MONTH` | yes | Stripe price ID for Enterprise monthly |
| `GEORGE_URL` | yes | `http://100.107.67.67:9850` (Mac2 Tailscale) |
| `GEORGE_ACCOUNT` | yes | `info` — routes email through info@nationalpaperhangers.com |
| `GEORGE_USER` | yes | `admin` (George basic auth) |
| `GEORGE_PASS` | yes | George basic auth password |
| `EMAIL_FROM` | yes | `info@nationalpaperhangers.com` |
| `EMAIL_FROM_NAME` | yes | `National Paper Hangers` |
| `GA4_MEASUREMENT_ID` | optional | GA4 G-XXXXXXX if analytics desired on prod |
---
## Step E — pm2 Start
### E1. Start / reload via ecosystem file
```bash
ssh root@45.61.58.125 "
cd /root/Projects/NationalPaperHangers
if pm2 describe national-paper-hangers > /dev/null 2>&1; then
pm2 reload national-paper-hangers --update-env
echo 'Reloaded existing pm2 process.'
else
pm2 start /root/Projects/NationalPaperHangers/ecosystem.kamatera.config.js --env production
echo 'Started new pm2 process.'
fi
"
```
Expected: `[PM2] Done` or `restarted` line, exit 0.
Failure: `Error: SESSION_SECRET must be set to a strong value in production` — the .env file is
missing or SESSION_SECRET is still the placeholder. Fix: update .env, then re-run `pm2 reload`.
Failure: `Error: Cannot find module './routes/public'` — rsync didn't complete cleanly.
Re-run Step C1.
### E2. pm2 save on Kamatera (persists across reboots via systemd pm2-root.service)
```bash
ssh root@45.61.58.125 "pm2 save"
```
Expected: `[PM2] Saving current process list... [PM2] Successfully saved in /root/.pm2/dump.pm2`
**IMPORTANT: Never run `pm2 save` on Mac2. Only run it on Kamatera.**
### E3. Verify process is healthy
```bash
ssh root@45.61.58.125 "pm2 show national-paper-hangers"
```
Expected: `status: online`, `restart time: 0` (or low), `uptime: Xs`.
```bash
ssh root@45.61.58.125 "curl -sf http://127.0.0.1:9765/health || curl -sf http://127.0.0.1:9765/"
```
Expected: HTTP 200 with HTML body.
### E4. nginx stale-config safety check
Per the 2026-04-30 Site Factory cross-routing incident, check for stale files before loading nginx:
```bash
ssh root@45.61.58.125 "find /etc/nginx/sites-enabled/ -name '*.bak' -o -name '*.old' 2>/dev/null && echo 'WARNING: stale configs found' || echo 'OK: no stale configs'"
```
If stale files are found: `rm` each one, then proceed.
---
## Step F — nginx Server Block
### F1. Write the config
```bash
ssh root@45.61.58.125 "cat > /etc/nginx/sites-available/nationalpaperhangers.com << 'NGINX_EOF'
# National Paper Hangers — nginx reverse proxy
# Proxy: 127.0.0.1:9765 (pm2 node process)
# SSL: managed by certbot (added in Step G; placeholders here until then)
# Real-IP: Cloudflare IP ranges so req.ip is the visitor, not CF's proxy.
# HTTP → HTTPS redirect (activated after certbot adds SSL in Step G)
server {
listen 80;
listen [::]:80;
server_name nationalpaperhangers.com www.nationalpaperhangers.com;
# Let certbot own /.well-known/acme-challenge/ for renewal.
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
# All other traffic: redirect to HTTPS after SSL is provisioned.
# During initial HTTP-only test, comment out the return line and add:
# proxy_pass http://127.0.0.1:9765;
return 301 https://\$host\$request_uri;
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name nationalpaperhangers.com www.nationalpaperhangers.com;
# SSL certs — certbot fills these in Step G.
# ssl_certificate /etc/letsencrypt/live/nationalpaperhangers.com/fullchain.pem;
# ssl_certificate_key /etc/letsencrypt/live/nationalpaperhangers.com/privkey.pem;
# include /etc/letsencrypt/options-ssl-nginx.conf;
# ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
# ── Real-IP from Cloudflare ──────────────────────────────────────────────
# Cloudflare publishes its IPv4 ranges. Update this list if CF adds new ranges.
# https://www.cloudflare.com/ips-v4
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
# IPv6 CF ranges
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2a06:98c0::/29;
set_real_ip_from 2c0f:f248::/32;
real_ip_header CF-Connecting-IP;
# ── Gzip ────────────────────────────────────────────────────────────────
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css application/json application/javascript
text/xml application/xml application/xml+rss text/javascript
image/svg+xml;
# ── Static assets (served by Express; nginx can short-circuit if needed) ─
# Express serves /public/ directly. Uncomment below for nginx-level caching
# if the Node process becomes a bottleneck.
# location /public/ {
# root /root/Projects/NationalPaperHangers;
# expires 30d;
# add_header Cache-Control "public, immutable";
# }
# ── Stripe webhook — raw body required ───────────────────────────────────
# express raw-body middleware handles this; no special nginx treatment needed.
# ── Main proxy ───────────────────────────────────────────────────────────
location / {
proxy_pass http://127.0.0.1:9765;
proxy_http_version 1.1;
# WebSocket support (booking slot polling uses SSE; future WS ready).
proxy_set_header Upgrade \$http_upgrade;
proxy_set_header Connection "upgrade";
# Pass real visitor IP + protocol to Express (app uses trust proxy 1).
proxy_set_header Host \$host;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto \$scheme;
# Timeouts — booking slot fetches can take a few seconds on cold DB.
proxy_connect_timeout 10s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
proxy_buffering off;
proxy_cache_bypass \$http_upgrade;
}
# ── Security headers not covered by helmet ────────────────────────────────
# Helmet handles most (HSTS, CSP, X-Frame, etc.) — nginx adds proto header
# so Express can enforce HTTPS redirects internally.
add_header X-Forwarded-Proto \$scheme always;
}
NGINX_EOF
echo 'nginx config written'"
```
### F2. Enable the site
```bash
ssh root@45.61.58.125 "
ln -sf /etc/nginx/sites-available/nationalpaperhangers.com \
/etc/nginx/sites-enabled/nationalpaperhangers.com
echo 'Symlink created.'
"
```
### F3. Validate and reload nginx
```bash
ssh root@45.61.58.125 "nginx -t && systemctl reload nginx && echo 'nginx reloaded OK'"
```
Expected: `nginx: the configuration file /etc/nginx/nginx.conf syntax is ok` and
`nginx: configuration file /etc/nginx/nginx.conf test is successful`, then `nginx reloaded OK`.
Failure: `nginx: [emerg] "server" directive is not allowed here` — usually a bad symlink pointing
to a non-config file. Check `ls -la /etc/nginx/sites-enabled/` for .bak/.old files and remove them.
---
## Step G — SSL (Let's Encrypt via certbot)
Run this **after** DNS has propagated to Kamatera (Step H) or during HTTP-only validation with
CF proxy temporarily disabled (orange cloud off).
```bash
ssh root@45.61.58.125 "
certbot --nginx \
-d nationalpaperhangers.com \
-d www.nationalpaperhangers.com \
--non-interactive \
--agree-tos \
-m info@nationalpaperhangers.com \
--redirect
"
```
Expected: certbot edits the nginx config, adds the ssl_certificate lines, and enables the
301 redirect block. `Congratulations! Your certificate and chain have been saved`.
Failure: `Challenge failed for domain nationalpaperhangers.com` (ACME HTTP-01 challenge).
This means DNS is not yet pointing at Kamatera. Complete Step H first, wait for propagation
(`dig nationalpaperhangers.com A` should return 45.61.58.125), then re-run certbot.
Failure: `too many certificates already issued` — Let's Encrypt rate limit. Use staging for
retries: add `--staging` flag. Remove staging certs and re-run without `--staging` when ready.
Auto-renew is handled by the certbot systemd timer already installed on Kamatera.
---
## Step H — DNS Cutover (GATED — requires new CF token)
**This step is blocked until a new Cloudflare token is minted.**
The active CF token is DNS-edit only (`Zone:DNS:Edit`) and **cannot create zones** (`Zone:Edit`
permission is required). Do NOT use the active token to attempt zone creation — it will fail.
### H1. Mint a new Cloudflare token (Steve must do this in CF dashboard)
1. Go to: https://dash.cloudflare.com/profile/api-tokens
2. Create token → "Edit zone DNS" template → Customize.
3. Permissions: `Zone — Zone — Edit` AND `Zone — DNS — Edit`.
4. Zone resources: Include — Specific zone — nationalpaperhangers.com (or "All zones" if preferred).
5. Copy the token. Route it through the secrets-manager skill.
### H2. Create the Cloudflare zone
```bash
curl -X POST "https://api.cloudflare.com/client/v4/zones" \
-H "Authorization: Bearer $CF_ZONE_CREATE_TOKEN" \
-H "Content-Type: application/json" \
--data '{"name":"nationalpaperhangers.com","jump_start":false}' | jq .
```
Note the `id` field from the response — this is the zone ID.
### H3. Add DNS records
Replace `ZONE_ID` with the zone ID from H2.
```bash
# A record — proxied through Cloudflare (orange cloud)
curl -X POST "https://api.cloudflare.com/client/v4/zones/ZONE_ID/dns_records" \
-H "Authorization: Bearer $CF_ZONE_CREATE_TOKEN" \
-H "Content-Type: application/json" \
--data '{"type":"A","name":"nationalpaperhangers.com","content":"45.61.58.125","proxied":true}' | jq .
# www CNAME — also proxied
curl -X POST "https://api.cloudflare.com/client/v4/zones/ZONE_ID/dns_records" \
-H "Authorization: Bearer $CF_ZONE_CREATE_TOKEN" \
-H "Content-Type: application/json" \
--data '{"type":"CNAME","name":"www","content":"nationalpaperhangers.com","proxied":true}' | jq .
# MX record for info@nationalpaperhangers.com (George Gmail)
# Set to match whatever MX George/Gmail uses — typically Google Workspace MX.
# Example (replace with actual MX values from DNS provider for info@):
# curl -X POST ... --data '{"type":"MX","name":"nationalpaperhangers.com","content":"aspmx.l.google.com","priority":1,"proxied":false}'
```
### H4. Get Cloudflare nameservers for this zone
```bash
curl -s "https://api.cloudflare.com/client/v4/zones/ZONE_ID" \
-H "Authorization: Bearer $CF_ZONE_CREATE_TOKEN" | jq '.result.name_servers'
```
Expected: two CF nameservers like `caden.ns.cloudflare.com` and `mia.ns.cloudflare.com`.
### H5. Update GoDaddy NS records
In the GoDaddy control panel for `nationalpaperhangers.com`:
- Change NS from `ns63.domaincontrol.com` / `ns64.domaincontrol.com`
- to the two Cloudflare nameservers from H4.
Or via GoDaddy API:
```bash
# Requires GODADDY_API_KEY and GODADDY_API_SECRET
curl -X PUT "https://api.godaddy.com/v1/domains/nationalpaperhangers.com/records/NS/@" \
-H "Authorization: sso-key $GODADDY_API_KEY:$GODADDY_API_SECRET" \
-H "Content-Type: application/json" \
--data '[{"data":"caden.ns.cloudflare.com"},{"data":"mia.ns.cloudflare.com"}]'
```
**PROTECTED DNS REMINDER: Never touch designerwallcoverings.com or studentdebtcrisis*.org
nameservers or DNS records in any script.**
### H6. Verify propagation
```bash
watch -n 10 "dig nationalpaperhangers.com NS +short && dig nationalpaperhangers.com A +short"
```
Wait until NS shows Cloudflare nameservers and A resolves to 45.61.58.125. Propagation is
typically 5–30 minutes for GoDaddy NS changes. Full TTL-based propagation can take up to 48h,
but CF's anycast means most users see it within minutes of GoDaddy acknowledging the NS change.
---
## Step I — Smoke Verification
### I1. Curl health checks
```bash
# Basic HTTP response
curl -I https://nationalpaperhangers.com/
# Expected: HTTP/2 200
# Check for: x-forwarded-proto: https (nginx header)
# strict-transport-security (helmet HSTS)
# content-type: text/html
# Find page (core directory)
curl -sf https://nationalpaperhangers.com/find | grep -i installer
# www redirect
curl -I https://www.nationalpaperhangers.com/
# Expected: HTTP/2 301 or 200 depending on CF www handling
```
### I2. Run the smoke test suite against the live URL
The smoke tests are wired to the local DB connection. For a prod smoke test, SSH to Kamatera
and run against the Kamatera DB:
```bash
ssh root@45.61.58.125 "
cd /root/Projects/NationalPaperHangers
NODE_ENV=production node --test tests/smoke.test.js
"
```
Expected: `10 passing` — all 10 tests green (DB ping, /find, /installer/:slug claimed + unclaimed,
404 slug, /book renders, /book for inactive 404, slots logic, claim token flow, teardown).
Failure on `DB: pool can query` — Postgres credentials in .env are wrong. Check PGHOST/PGUSER/PGPASSWORD.
Failure on `/find returns 200` — nginx proxy_pass misconfigured or pm2 process is down.
Check: `pm2 show national-paper-hangers` and `nginx -t`.
### I3. Stripe webhook endpoint registration
Register the production webhook endpoint in the Stripe dashboard:
- URL: `https://nationalpaperhangers.com/webhooks/stripe`
- Events: `customer.subscription.created`, `customer.subscription.updated`,
`customer.subscription.deleted`, `invoice.payment_succeeded`, `invoice.payment_failed`
- Copy the signing secret (`whsec_...`) into the `.env` `STRIPE_WEBHOOK_SECRET` field.
- Reload pm2: `ssh root@45.61.58.125 "pm2 reload national-paper-hangers --update-env"`
---
## Step J — Rollback Plan
If the deploy is broken and needs to be reverted:
### J1. Stop the Kamatera pm2 process
```bash
ssh root@45.61.58.125 "pm2 stop national-paper-hangers && pm2 save"
```
This takes the app offline. Traffic from CF will start 522-erroring.
### J2. Point Cloudflare A record back to Mac2
If Mac2 is publicly reachable (it needs a static IP or CF Tunnel):
Option A — Static IP: change the CF A record to Mac2's public IP.
Option B — CF Tunnel: if a Cloudflare Tunnel was pre-configured on Mac2, toggle the DNS record
to the tunnel's CNAME target.
```bash
curl -X PATCH "https://api.cloudflare.com/client/v4/zones/ZONE_ID/dns_records/A_RECORD_ID" \
-H "Authorization: Bearer $CF_ZONE_CREATE_TOKEN" \
-H "Content-Type: application/json" \
--data '{"content":"MAC2_PUBLIC_IP","proxied":true}' | jq .result.content
```
### J3. Restart Mac2 pm2 process
```bash
# Run on Mac2 — NOT Kamatera
pm2 restart national-paper-hangers
pm2 show national-paper-hangers
```
Mac2 pm2 `national-paper-hangers` at :9765 should come back online.
### J4. Verify rollback
```bash
curl -sf https://nationalpaperhangers.com/find | grep -i installer
```
### J5. Diagnose Kamatera failure
```bash
ssh root@45.61.58.125 "pm2 logs national-paper-hangers --lines 100"
ssh root@45.61.58.125 "tail -50 /root/.pm2/logs/national-paper-hangers-error.log"
```
Common root causes:
- Missing or malformed .env value (especially SESSION_SECRET or PGPASSWORD)
- PG role permissions not granted (Step B5 skipped)
- Port 9765 conflict with another process
- npm ci used wrong Node version — check `node --version` on Kamatera
---
## Appendix — Automated Deploy Script
`scripts/deploy-kamatera.sh` automates Steps B–E. It is DRY-RUN by default.
```bash
# Preview what it would do:
bash /Users/stevestudio2/Projects/NationalPaperHangers/scripts/deploy-kamatera.sh
# Execute for real (after Steve green-lights):
bash /Users/stevestudio2/Projects/NationalPaperHangers/scripts/deploy-kamatera.sh --commit
```
The script:
- Checks for protected-domain references before touching any nginx config
- Dumps PG, sccp's to Kamatera, pg_restore's
- rsync's source (excludes .env, node_modules, .git)
- npm ci --omit=dev on Kamatera
- pm2 reload (or start) on Kamatera
- pm2 save on Kamatera only
- Checks for stale .bak/.old nginx configs
- Logs everything to /tmp/deploy-kamatera-nph-*.log