`MORNING_CHECKLIST.md`

158 lines
# Morning checklist — 2026-05-01 ~6am

Read this first. Then `SUMMARY.md` for the full picture. Time estimates are rough.

## 🚨 0a. (15 min) — `hormuz-orchestrator :9801` is on PUBLIC IP with **NO AUTH**, including expensive POST endpoints

Auth audit on the 8 publicly-bound services found 6 properly gated (401), color-search returning 404 (probably fine), and **hormuz-orchestrator returning 200 unauthenticated**. This is on the public IP `45.61.58.125:9801`.

Exposed POST endpoints (anyone on the internet can hit them):
```
POST /api/launch/begin          # launches horsemen pipelines
POST /api/projects              # creates new projects in PG
POST /api/run-topic             # runs a topic through full 4-horseman pipeline (real $ in Paper+Figma+Canva+21st.dev API calls)
POST /api/stampede              # generates 3 wild graphic variants (real $)
POST /api/stampede/:id/promote
POST /api/chat
```

`POST /api/stampede` with empty body 500's (good — input validation is rejecting), but a malicious or accidental hit with a valid body would burn real API credits at Paper/Figma/Canva/21st.

**Action**: add basic-auth middleware to `~/kamatera-mirror/DW-Agents/hormuz-orchestrator/server.js` (and on Kamatera). One-paragraph fix:
```js
// Add right after `const app = express()`:
const HORMUZ_USER = process.env.HORMUZ_USER || 'admin';
const HORMUZ_PASS = process.env.HORMUZ_PASS;
if (!HORMUZ_PASS) throw new Error('HORMUZ_PASS env var required');
app.use((req, res, next) => {
  if (req.path === '/health') return next();
  const auth = req.headers.authorization || '';
  if (!auth.startsWith('Basic ')) return res.status(401).set('WWW-Authenticate', 'Basic realm="hormuz"').end();
  const [u,p] = Buffer.from(auth.slice(6), 'base64').toString().split(':');
  if (u === HORMUZ_USER && p === HORMUZ_PASS) return next();
  return res.status(401).end();
});
```
Then set `HORMUZ_PASS` in pm2 env on both Mac2 and Kamatera, restart.

Other services audited:
| port | service | auth state |
|---:|---|---|
| 7204 | slack-dm-viewer | ✓ 401 auth-gated |
| 7240 | dw-central | ✓ 401 |
| 9665 | dex-dedup-agent | ✓ 401 |
| 9670 | yolo-agent | ✓ 401 |
| 9674 | silas-sku-agent | ✓ 401 |
| 9870 | color-search | 404 on `/` (probably no UI; check API endpoints) |
| 9894 | sku-check-skill | ✓ 401 |
| 9801 | **hormuz-orchestrator** | **⚠️ 200 unauthenticated** |

(Also: MEMORY.md drift — `:9870` is `color-search`, not `norma-instagram`. Fix MEMORY when convenient.)

## 🚨 0. (10 min) — REVOKE & ROTATE LIVE SECRETS LEAKED IN SOURCE
Late-night Claude grep found multiple LIVE Anthropic, Gemini, and Shopify keys hardcoded as fallbacks in `.js`/`.ts` files (NOT just .env). Full list in `_SECRETS_INLINE.md`. Quick triage:

```bash
# Anthropic (5 distinct keys, 11 files including Kamatera/dear-bubbe-nextjs/ecosystem.config.js with no env fallback)
grep -rn "sk-ant-api03-" ~/Projects/ --include='*.js' --include='*.ts' --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=codex-review-2026-04-30 | head -20

# Gemini (1 key repeated in 13 files, mostly Designer-Wallcoverings)
grep -rn "AIzaSy" ~/Projects/ --include='*.js' --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=codex-review-2026-04-30 | head -20

# Shopify admin token in archive-agent
grep -n "shpat_" ~/Projects/archive-agent/server.js
```

Steps:
1. **Revoke** all 5 Anthropic keys at https://console.anthropic.com/settings/keys
2. **Revoke** the 2 Gemini keys at https://aistudio.google.com/app/apikey
3. **Rotate** the Shopify admin token at Shopify admin → Apps → archive-agent → rotate
4. Use `/secrets` skill to fan new keys into env files. Source files should reference `process.env.X` only (no fallback literals).
5. (Lower priority) The Roboflow `pk_live_bky...` key in `~/Projects/wine-finder/COMPLETE_SYSTEM_SUMMARY.md` is in a doc — rotate at https://app.roboflow.com/settings/api.

See `_PRIORITY_PATCHES.md` "🚨 P0" section for the explicit file:line list.

## 1. (5 min) — confirm OpenAI quota status
Codex hit "Quota exceeded" at ~23:43 PT. Round-3 was aborted; no source files changed by round-3. Check:
```bash
codex login status
codex exec --skip-git-repo-check --sandbox read-only "say hi" 2>&1 | head -3
```
If still quota-blocked, top up at https://platform.openai.com/account/billing or wait for the daily/monthly reset (resets are usually 00:00 UTC for daily limits, billing-cycle for monthly).

If quota is back: re-run round-3 by replaying the inline-bash loop in `all-night.log` (look for the `## round-3 sequential start` block).

## 2. (10 min) — eyeball pm2 fleet + the four operational saves
```bash
pm2 list | grep -E 'sf-admin|visual-factory|site-(wholivedthere|bubbesblock|claimmyaddress)|am-recrawl'
curl -s -o /dev/null -w 'sf-admin %{http_code}\n' http://127.0.0.1:9883/
curl -s -o /dev/null -w 'vf-orch %{http_code}\n' http://127.0.0.1:9892/
for p in 3001 3002 3003; do curl -s -o /dev/null -w "site:$p %{http_code}\n" http://127.0.0.1:$p/; done
```
Expect:
- `sf-admin` online + HTTP 302 (OAuth redirect, normal)
- `visual-factory-orchestrator` online + HTTP 200 (its restarts=14+ pattern is pre-existing — exits cleanly with rc=0, autorestart picks it back up; not a fire)
- `site-wholivedthere` `site-bubbesblock` `site-claimmyaddress` all online + HTTP 200
- `am-recrawl` STOPPED — leave it. See `_PM2_NOTES.md` for the codex-found patch (in `~/kamatera-mirror/...`, ambiguous sync direction — apply on Kamatera not Mac, then re-sync).

## 3. (15 min) — rebuild the 3 next.js sites to activate codex's Stripe webhook hardening
Codex modified `app/api/webhook/stripe/route.ts` in all 3 sites to fail-fast in production when `STRIPE_WEBHOOK_SECRET` missing. Source is updated; compiled `.next/` bundle still serves OLD code. To activate:
```bash
for site in wholivedthere.com bubbesblock.com claimmyaddress.com; do
  cd ~/Projects/site-factory/sites/$site/app
  npm run build 2>&1 | tail -5
done
pm2 restart site-wholivedthere site-bubbesblock site-claimmyaddress
sleep 5
for p in 3001 3002 3003; do curl -s -o /dev/null -w "site:$p %{http_code}\n" http://127.0.0.1:$p/; done
```
All 3 should still HTTP 200 after restart. Stripe webhook test: post a fake payload with no signature; should now fail-fast 500 in prod (was: silently 200 + `received:true,stub:true`).

## 4. (30 min) — rotate the secrets codex flagged
The big one. `SECRETS_AT_RISK.md` has the full per-project list. Top targets:
- **resize-it/server.py:22** — Spoonflower password `*Spoonaccess911*` was a hardcoded fallback. Already removed by codex round-1; verify by `grep -n Spoonaccess ~/Projects/resize-it/server.py` (should be empty). If empty: rotate the password at Spoonflower itself, then update `~/Projects/resize-it/.env`.
- **resize-it/SKILL.md:89** — Gemini API key was inline. Codex moved it to env; verify and rotate at https://aistudio.google.com/app/apikey.
- **Designer-Wallcoverings, jill-website, lawyer-directory-builder** — multiple `.env` files have live tokens (Stripe, Shopify, etc.). The `secrets` skill (`~/Projects/secrets-manager/cli.js`) routes per-key rotation; per memory there's already a 307-leak audit it can fold into.
- Also: `~/.claude/skills/setup-domains/SKILL.md` had a leaked Cloudflare token committed plaintext (per memory — separate issue, mint account-scoped replacement).

## 5. (60 min) — apply the highest-ROI deferred items
`CONSENSUS.md` has per-project deferred lists. The best ROI clusters:
1. **Auth on public POST endpoints** — `NEEDS_AUTH.md` lists projects. Start with the live ones: jill-website (nosarabeachfrontrentals), VictoryStays, Designer-Wallcoverings agents. A simple shared-secret header check is enough for internal tools.
2. **Stripe webhook signature verification** in projects beyond the 3 site-factory sites (lawyer-directory-builder also has Stripe).
3. **Schema/migration drift** — bankrupt-leads, lawyer-directory-builder, professional-directory each have schema-vs-code mismatches codex deferred (DDL is out of safe-fix scope but easy with a 30-min focused session).

## 6. (anytime) — re-run claude-codex skill as belt-and-suspenders
Once OpenAI quota is back, the new claude-codex skill scaffolding (`~/.claude/skills/claude-codex/`) is ready. Test on a single project first:
```bash
~/.claude/skills/claude-codex/scripts/start.sh --name resize-it-test --workdir ~/Projects/resize-it --rounds 3 --report-only
```
The `--report-only` mode produces findings without applying patches; `--apply` mode lets codex+claude debate AND apply.

## What was actually done overnight
- ✅ codex × claude review of all 25 tracked projects (review.md per project)
- ✅ codex round-1 fix pass: 117 patches applied across 17 projects
- ✅ codex re-review: 15 FIX BEFORE SHIP, 2 SHIP IT, 8 defer-only
- ✅ codex round-2 fix on the 15: ~40 additional patches
- ✅ POKE deep-dive on resize-it (regression in own pass-1 fix → fixed)
- ✅ POKE deep-dive on am-recrawl (root caused, codex's patch documented)
- ✅ Operational save: sf-admin SESSION_SECRET injected, restarted, verified
- ✅ Operational save: am-recrawl pm2-stopped (was 90 restarts/24h)
- ✅ Built CONSENSUS.md (1147 lines, per-project synthesis)
- ✅ Built TOP_FINDINGS.md, SECRETS_AT_RISK.md, NEEDS_AUTH.md cross-project worklists
- ✅ Matured the claude-codex skill: scripts/, claude_responder.md
- ❌ Round-3 POKE+APPLY: blocked by OpenAI quota at ~23:43 PT (no source files changed)

## The quick "did anything break overnight?" check
```bash
pm2 jlist | python3 -c "
import json,sys,time; d=json.load(sys.stdin); n=time.time()*1000
for p in d:
  e=p['pm2_env']; s=e.get('status'); r=e.get('restart_time',0) or 0
  u=int((n-e.get('pm_uptime',n))/1000) if e.get('pm_uptime') else 0
  if p['name']=='am-recrawl' and s=='stopped': continue  # intentional
  if s != 'online' or (r > 50) or (r > 5 and u < 120):
    print(f'⚠️  {p[\"name\"]}: status={s} restarts={r} uptime={u}s')
" || echo "all healthy"
```
If anything's flagged that wasn't yesterday, see `_PM2_NOTES.md` for the explanation pattern.