backfill-rooms-cron.sh: atomic designs.json write + catastrophe guard — fixes the recurring ~06:30 UTC wallco catalog blackout

0ed2c60b61707e0b2e1748b3bdd919cf3fc356e1 · 2026-05-30 08:21:11 -0700 · Steve Abrams

The inline python wrote designs.json via open(DJ,'w') (truncates the live 25MB
file to 0, then streams it back over several seconds) — any server read/reload in
that window saw 0 designs → ~4-min customer-facing blackout (observed 2026-05-30
06:30). It also bypassed refresh_designs_snapshot.py's >50%-shrink catastrophe
guard, so nothing caught the bad state. Fix: write to .tmp then os.replace() (atomic
rename — server only ever sees a complete file) + refuse to write if <1000 rows.
Was prod-only; now tracked on Mac2. Not in deploy-kamatera rsync set so no regress risk.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

A scripts/backfill-rooms-cron.sh

Diff

commit 0ed2c60b61707e0b2e1748b3bdd919cf3fc356e1
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Sat May 30 08:21:11 2026 -0700

    backfill-rooms-cron.sh: atomic designs.json write + catastrophe guard — fixes the recurring ~06:30 UTC wallco catalog blackout
    
    The inline python wrote designs.json via open(DJ,'w') (truncates the live 25MB
    file to 0, then streams it back over several seconds) — any server read/reload in
    that window saw 0 designs → ~4-min customer-facing blackout (observed 2026-05-30
    06:30). It also bypassed refresh_designs_snapshot.py's >50%-shrink catastrophe
    guard, so nothing caught the bad state. Fix: write to .tmp then os.replace() (atomic
    rename — server only ever sees a complete file) + refuse to write if <1000 rows.
    Was prod-only; now tracked on Mac2. Not in deploy-kamatera rsync set so no regress risk.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 scripts/backfill-rooms-cron.sh | 53 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/scripts/backfill-rooms-cron.sh b/scripts/backfill-rooms-cron.sh
new file mode 100755
index 0000000..017c3e5
--- /dev/null
+++ b/scripts/backfill-rooms-cron.sh
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+# backfill-rooms-cron.sh — recurring room-mockup backfill for prod.
+# Lock-file gated so a long-running render can't be double-fired.
+# Steve authorized recurring auto-backfill 2026-05-28 via AskUserQuestion.
+set -uo pipefail
+LOCK=/tmp/backfill-rooms.lock
+LOG=/tmp/backfill-rooms-cron.log
+cd /root/public-projects/wallco-ai || exit 1
+exec 200>$LOCK
+flock -n 200 || { echo "$(date -Iseconds) skip: prior run still active" >> $LOG; exit 0; }
+echo "$(date -Iseconds) ==== start ====" >> $LOG
+# Idempotent: /api/room refuses to re-render an already-rendered design.
+# Cap each run at --limit 50 to keep spend bounded ($5/run).
+CONCURRENCY=2 node scripts/backfill-rooms.js --limit 50 >> $LOG 2>&1
+# Patch designs.json with any IDs that got a fresh PNG in the last 2h.
+python3 <<'PY' >> $LOG 2>&1
+import json, os, re, glob, time, shutil
+DJ='data/designs.json'
+cutoff=time.time()-7200
+have=set()
+for p in glob.glob('data/rooms/design_*_living_room.png'):
+  if p.endswith('_hero.png'): continue
+  if os.path.getmtime(p)>=cutoff:
+    m=re.search(r'design_(\d+)_living_room\.png',p)
+    if m: have.add(int(m.group(1)))
+d=json.load(open(DJ))
+t=0
+for x in d:
+  if x.get('id') in have:
+    rm=x.get('room_mockups') or []
+    if 'living_room' not in rm:
+      rm.append('living_room'); x['room_mockups']=rm; t+=1
+if t:
+  # Catastrophe guard: never overwrite the live catalog with a suspiciously small
+  # set (mirrors refresh_designs_snapshot.py's >50%-shrink refuse, which this inline
+  # writer was bypassing — that's why the guard "didn't fire" on the 06:30 blackout).
+  if len(d) < 1000:
+    print(f'ABORT: refusing to write designs.json with only {len(d)} rows (catastrophe guard)')
+  else:
+    shutil.copy(DJ, DJ+'.bak.'+str(int(time.time())))
+    # ATOMIC write: stream to a temp file then os.replace() (atomic rename) so the
+    # server NEVER reads a half-written/0-byte file. open(DJ,'w') truncated the live
+    # 25MB file to zero and took seconds to refill — any read in that window saw 0
+    # designs → the recurring ~4-min catalog blackout. (fix 2026-05-30)
+    tmp=DJ+'.tmp'
+    with open(tmp,'w') as f: json.dump(d, f, separators=(',',':'))
+    os.replace(tmp, DJ)
+    print(f'patched {t} rows in designs.json (atomic)')
+else:
+  print('no new room_mockups to patch')
+PY
+curl -s -X POST -u admin:DWSecure2024! http://127.0.0.1:9905/admin/reload-designs >> $LOG 2>&1
+echo "" >> $LOG

← 12ce9a9 refresh_designs_snapshot: add byte-size guard (refuse <85% o · back to Wallco Ai · _evidence: snapshot bubbe RCE routes (hooks + upload-bubbe) a37f6a3 →