harden(TK-11962): catalog-cache guard survives corrupt/unreadable cache (Kimi review)

77ae6ab4e2c25fee7f8b696c2011341fa34fb1d9 · 2026-09-21 17:57:08 -0700 · Steve Abrams

Kimi (codex-check) found two holes in the size+mtime guard (39cda0e):
1. A torn/truncated cache made JSON.parse throw, so _diskCacheSig never
   updated → every 15-min tick re-parsed 332MB of garbage forever (the exact
   thrash the guard removes, reintroduced for the corrupt case).
2. The bare catch conflated ENOENT with transient errors → one hiccup could
   downgrade the healthy 252k in-memory catalog to the Fentucci-only fallback.

Fix: on a non-ENOENT error, record the file's signature (so the SAME bad file
isn't re-parsed until a real rewrite bumps mtime) and keep last-good CATALOG
instead of falling through. Unit-tested all 6 states (no-cache/valid/unchanged/
changed/corrupt#1/corrupt#2). node --check + eslint 0 errors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011vBEHxUrUejKV7JCQ3uYdg

Files touched

M server.js

Diff

commit 77ae6ab4e2c25fee7f8b696c2011341fa34fb1d9
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Mon Sep 21 17:57:08 2026 -0700

    harden(TK-11962): catalog-cache guard survives corrupt/unreadable cache (Kimi review)
    
    Kimi (codex-check) found two holes in the size+mtime guard (39cda0e):
    1. A torn/truncated cache made JSON.parse throw, so _diskCacheSig never
       updated → every 15-min tick re-parsed 332MB of garbage forever (the exact
       thrash the guard removes, reintroduced for the corrupt case).
    2. The bare catch conflated ENOENT with transient errors → one hiccup could
       downgrade the healthy 252k in-memory catalog to the Fentucci-only fallback.
    
    Fix: on a non-ENOENT error, record the file's signature (so the SAME bad file
    isn't re-parsed until a real rewrite bumps mtime) and keep last-good CATALOG
    instead of falling through. Unit-tested all 6 states (no-cache/valid/unchanged/
    changed/corrupt#1/corrupt#2). node --check + eslint 0 errors.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_011vBEHxUrUejKV7JCQ3uYdg
---
 server.js | 27 +++++++++++++++++++++------
 1 file changed, 21 insertions(+), 6 deletions(-)

diff --git a/server.js b/server.js
index 6593d12..7569885 100644
--- a/server.js
+++ b/server.js
@@ -2781,13 +2781,17 @@ async function buildCatalog() {
     console.log('all-dw feed unavailable:', e.message);
   }
   // 2) last-good disk cache
+  // TK-11962: the feed is down, so this branch runs on EVERY 15-min interval. The cache is a
+  // multi-hundred-MB / ~250k-row JSON; re-doing readFileSync+JSON.parse+map synchronously each
+  // time spikes the heap ~2.3GB and GC-thrashes the event loop for tens of seconds (scanner goes
+  // unresponsive). Skip the reparse when CATALOG already reflects this exact cache file (size+mtime).
+  // stat-BEFORE-read is deliberate: a concurrent writer that finishes after our read bumps mtime
+  // past our stored sig, so the next tick reparses (read-then-stat could tag torn content and skip
+  // forever).
+  let sig = null;
   try {
-    // TK-11962: the feed is down, so this branch runs on EVERY 15-min interval. The cache is a
-    // multi-hundred-MB / ~250k-row JSON; re-doing readFileSync+JSON.parse+map synchronously each
-    // time spikes the heap ~2.3GB and GC-thrashes the event loop for tens of seconds (scanner goes
-    // unresponsive). Skip the reparse when CATALOG already reflects this exact cache file (size+mtime).
     const st = fs.statSync(CATALOG_CACHE);
-    const sig = st.size + ':' + st.mtimeMs;
+    sig = st.size + ':' + st.mtimeMs;
     if (CATALOG.length && _diskCacheSig === sig) return;   // unchanged since last load → nothing to do
     const feed = JSON.parse(fs.readFileSync(CATALOG_CACHE, 'utf8'));
     if (feed && Array.isArray(feed.rows) && feed.rows.length) {
@@ -2796,7 +2800,18 @@ async function buildCatalog() {
       console.log(`catalog indexed: ${CATALOG.length} products from disk cache (feed down)`);
       return rebuildIndex();
     }
-  } catch (e) { /* no cache yet */ }
+  } catch (e) {
+    if (e.code !== 'ENOENT') {
+      // Readable-but-unparseable/erroring cache (torn write, disk full, EACCES/EIO): record its
+      // signature so the SAME bad file isn't re-parsed (and re-thrashed) every 15min — a genuine
+      // rewrite bumps mtime → sig differs → we retry. And never downgrade a healthy in-memory
+      // catalog to the Fentucci-only fallback on a transient read error.
+      if (sig) _diskCacheSig = sig;
+      console.error('catalog cache unreadable:', e.message);
+      if (CATALOG.length) return;
+    }
+    // ENOENT (no cache yet) or empty in-memory catalog → fall through to the legacy fallback
+  }
   // 3) legacy Fentucci-only Shopify crawl — never leave the scanner empty
   return buildCatalogFentucci();
 }

← 39cda0e fix(perf): guard redundant disk-cache reparse in buildCatalo · back to Dw Photo Capture · harden all-dw cache write: atomic tmp+rename, refresh disk-c 2bc0f06 →