fix(perf): guard redundant disk-cache reparse in buildCatalog (TK-11962)

39cda0efbee278ca995a3e39d4aac462dd9156ec · 2026-09-21 17:53:04 -0700 · Steve Abrams

Root cause of the local dwphoto event-loop wedge (reproduced 5x): when the
all-dw feed (:9958) is down, buildCatalog() falls back to synchronously
readFileSync+JSON.parse+map of data/all-catalog.json (332MB / 252,187 rows,
of which 137,433 are real STAGED "catalog-only" products, NOT junk) on the
MAIN THREAD — at boot AND every 15min via setInterval. That spikes the heap
~2.3GB and GC-thrashes the loop for tens of seconds; healthz + all requests
time out (scanner offline). It recurs every 15min, turning a healthy process
into a wedged one.

Fix: size+mtime guard — skip the reparse when CATALOG already reflects the
unchanged cache file. Eliminates the RECURRING 15-min reparse stall (the
common feed-down case). Verified: unchanged cache -> /api/reindex is a ~2ms
no-op (1 index, rss flat 2290MB); changed mtime -> correctly reloads.

PARTIAL by design: the one-time boot load and a genuine cache-change reload
still parse synchronously (a changed-cache reindex still stalls, negative
test confirmed). The dep-free deeper fix (worker_threads offload of the
parse) is a follow-up — rows can't be dropped (real staged products) and a
streaming parser would add a runtime dep (repo is zero-runtime-dep).

Deploy of this = external publish -> GATED (memo in pending-approval).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011vBEHxUrUejKV7JCQ3uYdg

Files touched

M server.js

Diff

commit 39cda0efbee278ca995a3e39d4aac462dd9156ec
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Mon Sep 21 17:53:04 2026 -0700

    fix(perf): guard redundant disk-cache reparse in buildCatalog (TK-11962)
    
    Root cause of the local dwphoto event-loop wedge (reproduced 5x): when the
    all-dw feed (:9958) is down, buildCatalog() falls back to synchronously
    readFileSync+JSON.parse+map of data/all-catalog.json (332MB / 252,187 rows,
    of which 137,433 are real STAGED "catalog-only" products, NOT junk) on the
    MAIN THREAD — at boot AND every 15min via setInterval. That spikes the heap
    ~2.3GB and GC-thrashes the loop for tens of seconds; healthz + all requests
    time out (scanner offline). It recurs every 15min, turning a healthy process
    into a wedged one.
    
    Fix: size+mtime guard — skip the reparse when CATALOG already reflects the
    unchanged cache file. Eliminates the RECURRING 15-min reparse stall (the
    common feed-down case). Verified: unchanged cache -> /api/reindex is a ~2ms
    no-op (1 index, rss flat 2290MB); changed mtime -> correctly reloads.
    
    PARTIAL by design: the one-time boot load and a genuine cache-change reload
    still parse synchronously (a changed-cache reindex still stalls, negative
    test confirmed). The dep-free deeper fix (worker_threads offload of the
    parse) is a follow-up — rows can't be dropped (real staged products) and a
    streaming parser would add a runtime dep (repo is zero-runtime-dep).
    
    Deploy of this = external publish -> GATED (memo in pending-approval).
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_011vBEHxUrUejKV7JCQ3uYdg
---
 server.js | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/server.js b/server.js
index 8e133a1..6593d12 100644
--- a/server.js
+++ b/server.js
@@ -2710,6 +2710,7 @@ async function createNewItem(p, b64, dryRun) {
 
 // In-memory index of ALL non-archived Fentucci products (for "find any SKU" lookup).
 let CATALOG = [], INDEX = [];
+let _diskCacheSig = null;   // TK-11962: size+mtime of the disk cache last loaded into CATALOG (skip redundant reparse)
 const nmfr = s => (s || '').trim().toUpperCase().replace(/T$/, ''); // normalize mfr (WOS3467T == wos3467)
 function rebuildIndex() {
   const haveSku = new Set(CATALOG.map(x => (x.dw_sku || '').toUpperCase()));
@@ -2781,9 +2782,17 @@ async function buildCatalog() {
   }
   // 2) last-good disk cache
   try {
+    // TK-11962: the feed is down, so this branch runs on EVERY 15-min interval. The cache is a
+    // multi-hundred-MB / ~250k-row JSON; re-doing readFileSync+JSON.parse+map synchronously each
+    // time spikes the heap ~2.3GB and GC-thrashes the event loop for tens of seconds (scanner goes
+    // unresponsive). Skip the reparse when CATALOG already reflects this exact cache file (size+mtime).
+    const st = fs.statSync(CATALOG_CACHE);
+    const sig = st.size + ':' + st.mtimeMs;
+    if (CATALOG.length && _diskCacheSig === sig) return;   // unchanged since last load → nothing to do
     const feed = JSON.parse(fs.readFileSync(CATALOG_CACHE, 'utf8'));
     if (feed && Array.isArray(feed.rows) && feed.rows.length) {
       CATALOG = feed.rows.map(feedRowToItem);
+      _diskCacheSig = sig;
       console.log(`catalog indexed: ${CATALOG.length} products from disk cache (feed down)`);
       return rebuildIndex();
     }

← b272ac4 auto-data-snapshot: 2026-09-21T17:27:28 (2 data files) — 5x/ · back to Dw Photo Capture · harden(TK-11962): catalog-cache guard survives corrupt/unrea 77ae6ab →