Live-scrape 504 fix: kill scrape at 52s (below the 60s CF/nginx proxy cap) → honest in-app 'timed out — use Email Vendor' JSON instead of a raw gateway 504; document the 3-step Kravet prod light-up recovery in build-coverage.js

81aa19fb057141b32da111f92dd691e31db933e7 · 2026-07-15 17:44:25 -0700 · Steve Abrams

Files touched

M scripts/build-coverage.js
M server.js

Diff

commit 81aa19fb057141b32da111f92dd691e31db933e7
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Wed Jul 15 17:44:25 2026 -0700

    Live-scrape 504 fix: kill scrape at 52s (below the 60s CF/nginx proxy cap) → honest in-app 'timed out — use Email Vendor' JSON instead of a raw gateway 504; document the 3-step Kravet prod light-up recovery in build-coverage.js
---
 scripts/build-coverage.js |  9 +++++++++
 server.js                 | 13 ++++++++++---
 2 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/scripts/build-coverage.js b/scripts/build-coverage.js
index 73ca1a5..23711eb 100644
--- a/scripts/build-coverage.js
+++ b/scripts/build-coverage.js
@@ -6,6 +6,15 @@
 // to drive each card's "Check Live" enable/dim (data-driven, not hardcoded per vendor),
 // and the live worker reads the per-vendor `adapter` + `catalog_table` to route a scrape.
 //
+// KRAVET LIGHT-UP RECOVERY (prod): the DB_AUTHORITATIVE tier needs the app role `repl_user`
+// to have SELECT on kravet_authoritative_pricing (owned by dw_admin). Without it, kapMatchCount
+// catches the permission error → 0 resolvable → Kravet dims to Email-Vendor (graceful, honest).
+// To light Kravet on prod after the grant lands, THREE steps in order (hot-reload alone is NOT
+// enough — it only re-READS this file, it does not regenerate it):
+//   1. GRANT:  ssh root@45.61.58.125 "sudo -u postgres psql -d dw_unified -c 'GRANT SELECT ON kravet_authoritative_pricing TO repl_user'"
+//   2. REGEN:  ssh root@45.61.58.125 'cd /root/Projects/all-designerwallcoverings && node scripts/build-coverage.js'
+//   3. WAIT:   the server's mtime hot-reload picks up the new file within ≤10 min (or pm2 reload).
+//
 // Tiers (Steve's audit spec):
 //   PORTAL_CREDS    — backing vendor has a login portal AND trade creds on file → live-capable (metered).
 //   PUBLIC_STOCK    — backing storefront exposes stock with no login (Shopify .js / Woo Store API) → live-capable (metered headless).
diff --git a/server.js b/server.js
index aeb419e..e4e2a72 100644
--- a/server.js
+++ b/server.js
@@ -174,7 +174,12 @@ const LIVE_WORKER = path.join(__dirname, 'scripts', 'live-scrape.js');
 const LIVE_CACHE_MS = 10 * 60 * 1000;        // repeat clicks within 10 min = cached, $0
 const LIVE_MAX_INFLIGHT = 2;                 // global burst cap on concurrent portal scrapes
 const LIVE_IP_MAX = 6, LIVE_IP_WIN_MS = 60 * 1000; // per-IP: ≤6 live calls / 60s
-const LIVE_TIMEOUT_MS = 105 * 1000;          // hard kill a hung portal scrape
+// Kill the scrape BELOW the ~60s Cloudflare/nginx proxy cap (verified 2026-07-15: a ~63s
+// scrape 504'd through the public URL). At 52s the worker is killed and we return honest
+// in-app JSON ("timed out — use Email Vendor") that reaches the browser under the proxy cap,
+// instead of a raw gateway 504 that also hides that the session billed. Genuinely->52s portal
+// scrapes can't complete via the public URL anyway — async/poll is the real fix (deferred).
+const LIVE_TIMEOUT_MS = 52 * 1000;           // hard kill a hung/slow portal scrape (< proxy cap)
 const BB_RATE_PER_MIN = 0.15;                // pricing.json apis.browserbase.rates.session_min
 const BB_MIN_SESSION_MIN = 0.2;              // minimum billable Browserbase session (minutes)
 const COST_LOG = path.join(os.homedir(), '.claude', 'skills', 'cost-tracker', 'scripts', 'log.js');
@@ -864,8 +869,10 @@ function runLiveScrape(sku, adapterKey, catalogTable) {
         let parsed = null;
         try { parsed = JSON.parse((stdout || '').trim().split('\n').filter(Boolean).pop()); } catch { /* no JSON */ }
         if (!parsed) {
-          return resolve({ ok: false, live: true, live_available: true, sku,
-            reason: err ? ('live worker ' + (err.killed ? 'timed out' : 'error')) : 'no result from live worker', cost_usd: 0 });
+          const reason = err
+            ? (err.killed ? 'live check timed out — use Email Vendor to confirm stock' : 'live check error — use Email Vendor')
+            : 'no result from live worker';
+          return resolve({ ok: false, live: true, live_available: true, sku, reason, cost_usd: 0 });
         }
         if (parsed.available === false) {
           // A Browserbase session may have opened before failing → that IS billable, so log it.

← 8a79e7b Backfill mfr_sku for 163 mfr-less Kravet-family SKUs at $0 ( · back to All Designerwallcoverings · all-dw: add schumacher to BANNED regex (internal-only vendor fc6f2ba →