all.dw: raise live-scrape worker timeout 52s->75s (async/poll decoupled from proxy cap; genuine ~63s scrapes now complete)

76b03727337e78a0fe28bcd7c76c2f857f28fd29 · 2026-07-27 14:45:14 -0700 · Steve

Files touched

M server.js

Diff

commit 76b03727337e78a0fe28bcd7c76c2f857f28fd29
Author: Steve <steve@designerwallcoverings.com>
Date:   Mon Jul 27 14:45:14 2026 -0700

    all.dw: raise live-scrape worker timeout 52s->75s (async/poll decoupled from proxy cap; genuine ~63s scrapes now complete)
---
 server.js | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/server.js b/server.js
index 696bf91..1c8505b 100644
--- a/server.js
+++ b/server.js
@@ -174,12 +174,16 @@ const LIVE_WORKER = process.env.LIVE_WORKER || path.join(__dirname, 'scripts', '
 const LIVE_CACHE_MS = 10 * 60 * 1000;        // repeat clicks within 10 min = cached, $0
 const LIVE_MAX_INFLIGHT = 2;                 // global burst cap on concurrent portal scrapes
 const LIVE_IP_MAX = 6, LIVE_IP_WIN_MS = 60 * 1000; // per-IP: ≤6 live calls / 60s
-// Kill the scrape BELOW the ~60s Cloudflare/nginx proxy cap (verified 2026-07-15: a ~63s
-// scrape 504'd through the public URL). At 52s the worker is killed and we return honest
-// in-app JSON ("timed out — use Email Vendor") that reaches the browser under the proxy cap,
-// instead of a raw gateway 504 that also hides that the session billed. Genuinely->52s portal
-// scrapes can't complete via the public URL anyway — async/poll is the real fix (deferred).
-const LIVE_TIMEOUT_MS = 52 * 1000;           // hard kill a hung/slow portal scrape (< proxy cap)
+// Worker hard-kill for a hung/slow portal scrape. Was 52s to stay UNDER the ~60s Cloudflare/
+// nginx proxy cap (2026-07-15: a ~63s scrape 504'd through the public URL). That cap no longer
+// binds the worker — the async job+poll path (2026-07-27) decouples the HTTP request from the
+// scrape, so the initiating request returns instantly and the client polls. Raised to 75s (DTD
+// 5/5, 2026-07-27) so a genuine ~63s WallQuest/Romo scrape actually COMPLETES and returns a live
+// answer instead of being killed at 52s and falsely surfaced as "timed out — use Email Vendor".
+// Cost stays bounded: only the slowest scrapes use the extra ~20s (~$0.03 more at $0.15/session-
+// min), under the $5/day kill-switch + the global burst cap of 2. The client poll budget (30×3s
+// = ~90s) already exceeds 75s, so a completing scrape resolves before the poll loop gives up.
+const LIVE_TIMEOUT_MS = 75 * 1000;           // hard kill a hung/slow portal scrape (async/poll decoupled from proxy cap)
 const BB_RATE_PER_MIN = 0.15;                // pricing.json apis.browserbase.rates.session_min
 const BB_MIN_SESSION_MIN = 0.2;              // minimum billable Browserbase session (minutes)
 const COST_LOG = path.join(os.homedir(), '.claude', 'skills', 'cost-tracker', 'scripts', 'log.js');

← ad479fd all.dw: async job+poll for metered live check (never 504s th · back to All Designerwallcoverings · auto-save: 2026-07-27T14:51:27 (1 files) — server.js ae33f6d →