[object Object]

← back to Designer Wallcoverings

Fix importer throttle bug (require real pid for success) + batched Schumacher re-import

fd28444b107e2c212ba25417fcf8ea63ef973a77 · 2026-06-11 13:45:35 -0700 · SteveStudio2

createProduct now returns ok:false when no product id comes back (throttle-exhausted/empty
response previously mis-counted as created, inflating CREATED N while making nothing).
gqlRetry: 8 attempts, 3s*n backoff, proactive 1.8s backoff when leaky-bucket < 400.
schu-reimport.sh re-creates the ~2,558 genuinely-uncreated Schumacher SKUs in 200-batches
with 20s pauses; idempotent (net-new filter), no-progress guard. Not a dup risk.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Files touched

Diff

commit fd28444b107e2c212ba25417fcf8ea63ef973a77
Author: SteveStudio2 <stevestudio2@SteveStudio2s-Mac-Studio.local>
Date:   Thu Jun 11 13:45:35 2026 -0700

    Fix importer throttle bug (require real pid for success) + batched Schumacher re-import
    
    createProduct now returns ok:false when no product id comes back (throttle-exhausted/empty
    response previously mis-counted as created, inflating CREATED N while making nothing).
    gqlRetry: 8 attempts, 3s*n backoff, proactive 1.8s backoff when leaky-bucket < 400.
    schu-reimport.sh re-creates the ~2,558 genuinely-uncreated Schumacher SKUs in 200-batches
    with 20s pauses; idempotent (net-new filter), no-progress guard. Not a dup risk.
    
    Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 shopify/scripts/cadence/cadence-import.js | 20 ++++++++++++++++----
 shopify/scripts/cadence/schu-reimport.sh  | 25 +++++++++++++++++++++++++
 2 files changed, 41 insertions(+), 4 deletions(-)

diff --git a/shopify/scripts/cadence/cadence-import.js b/shopify/scripts/cadence/cadence-import.js
index 7ad4e029..b80cd7f5 100644
--- a/shopify/scripts/cadence/cadence-import.js
+++ b/shopify/scripts/cadence/cadence-import.js
@@ -106,9 +106,17 @@ function gql(query, variables) {
   });
 }
 async function gqlRetry(q, v) {
-  for (let a=0;a<5;a++){ const r=await gql(q,v); const t=r.json&&r.json.errors&&JSON.stringify(r.json.errors).includes('THROTTLED');
-    if (r.status===429||t){await sleep(2000*(a+1));continue;} await sleep(550); return r; }
-  return { status: 429, raw: 'throttled' };
+  for (let a=0;a<8;a++){
+    const r=await gql(q,v);
+    const errs = r.json && r.json.errors ? JSON.stringify(r.json.errors) : '';
+    if (r.status===429 || errs.includes('THROTTLED')) { await sleep(3000*(a+1)); continue; }
+    // proactive backoff when Shopify's leaky-bucket is running low
+    const avail = r.json && r.json.extensions && r.json.extensions.cost && r.json.extensions.cost.throttleStatus
+      ? r.json.extensions.cost.throttleStatus.currentlyAvailable : null;
+    await sleep(avail != null && avail < 400 ? 1800 : 450);
+    return r;
+  }
+  return { status: 429, raw: 'throttled-exhausted' };
 }
 
 // ---- ensure dw_unified logs our computed price FIRST (Steve's hard rule) ----
@@ -208,7 +216,11 @@ async function createProduct(table, row, payload) {
   const ue = r.json?.data?.productSet?.userErrors;
   if (ue && ue.length) return { ok:false, err: ue };
   const pid = r.json?.data?.productSet?.product?.id;
-  if (pid) psql(`UPDATE ${table} SET shopify_product_id='${String(pid).replace(/.*\//,'')}' WHERE dw_sku='${row.dw_sku.replace(/'/g,"''")}';`);
+  // CRITICAL: only count as created when a real product id comes back. A throttle-exhausted
+  // or empty response has no userErrors AND no pid — previously that was mis-counted as ok,
+  // inflating "CREATED N" while nothing was made (and the row stayed net-new → re-run safe).
+  if (!pid) return { ok:false, err: r.raw || 'no-pid (throttled/empty response)' };
+  psql(`UPDATE ${table} SET shopify_product_id='${String(pid).replace(/.*\//,'')}' WHERE dw_sku='${row.dw_sku.replace(/'/g,"''")}';`);
   return { ok:true, pid };
 }
 
diff --git a/shopify/scripts/cadence/schu-reimport.sh b/shopify/scripts/cadence/schu-reimport.sh
new file mode 100755
index 00000000..0c40ea42
--- /dev/null
+++ b/shopify/scripts/cadence/schu-reimport.sh
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# Re-create the genuinely-uncreated Schumacher SKUs in SMALL batches with throttle-safe
+# pacing. The cadence importer now (a) only counts a create when a real product id comes
+# back, (b) backs off proactively on a low leaky-bucket. So each batch creates what it can;
+# throttle-failed rows stay net-new and the next batch picks them up. Idempotent + safe.
+set -u
+export PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin
+DIR="$HOME/Projects/Designer-Wallcoverings/shopify/scripts/cadence"
+LOG="$DIR/data/schu-reimport.log"
+cd "$DIR" || exit 1
+NETNEW_SQL="SELECT count(*) FROM schumacher_catalog WHERE coalesce(shopify_product_id::text,'')='' AND cost>0 AND NOT coalesce(excluded,false) AND dw_sku IS NOT NULL AND dw_sku<>'';"
+
+echo "$(date '+%F %T') ===== schumacher re-import start =====" >> "$LOG"
+prev=999999
+for i in $(seq 1 25); do
+  nn=$(psql -At -d dw_unified -c "$NETNEW_SQL" 2>/dev/null)
+  echo "$(date '+%F %T') batch $i — net-new=$nn" >> "$LOG"
+  [ "${nn:-0}" -lt 5 ] && { echo "$(date '+%F %T') net-new cleared ($nn) — done." >> "$LOG"; break; }
+  # no-progress guard: if a full batch created nothing (all genuinely un-creatable), stop
+  if [ "${nn:-0}" -ge "$prev" ] && [ "$i" -gt 1 ]; then echo "$(date '+%F %T') no progress (still $nn) — stopping." >> "$LOG"; break; fi
+  prev=$nn
+  node cadence-import.js --only Schumacher --skus 200 --commit >> "$LOG" 2>&1
+  sleep 20   # let the rate-limit bucket refill between batches
+done
+echo "$(date '+%F %T') ===== schumacher re-import done — final net-new=$(psql -At -d dw_unified -c "$NETNEW_SQL" 2>/dev/null) =====" >> "$LOG"

← 70d046b0 Add by-sku reconcile; confirmed missing Schumacher SKUs genu  ·  back to Designer Wallcoverings  ·  Romo 339 cost backfill: pulled current trade prices from the d05c6b58 →