thibaut-refresh: use repeat-scraper (standalone refresher) — live-verified writes to local dw_unified, 0 scrape errors; run.sh tolerates the pre-existing Slack-webhook non-fatal exit

30ee316b5caf51ad969c83405ae7c0233214ebc5 · 2026-07-07 05:57:07 -0700 · steve

Files touched

M scripts/thibaut-refresh/run.sh

Diff

commit 30ee316b5caf51ad969c83405ae7c0233214ebc5
Author: steve <steve@designerwallcoverings.com>
Date:   Tue Jul 7 05:57:07 2026 -0700

    thibaut-refresh: use repeat-scraper (standalone refresher) — live-verified writes to local dw_unified, 0 scrape errors; run.sh tolerates the pre-existing Slack-webhook non-fatal exit
---
 scripts/thibaut-refresh/run.sh | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

diff --git a/scripts/thibaut-refresh/run.sh b/scripts/thibaut-refresh/run.sh
index 2bf774fb..a6bbb1cc 100755
--- a/scripts/thibaut-refresh/run.sh
+++ b/scripts/thibaut-refresh/run.sh
@@ -1,16 +1,14 @@
 #!/usr/bin/env bash
-# Thibaut refresh — REFERENCE wiring for the vendor-recrawl-dispatcher.
-# Pattern (WallQuest-style): scrape -> write LOCAL dw_unified staging -> price-finder -> CNCP.
-# NO PUBLISH. Writes go to postgresql://dw_admin@127.0.0.1:5432/dw_unified.
-# NOTE: uses thibaut-scrape-missing.js (writes thibaut_catalog + dw_sku_registry). A fuller
-# "refresh existing SKUs" scrape can replace the scrape line per Thibaut's needs (double-roll rule).
+# Thibaut refresh — REFERENCE wiring for vendor-recrawl-dispatcher.
+# Uses thibaut-repeat-scraper.js: a STANDALONE refresher that reads existing Thibaut
+# products from local dw_unified and re-scrapes price/specs (built-in politeness:
+# batch 5, 1.5s delay). Writes to postgresql://dw_admin@127.0.0.1:5432/dw_unified.
+# NO PUBLISH. Then price-finder + CNCP.
 set -uo pipefail
-DWS="$HOME/Projects/Designer-Wallcoverings/scripts"
-LOG="$DWS/thibaut-refresh/refresh.log"
+REPEAT="$HOME/Projects/Designer-Wallcoverings/shopify/scripts/thibaut-repeat-scraper.js"
+LOG="$HOME/Projects/Designer-Wallcoverings/scripts/thibaut-refresh/refresh.log"
 TS(){ date +%FT%T%z; }
 echo "[$(TS)] === Thibaut refresh START ===" >> "$LOG"
-cd "$DWS"
-node thibaut-scrape-missing.js >> "$LOG" 2>&1 || echo "[$(TS)] scrape rc=$?" >> "$LOG"
-# recompute costs/prices (no publish)
+node "$REPEAT" --limit "${THIBAUT_REFRESH_LIMIT:-200}" >> "$LOG" 2>&1 || echo "[$(TS)] scrape rc=$?" >> "$LOG"
 ( cd "$HOME/.claude/skills/price-finder" && node scripts/verify-price.js --vendor "Thibaut" --write 2>&1 | grep -iE "SKUs:|WROTE" ) >> "$LOG" 2>&1 || true
 echo "[$(TS)] === Thibaut refresh DONE ===" >> "$LOG"

← df78838a thibaut-refresh: reference vendor refresh wiring for the rec · back to Designer Wallcoverings · fix Slack webhook crash: 66 DW scripts had literal '${SLACK_ 98dd515a →