← back to Sanderson Onboard
TK-10400: durable Sanderson trade harvest via openclaw (kills daily puppeteer bot-block)
cc268fdaa486cfe7d4c7fcf39d32592ee4df1390 · 2026-08-10 09:14:01 -0700 · Steve Abrams
harvest_trade_openclaw.sh drives the persistent authed openclaw real-Chrome session (SDG
anti-bot wall lets it through) instead of bundled puppeteer (fingerprint-flagged every
unattended run -> daily NEEDS_REAUTH). Same output contract (data/*_raw.txt + trade_summary.json)
so parse_trade.py -> load_trade.sql -> reconcile.sql is unchanged. Fresh-tab auth (avoids the
heavy-PLP innerText gateway timeout), data-driven reauth detection (0 products both cats),
self-contained PATH for cron. refresh.sh Phase B now openclaw-primary + puppeteer auto-fallback.
Verified live: wallpaper 2433/2433 harvested, parse parsed 615 rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Files touched
M refresh.shA scripts/harvest_trade_openclaw.sh
Diff
commit cc268fdaa486cfe7d4c7fcf39d32592ee4df1390
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Mon Aug 10 09:14:01 2026 -0700
TK-10400: durable Sanderson trade harvest via openclaw (kills daily puppeteer bot-block)
harvest_trade_openclaw.sh drives the persistent authed openclaw real-Chrome session (SDG
anti-bot wall lets it through) instead of bundled puppeteer (fingerprint-flagged every
unattended run -> daily NEEDS_REAUTH). Same output contract (data/*_raw.txt + trade_summary.json)
so parse_trade.py -> load_trade.sql -> reconcile.sql is unchanged. Fresh-tab auth (avoids the
heavy-PLP innerText gateway timeout), data-driven reauth detection (0 products both cats),
self-contained PATH for cron. refresh.sh Phase B now openclaw-primary + puppeteer auto-fallback.
Verified live: wallpaper 2433/2433 harvested, parse parsed 615 rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
refresh.sh | 14 +++++-
scripts/harvest_trade_openclaw.sh | 89 +++++++++++++++++++++++++++++++++++++++
2 files changed, 101 insertions(+), 2 deletions(-)
diff --git a/refresh.sh b/refresh.sh
index 9d025c9..c0e10c9 100644
--- a/refresh.sh
+++ b/refresh.sh
@@ -19,8 +19,18 @@ python3 "$SC/palette.py" >> "$LOG" 2>&1 || true
[ -s palette_updates.sql ] && PGc -f palette_updates.sql >> "$LOG" 2>&1 || true
# --- Phase B: trade portal (auth, best-effort) ---
-TRADE=skipped
-if node "$SC/harvest_trade.js" >> "$LOG" 2>&1; then
+# DURABLE (TK-10400): harvest through the persistent authed openclaw real-Chrome session
+# (SDG's anti-bot wall lets it through). Bundled puppeteer (harvest_trade.js) is fingerprint-
+# flagged on every unattended run -> daily NEEDS_REAUTH; kept only as an automatic fallback
+# for a non-auth openclaw failure. Both write the SAME data/*_raw.txt, so the downstream
+# parse_trade.py -> load_trade.sql -> reconcile.sql is unchanged.
+TRADE=skipped; HARVESTED=0
+if bash "$SC/harvest_trade_openclaw.sh" >> "$LOG" 2>&1; then
+ HARVESTED=1
+elif [ ! -f "$DATA/NEEDS_REAUTH" ] && node "$SC/harvest_trade.js" >> "$LOG" 2>&1; then
+ echo "$(ts) openclaw harvest failed (non-auth) -> puppeteer fallback used" >> "$LOG"; HARVESTED=1
+fi
+if [ "$HARVESTED" = "1" ]; then
python3 "$SC/parse_trade.py" >> "$LOG" 2>&1 \
&& PGc -f "$SC/load_trade.sql" >> "$LOG" 2>&1 \
&& PGc -f "$SC/reconcile.sql" >> "$LOG" 2>&1 && TRADE=ok || TRADE=parse_or_load_fail
diff --git a/scripts/harvest_trade_openclaw.sh b/scripts/harvest_trade_openclaw.sh
new file mode 100755
index 0000000..9aa4f86
--- /dev/null
+++ b/scripts/harvest_trade_openclaw.sh
@@ -0,0 +1,89 @@
+#!/bin/bash
+# Daily trade-portal harvest THROUGH the persistent authed openclaw real-Chrome session.
+#
+# DURABLE replacement for harvest_trade.js: that script launched BUNDLED puppeteer, which
+# SDG's anti-bot wall fingerprint-flags on every unattended run -> NEEDS_REAUTH set daily
+# (TK-10400 / root cause behind TK-00058). openclaw drives Steve's real Chrome (persistent
+# 'openclaw' profile, CDP :18800), which the wall lets through, and the session persists
+# across daily runs until it naturally expires — so the cadence keeps flowing.
+#
+# Same output contract as harvest_trade.js (downstream parse_trade.py -> load_trade.sql ->
+# reconcile.sql is UNCHANGED): data/sanderson_<cat>_raw.txt + data/trade_summary.json,
+# and the same NEEDS_REAUTH marker on auth loss.
+#
+# Env: CAP_WALLPAPER / CAP_FABRIC override the load-more iteration caps (for a quick smoke test).
+set -uo pipefail
+export PATH="$HOME/.npm-global/bin:/opt/homebrew/bin:$PATH" # openclaw CLI + node under cron
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+DATA="$ROOT/data"; SC="$ROOT/scripts"
+BASE="https://trade.sandersondesigngroup.com/us"
+CAP_WALLPAPER="${CAP_WALLPAPER:-120}"
+CAP_FABRIC="${CAP_FABRIC:-260}"
+log(){ echo "$1"; echo "$(date -u +%FT%TZ) $1" >> "$DATA/trade_harvest.log"; }
+
+# 0. openclaw browser must be up (persistent authed profile)
+if ! openclaw browser status 2>/dev/null | grep -q "running: true"; then
+ log "AUTH/BROWSER: openclaw browser not running -> NEEDS_REAUTH (start via scripts/reauth.sh)"
+ echo "$(date -u +%FT%TZ) openclaw browser down -> openclaw reauth" > "$DATA/NEEDS_REAUTH"
+ exit 3
+fi
+
+# 1. Open a FRESH light tab for the run. Reusing a pre-loaded PLP tab (e.g. fabric
+# #limit=3120) makes document.body.innerText huge, and the openclaw CDP gateway
+# times out (~20s) reading it -> a FALSE "auth lost". A fresh tab starts light;
+# openclaw_harvest_cat.sh then navigates it per category and loads incrementally.
+list_sand(){ openclaw browser tabs --json 2>/dev/null | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{const j=JSON.parse(s);for(const t of (j.tabs||[]))if(t.type==="page"&&/trade\.sandersondesigngroup\.com/.test(t.url||""))console.log(t.targetId);}catch(e){}})'; }
+before="$(list_sand)"
+openclaw browser open "$BASE/" >/dev/null 2>&1; sleep 6
+SAND="$(comm -13 <(echo "$before" | sort) <(list_sand | sort) | head -1)"
+[ -z "$SAND" ] && SAND="$(list_sand | head -1)" # fallback: reuse an existing tab
+[ -z "$SAND" ] && { log "AUTH: no Sanderson tab and open failed -> NEEDS_REAUTH"; echo "$(date -u +%FT%TZ) no sanderson tab -> openclaw reauth" > "$DATA/NEEDS_REAUTH"; exit 3; }
+FRESH_TAB="$([ -n "$(comm -13 <(echo "$before" | sort) <(list_sand | sort))" ] && echo 1 || echo 0)"
+log "using openclaw target $SAND (fresh=$FRESH_TAB)"
+
+# 2. auth check — cheap selector query (NOT a full innerText scan, which times out on
+# a heavy PLP). Retries once, and treats a timeout/empty read as UNKNOWN -> proceeds
+# to harvest rather than false-failing; the harvest ('Viewing X of Y' vs login text)
+# is the real auth proof, and only a definite 'NO' writes NEEDS_REAUTH.
+authcheck(){ openclaw browser evaluate --timeout 30000 --target-id "$SAND" --fn "() => { const q=document.querySelector('a[href*=\"logout\" i],a[href*=\"signout\" i],a[href*=\"account\" i],[class*=\"account\" i]'); const h=(document.body.innerText||'').slice(0,3000).toLowerCase(); return (q||h.includes('my account')||h.includes('quick order')||h.includes('logout'))?'YES':'NO'; }" 2>/dev/null | tr -d '[:space:]"'; }
+AUTH="$(authcheck)"; [ "$AUTH" != "YES" ] && { sleep 3; AUTH="$(authcheck)"; }
+if [ "$AUTH" = "NO" ]; then
+ log "AUTH LOST -> NEEDS_REAUTH written; skipping trade harvest (rerun scripts/reauth.sh openclaw login)"
+ echo "$(date -u +%FT%TZ) trade portal session expired -> openclaw reauth" > "$DATA/NEEDS_REAUTH"
+ exit 3
+fi
+[ "$AUTH" = "YES" ] && rm -f "$DATA/NEEDS_REAUTH" || log "AUTH UNKNOWN (evaluate empty/timeout) — proceeding; harvest is the real test"
+
+# 3. harvest each category through the authed tab (existing per-cat primitive)
+for cat in wallpaper fabric; do
+ cap_var="CAP_${cat^^}"; cap="${!cap_var}"
+ log "harvesting $cat via openclaw (cap=$cap)..."
+ bash "$SC/openclaw_harvest_cat.sh" "$SAND" "$cat" "$cap" >> "$DATA/trade_harvest.log" 2>&1 \
+ || log "$cat harvest returned nonzero (partial raw kept — dump guard preserves last good data)"
+done
+
+# 4. summary from the raw dumps (Viewing X of Y)
+node -e '
+const fs=require("fs");const DATA=process.argv[1];const out={};
+for(const cat of ["wallpaper","fabric"]){
+ let shown=0,total=0;
+ try{const t=fs.readFileSync(DATA+"/sanderson_"+cat+"_raw.txt","utf8");const m=t.match(/Viewing\s+(\d+)\s+of\s+(\d+)/i);if(m){shown=+m[1];total=+m[2];}}catch(e){}
+ out[cat]={shown,total};
+}
+fs.writeFileSync(DATA+"/trade_summary.json",JSON.stringify(out,null,2));
+// Data-driven auth-loss detection: a logged-out session shows NO "Viewing X of Y" on
+// any product page. If BOTH categories harvested 0, treat it as a real reauth need
+// (backstop for the flaky live auth-evaluate).
+if(out.wallpaper.total===0 && out.fabric.total===0){
+ fs.writeFileSync(DATA+"/NEEDS_REAUTH", new Date().toISOString()+" 0 products across both categories -> likely logged out; rerun scripts/reauth.sh openclaw login\n");
+ console.log("TRADE HARVEST: 0/0 both cats -> NEEDS_REAUTH written");
+ process.exit(3);
+}
+console.log("TRADE HARVEST OK "+JSON.stringify(out));
+' "$DATA"
+HARVEST_RC=$?
+
+# 5. close the fresh tab we opened (don't leak a tab per daily run); leave a reused tab alone
+[ "${FRESH_TAB:-0}" = "1" ] && openclaw browser close "$SAND" >/dev/null 2>&1 && log "closed fresh openclaw tab $SAND"
+log "openclaw trade harvest complete (rc=${HARVEST_RC:-0})"
+exit "${HARVEST_RC:-0}"
← 85f2103 auto-data-snapshot: 2026-08-10T08:54:43 (1 data files) — dat
·
back to Sanderson Onboard
·
openclaw real-Chrome trade-harvester: crash-safe cumulative d174e0d →