← back to Sanderson Onboard
TK-10400 Cody-gate fixes: reauth->openclaw, tab-race, freshness gate (kill silent stale reload)
c895b5675c77b79475cd104b97e25d3c2c6526e8 · 2026-08-10 09:24:52 -0700 · Steve Abrams
- HOLE 1 (critical): reauth.sh was logging into the DEAD puppeteer chrome-profile, not the
openclaw session the harvester now uses -> reauth theater. Rewrote reauth.sh to open the
SDG login in the openclaw Chrome + confirm via openclaw evaluate.
- HOLE 2: collapsed the two racing comm -13 tab diffs into one (was closing a tab it didn't open).
- HOLE 3 (silent data corruption): the dump guard only overwrites when the new dump is bigger,
so an expired session left the prior full raw file untouched and a naive summary reloaded stale
prices. Added an mtime freshness gate: a category is trusted ONLY if its raw file was written
THIS run; 0 fresh -> NEEDS_REAUTH, stale never reloaded. Auth-signal-independent (also covers
HOLE 4's unverified public-teaser-PLP assumption).
- Verified: cap=6 empty-slate run writes fresh files + passes the gate (no false reauth);
cap=2 (no dump) correctly writes NEEDS_REAUTH; openclaw evaluate reads a fresh light PLP fine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Files touched
M scripts/harvest_trade_openclaw.shM scripts/reauth.sh
Diff
commit c895b5675c77b79475cd104b97e25d3c2c6526e8
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Mon Aug 10 09:24:52 2026 -0700
TK-10400 Cody-gate fixes: reauth->openclaw, tab-race, freshness gate (kill silent stale reload)
- HOLE 1 (critical): reauth.sh was logging into the DEAD puppeteer chrome-profile, not the
openclaw session the harvester now uses -> reauth theater. Rewrote reauth.sh to open the
SDG login in the openclaw Chrome + confirm via openclaw evaluate.
- HOLE 2: collapsed the two racing comm -13 tab diffs into one (was closing a tab it didn't open).
- HOLE 3 (silent data corruption): the dump guard only overwrites when the new dump is bigger,
so an expired session left the prior full raw file untouched and a naive summary reloaded stale
prices. Added an mtime freshness gate: a category is trusted ONLY if its raw file was written
THIS run; 0 fresh -> NEEDS_REAUTH, stale never reloaded. Auth-signal-independent (also covers
HOLE 4's unverified public-teaser-PLP assumption).
- Verified: cap=6 empty-slate run writes fresh files + passes the gate (no false reauth);
cap=2 (no dump) correctly writes NEEDS_REAUTH; openclaw evaluate reads a fresh light PLP fine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
scripts/harvest_trade_openclaw.sh | 29 ++++++++++++++++++---
scripts/reauth.sh | 55 ++++++++++++++++++++++++++-------------
2 files changed, 63 insertions(+), 21 deletions(-)
diff --git a/scripts/harvest_trade_openclaw.sh b/scripts/harvest_trade_openclaw.sh
index 9aa4f86..9ab2bbc 100755
--- a/scripts/harvest_trade_openclaw.sh
+++ b/scripts/harvest_trade_openclaw.sh
@@ -33,12 +33,16 @@ fi
# times out (~20s) reading it -> a FALSE "auth lost". A fresh tab starts light;
# openclaw_harvest_cat.sh then navigates it per category and loads incrementally.
list_sand(){ openclaw browser tabs --json 2>/dev/null | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{const j=JSON.parse(s);for(const t of (j.tabs||[]))if(t.type==="page"&&/trade\.sandersondesigngroup\.com/.test(t.url||""))console.log(t.targetId);}catch(e){}})'; }
+RUN_START="$(date +%s)"
before="$(list_sand)"
openclaw browser open "$BASE/" >/dev/null 2>&1; sleep 6
-SAND="$(comm -13 <(echo "$before" | sort) <(list_sand | sort) | head -1)"
-[ -z "$SAND" ] && SAND="$(list_sand | head -1)" # fallback: reuse an existing tab
+# Compute the new-tab diff ONCE (two independent diffs race: the delayed tab could appear
+# between them, making FRESH_TAB=1 while SAND is a pre-existing tab -> we'd close a tab we
+# never opened). Single source of truth: new_tabs.
+new_tabs="$(comm -13 <(echo "$before" | sort) <(list_sand | sort))"
+SAND="$(echo "$new_tabs" | grep -v '^$' | head -1)"
+if [ -n "$SAND" ]; then FRESH_TAB=1; else SAND="$(list_sand | head -1)"; FRESH_TAB=0; fi # fallback: reuse
[ -z "$SAND" ] && { log "AUTH: no Sanderson tab and open failed -> NEEDS_REAUTH"; echo "$(date -u +%FT%TZ) no sanderson tab -> openclaw reauth" > "$DATA/NEEDS_REAUTH"; exit 3; }
-FRESH_TAB="$([ -n "$(comm -13 <(echo "$before" | sort) <(list_sand | sort))" ] && echo 1 || echo 0)"
log "using openclaw target $SAND (fresh=$FRESH_TAB)"
# 2. auth check — cheap selector query (NOT a full innerText scan, which times out on
@@ -62,6 +66,25 @@ for cat in wallpaper fabric; do
|| log "$cat harvest returned nonzero (partial raw kept — dump guard preserves last good data)"
done
+# 3b. FRESHNESS GATE (Cody HOLE 3/4): the dump guard in openclaw_harvest_cat.sh only
+# overwrites the raw file when the new dump is BIGGER. So an expired session (small login
+# page) leaves the PRIOR run's large raw file untouched — and a naive summary would read
+# that stale file, look healthy, and silently reload old prices. Trust a category ONLY if
+# its raw file was actually written THIS run (mtime >= RUN_START). This is auth-signal-
+# independent, so it also covers the unverified "public teaser PLP" case.
+mt(){ stat -f %m "$1" 2>/dev/null || echo 0; }
+FRESH=0
+for cat in wallpaper fabric; do
+ f="$DATA/sanderson_${cat}_raw.txt"
+ if [ -f "$f" ] && [ "$(mt "$f")" -ge "$RUN_START" ]; then FRESH=$((FRESH+1)); else log "STALE: $cat raw not updated this run (dump rejected / auth loss) — will NOT be trusted"; fi
+done
+if [ "$FRESH" -eq 0 ]; then
+ log "NO category harvested fresh data this run -> NEEDS_REAUTH; stale data NOT reloaded downstream"
+ echo "$(date -u +%FT%TZ) no fresh harvest (both raw files stale) -> openclaw reauth" > "$DATA/NEEDS_REAUTH"
+ [ "${FRESH_TAB:-0}" = "1" ] && openclaw browser close "$SAND" >/dev/null 2>&1
+ exit 3
+fi
+
# 4. summary from the raw dumps (Viewing X of Y)
node -e '
const fs=require("fs");const DATA=process.argv[1];const out={};
diff --git a/scripts/reauth.sh b/scripts/reauth.sh
index 8ae17ea..5f1f5f5 100644
--- a/scripts/reauth.sh
+++ b/scripts/reauth.sh
@@ -1,20 +1,39 @@
#!/bin/bash
# Headed re-login for the SDG trade portal when the daily cadence flags NEEDS_REAUTH.
-# Opens a browser; Steve types the login; cookies persist to chrome-profile so the headless daily job resumes.
-export ROOT="$(cd "$(dirname "$0")/.." && pwd)"
-node -e '
-const puppeteer=require("/Users/macstudio3/Projects/Designer-Wallcoverings/DW-Programming/ImportNewSkufromURL/node_modules/puppeteer");
-const ROOT=process.env.ROOT, PROFILE=ROOT+"/chrome-profile", DATA=ROOT+"/data";
-const sleep=ms=>new Promise(r=>setTimeout(r,ms));
-(async()=>{
- const b=await puppeteer.launch({headless:false,defaultViewport:null,userDataDir:PROFILE,args:["--no-sandbox","--start-maximized","--disable-blink-features=AutomationControlled"]});
- const p=(await b.pages())[0]||await b.newPage();
- await p.goto("https://trade.sandersondesigngroup.com/us/login",{waitUntil:"networkidle2"});
- console.log("Log in in the window. Waiting up to 5 min...");
- for(let i=0;i<100;i++){ await sleep(3000);
- const inz=await p.evaluate(()=>{const t=document.body.innerText.toLowerCase();return t.includes("quick order")||t.includes("my account")||t.includes("sign out");});
- if(inz){ require("fs").rmSync(DATA+"/NEEDS_REAUTH",{force:true}); console.log("LOGGED IN — session persisted."); break; }
- }
- await sleep(2000); await b.close();
-})();'
-echo "reauth complete (if you logged in). NEEDS_REAUTH cleared if session detected."
+#
+# TK-10400: the daily harvest now runs through the PERSISTENT openclaw real-Chrome session
+# (harvest_trade_openclaw.sh), NOT bundled puppeteer. So re-auth MUST log into that same
+# openclaw session — logging into puppeteer's old ROOT/chrome-profile is theater (it never
+# touches the session the harvester uses, so the next cron run re-flags NEEDS_REAUTH).
+# This opens the SDG login page IN the openclaw Chrome and waits for Steve to sign in there;
+# the cookie then persists in the openclaw profile across daily runs until it naturally expires.
+set -uo pipefail
+export PATH="$HOME/.npm-global/bin:/opt/homebrew/bin:$PATH"
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"; DATA="$ROOT/data"
+BASE="https://trade.sandersondesigngroup.com/us"
+
+if ! openclaw browser status 2>/dev/null | grep -q "running: true"; then
+ echo "openclaw browser is not running. Start it first (openclaw browser start), then rerun this."
+ exit 1
+fi
+
+echo "Opening the SDG trade login in the openclaw Chrome window — log in there."
+openclaw browser open "$BASE/login" >/dev/null 2>&1
+sleep 4
+
+find_target(){ openclaw browser tabs --json 2>/dev/null | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{const j=JSON.parse(s);const t=(j.tabs||[]).find(x=>x.type==="page"&&/trade\.sandersondesigngroup\.com/.test(x.url||""));process.stdout.write(t?t.targetId:"");}catch(e){}})'; }
+SAND="$(find_target)"
+[ -z "$SAND" ] && { echo "Could not find/open the SDG tab in openclaw. Open $BASE/login manually in the openclaw window, log in, then rerun."; exit 1; }
+
+echo "Waiting up to 5 min for a logged-in session (openclaw target $SAND)..."
+for i in $(seq 1 100); do
+ sleep 3
+ IN="$(openclaw browser evaluate --timeout 20000 --target-id "$SAND" --fn "() => { const q=document.querySelector('a[href*=\"logout\" i],a[href*=\"signout\" i],a[href*=\"account\" i]'); const h=(document.body.innerText||'').slice(0,3000).toLowerCase(); return (q||h.includes('my account')||h.includes('quick order')||h.includes('sign out'))?'YES':'NO'; }" 2>/dev/null | tr -d '[:space:]"')"
+ if [ "$IN" = "YES" ]; then
+ rm -f "$DATA/NEEDS_REAUTH"
+ echo "LOGGED IN — openclaw session persisted; NEEDS_REAUTH cleared. Daily cadence will resume."
+ exit 0
+ fi
+done
+echo "Timed out waiting for login. If you did log in, the daily run will detect it; otherwise rerun this."
+exit 1
← d174e0d openclaw real-Chrome trade-harvester: crash-safe cumulative
·
back to Sanderson Onboard
·
Sanderson: harvest product width from public retail API ($0, fa05ca1 →