← back to Designer Wallcoverings

TK-10717: harden stage-only tripwire — scan full new-products scraper tree + shared libs

dc7e304a5ec2e190bda4c5f22a9ef4c16d6e2d22 · 2026-08-22 11:11:33 -0700 · Steve

Extend vendor-overnight-refresh.sh's forward-drift guard beyond the orchestrator to
the full dynamic-import-reachable surface (all 200 lib/scrapers/*-new-products-scraper.ts
+ the shared ../lib helpers they import), using the full write/publish primitive set
(designer-laboratory-sandbox, SHOPIFY_ADMIN_TOKEN, admin/api/2024, productCreate,
productUpdate, productVariantsBulk, go-live/golive, myshopify*admin, --apply). Uses a
bash array (not a word-split ls-string) so the scan is shell-robust — a string relied on
IFS word-splitting that zsh omits, which would have silently passed 200 files as one arg
and disabled the guard. Verified clean on the current tree, trips on a planted
productCreate line (exit 3), and no false positive on fortuny (reads its own
.myshopify.com storefront) or scalamandre (PRODUCTSTATUS:'active' search filter).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

• A DW-Programming/ImportNewSkufromURL/scripts/vendor-overnight-refresh.sh

Diff

commit dc7e304a5ec2e190bda4c5f22a9ef4c16d6e2d22
Author: Steve <steve@designerwallcoverings.com>
Date:   Sat Aug 22 11:11:33 2026 -0700

    TK-10717: harden stage-only tripwire — scan full new-products scraper tree + shared libs
    
    Extend vendor-overnight-refresh.sh's forward-drift guard beyond the orchestrator to
    the full dynamic-import-reachable surface (all 200 lib/scrapers/*-new-products-scraper.ts
    + the shared ../lib helpers they import), using the full write/publish primitive set
    (designer-laboratory-sandbox, SHOPIFY_ADMIN_TOKEN, admin/api/2024, productCreate,
    productUpdate, productVariantsBulk, go-live/golive, myshopify*admin, --apply). Uses a
    bash array (not a word-split ls-string) so the scan is shell-robust — a string relied on
    IFS word-splitting that zsh omits, which would have silently passed 200 files as one arg
    and disabled the guard. Verified clean on the current tree, trips on a planted
    productCreate line (exit 3), and no false positive on fortuny (reads its own
    .myshopify.com storefront) or scalamandre (PRODUCTSTATUS:'active' search filter).
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 .../scripts/vendor-overnight-refresh.sh            | 115 +++++++++++++++++++++
 1 file changed, 115 insertions(+)

diff --git a/DW-Programming/ImportNewSkufromURL/scripts/vendor-overnight-refresh.sh b/DW-Programming/ImportNewSkufromURL/scripts/vendor-overnight-refresh.sh
new file mode 100755
index 00000000..c2dc5899
--- /dev/null
+++ b/DW-Programming/ImportNewSkufromURL/scripts/vendor-overnight-refresh.sh
@@ -0,0 +1,115 @@
+#!/usr/bin/env bash
+# ============================================================================
+# All-Vendor OVERNIGHT STAGE-ONLY refresh  (TK-10717, 2026-08-20)
+# ----------------------------------------------------------------------------
+# REVIVES the dead all-vendor overnight cron. The engine (run-all-vendor-scrapers.ts,
+# driven by REFRESH_MODE=monthly) already existed and is STAGE-ONLY, but nothing
+# scheduled it (no plist, no log) so the fleet-wide refresh never ran.
+#
+# What it does: iterates the vendors whose stable day-of-month hash == today
+# (drawn from the healthcheck's VERIFIED set — ~4-5 vendors/day, so the 191
+# catalog_table vendors spread across the month and no single night hammers the
+# box), re-scrapes each through its OWN <vendor>-new-products-scraper.ts, and
+# upserts the results into dw_unified STAGING tables ONLY (vendors / products /
+# scraper_test_results).
+#
+# HARD INVARIANT — STAGE ONLY (enforced two ways):
+#   1) The engine run-all-vendor-scrapers.ts contains ZERO Shopify / go-live /
+#      productCreate / productUpdate / --apply references (proven by grep, TK-10717).
+#   2) THIS wrapper RE-GREPS the engine at runtime and ABORTS before scraping if
+#      any publish path was ever added — a stage-only tripwire so a future edit
+#      can never silently turn the overnight cron into a live publisher.
+#
+# Cadence: nightly 02:30 via com.steve.dw-vendor-overnight-refresh.plist; the
+# monthly-day hash inside the engine means only ~4-5 vendors actually scrape each
+# night. Cost: $0 (local scrapers + local PG). Some vendor scrapers may use
+# Browserbase/2captcha (metered) — but this refresh runs only the day's small
+# slice, never a 191-vendor blast.
+# ============================================================================
+set -uo pipefail
+export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin"
+export REFRESH_MODE=monthly   # engine → only today's day-hash vendors from the verified set
+
+ENGINE="$HOME/Projects/Designer-Wallcoverings/DW-Programming/ImportNewSkufromURL"
+RUNNER="scripts/run-all-vendor-scrapers.ts"
+LOG="/tmp/dw-vendor-overnight-refresh.log"
+# Dedicated kill-switch (TK-10717). Was ~/.dw-fixer-stop, but that flag is scoped to the
+# hollywood-create-resume generator (set 2026-08-18 for the DWHW2-* re-mint regression) and
+# collaterally froze this UNRELATED stage-only sweep every night. This sweep only writes
+# *_catalog staging (double-tripwire-guarded — no Shopify/publish/mint path), so it owns its
+# own stop flag. To pause it:  touch ~/.dw-overnight-refresh-stop
+KILL="$HOME/.dw-overnight-refresh-stop"
+
+cd "$ENGINE" || { echo "$(date '+%F %T') FATAL cd $ENGINE failed" >>"$LOG"; exit 1; }
+exec >>"$LOG" 2>&1
+echo "===== $(date '+%F %T') vendor-overnight-refresh START (day $(date '+%d'), STAGE-ONLY) ====="
+[ -f "$KILL" ] && { echo "kill-switch $KILL present — skip"; exit 0; }
+
+# --- STAGE-ONLY TRIPWIRE (orchestrator): refuse to run if the engine gained any publish path ---
+if grep -nEi "shopify|productCreate|productUpdate|go-live|golive|--apply|admin/api|graphql\.json|myshopify" "$RUNNER" >/dev/null 2>&1; then
+  echo "ABORT: stage-only tripwire — $RUNNER now contains a Shopify/publish reference."
+  echo "       Refusing to run so the overnight cron cannot publish. Inspect $RUNNER."
+  grep -nEi "shopify|productCreate|productUpdate|go-live|golive|--apply|admin/api|graphql\.json|myshopify" "$RUNNER"
+  exit 2
+fi
+echo "stage-only tripwire OK: $RUNNER has no Shopify/publish path."
+
+# --- STAGE-ONLY TRIPWIRE (whole scraper tree + shared libs): forward-drift guard ---
+# (TK-10717 CAVEAT #1 close — 2026-08-22 hardening)
+# The orchestrator grep above only covers run-all-vendor-scrapers.ts. runVendorScraper()
+# dynamically imports ${vendorId}-new-products-scraper.ts (run-all-vendor-scrapers.ts:361),
+# and those ~200 scrapers in turn import a small set of shared ../lib/* helpers. A FUTURE edit
+# could add a direct DW-STORE write inside ANY of those files and never trip the orchestrator
+# guard. So we scan the FULL dynamic-import-reachable surface: every *-new-products-scraper.ts
+# PLUS the shared helper modules they import.
+#
+# We match DW-STORE *WRITE*/publish primitives only — NOT the bare word "shopify" and NOT a bare
+# "*.myshopify.com" (which legitimately appear when a scraper READS a vendor's OWN Shopify
+# storefront, e.g. fortuny reads fortuny-4872.myshopify.com via a Storefront token, and
+# scalamandre passes PRODUCTSTATUS:'active' as a SEARCH FILTER — both vendor-side reads, not DW
+# publishes). The 'myshopify.*admin' clause requires an admin surface AFTER the myshopify domain,
+# so fortuny's read '.myshopify.com' does not trip it. Verified CLEAN across all 200 scrapers +
+# the shared helpers on the current tree (TK-10717), zero false positives on fortuny/scalamandre.
+SCRAPERS_DIR="lib/scrapers"
+# Shared ../lib helpers the new-products scrapers import (dynamic-import-reachable, extend if a
+# scraper starts importing a new shared module). Missing files are skipped harmlessly.
+SHARED_LIBS=(
+  lib/browserbase-connect.ts
+  lib/browserbase-helper.ts
+  lib/brightdata-proxy.ts
+  lib/vendor-timeouts.ts
+  lib/scraper-wrapper.ts
+  lib/schema-helper.ts
+  lib/puppeteer-preflight.ts
+  lib/database/postgres-client.ts
+)
+# Full write/publish primitive set (per TK-10717 spec): live store domain, admin token, admin
+# write API, product-write mutations, go-live marker, myshopify-ADMIN surface, and the --apply flag.
+DW_WRITE_RE='designer-laboratory-sandbox|SHOPIFY_ADMIN_TOKEN|SHOPIFY_ADMIN_ACCESS_TOKEN|admin/api/2024|productCreate|productUpdate|productVariantsBulk|go-live|golive|myshopify[^ ]*admin|--apply'
+# Build the file list into an ARRAY (glob, then drop *.backup* entries). An array — not a
+# word-split string — is used deliberately: a `$(ls …)`-string relies on IFS word-splitting,
+# which zsh does NOT do by default (a real Steve-fleet gotcha) and which also mishandles paths
+# with spaces/globs, silently passing 200 files as ONE argument so grep finds NOTHING and the
+# guard never trips. The array makes the scan shell-robust and space-safe.
+SCAN_FILES=()
+for _f in "$SCRAPERS_DIR"/*-new-products-scraper.ts; do
+  [ -f "$_f" ] || continue          # nullglob-safe: skip the literal pattern if no matches
+  case "$_f" in *backup*) continue;; esac
+  SCAN_FILES+=("$_f")
+done
+for _lib in "${SHARED_LIBS[@]}"; do
+  [ -f "$_lib" ] && SCAN_FILES+=("$_lib")   # missing shared libs skipped harmlessly
+done
+DW_WRITE_HITS=$(grep -HnE "$DW_WRITE_RE" "${SCAN_FILES[@]}" 2>/dev/null || true)
+if [ -n "$DW_WRITE_HITS" ]; then
+  echo "ABORT: stage-only tripwire — a scraper or shared lib now contains a DW-STORE write/publish primitive:"
+  echo "$DW_WRITE_HITS"
+  echo "       Refusing to run so the overnight cron cannot publish to designer-laboratory-sandbox."
+  exit 3
+fi
+echo "stage-only tripwire OK: no per-vendor scraper (or shared lib) writes to the DW store."
+
+# --- run the day's vendor slice through the canonical per-vendor scrapers ---
+timeout 5400 npx tsx "$RUNNER"
+RC=$?
+echo "===== $(date '+%F %T') vendor-overnight-refresh END rc=$RC (staging tables only) ====="

← dcd45908 auto-data-snapshot: 2026-08-22T09:11:56 (1 data files) — DW-  ·  back to Designer Wallcoverings  ·  TK-10717: harden tripwire regex + widen coverage (contrarian 934eff2b →