TK-10717: harden tripwire regex + widen coverage (contrarian pass)

934eff2b24fb619d44fb15cd08ff62c14fe67231 · 2026-08-22 11:16:37 -0700 · Steve

Contrarian FIX-FIRST fixes on the stage-only tripwire:
- Regex made YEAR-AGNOSTIC (admin/api/2024 -> admin/api/20[0-9]{2}) so it keeps
  matching admin/api/2025+; add the current write verbs it was missing
  (productSet, publishablePublish, admin/graphql) that a realistic future
  Shopify write would use. Deliberately does NOT match a bare products.json/
  variants.json — that is the ubiquitous vendor-feed READ (42 scrapers fetch the
  vendor's own /products.json); a DW WRITE is always an admin surface / DW domain /
  admin token / write verb, all matched.
- Coverage widened from a flat 208-file subset to the real reachable tree (803
  files): the Skills/*/*.skill.ts catalog tree (24 scrapers import it) + the
  color-variant-detector/tracker + vendor-schema + skill-scraper-bridge helpers.

Re-proven under bash: clean today (exit 0, reaches scrape); trips on a realistic
productSet write AND on admin/api/2025 (the exact leak found) => exit 3, no scrape;
zero false positives on fortuny, scalamandre, and all 42 vendor-feed products.json
scrapers; a planted write inside Skills/kravet/catalog.skill.ts is caught.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

M DW-Programming/ImportNewSkufromURL/scripts/vendor-overnight-refresh.sh

Diff

commit 934eff2b24fb619d44fb15cd08ff62c14fe67231
Author: Steve <steve@designerwallcoverings.com>
Date:   Sat Aug 22 11:16:37 2026 -0700

    TK-10717: harden tripwire regex + widen coverage (contrarian pass)
    
    Contrarian FIX-FIRST fixes on the stage-only tripwire:
    - Regex made YEAR-AGNOSTIC (admin/api/2024 -> admin/api/20[0-9]{2}) so it keeps
      matching admin/api/2025+; add the current write verbs it was missing
      (productSet, publishablePublish, admin/graphql) that a realistic future
      Shopify write would use. Deliberately does NOT match a bare products.json/
      variants.json — that is the ubiquitous vendor-feed READ (42 scrapers fetch the
      vendor's own /products.json); a DW WRITE is always an admin surface / DW domain /
      admin token / write verb, all matched.
    - Coverage widened from a flat 208-file subset to the real reachable tree (803
      files): the Skills/*/*.skill.ts catalog tree (24 scrapers import it) + the
      color-variant-detector/tracker + vendor-schema + skill-scraper-bridge helpers.
    
    Re-proven under bash: clean today (exit 0, reaches scrape); trips on a realistic
    productSet write AND on admin/api/2025 (the exact leak found) => exit 3, no scrape;
    zero false positives on fortuny, scalamandre, and all 42 vendor-feed products.json
    scrapers; a planted write inside Skills/kravet/catalog.skill.ts is caught.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 .../scripts/vendor-overnight-refresh.sh            | 35 ++++++++++++++++++----
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/DW-Programming/ImportNewSkufromURL/scripts/vendor-overnight-refresh.sh b/DW-Programming/ImportNewSkufromURL/scripts/vendor-overnight-refresh.sh
index c2dc5899..b52164d0 100755
--- a/DW-Programming/ImportNewSkufromURL/scripts/vendor-overnight-refresh.sh
+++ b/DW-Programming/ImportNewSkufromURL/scripts/vendor-overnight-refresh.sh
@@ -71,32 +71,57 @@ echo "stage-only tripwire OK: $RUNNER has no Shopify/publish path."
 # so fortuny's read '.myshopify.com' does not trip it. Verified CLEAN across all 200 scrapers +
 # the shared helpers on the current tree (TK-10717), zero false positives on fortuny/scalamandre.
 SCRAPERS_DIR="lib/scrapers"
-# Shared ../lib helpers the new-products scrapers import (dynamic-import-reachable, extend if a
-# scraper starts importing a new shared module). Missing files are skipped harmlessly.
+# Shared ../lib helpers + the Skills/ catalog modules the new-products scrapers import
+# (dynamic-import-reachable). 24 scrapers import ../../Skills/<vendor>/catalog.skill.ts and
+# scraper-wrapper.ts pulls in color-variant-detector/tracker — all globbed in below so a write
+# added to any of them trips too. Missing files are skipped harmlessly.
 SHARED_LIBS=(
   lib/browserbase-connect.ts
   lib/browserbase-helper.ts
   lib/brightdata-proxy.ts
   lib/vendor-timeouts.ts
   lib/scraper-wrapper.ts
+  lib/color-variant-detector.ts
+  lib/color-variant-tracker.ts
   lib/schema-helper.ts
+  lib/vendor-schema.ts
+  lib/skill-scraper-bridge.ts
   lib/puppeteer-preflight.ts
   lib/database/postgres-client.ts
 )
-# Full write/publish primitive set (per TK-10717 spec): live store domain, admin token, admin
-# write API, product-write mutations, go-live marker, myshopify-ADMIN surface, and the --apply flag.
-DW_WRITE_RE='designer-laboratory-sandbox|SHOPIFY_ADMIN_TOKEN|SHOPIFY_ADMIN_ACCESS_TOKEN|admin/api/2024|productCreate|productUpdate|productVariantsBulk|go-live|golive|myshopify[^ ]*admin|--apply'
+# Full write/publish primitive set (per TK-10717 spec, hardened after the contrarian pass):
+# live store domain, admin token, admin write API (YEAR-AGNOSTIC — was hardcoded to 2024, which
+# would silently stop matching admin/api/2025+), an admin GraphQL surface, the current product-write
+# mutations (productCreate/Update/Set + productVariantsBulk + publishablePublish), the go-live
+# marker, the myshopify-ADMIN surface, and the --apply flag.
+#
+# DELIBERATELY NOT matched — a bare "products.json"/"variants.json": that is the ubiquitous
+# vendor-side READ (the feed-first strategy — 42 scrapers legitimately fetch the VENDOR's own
+# Shopify /products.json storefront feed). A DW-store WRITE is always distinguishable by an ADMIN
+# surface (admin/api, admin/graphql), the DW store domain, the admin token, or a write-mutation
+# verb — all matched above — so a bare .json read is safely ignored. Verified 0 false positives on
+# the real tree: fortuny's Storefront read + scalamandre's PRODUCTSTATUS filter + all 42 vendor-feed
+# scrapers stay clean; productSet/publishablePublish/admin-graphql/admin-api-20xx match nothing benign.
+DW_WRITE_RE='designer-laboratory-sandbox|SHOPIFY_ADMIN_TOKEN|SHOPIFY_ADMIN_ACCESS_TOKEN|admin/api/20[0-9]{2}|admin/graphql|productCreate|productUpdate|productSet|productVariantsBulk|publishablePublish|go-live|golive|myshopify[^ ]*admin|--apply'
 # Build the file list into an ARRAY (glob, then drop *.backup* entries). An array — not a
 # word-split string — is used deliberately: a `$(ls …)`-string relies on IFS word-splitting,
 # which zsh does NOT do by default (a real Steve-fleet gotcha) and which also mishandles paths
 # with spaces/globs, silently passing 200 files as ONE argument so grep finds NOTHING and the
 # guard never trips. The array makes the scan shell-robust and space-safe.
 SCAN_FILES=()
+# (i) every new-products scraper the orchestrator can dynamically import
 for _f in "$SCRAPERS_DIR"/*-new-products-scraper.ts; do
   [ -f "$_f" ] || continue          # nullglob-safe: skip the literal pattern if no matches
   case "$_f" in *backup*) continue;; esac
   SCAN_FILES+=("$_f")
 done
+# (ii) the Skills/ catalog + skill modules those scrapers import (24 scrapers reach this tree)
+for _sk in Skills/*/catalog.skill.ts Skills/*/*.skill.ts; do
+  [ -f "$_sk" ] || continue
+  case "$_sk" in *backup*) continue;; esac
+  SCAN_FILES+=("$_sk")
+done
+# (iii) the enumerated shared ../lib helpers
 for _lib in "${SHARED_LIBS[@]}"; do
   [ -f "$_lib" ] && SCAN_FILES+=("$_lib")   # missing shared libs skipped harmlessly
 done

← dc7e304a TK-10717: harden stage-only tripwire — scan full new-product · back to Designer Wallcoverings · auto-data-snapshot: 2026-08-22T11:19:22 (1 data files) — DW- ddbc12bc →