trending: build live-refresh wiring — scout→bestsellers mapper (DTD verdict A)

6ab84eff51b8dc006074f2e13223a158be5a2ba1 · 2026-07-06 21:28:38 -0700 · Steve Abrams

Closes the contrarian's core honesty gap (a 'trending' board whose data can't move)
WITHOUT new credentials or unattended spend — Exa is reachable only via the sanctioned
design-trend-scout agent, so the buildable-now $0 piece is the deterministic mapper:

- scripts/scout-to-bestsellers.js: pure/offline mapper. Accepts the scout's per-item feed
  OR its lane-level trend-board.json, validates required fields, stamps TODAY's spottedAt,
  emits the exact board shape. --self-test proves it end-to-end with zero network. Unknown
  colors get a deterministic (non-random) muted-hex fallback so every card renders.
- refresh.sh: rewritten into explicit paths — '--scout <feed>' (LIVE, $0 map+attach+reload)
  vs '--seed' (static dev re-bake); no-arg prints guidance and does NOTHING (can't silently
  re-bake stale data). refresh.sh NEVER runs the metered Exa pull — that stays a separate
  gated step.
- tick.sh refresh memo: updated — wiring is now BUILT; only the metered design-trend-scout
  pull remains gated (then $0 map, then deploy).

DTD panel 3/3 verdict A. The metered scout pull + deploy stay Steve-gated.

Files touched

M scripts/refresh.sh
A scripts/scout-to-bestsellers.js
M scripts/tick.sh

Diff

commit 6ab84eff51b8dc006074f2e13223a158be5a2ba1
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Mon Jul 6 21:28:38 2026 -0700

    trending: build live-refresh wiring — scout→bestsellers mapper (DTD verdict A)
    
    Closes the contrarian's core honesty gap (a 'trending' board whose data can't move)
    WITHOUT new credentials or unattended spend — Exa is reachable only via the sanctioned
    design-trend-scout agent, so the buildable-now $0 piece is the deterministic mapper:
    
    - scripts/scout-to-bestsellers.js: pure/offline mapper. Accepts the scout's per-item feed
      OR its lane-level trend-board.json, validates required fields, stamps TODAY's spottedAt,
      emits the exact board shape. --self-test proves it end-to-end with zero network. Unknown
      colors get a deterministic (non-random) muted-hex fallback so every card renders.
    - refresh.sh: rewritten into explicit paths — '--scout <feed>' (LIVE, $0 map+attach+reload)
      vs '--seed' (static dev re-bake); no-arg prints guidance and does NOTHING (can't silently
      re-bake stale data). refresh.sh NEVER runs the metered Exa pull — that stays a separate
      gated step.
    - tick.sh refresh memo: updated — wiring is now BUILT; only the metered design-trend-scout
      pull remains gated (then $0 map, then deploy).
    
    DTD panel 3/3 verdict A. The metered scout pull + deploy stay Steve-gated.
---
 scripts/refresh.sh              |  56 +++++++++++++++---
 scripts/scout-to-bestsellers.js | 128 ++++++++++++++++++++++++++++++++++++++++
 scripts/tick.sh                 |  13 ++--
 3 files changed, 183 insertions(+), 14 deletions(-)

diff --git a/scripts/refresh.sh b/scripts/refresh.sh
index 130cd60..ce2496c 100755
--- a/scripts/refresh.sh
+++ b/scripts/refresh.sh
@@ -1,11 +1,51 @@
 #!/bin/bash
-# trending-dw refresh — regenerate the bestseller seed + reload.
-# NOTE: this regenerates from the committed research seed. A TRUE live refresh
-# (re-scraping Google/POD via Exa) runs through the design-trend-scout agent,
-# which is metered — wire that as the gated next step (show + log cost per run).
+# trending-dw refresh. Two paths, clearly separated so spend is never accidental:
+#
+#   LIVE  (data actually moves):
+#     1. GATED, metered — run the design-trend-scout agent (Exa) to research current bestsellers.
+#        It is the ONLY step that costs money; it is NOT run here. Have it write its per-item
+#        research to a feed JSON (shape: {scannedAt, items:[{title,company,marketplace,style,
+#        color,priceBand,signal,signalRank,url}]}) — e.g. data/scout-feed.json.
+#     2. $0, offline — map that feed into the board + re-attach our-own images + reload:
+#            bash scripts/refresh.sh --scout data/scout-feed.json
+#
+#   SEED  (static/dev only — re-bakes the committed 2026-07-02 research; data does NOT move):
+#            bash scripts/refresh.sh --seed
+#
+# Running with NO args prints this guidance and does nothing (so a stray cron/hand-run can't
+# silently re-bake stale data and pretend it refreshed).
 set -e
 cd "$(dirname "$0")/.."
-node scripts/seed.js
-node scripts/attach-images.js   # re-attach our-own images (seed.js rewrites the file, wiping them) + flag GAP lanes
-if command -v pm2 >/dev/null 2>&1 && pm2 id trending-dw >/dev/null 2>&1; then pm2 restart trending-dw; fi
-echo "refreshed $(date)"
+
+reload(){ if command -v pm2 >/dev/null 2>&1 && pm2 id trending-dw >/dev/null 2>&1; then pm2 restart trending-dw; fi; }
+
+case "${1:-}" in
+  --scout)
+    FEED="${2:?usage: refresh.sh --scout <scout-feed.json>}"
+    node scripts/scout-to-bestsellers.js "$FEED"     # $0 deterministic mapper → data/bestsellers.json (today's date)
+    node scripts/attach-images.js                    # re-attach our-own images (mapper rewrites the file) + GAP flags
+    reload
+    echo "LIVE refresh done from $FEED — $(date)"
+    ;;
+  --seed)
+    node scripts/seed.js                             # re-bake committed static research (spottedAt hardcoded)
+    node scripts/attach-images.js
+    reload
+    echo "SEED re-bake done (STATIC data — does not move the vintage) — $(date)"
+    ;;
+  *)
+    cat <<'USAGE'
+trending-dw refresh — pick a path:
+
+  LIVE (data moves):  bash scripts/refresh.sh --scout <scout-feed.json>
+      where <scout-feed.json> was produced by a GATED, metered design-trend-scout run.
+      This script does NOT run the metered Exa pull — that stays a separate approved step.
+
+  SEED (static/dev):  bash scripts/refresh.sh --seed
+      re-bakes the committed 2026-07-02 research; the freshness badge will NOT clear.
+
+No path given → nothing done (refusing to silently re-bake stale data).
+USAGE
+    exit 0
+    ;;
+esac
diff --git a/scripts/scout-to-bestsellers.js b/scripts/scout-to-bestsellers.js
new file mode 100644
index 0000000..06d46ae
--- /dev/null
+++ b/scripts/scout-to-bestsellers.js
@@ -0,0 +1,128 @@
+#!/usr/bin/env node
+// scout-to-bestsellers — the LIVE-REFRESH MAPPER (DTD verdict A, 2026-07-06).
+//
+// Closes the "trending board with frozen data" honesty gap WITHOUT new credentials or
+// unattended spend. Exa is reachable only via the design-trend-scout agent (MCP-backed),
+// so the live-refresh path is:
+//     1. run design-trend-scout  (metered Exa pull — GATED, Steve-approved)  → writes an intermediate JSON
+//     2. node scripts/scout-to-bestsellers.js <intermediate.json>            → this mapper ($0, deterministic)
+//     3. node scripts/attach-images.js                                        → re-attach our-own images + GAP flags
+// This file is step 2: it is pure, offline, and never touches the network — so it is $0 and
+// safe to run/test anytime. The ONLY spend is step 1, which stays gated.
+//
+// INPUT (either shape is accepted):
+//   (a) per-item scout feed  { scannedAt, items:[ {title, company, marketplace, style, color,
+//                              priceBand, signal, signalRank, url} ] }   ← preferred, richest
+//   (b) lane-level trend-board.json { scannedAt, trends:[ {lane, why, priceBand, haveCoverage,
+//                              gap, briefs:[...] } ] }                     ← the scout's persisted artifact;
+//                              flattened best-effort (one item per lane) so the pipeline still works.
+// OUTPUT: data/bestsellers.json  { generatedAt, source, count, items:[ ...board shape ] }
+//
+// Usage:
+//   node scripts/scout-to-bestsellers.js path/to/scout.json          # write bestsellers.json
+//   node scripts/scout-to-bestsellers.js --self-test                 # prove the mapper works, no input needed
+//   node scripts/scout-to-bestsellers.js path/to/scout.json --dry    # validate + print, don't write
+
+const fs = require('fs');
+const path = require('path');
+
+// color family -> representative swatch hex (superset of seed.js's HEX; unknown colors get a
+// deterministic muted fallback so every card still renders a swatch tile).
+const HEX = {
+  'Forest Green':'#2f5d3a','Oxblood':'#6e2b2b','Plum':'#5a3a55','Chartreuse':'#b6c33a',
+  'Cloud White':'#efece3','Blush':'#e7c3c0','Terracotta':'#c07a4e','Indigo':'#28324f',
+  'Greige':'#9a9488','Mustard':'#c99a3a','Teal':'#2b6c6b','Black':'#232323','Sage':'#9bab8b',
+  'Claret':'#76322f','Gold':'#b49555','Navy':'#26364d','Cream Gold':'#cdb87a','Mushroom':'#8a6f5a',
+  'Pop Pink':'#c85a86','Ombre Blue':'#5b7fa6','Cream':'#eee6d3','Rust':'#a1522d','Olive':'#6b6b3a',
+  'Charcoal':'#333330','Powder Blue':'#a9c3d6','Burgundy':'#5c2230','Multi':'#8a7f9a'
+};
+function hexFor(color){
+  if (HEX[color]) return HEX[color];
+  // deterministic muted fallback: hash the name to a low-saturation hex (never random — reproducible)
+  const s = String(color||'');
+  let h = 0; for (let i=0;i<s.length;i++) h = (h*31 + s.charCodeAt(i)) & 0xffffff;
+  const r = 90 + (h & 0x3f), g = 90 + ((h>>6) & 0x3f), b = 90 + ((h>>12) & 0x3f);
+  return '#' + [r,g,b].map(v=>v.toString(16).padStart(2,'0')).join('');
+}
+
+const REQ = ['title','style','color','priceBand'];   // minimum viable board item
+function normItem(o, i, scannedAt){
+  const missing = REQ.filter(k => !o[k]);
+  if (missing.length) throw new Error(`item ${i} missing required field(s): ${missing.join(', ')} — ${JSON.stringify(o).slice(0,120)}`);
+  return {
+    id: 'TR-' + String(i+1).padStart(3,'0'),
+    dominantHex: hexFor(o.color),
+    spottedAt: (scannedAt || new Date().toISOString()).slice(0,10),
+    isNew: o.isNew !== false,
+    title: o.title,
+    company: o.company || 'Independent maker',
+    marketplace: o.marketplace || 'Google / Retail',
+    style: o.style,
+    color: o.color,
+    priceBand: o.priceBand,
+    signal: o.signal || 'Trending',
+    signalRank: Number.isFinite(o.signalRank) ? o.signalRank : 50,
+    url: o.url || ''
+  };
+}
+
+// flatten the lane-level trend-board.json into per-item board rows (shape (b) fallback)
+function flattenTrendBoard(board){
+  const out = [];
+  for (const t of (board.trends || [])){
+    // one representative item per lane; briefs (if present) become extra items so a rich scan yields more rows
+    const base = { title: t.lane, company: 'Trend lane', marketplace: 'Multiple', style: t.lane,
+                   color: 'Multi', priceBand: t.priceBand || '$$', signal: t.why || 'Selling now',
+                   signalRank: t.gap ? 85 : 60 };
+    out.push(base);
+    for (const b of (t.briefs || [])){
+      out.push({ title: b.title || (t.lane+' original'), company: 'DW original (brief)', marketplace:'Pattern Vault',
+                 style: t.lane, color: b.colorway || b.color || 'Multi', priceBand: b.tier || t.priceBand || '$$',
+                 signal: 'Gap-lane brief', signalRank: 88 });
+    }
+  }
+  return out;
+}
+
+function buildItems(input){
+  const scannedAt = input.scannedAt || input.generatedAt || new Date().toISOString();
+  let raw;
+  if (Array.isArray(input.items)) raw = input.items;              // shape (a)
+  else if (Array.isArray(input.trends)) raw = flattenTrendBoard(input); // shape (b)
+  else throw new Error('input has neither .items[] (per-item feed) nor .trends[] (trend-board)');
+  if (!raw.length) throw new Error('input produced 0 items — refusing to overwrite the board with an empty set');
+  return { scannedAt, items: raw.map((o,i)=>normItem(o,i,scannedAt)) };
+}
+
+// ---- self-test: proves the mapper end-to-end with a synthetic feed, zero network, zero deps ----
+function selfTest(){
+  const sample = { scannedAt: new Date().toISOString(), items: [
+    { title:'Test Heritage Floral', company:'Test Co', marketplace:'Etsy', style:'Heritage Floral', color:'Forest Green', priceBand:'$$$', signal:'bestseller', signalRank:95, url:'https://example.com' },
+    { title:'Unknown-Color Item', style:'Boho', color:'Zzz Nonexistent', priceBand:'$$' }   // exercises defaults + hex fallback
+  ]};
+  const built = buildItems(sample);
+  const ok = built.items.length === 2
+    && built.items[0].id === 'TR-001'
+    && built.items[1].company === 'Independent maker'          // default applied
+    && /^#[0-9a-f]{6}$/.test(built.items[1].dominantHex)       // fallback hex valid
+    && built.items[0].spottedAt === sample.scannedAt.slice(0,10);
+  console.log(ok ? 'SELF-TEST PASS' : 'SELF-TEST FAIL');
+  console.log(JSON.stringify(built.items, null, 1));
+  process.exit(ok ? 0 : 1);
+}
+
+function main(){
+  const args = process.argv.slice(2);
+  if (args.includes('--self-test')) return selfTest();
+  const dry = args.includes('--dry');
+  const inPath = args.find(a => !a.startsWith('--'));
+  if (!inPath){ console.error('usage: scout-to-bestsellers.js <scout.json> [--dry]  |  --self-test'); process.exit(2); }
+  const input = JSON.parse(fs.readFileSync(inPath, 'utf8'));
+  const built = buildItems(input);
+  const out = { generatedAt: built.scannedAt.slice(0,10), source: 'design-trend-scout (Exa research) → scout-to-bestsellers mapper', count: built.items.length, items: built.items };
+  if (dry){ console.log(`[dry] ${out.count} items, generatedAt ${out.generatedAt} — NOT written`); console.log(JSON.stringify(out.items.slice(0,3), null, 1)); return; }
+  const dest = path.join(__dirname, '..', 'data', 'bestsellers.json');
+  fs.writeFileSync(dest, JSON.stringify(out, null, 2));
+  console.log(`wrote ${dest} — ${out.count} items, generatedAt ${out.generatedAt}. Next: node scripts/attach-images.js`);
+}
+main();
diff --git a/scripts/tick.sh b/scripts/tick.sh
index 6fb3f79..0cfa1b9 100755
--- a/scripts/tick.sh
+++ b/scripts/tick.sh
@@ -60,15 +60,16 @@ if [ "$STALE_DAYS" -ge "$STALE_THRESHOLD" ] 2>/dev/null; then
 **Officer of record:** vp-dw-commerce
 **Why:** the trending board's data is **$STALE_DAYS days old** (as of $AS_OF), past the ${STALE_THRESHOLD}-day cadence. A *trending* board is only as credible as its freshness.
 
-## HONEST NOTE (read before approving)
-\`scripts/refresh.sh\` today just re-seeds the **committed** \`seed.js\` (spottedAt hardcoded to $AS_OF) — running it as-is will NOT move the data or clear this memo. The real fix is a two-parter:
-1. **Build** the live wiring: \`refresh.sh\` → call the metered \`design-trend-scout\` (Exa) agent → write fresh items with today's spottedAt. (code-only, reversible, \$0 to build)
-2. **Run** it (metered — the gated spend).
+## Status: wiring is BUILT — only the metered pull remains gated
+The \$0 mapper (\`scripts/scout-to-bestsellers.js\`) + \`refresh.sh --scout\` path exist and are tested; a scout feed maps to the board with TODAY's date. What's left is the one metered step:
+1. **GATED** — run the \`design-trend-scout\` agent (Exa, metered) → write \`data/scout-feed.json\`.
+2. \$0 — \`bash ~/Projects/trending-dw/scripts/refresh.sh --scout data/scout-feed.json\` (maps + re-attaches images + reloads).
+3. GATED — deploy to prod.
 
 ## The ask
-Approve step 2 (the metered Exa pull) — and, if not yet built, approve step 1 first.
+Approve step 1 (the metered Exa pull). Steps 2 is \$0; step 3 is a normal deploy.
 
-- **Cost:** metered Exa research pull — estimate a few \$ per run (show + log actual to cost-tracker before running). The tick + the wiring build are \$0; only the pull spends.
+- **Cost:** metered Exa research pull — estimate a few \$ per run (show + log actual to cost-tracker before running). The tick, the mapper, and \`refresh.sh --scout\` are all \$0; only the scout pull spends.
 
 ## APPROVE / REVISE / BLOCK
 - [ ] APPROVE — build the live-Exa wiring (if needed) + run the metered pull + draft the deploy