← back to Vendor Recrawl Dispatcher
add README + launchd plist draft + pending-approval go-live memo (seed SQL + gated job; honest scraper-wiring caveat)
bec8dba7e2ae232e2e8d933cf08bad4fe8585560 · 2026-07-06 22:00:00 -0700 · steve
Files touched
A README.mdA deploy/com.steve.vendor-recrawl-dispatcher.plist
Diff
commit bec8dba7e2ae232e2e8d933cf08bad4fe8585560
Author: steve <steve@designerwallcoverings.com>
Date: Mon Jul 6 22:00:00 2026 -0700
add README + launchd plist draft + pending-approval go-live memo (seed SQL + gated job; honest scraper-wiring caveat)
---
README.md | 48 ++++++++++++++++++++++++
deploy/com.steve.vendor-recrawl-dispatcher.plist | 22 +++++++++++
2 files changed, 70 insertions(+)
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..8e99b72
--- /dev/null
+++ b/README.md
@@ -0,0 +1,48 @@
+# vendor-recrawl-dispatcher
+
+Registry-driven nightly recrawl of the ~463 active DW vendors. The initial crawls
+went stale (~2 months); this keeps `dw_unified` catalog data fresh on a rolling wave.
+**Read-only DATA refresh into staging — NO publish.** Local models only.
+
+## Design (DTD verdict — committed)
+One nightly launchd job (`com.steve.vendor-recrawl-dispatcher`, 2:00 AM):
+1. Reads `dw_unified.vendor_registry`, selects vendors **DUE** tonight
+ (`next_scheduled_crawl` passed or NULL).
+2. Orders **HIGH → MEDIUM → LOW**, then `last_product_update ASC NULLS FIRST`
+ (stalest first), then largest catalog first.
+3. Refreshes each resolvable vendor: **scrape → write dw_unified staging →
+ price-finder → CNCP win**. No publish.
+4. Stamps `last_product_update` + `next_scheduled_crawl` after each success
+ (cadence by priority: HIGH 7d / MEDIUM 21d / LOW 28d).
+5. **Batch = 10/night** (tunable), 7h wall-clock cap, 20–90s politeness jitter
+ between vendors. Full sweep rolls over ~a few weeks, then repeats.
+
+## Enrichment — LOCAL MODELS ONLY (`lib/enrich-router.js`)
+Any AI enrichment routes: **exo `:52415` → Mac1 Ollama `:11434` → skip.** Never a
+paid API. The exo check is a *real completion probe* (not just daemon-up), so while
+exo inference is Xcode-blocked it correctly falls through to Ollama (`qwen3:14b`
+text, `qwen2.5vl:7b` vision) and **auto-upgrades to exo** the moment exo can serve.
+Every path logs `$0 (local)`.
+
+## Run
+```
+npm install
+node dispatcher.js --dry-run # print the wave ordering, scrape NOTHING (safe)
+node dispatcher.js # run tonight's batch (only via the gated job)
+node dispatcher.js --batch=12 # override batch
+```
+
+## Status / the scraper-wiring frontier
+The **scheduling framework is complete and dry-run-verified** (462 due vendors,
+correct order). Per-vendor **scraper resolution** (`lib/resolve-pipeline.js`) currently
+only auto-discovers `scripts/<code>-refresh/run.sh` dirs (WallQuest is the reference,
+and is *excluded* here because it has its own weekly job). So today **0 other vendors
+are wired** — the dispatcher skips + logs them (`logs/unresolved.jsonl`) and never
+crashes. As each vendor's `*-scraper-manager` is wrapped into a `<code>-refresh/` dir
+(scrape → write staging → price-finder, no publish), it **auto-activates** in the wave.
+Wiring the ~39 dedicated skills + the shared `product-import-from-url` engine (108
+vendors, AI-orchestrated) is the ongoing integration.
+
+## Gated install (Steve runs — see deploy/ + the pending-approval memo)
+- `sql/seed-crawl-schedule.sql` — one-time stagger of `next_scheduled_crawl` (canonical write).
+- `deploy/com.steve.vendor-recrawl-dispatcher.plist` — the nightly job.
diff --git a/deploy/com.steve.vendor-recrawl-dispatcher.plist b/deploy/com.steve.vendor-recrawl-dispatcher.plist
new file mode 100644
index 0000000..23e99f0
--- /dev/null
+++ b/deploy/com.steve.vendor-recrawl-dispatcher.plist
@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- DRAFT — NOT installed. Steve bootstraps this (gated: scheduled job).
+ Nightly 2:00 AM registry-driven vendor recrawl wave. NO publish. -->
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0"><dict>
+ <key>Label</key><string>com.steve.vendor-recrawl-dispatcher</string>
+ <key>ProgramArguments</key><array>
+ <string>/bin/bash</string>
+ <string>/Users/macstudio3/Projects/vendor-recrawl-dispatcher/run.sh</string>
+ </array>
+ <key>EnvironmentVariables</key><dict>
+ <key>HOME</key><string>/Users/macstudio3</string>
+ <key>PATH</key><string>/Users/macstudio3/.local/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
+ </dict>
+ <key>StartCalendarInterval</key><dict>
+ <key>Hour</key><integer>2</integer>
+ <key>Minute</key><integer>0</integer>
+ </dict>
+ <key>RunAtLoad</key><false/>
+ <key>StandardOutPath</key><string>/Users/macstudio3/Projects/vendor-recrawl-dispatcher/logs/launchd.out.log</string>
+ <key>StandardErrorPath</key><string>/Users/macstudio3/Projects/vendor-recrawl-dispatcher/logs/launchd.err.log</string>
+</dict></plist>
← 2c413c7 vendor-recrawl-dispatcher: registry-driven wave framework +
·
back to Vendor Recrawl Dispatcher
·
DTD Option C: generic product_url refresher (fetch HTML free 0dc0526 →