`README.md`

49 lines
# vendor-recrawl-dispatcher

Registry-driven nightly recrawl of the ~463 active DW vendors. The initial crawls
went stale (~2 months); this keeps `dw_unified` catalog data fresh on a rolling wave.
**Read-only DATA refresh into staging — NO publish.** Local models only.

## Design (DTD verdict — committed)
One nightly launchd job (`com.steve.vendor-recrawl-dispatcher`, 2:00 AM):
1. Reads `dw_unified.vendor_registry`, selects vendors **DUE** tonight
   (`next_scheduled_crawl` passed or NULL).
2. Orders **HIGH → MEDIUM → LOW**, then `last_product_update ASC NULLS FIRST`
   (stalest first), then largest catalog first.
3. Refreshes each resolvable vendor: **scrape → write dw_unified staging →
   price-finder → CNCP win**. No publish.
4. Stamps `last_product_update` + `next_scheduled_crawl` after each success
   (cadence by priority: HIGH 7d / MEDIUM 21d / LOW 28d).
5. **Batch = 10/night** (tunable), 7h wall-clock cap, 20–90s politeness jitter
   between vendors. Full sweep rolls over ~a few weeks, then repeats.

## Enrichment — LOCAL MODELS ONLY (`lib/enrich-router.js`)
Any AI enrichment routes: **exo `:52415` → Mac1 Ollama `:11434` → skip.** Never a
paid API. The exo check is a *real completion probe* (not just daemon-up), so while
exo inference is Xcode-blocked it correctly falls through to Ollama (`qwen3:14b`
text, `qwen2.5vl:7b` vision) and **auto-upgrades to exo** the moment exo can serve.
Every path logs `$0 (local)`.

## Run
```
npm install
node dispatcher.js --dry-run     # print the wave ordering, scrape NOTHING (safe)
node dispatcher.js               # run tonight's batch (only via the gated job)
node dispatcher.js --batch=12    # override batch
```

## Status / the scraper-wiring frontier
The **scheduling framework is complete and dry-run-verified** (462 due vendors,
correct order). Per-vendor **scraper resolution** (`lib/resolve-pipeline.js`) currently
only auto-discovers `scripts/<code>-refresh/run.sh` dirs (WallQuest is the reference,
and is *excluded* here because it has its own weekly job). So today **0 other vendors
are wired** — the dispatcher skips + logs them (`logs/unresolved.jsonl`) and never
crashes. As each vendor's `*-scraper-manager` is wrapped into a `<code>-refresh/` dir
(scrape → write staging → price-finder, no publish), it **auto-activates** in the wave.
Wiring the ~39 dedicated skills + the shared `product-import-from-url` engine (108
vendors, AI-orchestrated) is the ongoing integration.

## Gated install (Steve runs — see deploy/ + the pending-approval memo)
- `sql/seed-crawl-schedule.sql` — one-time stagger of `next_scheduled_crawl` (canonical write).
- `deploy/com.steve.vendor-recrawl-dispatcher.plist` — the nightly job.