`DEPLOY.md`

109 lines
# enrich-ai-tags → local hybrid (build + you-deploy)

**What:** swaps DW image enrichment from paid Gemini to a HYBRID local pipeline —
exact hex+% from real pixels (Pillow, ground-truth, beats Gemini) + a vision LLM
(exo ring, Gemini fallback) for color names / styles / patterns / material /
imageType / dims / description. Same `aiData` output contract; ~$0 when the ring
is live, cost-ledgered Gemini calls otherwise.

**Safety:** `ENRICH_PROVIDER` defaults to `gemini`, so dropping the files changes
nothing until you set the env. Any local failure auto-falls-back to Gemini.

## ✅ FIXED + DEPLOYED 2026-09-23 (TK-12090 / TK-12113)
Repoint LIVE on Kamatera: `enrich-local.js` + `_shared/lib/{exo-vision.mjs,exo_vision.py}`
deployed to `/root/DW-Agents/{vendor-scrapers,_shared/lib}/`, `.env` `VISION_URL=http://100.82.17.107:52415`,
and `GEMINI_API_KEY` switched off the depleted prepaid account to the working master key.
Verified: exoPreflight from Kamatera returns a live `Qwen3-VL-4B-Instruct-4bit` instance; a real
Sanderson swatch enriched via exo `provider=local-hybrid cost=0`; a live milton_king batch wrote
2 rows (`Updated:2 Errors:0`); `enrichment-health` heartbeat flipped `WARN(130h) → PASS(stale_h=0)`.
Throughput hardening added: exo attempt bounded to 45s (was a hardcoded 120s shared with the
fallback → 1500 img/cron ≈ 50h) + a 5-consecutive-failure circuit breaker skips a dead/cold ring
for the rest of a run. On a loaded host, slow exo calls fall to the (now working) Gemini fallback
at ~$0.0006/img, bounded by the script's $25/run cap; on a normal-load host exo serves at $0.
Backups + exact undo in `/root/backups/tk-12090-enrichment/` and the reversible ledger.

The history below is retained for context.

## ⚠️ (HISTORICAL) LIVE PRODUCTION WAS SILENTLY GEMINI-ONLY (found 2026-09-23, TK-12090)

`ENRICH_PROVIDER=local` is **already set** in `/root/DW-Agents/full-monte/.env` on
Kamatera, and the patched `enrich-ai-tags.js` **is already deployed** there
(`localAnalyze` present) — this pipeline has been running in production every
Phase-3 cron since it was deployed. But `ENRICH_OLLAMA_URL` in that same `.env`
points at `http://100.82.17.107:11434` (Mac2's tailnet IP — a failover applied
2026-06-30 when Mac1 went offline, noted inline in that file). **Ollama on port
11434 is retired** (zombie — the port accepts the connection, zero models are
loaded, every `/api/generate` call fails). `analyze()`'s `catch` swallows that
failure and falls through to Gemini on *every single image*, so the "local
hybrid, ~$0" pipeline has actually been 100% Gemini-billed since Ollama was
retired, with nothing alerting on it (the `console.log` fallback line is the
only trace, and nothing reads Phase-3 logs for it).

Verified 2026-09-23 (TK-12090): Kamatera → `100.94.103.98:11434` (the original
Mac1 tailnet address) is now **completely unroutable** — that device is no
longer even a member of the tailnet (not in `tailscale status` at all, 100%
ping loss). Kamatera → `100.82.17.107:52415` (this Mac2/macstudio3's exo ring,
over the SAME tailnet route the failover already uses) returns **HTTP 200** —
the exo ring IS reachable from Kamatera today, just not on the port the live
`.env` names.

**The fix below (this repo, already applied) plus a redeploy + one `.env` line
change on Kamatera closes the gap.** The redeploy is a live-production script
replacement + env edit on a revenue-adjacent pipeline — left for Steve to run
or explicitly approve, per standing gate rules; not executed by this ticket.

## Files
- `enrich-palette.py`     — Pillow ground-truth palette (exact hex + %), unchanged
- `enrich-local.js`       — hybrid analyzer, exports localAnalyze() (geminiAnalyze shape).
  TK-12090 Lane E: now routes vision through `~/Projects/_shared/lib/exo-vision.mjs`
  instead of a direct Ollama call.
- `enrich-ai-tags.patched.js` — prod script + analyze() dispatcher (off by default)

## Deploy (run from Mac2)
```bash
# 1. Back up the live prod script
ssh my-server 'cp /root/DW-Agents/vendor-scrapers/enrich-ai-tags.js /root/DW-Agents/vendor-scrapers/enrich-ai-tags.js.bak-pre-exo-$(date +%Y%m%d)'

# 2. Copy the 3 files up (enrich-local.js now needs ~/Projects/_shared/lib/exo-vision.mjs
#    reachable at the SAME relative path on Kamatera — ../_shared/lib/exo-vision.mjs from
#    vendor-scrapers/ — deploy that lib dir too if it isn't already on the box)
scp enrich-local.js enrich-palette.py enrich-ai-tags.patched.js my-server:/root/DW-Agents/vendor-scrapers/
ssh my-server 'mkdir -p /root/DW-Agents/_shared/lib' # if not already present
scp ../_shared/lib/exo-vision.mjs my-server:/root/DW-Agents/_shared/lib/
ssh my-server 'cd /root/DW-Agents/vendor-scrapers && mv enrich-ai-tags.patched.js enrich-ai-tags.js'

# 3. Fix the .env — replace the dead ENRICH_OLLAMA_URL:11434 line with VISION_URL:52415
ssh my-server "sed -i 's#^ENRICH_OLLAMA_URL=.*#VISION_URL=http://100.82.17.107:52415#' /root/DW-Agents/full-monte/.env"

# 4. DRY-RUN with local provider (NO DB writes) — proves Kamatera→exo-ring + quality
ssh my-server 'cd /root/DW-Agents/vendor-scrapers && ENRICH_PROVIDER=local node enrich-ai-tags.js thibaut --dry-run --limit 3'
```

## GO LIVE (the gated step — only after the dry-run looks good)
The nightly enrichment is NOT `node enrich-ai-tags.js` directly — it's three Phase-3
crons (tier 1/2/3 @ 9/10/11 UTC ≈ 2-4am PT) running `full-monte-batch.js --phase 3`,
which SPAWNS `enrich-ai-tags.js` with `env: {...process.env}`. All three source
`/root/DW-Agents/full-monte/.env`, so once `VISION_URL` is set per step 3 above,
`ENRICH_PROVIDER=local` (already set) takes effect next Phase-3 run — cron re-sources
`.env` each run, no restart needed. Covers all tiers.

**Rollback (instant):**
```bash
ssh my-server "sed -i '/^ENRICH_PROVIDER=/d' /root/DW-Agents/full-monte/.env"
```
Or restore `enrich-ai-tags.js.bak-pre-exo-*`. Per-image Gemini fallback also auto-covers
any local outage (exo ring unreachable, or up with no live vision instance).

## Env knobs (canonical exo-vision-lib names — shared with every Lane in TK-12090)
- `ENRICH_PROVIDER` = local | hybrid | gemini (default gemini)
- `VISION_URL` = exo ring base. **On Kamatera this must be a tailnet/LAN address that
  can reach a Mac in the ring** — verified working 2026-09-23:
  `http://100.82.17.107:52415` (this Mac2/macstudio3). The shared lib's own default
  (`127.0.0.1:52415`) is only correct for local Mac testing via `test-local.js`.
- `VISION_MODEL` = vision model id (default from the shared lib:
  `mlx-community/Qwen3-VL-4B-Instruct-4bit`). `ENRICH_VL_MODEL` accepted as a legacy alias.
- `VISION_FALLBACK` = gemini (default) | none — vision-only; text has no fallback in the
  shared lib (this pipeline is vision-only, so that doesn't apply here).
- `GEMINI_API_KEY` — read from env or `~/Projects/secrets-manager/.env` by the shared lib;
  every Gemini fallback call is logged to `~/.claude/cost-ledger.jsonl`.