← back to Dw Photo Capture

identify: per-request model toggle (moondream fast path) + API.md (cycle 7)

d76cc5acce717539819a2a8036cb057c41244cb9 · 2026-06-26 00:48:27 -0700 · Steve Abrams

/api/identify can now trade accuracy for speed per request. ollamaVision() gains a
model arg; a VISION_MODELS allow-list maps {fast:true} or {model:'moondream'|...} to a
faster local model (anything off-list falls back to default — no arbitrary model pulls).
The chosen model is echoed in the response. Default stays qwen2.5vl:7b (most accurate).

Verified: default qwen2.5vl 21.5s (resolves Kravet), {fast:true} moondream 12.2s (~2x
faster, lower accuracy as documented); lint + 21 unit tests green. Added API.md
documenting /api/ocr, /api/identify + the model toggle table, and the learn surface.
$0 (local VLM).

Files touched

• A API.md
• M OVERNIGHT_LEDGER.md
• M server.js

Diff

commit d76cc5acce717539819a2a8036cb057c41244cb9
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Fri Jun 26 00:48:27 2026 -0700

    identify: per-request model toggle (moondream fast path) + API.md (cycle 7)
    
    /api/identify can now trade accuracy for speed per request. ollamaVision() gains a
    model arg; a VISION_MODELS allow-list maps {fast:true} or {model:'moondream'|...} to a
    faster local model (anything off-list falls back to default — no arbitrary model pulls).
    The chosen model is echoed in the response. Default stays qwen2.5vl:7b (most accurate).
    
    Verified: default qwen2.5vl 21.5s (resolves Kravet), {fast:true} moondream 12.2s (~2x
    faster, lower accuracy as documented); lint + 21 unit tests green. Added API.md
    documenting /api/ocr, /api/identify + the model toggle table, and the learn surface.
    $0 (local VLM).
---
 API.md              | 42 ++++++++++++++++++++++++++++++++++++++++++
 OVERNIGHT_LEDGER.md |  3 ++-
 server.js           | 15 +++++++++++----
 3 files changed, 55 insertions(+), 5 deletions(-)

diff --git a/API.md b/API.md
new file mode 100644
index 0000000..4157926
--- /dev/null
+++ b/API.md
@@ -0,0 +1,42 @@
+# dw-photo-capture — API notes
+
+Local sample scanner. All recognition is on-device, $0 (macOS Vision OCR + local Ollama VLM).
+
+## `POST /api/ocr`
+Body: `{ dataUrl }` (a base64 image data URL). Returns ranked SKU candidates:
+`{ ok, text, vendor, candidates[], top, topStrong, barcode }`.
+- Barcodes/QR (read in the same Vision pass) are ground truth — they outrank OCR codes and force `topStrong`.
+- `vendor` is detected from a brand name printed on the swatch (180 learned vendor profiles).
+- `topStrong` = safe to lock on the first read (known prefix, brand+font-dominant, or a barcode).
+
+## `POST /api/identify`
+VLM logo/typeface recognition — the assist for when OCR can't read a stylized wordmark.
+Body: `{ dataUrl, model?, fast? }`. Returns
+`{ ok, model, brand, confidence, logo, typeface, code, vendor, match_via }`.
+
+### Model toggle (speed vs accuracy)
+By default `/api/identify` uses **qwen2.5vl:7b** (~15-20s, most accurate). To trade accuracy for speed,
+pick a faster model per request:
+
+| Request body | Model used | Notes |
+|---|---|---|
+| *(none)* | `qwen2.5vl:7b` | default — most accurate |
+| `{ "fast": true }` | `moondream:latest` | ~2x faster, lower accuracy (weaker at structured JSON) |
+| `{ "model": "moondream" }` | `moondream:latest` | explicit |
+| `{ "model": "qwen2.5vl:7b" }` | `qwen2.5vl:7b` | explicit |
+| `{ "model": "llava:13b" }` | `llava:13b` | |
+
+Only **allow-listed** models are honored (`VISION_MODELS` in server.js); anything off-list falls back to the
+default — a caller can't request an arbitrary model to pull/run. The chosen model is echoed back in `model`.
+
+`vendor` is resolved from the VLM's brand guess by: (1) lexicon regex, (2) fuzzy name match, then
+(3) **fingerprint tiebreaker** (only if 1+2 are inconclusive) — comparing the live read to the validated
+reference-logo fingerprints. `match_via` reports which path resolved it (`name` | `fingerprint`).
+
+## `GET /api/vendor-profiles` · `POST /api/learn` · `/learn`
+What the scanner has learned about each vendor's sample labeling (prefixes, examples, logo fingerprint).
+`/api/learn` refines a vendor's profile from a scan that resolved to a real product. `/learn` is the viewer.
+
+## Default vision model via env
+`OLLAMA_VISION_MODEL` overrides the default for the whole process (still must be an installed model).
+`OLLAMA_URL` (shared with the voice feature) points at the Ollama host.
diff --git a/OVERNIGHT_LEDGER.md b/OVERNIGHT_LEDGER.md
index 3267951..6bd0cba 100644
--- a/OVERNIGHT_LEDGER.md
+++ b/OVERNIGHT_LEDGER.md
@@ -18,7 +18,7 @@ the single highest-value safe one, the officer executes it, and the loop resched
 - [x] analyzeOcr unit tests — 21 assertions, npm test + selfcheck gate
 - [x] Barcode UX: distinct "▍▍ BARCODE" lock label on a barcode read
 - [x] /learn: shows logo fingerprint + flags rejected logos with their read-as
-- [ ] Faster VLM path: moondream fast-mode toggle for /api/identify (env already supports model)
+- [x] moondream fast-toggle for /api/identify ({fast}/{model}, allow-listed, documented in API.md)
 - [ ] Auto-fire /api/identify in background on a lock-with-code-but-no-vendor (non-blocking)
 - [ ] /code-review (or claude-codex) pass on the full session diff; fix findings
 - [x] GATED (drafted to pending-approval): weekly launchd refresh — plist staged
@@ -33,3 +33,4 @@ the single highest-value safe one, the officer executes it, and the loop resched
 | 4 | — (clear win) | /learn surfaces logo fingerprint (reads-as + typeface) and flags the 6 rejected logos with what they actually read as | HTTP 200, 6 rejects visible (Versace→Savannah etc.) | $0 |
 | 5 | **DTD 3/3 → A** | wire logo fingerprints into /api/identify as a TIEBREAKER (only when name-match is inconclusive) | clean name still via=name (no regression); fingerprint resolves Mind the Gap/Ralph Lauren in unit test | $0.003 (DTD) + $0 |
 | 6 | — (clear win) | analyzeOcr/vendor unit tests — harness slices the REAL fn source from server.js + eval-tests vs crafted inputs & real profile data; wired into npm test + selfcheck | 21/21 pass; lint clean; covers parseOcrRows/analyzeOcr(brand,barcode,prefix,weak,empty)/fuzzyVendor/fingerprintVendor | $0 |
+| 7 | — (clear win) | moondream fast-toggle for /api/identify: ollamaVision gains a model arg, allow-listed VISION_MODELS, {fast:true}/{model} per-request override (echoed back), API.md docs | qwen 21.5s (Kravet ✓) vs moondream 12.2s; lint+21 tests green; off-list ignored (no arbitrary pulls) | $0 |
diff --git a/server.js b/server.js
index dcde82a..4c3a143 100644
--- a/server.js
+++ b/server.js
@@ -383,11 +383,16 @@ rebuildLexicon();
 // Reads a STYLIZED logo/wordmark + typesetting that the deterministic OCR can't, to name
 // the vendor when no brand text was cleanly recognized. An on-lock assist, not per-frame.
 const OLLAMA_VISION_MODEL = process.env.OLLAMA_VISION_MODEL || 'qwen2.5vl:7b';   // reuses OLLAMA_URL declared with ollamaAsk
-function ollamaVision(b64, prompt, timeoutMs) {
+// Allow-listed vision models a request may pick. 'fast' => moondream (~3-5s, lower accuracy);
+// default qwen2.5vl is more accurate (~15-20s). Anything off-list is ignored (no arbitrary pulls).
+const VISION_MODELS = { 'qwen2.5vl:7b': 'qwen2.5vl:7b', 'moondream:latest': 'moondream:latest',
+  'moondream': 'moondream:latest', 'fast': 'moondream:latest', 'accurate': 'qwen2.5vl:7b', 'llava:13b': 'llava:13b' };
+function pickVisionModel(req) { return VISION_MODELS[String(req || '').toLowerCase().trim()] || OLLAMA_VISION_MODEL; }
+function ollamaVision(b64, prompt, timeoutMs, model) {
   return new Promise(resolve => {
     let target; try { target = new URL('/api/generate', OLLAMA_URL); } catch (e) { return resolve({ error: 'bad OLLAMA_URL' }); }
     const lib = target.protocol === 'https:' ? https : http;
-    const payload = JSON.stringify({ model: OLLAMA_VISION_MODEL, prompt, images: [b64],
+    const payload = JSON.stringify({ model: model || OLLAMA_VISION_MODEL, prompt, images: [b64],
       stream: false, format: 'json', options: { temperature: 0 } });
     // default 35s is fine once the model is warm; callers expecting a possible COLD model
     // load (first call after idle ≈ 50s here) pass a larger budget so they don't false-timeout.
@@ -604,7 +609,9 @@ const appHandler = (req, res) => {
       if (!p.dataUrl) return send(res, 400, { err: 'dataUrl required' });
       const b64 = p.dataUrl.replace(/^data:image\/\w+;base64,/, '');
       const prompt = 'This is a wallcovering or fabric SAMPLE label. Identify the BRAND from its logo or wordmark and note the typesetting. Reply ONLY as compact JSON: {"brand":"<manufacturer/brand or empty if unsure>","confidence":<0-1>,"logo":"<short logo/wordmark description>","typeface":"<e.g. serif wordmark / sans caps / script>","code":"<any SKU or model number visible, else empty>"}';
-      const r = await ollamaVision(b64, prompt);
+      // model toggle: {model:"fast"|"moondream"|"qwen2.5vl:7b"|...} or {fast:true} → moondream (faster, less accurate)
+      const useModel = pickVisionModel(p.fast ? 'fast' : p.model);
+      const r = await ollamaVision(b64, prompt, p.fast ? 25000 : undefined, useModel);
       let out = {}; try { out = JSON.parse(r.response || '{}'); } catch (e) { out = {}; }
       const brand = (out.brand || '').toString().trim();
       // primary: cross-check the VLM's brand guess against the learned lexicon, then fuzzy name.
@@ -616,7 +623,7 @@ const appHandler = (req, res) => {
         const fp = fingerprintVendor(brand, out.typeface);
         if (fp) { matched = fp.vendor; matchVia = 'fingerprint'; }
       }
-      return send(res, 200, { ok: !r.error, model: OLLAMA_VISION_MODEL,
+      return send(res, 200, { ok: !r.error, model: useModel,
         brand: brand || null, confidence: (out.confidence ?? null), logo: out.logo || null,
         typeface: out.typeface || null, code: (out.code || '').toString().toUpperCase().replace(/\s+/g, '') || null,
         vendor: matched, match_via: matchVia, err: r.error || null });

← 934a758 auto-save: 2026-06-26T00:38:23 (2 files) — data/build.json d  ·  back to Dw Photo Capture  ·  scan: background logo-ID when a code locks but OCR names no 64ab508 →