observability: add /api/stats; backlog O/0 OCR fallback with evidence (cycle 19)

e3483ef8aacb60e838b56393d707fc3fa8d6f6a7 · 2026-06-26 08:20:13 -0700 · Steve Abrams

Investigated the OCR O-vs-0 / I-vs-1 normalization idea empirically: the ambiguity is REAL
(of 1033 sample catalog codes, 122 contain a letter-O and 47 contain both O and 0, e.g.
SNPBOBJML11I3, BMGI006). But a recovery fallback touches the hot resolve path and carries a
false-match risk that can't be tested headlessly (resolve hits live endpoints), so I
BACKLOGGED it with the evidence + a safe design (exact-first, variant-only-on-miss, capped)
rather than ship speculative hot-path complexity.

Shipped the clean additive win instead: GET /api/stats — a read-only observability snapshot
for the self-teaching scanner (version, vendor count, seeded, fingerprinted, total learned
scans, last-learn timestamp, recents, catalog indexed, vision model). Auth-gated, in-memory,
zero hot-path risk. Verified: 200 with creds / 401 without; lint + 51 tests green. $0 (local).

Files touched

M OVERNIGHT_LEDGER.md
M server.js

Diff

commit e3483ef8aacb60e838b56393d707fc3fa8d6f6a7
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Fri Jun 26 08:20:13 2026 -0700

    observability: add /api/stats; backlog O/0 OCR fallback with evidence (cycle 19)
    
    Investigated the OCR O-vs-0 / I-vs-1 normalization idea empirically: the ambiguity is REAL
    (of 1033 sample catalog codes, 122 contain a letter-O and 47 contain both O and 0, e.g.
    SNPBOBJML11I3, BMGI006). But a recovery fallback touches the hot resolve path and carries a
    false-match risk that can't be tested headlessly (resolve hits live endpoints), so I
    BACKLOGGED it with the evidence + a safe design (exact-first, variant-only-on-miss, capped)
    rather than ship speculative hot-path complexity.
    
    Shipped the clean additive win instead: GET /api/stats — a read-only observability snapshot
    for the self-teaching scanner (version, vendor count, seeded, fingerprinted, total learned
    scans, last-learn timestamp, recents, catalog indexed, vision model). Auth-gated, in-memory,
    zero hot-path risk. Verified: 200 with creds / 401 without; lint + 51 tests green. $0 (local).
---
 OVERNIGHT_LEDGER.md |  2 ++
 server.js           | 20 ++++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/OVERNIGHT_LEDGER.md b/OVERNIGHT_LEDGER.md
index 75d3dc3..1473050 100644
--- a/OVERNIGHT_LEDGER.md
+++ b/OVERNIGHT_LEDGER.md
@@ -13,6 +13,7 @@ explicit backlog empties, each cycle BRAINSTORMS fresh refinement ideas (local L
 the single highest-value safe one, the officer executes it, and the loop reschedules — it does NOT stop.
 
 ## Backlog (safe / reversible unless flagged GATED)
+- [ ] OCR O/0 + I/1 variant fallback (cycle-19 evidence: 122/1033 codes contain letter-O, 47 contain both O+0 — ambiguity is REAL). SAFE design only: generate O↔0/I↔1 variants and try them ONLY after the exact code fails to resolve, never replacing the exact match; cap variants (≤8) to bound lookup calls; first check no two catalog codes are pure O/0-variants of each other (false-match risk). Touches scanResolve (hot path) — pure variant-generator is unit-testable; wiring needs care. Possibly /dtd the approach.
 - [ ] GATED/Steve: Data volume ~100% full — review big consumers: ~/.ollama 45G (16 models, prune unused?), ~/Projects 83G (node_modules dupes?), ~250G in sudo-gated system/var areas needs Steve to inspect (`sudo du -xh -d1 /System/Volumes/Data | sort -rh | head`). Loop kept tripping ENOSPC.
 - [x] A-verdict: tighten logo matcher + validate fingerprints + re-harvest  (DTD 3/3)
 - [x] Wire logo fingerprints into /api/identify as a TIEBREAKER (DTD-A, name-match primary)
@@ -64,3 +65,4 @@ the single highest-value safe one, the officer executes it, and the loop resched
 ### Deferred known-minor (cycle 17, low value)
 - scanResolve runs ALL candidates through TWIL (pass 1) before ANY through Shopify (pass 2), so a weak candidate that hits TWIL beats a barcode that only resolves in Shopify. Narrow edge (barcode present + fails TWIL + weak code hits TWIL); the TWIL-first source preference is deliberate. Revisit only if it surfaces in real use.
 | 18 | brainstorm #2 (ROBUSTNESS) | atomic saveJSON (temp-file + rename) — a crash/ENOSPC mid-write can no longer truncate vendor_profiles.json (180 learned profiles) or progress/recents/favorites; single shared-primitive fix protects all persistence | df 29G; proved via REAL saveJSON: forced write-failure left original intact, 0 stale temps; learn persists 200; lint+51 tests; selfcheck 200 | $0 |
+| 19 | investigate O/0 (real, backlogged) + ship /api/stats | O/0/I/1 ambiguity CONFIRMED real (122 of 1033 codes have letter-O, 47 have both O+0) but touches hot resolve path w/ untestable false-match risk → BACKLOGGED w/ evidence; shipped additive /api/stats (version, vendors, fingerprinted, learn total, last-learn, model) — read-only, auth-gated, zero hot-path risk | df 27G; stats 200 + 401 no-creds; lint+51 tests; selfcheck implied | $0 |
diff --git a/server.js b/server.js
index c608e88..42cb92a 100644
--- a/server.js
+++ b/server.js
@@ -586,6 +586,26 @@ const appHandler = (req, res) => {
       .sort((a, b) => (b.learned_scans || 0) - (a.learned_scans || 0) || (b.n || 0) - (a.n || 0));
     return send(res, 200, { total: vendors.length, seeded: VENDOR_LEX.map(v => v.name), vendors });
   }
+  // Observability snapshot for the self-teaching scanner — read-only, in-memory. $0.
+  if (u.pathname === '/api/stats' && req.method === 'GET') {
+    const profs = Object.values(VENDOR_PROFILES);
+    let lastLearn = null, learnTotal = 0;
+    for (const p of profs) {
+      learnTotal += (p.learned_scans || 0);
+      if (p.learned_scans > 0 && p.updated_at && (!lastLearn || p.updated_at > lastLearn)) lastLearn = p.updated_at;
+    }
+    return send(res, 200, {
+      version: buildLabel(),
+      vendors: profs.length,
+      seeded: VENDOR_LEX.length,
+      fingerprinted: profs.filter(p => p.logo_valid === true).length,
+      learned_scans_total: learnTotal,
+      last_learn: lastLearn,
+      recents: Object.keys(recents).length,
+      catalog_indexed: CATALOG.length,
+      vision_model: OLLAMA_VISION_MODEL,
+    });
+  }
   // Refine a vendor's profile from a scan that resolved to a real product. Conservative:
   // we only fold in structured signal (the on-swatch code prefix), never raw OCR prose.
   if (u.pathname === '/api/learn' && req.method === 'POST') {

← c3b1b21 auto-save: 2026-06-26T08:13:39 (2 files) — data/build.json d · back to Dw Photo Capture · test: reject O/0 OCR fallback (real collisions) + lock verba eb1e546 →