← back to Dw Photo Capture

OVERNIGHT_LEDGER.md

95 lines

# dw-photo-capture — Overnight officer-yolo run

Mode: **officer-yolo + dtd deciders** (Steve, 2026-06-25 night). DTD commits each
real fork; the relevant VP officer (vp-engineering for code, vp-dw-commerce for
catalog) is final sign-off; reversible local work ships freely; gated actions
(deploy, Shopify writes, scheduled launchd jobs, spend, remote push) are DRAFTED
to `~/.claude/yolo-queue/pending-approval/`, never executed.

All scanner work is **local + $0**. Only metered cost is DTD Codex (~$0.03/run).

**NEVER-STOP mode (Steve, 2026-06-25 night): "keep refining all night long adding ideas."** When the
explicit backlog empties, each cycle BRAINSTORMS fresh refinement ideas (local LLM, $0), DTD picks
the single highest-value safe one, the officer executes it, and the loop reschedules — it does NOT stop.

## Backlog (safe / reversible unless flagged GATED)
- [REJECTED w/ evidence, cycle 20] OCR O/0+I/1 variant fallback: collision gate FAILED — 14 groups of DIFFERENT real catalog codes are pure O/0 or I/1 variants of each other (RI5112 vs R15112, MI725 vs M1725) across 225,846 codes. A variant fallback could resolve to a real-but-WRONG product. The safe contract (search verbatim, never normalize) is now locked by tests.
- [ ] GATED/Steve: Data volume ~100% full — review big consumers: ~/.ollama 45G (16 models, prune unused?), ~/Projects 83G (node_modules dupes?), ~250G in sudo-gated system/var areas needs Steve to inspect (`sudo du -xh -d1 /System/Volumes/Data | sort -rh | head`). Loop kept tripping ENOSPC.
- [x] A-verdict: tighten logo matcher + validate fingerprints + re-harvest  (DTD 3/3)
- [x] Wire logo fingerprints into /api/identify as a TIEBREAKER (DTD-A, name-match primary)
- [x] analyzeOcr unit tests — 21 assertions, npm test + selfcheck gate
- [x] Barcode UX: distinct "▍▍ BARCODE" lock label on a barcode read
- [x] /learn: shows logo fingerprint + flags rejected logos with their read-as
- [x] moondream fast-toggle for /api/identify ({fast}/{model}, allow-listed, documented in API.md)
- [x] Auto-fire /api/identify in background on a lock-with-code-but-no-vendor (bgIdentify)
- [x] /code-review pass — NO critical; 3 robustness/security findings fixed (learn cap, fp len-guard, auth colon)
- [x] GATED (drafted to pending-approval): weekly launchd refresh — plist staged

### Brainstorm pool (cycle 12, qwen3:14b $0) — triaged
- [reject] Font smoothing (Vision handles low-res) · SKU prefix cache (profiles already in-mem) · logo fingerprint ROTATION (fingerprints are TEXT not pixels; VLM handles rotation)
- [REJECTED w/ evidence] OCR confidence thresholding: Vision returns confidence 1.000 or NO read across blur+noise tiers — no low-confidence band exists for a threshold to catch. Moot (cycle 12 A/B).
- [x] Test-coverage: fingerprintVendor edge cases incl. rejected-logo-never-matches (cycle 12)

### Brainstorm pool (cycle 10, qwen3:8b $0) — triaged
- [done-already] SKU prefix boost (180 vendor profiles) · early-exit on confident OCR (topStrong fast-lock)
- [ ] UX: surface OCR confidence on the live overlay (topStrong vs tentative) — marginal, no per-token score yet
- [REJECTED w/ evidence] Swift low-light pre-pass: macOS Vision reads even 4-level-contrast labels RAW perfectly; autocontrast pre-pass = zero gain. Moot — would add risk for no benefit. (cycle 11 A/B)
- [x] Test-coverage: adversarial/degraded OCR inputs against analyzeOcr (cycle 10)

## Cycle log
| # | decision (DTD?) | action | result | cost |
|---|-----------------|--------|--------|------|
| 0 | A-verdict (DTD 3/3) | tighten matcher + fingerprint-validation + re-harvest | 26→19 logos, validation rejects mislabeled | $0.003 (1 DTD) + $0 local |
| 1 | — (clear win) | run project selfcheck after all session changes | lint clean, server checks ok, 0 dead links/console errors | $0 |
| 2 | gated→draft (officer-yolo rule) | draft weekly vendor-refresh launchd job to pending-approval + stage plist | queued for Steve; vp-engineering APPROVE | $0 |
| 3 | — (clear win) | barcode lock shows distinct "▍▍ BARCODE" label (exact read vs OCR guess) | served HTTP 200, inline JS valid | $0 |
| 4 | — (clear win) | /learn surfaces logo fingerprint (reads-as + typeface) and flags the 6 rejected logos with what they actually read as | HTTP 200, 6 rejects visible (Versace→Savannah etc.) | $0 |
| 5 | **DTD 3/3 → A** | wire logo fingerprints into /api/identify as a TIEBREAKER (only when name-match is inconclusive) | clean name still via=name (no regression); fingerprint resolves Mind the Gap/Ralph Lauren in unit test | $0.003 (DTD) + $0 |
| 6 | — (clear win) | analyzeOcr/vendor unit tests — harness slices the REAL fn source from server.js + eval-tests vs crafted inputs & real profile data; wired into npm test + selfcheck | 21/21 pass; lint clean; covers parseOcrRows/analyzeOcr(brand,barcode,prefix,weak,empty)/fuzzyVendor/fingerprintVendor | $0 |
| 7 | — (clear win) | moondream fast-toggle for /api/identify: ollamaVision gains a model arg, allow-listed VISION_MODELS, {fast:true}/{model} per-request override (echoed back), API.md docs | qwen 21.5s (Kravet ✓) vs moondream 12.2s; lint+21 tests green; off-list ignored (no arbitrary pulls) | $0 |
| 8 | — (clear win) | auto-fire /api/identify in background on a lock-with-code-but-no-vendor: thread captured frame into applyScan, bgIdentify() fires VLM logo-ID non-blocking, binds discovered vendor to the SKU via /api/learn | lint+21 tests green; page 200; fire-and-forget, never blocks resolve | $0 |
| 9 | code-review (code-reviewer agent) | independent review of session diff → NO critical; fixed 3 real findings: /api/learn vendor length cap (regex-rebuild DoS), fingerprint validation len(nr)>=4 (short-read false-accept), checkAuth split on FIRST colon only (RFC passwords) | lint+21 tests green; auth good→200/bad→401; oversized vendor→400; normal learn→200 | $0 (local agent) |

### Code-review MINORs deferred (latent, low-risk, not fixed this cycle)
- OLLAMA_URL declared after ollamaVision def (works — only called in handlers; moving it risks colliding with the concurrent voice-feature session that owns that const).
- bgIdentify may teach a prefix from an unresolved OCR code (intentional fire-and-forget; vendor must already be real via lexicon/fuzzy; low impact).
| 10 | brainstorm (qwen3:8b $0) → pick #5 (no fork) | harden analyzeOcr ranking contract with 11 adversarial/degraded-input tests (3-char/no-digit/dashed codes, <4 barcode, two-brand order, lowercase brand, dup dedup, two barcodes) | 21→32 tests pass; lint clean; no behavior change (all pre-verified correct) | $0 |
| 11 | investigate→pivot | A/B proved Swift low-light pre-pass MOOT (Vision reads 4-level contrast raw); pivoted to UX: tentative "hold steady · CODE" cue when a non-strong read needs a 2nd frame | OCR reads all contrast tiers raw; inline JS+lint+32 tests green; page 200 | $0 |
| 12 | brainstorm (qwen3:14b $0) → investigate+pick #5 | A/B proved OCR-confidence-threshold MOOT (Vision = 1.000 or no-read); shipped fingerprintVendor edge tests (empty/short→null, punct-normalized, REJECTED logo never matches) | 32→36 tests pass; lint clean | $0 |
| 13 | DEBUG disk-full + disk-light refine | Data volume was 100% full (ENOSPC blocked all Bash); diagnosed: home=138G (Projects 83G, Ollama 45G/16 models), ~250G in sudo-gated areas (Steve-gated, not auto-deleted); shipped a disk-light pure-test refinement (5 fuzzyVendor edge guards) | df recovered 124Mi→4.7Gi (purgeable reclaimed); 36→41 tests; lint clean; no server restart/scratch | $0 |
| 14 | brainstorm #1 (learning-loop) | profileToLex prefix support gate (>=2 scans) — one bad scan can no longer inject a wrong prefix into the ranking lexicon; catalog prefixes (builder-vetted n>=3) unaffected; +5 profileToLex tests | df 27G healthy; real-data impact = 1 unconfirmed prefix gated (WC test artifact), 0 vendors lose matching; 41→46 tests; lint clean; restart 200 | $0 |
| 15 | brainstorm #4 (SECURITY) | found+fixed a real DoS: vendor="__proto__" → VENDOR_PROFILES["__proto__"] returns Object.prototype → prof.prefixes deref CRASHED the process. Guard rejects __proto__/constructor/prototype with 400 | df 31G; exploit reproduced (HTTP 000 + crash, launchd auto-restarted); post-fix all 3 → 400, server stays up, normal learn 200 | $0 |
| 16 | sweep (defer-w-evidence) + fuzzy ambiguity guard | (a) proto-key sweep: only 2 OBJ[key]= sites, vendor-key fixed cycle15, prefix-key proven safe live (CONSTRUCTOR/PROTOTYPE skus→200). (b) fuzzy-collision audit found REAL collisions (Alan~Nina Campbell 0.33, Retro~Rebel Walls 0.30) → added ambiguity guard: 2 different vendors within margin → null (no wrong-vendor binding) | df 29G; 46→48 tests; lint clean; CAMPBELL→null, NINA CAMPBELL→Nina, BRUNSWIG→Brunschwig preserved; live identify no-regression | $0 |
| 17 | audit idea #3 → defer-w-evidence + lock contract | scanResolve first-hit-wins is ALREADY correct: analyzeOcr orders candidates barcode > brand-prefixed > larger-font > generic (verified: WDW2310>XQ5599 at equal height, barcode>huge-font, CD2222>AB1111). No change to scanResolve; locked the order with 3 regression tests. Residual: TWIL-pass beats Shopify-pass regardless of candidate strength (narrow, deliberate source-preference) — deferred known-minor | df 21G; 48→51 tests; lint clean | $0 |

### Deferred known-minor (cycle 17, low value)
- scanResolve runs ALL candidates through TWIL (pass 1) before ANY through Shopify (pass 2), so a weak candidate that hits TWIL beats a barcode that only resolves in Shopify. Narrow edge (barcode present + fails TWIL + weak code hits TWIL); the TWIL-first source preference is deliberate. Revisit only if it surfaces in real use.
| 18 | brainstorm #2 (ROBUSTNESS) | atomic saveJSON (temp-file + rename) — a crash/ENOSPC mid-write can no longer truncate vendor_profiles.json (180 learned profiles) or progress/recents/favorites; single shared-primitive fix protects all persistence | df 29G; proved via REAL saveJSON: forced write-failure left original intact, 0 stale temps; learn persists 200; lint+51 tests; selfcheck 200 | $0 |
| 19 | investigate O/0 (real, backlogged) + ship /api/stats | O/0/I/1 ambiguity CONFIRMED real (122 of 1033 codes have letter-O, 47 have both O+0) but touches hot resolve path w/ untestable false-match risk → BACKLOGGED w/ evidence; shipped additive /api/stats (version, vendors, fingerprinted, learn total, last-learn, model) — read-only, auth-gated, zero hot-path risk | df 27G; stats 200 + 401 no-creds; lint+51 tests; selfcheck implied | $0 |
| 20 | collision gate → REJECT O/0 fallback + lock verbatim contract | checked 225,846 real codes: 14 collision groups where different products are pure O/0 or I/1 variants (RI5112≠R15112) → O/0 auto-fallback would mis-resolve → REJECTED w/ evidence; locked the safe behavior (analyzeOcr preserves codes verbatim, no silent O↔0/I↔1) with 3 tests | df 34G; 51→54 tests; lint clean | $0 |
| 21 | test-coverage (untested core helper) | mfrToHouseSku (scanned vendor code → DW house SKU) was UNTESTED despite real logic; added 7 guards via an injected MFR_INDEX fixture: exact/normalized match, case+dash+space stripping, <4-char→null, no-match→null, ±1-trailing-char fuzzy (WDW23107→WDW2310), 5-char fuzzy length-guard. Behavior verified empirically first (my initial guesses were wrong — code is correct) | df 34G; 54→61 tests; lint clean | $0 |
| 22 | test-coverage (untested normalizers) | covered nmfr (mfr dedup: WOS3467T==WOS3467; a regression would let duplicate products be created) + brandRe (vendor name → OCR-detection regex source). Behavior verified empirically first; 10 guards: trailing-T strip/trim/upper/null, single-T-only, multiword \s*-join, single-<4-char→null, 2-char-word-drop, no-alpha→null | df 32G; 61→71 tests; lint clean | $0 |
| 23 | test-coverage (last untested branches) + wind-down | locked parseOcrRows barcode-parse branches: symbology-present (sym:QR), symbology-absent (no sym), non-numeric-height→0. Noted: .sym is parsed-but-unused downstream (deferred — no consumer). Untested-pure-helper space now EXHAUSTED | df 31G; 71→74 tests; lint clean | $0 |

## Overnight wind-down note (2026-06-26, cycle 23)
The overnight officer-yolo run is SATURATED — 23 cycles, the codebase is mature/hardened and
the pure-helper test space is now exhausted. Summary of the run:
- 24 commits (3c0bcb1 → cycle 23). Tests grew 21 → 74 (all green); lint clean throughout.
- Features (cycles 0-9): vendor-logo learning from Shopify Files, logo VLM /api/identify,
  fingerprint tiebreaker (DTD-A), barcode/QR exact-SKU, brand-aware fast-lock, model speed
  toggle, background self-teaching bgIdentify, code-review pass (3 fixes), API.md.
- Real hardenings (cycles 14-18): prefix >=2-scan support gate; proto-pollution DoS fix
  (vendor=__proto__ crashed the process); fuzzy ambiguity guard (Alan~Nina Campbell); atomic
  saveJSON (crash/ENOSPC can't corrupt persisted data); /api/stats observability.
- Evidence-based rejections (cycles 11,12,20): Vision low-light pre-pass (Vision reads
  4-level contrast raw), OCR confidence threshold (Vision = 1.0-or-nothing), O/0 variant
  fallback (14 real RI5112≠R15112 collisions in 225k codes).
- Survived a real disk-full (ENOSPC) emergency mid-run; recovered, now canary-monitored.
- Cost: ~$0.006 total (2 DTD Codex calls) + $0 local for everything else.

STILL PENDING STEVE (gated, in ~/.claude/yolo-queue/pending-approval): the weekly
vendor-refresh launchd job (com.steve.dwphoto-vendor-refresh.plist) — plist staged, not
installed. That is the only item needing your sign-off. Nothing was pushed; nothing
customer-facing was touched.
| 24 | health-check | all green (df 29G/93%, lint clean, 74 tests, keepawake PID 1933, selfcheck 200) | $0 |