`5x/REPORT.md`

68 lines
# /5x REPORT — photo.designerwallcoverings.com

Target: https://photo.designerwallcoverings.com (auth admin:DW2024!) — the dwphoto scanner, after
the full session: spec capture, per-field mics, 10/6 media gallery → Shopify (draft-create +
confirmation-gated live-update), video transcode poll, visual pattern-ID, calibrate-once measure,
**fine-tuned DW CLIP wired in + catalog fully re-embedded**, and **Add-mode visual dedup**.

## Sweep ledger
| sweep | runner | flow errors | dedup | caught | app defects | fixed |
|---|---|---|---|---|---|---|
| 1 | 3/4 (M3 auth artifact) | 0 | ✓ | 1 (M3, proven non-app) | **0** | 0 |
| 2 | **4/4** (M3 passed) | 0 | ✓ + calibrate ✓ | 0 | **0** | 0 |

`sweep 1: 3/4 · caught 1 (M3 harness auth artifact) · fixed 0`
`sweep 2: 4/4 · caught 0 · fixed 0`

**Stop condition met: two consecutive sweeps with ZERO app defects → DONE.** No commits — nothing
in the app needed fixing this run. (The substantive fixes happened earlier this session under two
prior /yolo hardening loops — 8+1 review defects and 2 dedup defects — all already committed.)

## The one non-green — M3 sweep 1 — proven NON-app
The /3x M3 method sends no basic-auth → hits the site's 401 gate → `ERR_INVALID_AUTH_CREDENTIALS`.
Proof: authed page load = **0.24s**; no-auth load = the exact M3 error. M3 passes when the run lands
clean (sweep 2). Not fixable in the app because the app isn't broken — it's the runner omitting creds.

## What was exercised beyond the runner (auth-gated SPA — real surface is endpoints + flows)
- **Endpoints (all 200):** /healthz, /api/vendors, /api/resolve-product, /api/media-status, /caps.
- **Headless flow (Playwright/system Chrome), 0 console + 0 page errors, both sweeps:** camera-first
  Add, 10/6 media counters, Update mode, Check-Sample fb modal.
- **Dedup + FT visual-ID:** "⚠ may already exist as Atlas (DWWG-535131) · high" + Update-instead switch.
- **Calibrate-once measure:** reopen → no card.
- **FT visual-ID at full coverage:** exact self-match 1.0000 / high (verified after re-embed completed).

## Final six-way state
M1 ✓ · M2 ✓ · M3 ✓ (sweep 2) · B4 ✓ · (B5/B6 GUI-open skipped in --no-open). App loads in 0.24s authed.

## ⚠ CONTRARIAN GATE — verdict FIX-FIRST, and it was RIGHT. Honest re-scope below.
The two "clean sweeps" cover the **server + DOM layer only**. A headless harness structurally
cannot exercise `getUserMedia` (the camera), HEIC decode, iOS Safari, or a real Shopify media write.
So "0 app defects" means **0 defects in the paths the harness can reach** — NOT a clean app.

### The real finding the contrarian forced (FT visual-ID on the ACTUAL query distribution)
The FT model was validated on CLEAN catalog images (self-match 1.0000, held-out image→text 52.9%).
The deployed task is **image→image**: a degraded handheld iPhone photo → the clean catalog image.
`robustness_test.py` simulated that (rotate/crop/½-res/blur/glare/JPEG-q55) and queried the LIVE index,
n=40:
- **top-1 correct: 28%** (top-5: 38%). NOT the ~100% the self-match implied.
- On degraded input most results score **'low'** (high=0, medium=5, low=35).
- Correct top-scores avg **0.770**; WRONG top-scores reach **0.865** → the 0.85/0.92 thresholds don't
  cleanly separate correct from wrong on real-world input; the distributions overlap.

**Why the app is still SAFE despite this** (not dangerous, just weaker than hoped):
- Update mode's PRIMARY path is label-OCR (mfr#) → exact resolve; visual is a fallback that requires
  confidence≠'low' AND a 2nd-Push confirm. Degraded photos mostly return 'low' → the visual auto-bind
  rarely fires on a bad photo → it falls through to "fix the mfr#", not a wrong live write.
- Dedup only warns on confidence≠'low' → rarely false-warns on a degraded photo (non-blocking anyway).

**The real fix (recommended, NOT auto-run — it's a big compute decision):** re-fine-tune with data
AUGMENTATION (blur/crop/brightness/JPEG on the training images) so the model learns degraded→clean
matching. That's another overnight run; surfaced for Steve, not executed. Threshold-tuning alone
won't fix it (correct 0.77 vs wrong 0.74 overlap too much).

## Still open (not app defects — hardware-dependent, Steve-owned)
- **Real-device iPhone-Safari pass (the app's core purpose) — untested. See DEVICE-TEST.md.** This is
  the contrarian's single highest-leverage ask and it needs Steve's physical phone.
- One live end-to-end video upload to a real Shopify product.
- (Recommended) augmentation re-fine-tune to lift real-photo visual-ID above 28% top-1.