5x contrarian gate: honest re-scope — FT visual-ID validated on the REAL task (degraded phone-like image→image) = only 28% top-1 (n=40), thresholds overlap; clean-image self-match hid this. App still safe (label-OCR primary + confidence+2nd-push gates). Real fix = augmentation re-train (surfaced, gated). Camera/iOS-Safari still untested (headless can't reach getUserMedia)

6a137d9d876284316508f757c3e096f2b1484a23 · 2026-07-07 12:52:50 -0700 · Steve Abrams
Files touched

M 5x/REPORT.md
A visual-search/train/robustness_test.py
Diff

commit 6a137d9d876284316508f757c3e096f2b1484a23
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Tue Jul 7 12:52:50 2026 -0700

    5x contrarian gate: honest re-scope — FT visual-ID validated on the REAL task (degraded phone-like image→image) = only 28% top-1 (n=40), thresholds overlap; clean-image self-match hid this. App still safe (label-OCR primary + confidence+2nd-push gates). Real fix = augmentation re-train (surfaced, gated). Camera/iOS-Safari still untested (headless can't reach getUserMedia)
---
 5x/REPORT.md                           | 32 ++++++++++++++-
 visual-search/train/robustness_test.py | 73 ++++++++++++++++++++++++++++++++++
 2 files changed, 103 insertions(+), 2 deletions(-)

diff --git a/5x/REPORT.md b/5x/REPORT.md
index c24224d..4f7d088 100644
--- a/5x/REPORT.md
+++ b/5x/REPORT.md
@@ -34,6 +34,34 @@ clean (sweep 2). Not fixable in the app because the app isn't broken — it's th
 ## Final six-way state
 M1 ✓ · M2 ✓ · M3 ✓ (sweep 2) · B4 ✓ · (B5/B6 GUI-open skipped in --no-open). App loads in 0.24s authed.
 
-## Still open (not app defects — hardware-dependent)
-- Real-device iPhone-Safari pass (HEIC/camera path) — see DEVICE-TEST.md.
+## ⚠ CONTRARIAN GATE — verdict FIX-FIRST, and it was RIGHT. Honest re-scope below.
+The two "clean sweeps" cover the **server + DOM layer only**. A headless harness structurally
+cannot exercise `getUserMedia` (the camera), HEIC decode, iOS Safari, or a real Shopify media write.
+So "0 app defects" means **0 defects in the paths the harness can reach** — NOT a clean app.
+
+### The real finding the contrarian forced (FT visual-ID on the ACTUAL query distribution)
+The FT model was validated on CLEAN catalog images (self-match 1.0000, held-out image→text 52.9%).
+The deployed task is **image→image**: a degraded handheld iPhone photo → the clean catalog image.
+`robustness_test.py` simulated that (rotate/crop/½-res/blur/glare/JPEG-q55) and queried the LIVE index,
+n=40:
+- **top-1 correct: 28%** (top-5: 38%). NOT the ~100% the self-match implied.
+- On degraded input most results score **'low'** (high=0, medium=5, low=35).
+- Correct top-scores avg **0.770**; WRONG top-scores reach **0.865** → the 0.85/0.92 thresholds don't
+  cleanly separate correct from wrong on real-world input; the distributions overlap.
+
+**Why the app is still SAFE despite this** (not dangerous, just weaker than hoped):
+- Update mode's PRIMARY path is label-OCR (mfr#) → exact resolve; visual is a fallback that requires
+  confidence≠'low' AND a 2nd-Push confirm. Degraded photos mostly return 'low' → the visual auto-bind
+  rarely fires on a bad photo → it falls through to "fix the mfr#", not a wrong live write.
+- Dedup only warns on confidence≠'low' → rarely false-warns on a degraded photo (non-blocking anyway).
+
+**The real fix (recommended, NOT auto-run — it's a big compute decision):** re-fine-tune with data
+AUGMENTATION (blur/crop/brightness/JPEG on the training images) so the model learns degraded→clean
+matching. That's another overnight run; surfaced for Steve, not executed. Threshold-tuning alone
+won't fix it (correct 0.77 vs wrong 0.74 overlap too much).
+
+## Still open (not app defects — hardware-dependent, Steve-owned)
+- **Real-device iPhone-Safari pass (the app's core purpose) — untested. See DEVICE-TEST.md.** This is
+  the contrarian's single highest-leverage ask and it needs Steve's physical phone.
 - One live end-to-end video upload to a real Shopify product.
+- (Recommended) augmentation re-fine-tune to lift real-photo visual-ID above 28% top-1.
diff --git a/visual-search/train/robustness_test.py b/visual-search/train/robustness_test.py
new file mode 100644
index 0000000..a5b66d2
--- /dev/null
+++ b/visual-search/train/robustness_test.py
@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+# Real-world-ish validation of the fine-tuned visual-ID: does a DEGRADED "phone-like" photo of a
+# catalog item still retrieve the CORRECT product from the live FT index — and do the confidence
+# thresholds (>=0.92 high, >=0.85 medium) mean anything on that query distribution?
+# We can't shoot real samples autonomously, so we simulate handheld capture: downscale, JPEG-recompress,
+# blur, brightness/glare, small rotate+crop — then query the LIVE /api/identify-multi endpoint.
+import os, io, sys, json, base64, subprocess, urllib.request, urllib.parse, ssl
+from PIL import Image, ImageFilter, ImageEnhance
+
+APP = "https://photo.designerwallcoverings.com/api/identify-multi"
+AUTH = base64.b64encode(b"admin:DW2024!").decode()
+N = 40
+ctx = ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
+
+# pull N random rows that ARE embedded on KAMATERA (the live FT index) — so the correct answer is in it
+remote_sql = ("select e.dw_sku, e.image_url from image_embeddings e where e.embedding is not null "
+              "and left(e.image_url,4)='http' order by random() limit " + str(N))
+cmd = ('DB="$(grep ^DW_UNIFIED_DB= /root/public-projects/dwphoto/.env|cut -d= -f2-)"; '
+       'psql "$DB" -F "|" -tA -c "' + remote_sql + '"')
+out = subprocess.check_output(['ssh','root@45.61.58.125', cmd]).decode()
+rows = [l.split('|',1) for l in out.splitlines() if '|' in l]
+print(f"pulled {len(rows)} live-indexed rows", flush=True)
+
+def small(u):
+    if 'shopify' in u:
+        pr=urllib.parse.urlparse(u);q=urllib.parse.parse_qs(pr.query);q['width']=['512']
+        return urllib.parse.urlunparse(pr._replace(query=urllib.parse.urlencode(q,doseq=True)))
+    return u
+
+def degrade(im):
+    # mimic a handheld iPhone shot of a physical sample
+    w,h = im.size
+    im = im.rotate(4, expand=False, fillcolor=(240,240,240))        # slight tilt
+    im = im.crop((int(w*0.06), int(h*0.06), int(w*0.94), int(h*0.94)))  # off-center crop
+    im = im.resize((max(64,im.size[0]//2), max(64,im.size[1]//2)))   # lower res (distance)
+    im = im.filter(ImageFilter.GaussianBlur(1.2))                    # hand shake / focus
+    im = ImageEnhance.Brightness(im).enhance(1.18)                   # glare/overexposure
+    im = ImageEnhance.Contrast(im).enhance(0.92)
+    b=io.BytesIO(); im.save(b,'JPEG',quality=55); b.seek(0)          # phone JPEG compression
+    return Image.open(b).convert('RGB')
+
+def query(im):
+    bio=io.BytesIO(); im.save(bio,'JPEG',quality=80)
+    b64='data:image/jpeg;base64,'+base64.b64encode(bio.getvalue()).decode()
+    req=urllib.request.Request(APP, data=json.dumps({'front':b64}).encode(),
+        headers={'Content-Type':'application/json','Authorization':'Basic '+AUTH,'User-Agent':'rt/1'})
+    return json.load(urllib.request.urlopen(req, timeout=30, context=ctx))
+
+top1=0; top5=0; conf={'high':0,'medium':0,'low':0}; scored=[]; n=0
+for dw_sku,url in rows:
+    try:
+        data=urllib.request.urlopen(urllib.request.Request(small(url),headers={'User-Agent':'rt/1'}),timeout=20).read()
+        q=degrade(Image.open(io.BytesIO(data)).convert('RGB'))
+        r=query(q); vis=r.get('visual') or []
+        if not vis: continue
+        n+=1
+        skus=[str(v.get('dw_sku')) for v in vis]
+        if skus and skus[0]==dw_sku: top1+=1
+        if dw_sku in skus[:5]: top5+=1
+        conf[r.get('confidence','low')] = conf.get(r.get('confidence','low'),0)+1
+        scored.append((dw_sku, round(vis[0].get('score',0),3), skus[0]==dw_sku, r.get('confidence')))
+    except Exception as e:
+        print('skip',e, flush=True)
+
+print(f"\n=== DEGRADED (phone-like) image→image retrieval on the LIVE FT index, n={n} ===")
+print(f"top-1 correct: {top1}/{n} ({100*top1/max(1,n):.0f}%)   top-5: {top5}/{n} ({100*top5/max(1,n):.0f}%)")
+print(f"confidence buckets: {conf}")
+correct_scores=[s for _,s,c,_ in scored if c]; wrong_scores=[s for _,s,c,_ in scored if not c]
+if correct_scores: print(f"top score when CORRECT: min={min(correct_scores):.3f} avg={sum(correct_scores)/len(correct_scores):.3f}")
+if wrong_scores:   print(f"top score when WRONG:   max={max(wrong_scores):.3f} avg={sum(wrong_scores)/len(wrong_scores):.3f}")
+# threshold sanity: of matches labeled high (>=0.92), how many were actually correct?
+high=[c for _,s,c,cf in scored if cf=='high'];
+if high: print(f"of 'high'-confidence results, {sum(high)}/{len(high)} were the CORRECT product ({100*sum(high)/len(high):.0f}%)")
← 8694d87 5x: 2 consecutive clean sweeps (0 app defects) post FT-model · back to Dw Photo Capture · Augmentation re-fine-tune: warm-start from dw_clip_ft.pt + h 54e085f →