Handoff doc: how to wire the fine-tuned DW CLIP weights into visual search (sanity-check, re-embed both sides, gate the reindex, rollback)

b0a3eccebc5096890f2eaf020874ceeec38abe29 · 2026-07-06 21:22:44 -0700 · Steve Abrams
Files touched

A visual-search/train/HANDOFF.md
Diff

commit b0a3eccebc5096890f2eaf020874ceeec38abe29
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Mon Jul 6 21:22:44 2026 -0700

    Handoff doc: how to wire the fine-tuned DW CLIP weights into visual search (sanity-check, re-embed both sides, gate the reindex, rollback)
---
 visual-search/train/HANDOFF.md | 63 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/visual-search/train/HANDOFF.md b/visual-search/train/HANDOFF.md
new file mode 100644
index 0000000..ab7e6ad
--- /dev/null
+++ b/visual-search/train/HANDOFF.md
@@ -0,0 +1,63 @@
+# Handoff — wire the fine-tuned DW CLIP weights into visual search
+
+The overnight run fine-tunes `open_clip ViT-B-32` (laion2b) on the DW catalog and saves the
+weights to `visual-search/train/ckpt/dw_clip_ft.pt` (per-epoch: `dw_clip_ft_ep<N>.pt`).
+Same architecture + pretrained tag as the live search service, so the fine-tuned weights are a
+**drop-in replacement** — no code rewrite, just load the checkpoint over the base model.
+
+## 0. Confirm the run finished
+```sh
+cd ~/Projects/dw-photo-capture/visual-search/train
+tail -20 logs/overnight.log          # look for "OVERNIGHT RUN COMPLETE" or "DONE — final weights"
+ls -la ckpt/                          # dw_clip_ft.pt should exist (+ dw_clip_ft_ep*.pt)
+```
+If it stopped early (`max_hours`), `dw_clip_ft.pt` is still saved from the last step — usable.
+
+## 1. Sanity-check the fine-tuned model BEFORE shipping it
+The fine-tune can help OR (if LR was too high / too few images) drift. Verify it retrieves DW
+patterns at least as well as base CLIP. Quick check on this Mac:
+```sh
+./venv/bin/python - <<'PY'
+import torch, open_clip
+m,_,pp = open_clip.create_model_and_transforms("ViT-B-32", pretrained="laion2b_s34b_b79k")
+sd = torch.load("ckpt/dw_clip_ft.pt", map_location="cpu")
+print("missing/unexpected:", m.load_state_dict(sd, strict=False))   # should be [] / []
+print("loaded OK — logit_scale:", float(m.logit_scale))
+PY
+```
+`strict=False` must report no missing/unexpected keys. If it errors, the checkpoint arch drifted —
+stop and inspect (don't ship).
+
+## 2. Two ways to deploy (pick one)
+
+### A. Re-embed the catalog with the fine-tuned model (BEST — full benefit)
+The retrieval quality gain only shows if BOTH the query image AND the catalog are embedded by the
+SAME model. So re-embed with the fine-tuned weights:
+1. Copy `ckpt/dw_clip_ft.pt` to Kamatera `/root/public-projects/dwphoto/visual-search/dw_clip_ft.pt`.
+2. In `embed_catalog.py` AND `search_service.py`, after `create_model_and_transforms(...)`, add:
+   ```python
+   import torch, os
+   _ft = os.path.join(os.path.dirname(__file__), "dw_clip_ft.pt")
+   if os.path.exists(_ft): model.load_state_dict(torch.load(_ft, map_location="cpu"), strict=False)
+   ```
+3. Truncate + re-embed: this is a full re-embed (~255k images, ~4h on Kamatera CPU). Either
+   `TRUNCATE image_embeddings` then let `embed-daemon.sh` refill, or embed into a new table and
+   swap. **Gated** — it rewrites the whole visual index; do it deliberately, off-peak.
+4. `pm2 restart dwphoto-vsearch` + `curl :9914/reload`.
+
+### B. Query-side only (fast, partial benefit)
+Load the fine-tuned weights in `search_service.py` ONLY (step 2's snippet). The query image gets the
+DW-specialized embedding but the catalog is still base-CLIP → mismatched spaces, retrieval may be
+WORSE, not better. **Not recommended** unless you also do A. Mixing embedding models is the classic
+foot-gun here — same-model-both-sides is the rule.
+
+## 3. Rollback
+Keep the base model one line away: comment out the `load_state_dict` line and restart. The base
+embeddings in `image_embeddings` are unaffected unless you did A's re-embed — so before A, snapshot:
+`pg_dump ... -t image_embeddings > image_embeddings.bak.sql`.
+
+## TL;DR
+- Weights: `visual-search/train/ckpt/dw_clip_ft.pt` (drop-in ViT-B-32).
+- Right way: load them in BOTH `embed_catalog.py` + `search_service.py`, re-embed the catalog, restart vsearch.
+- Never mix: query and catalog must be embedded by the same model or retrieval degrades.
+- Gate the full re-embed (rewrites the index); snapshot `image_embeddings` first.
← da94b6f Card-free measurement groundwork: lock camera zoom + /caps c · back to Dw Photo Capture · Camera-unavailable UX: detect Chrome/3rd-party iOS browsers 8cff6c4 →