`visual-search/train/HANDOFF.md`

64 lines
# Handoff — wire the fine-tuned DW CLIP weights into visual search

The overnight run fine-tunes `open_clip ViT-B-32` (laion2b) on the DW catalog and saves the
weights to `visual-search/train/ckpt/dw_clip_ft.pt` (per-epoch: `dw_clip_ft_ep<N>.pt`).
Same architecture + pretrained tag as the live search service, so the fine-tuned weights are a
**drop-in replacement** — no code rewrite, just load the checkpoint over the base model.

## 0. Confirm the run finished
```sh
cd ~/Projects/dw-photo-capture/visual-search/train
tail -20 logs/overnight.log          # look for "OVERNIGHT RUN COMPLETE" or "DONE — final weights"
ls -la ckpt/                          # dw_clip_ft.pt should exist (+ dw_clip_ft_ep*.pt)
```
If it stopped early (`max_hours`), `dw_clip_ft.pt` is still saved from the last step — usable.

## 1. Sanity-check the fine-tuned model BEFORE shipping it
The fine-tune can help OR (if LR was too high / too few images) drift. Verify it retrieves DW
patterns at least as well as base CLIP. Quick check on this Mac:
```sh
./venv/bin/python - <<'PY'
import torch, open_clip
m,_,pp = open_clip.create_model_and_transforms("ViT-B-32", pretrained="laion2b_s34b_b79k")
sd = torch.load("ckpt/dw_clip_ft.pt", map_location="cpu")
print("missing/unexpected:", m.load_state_dict(sd, strict=False))   # should be [] / []
print("loaded OK — logit_scale:", float(m.logit_scale))
PY
```
`strict=False` must report no missing/unexpected keys. If it errors, the checkpoint arch drifted —
stop and inspect (don't ship).

## 2. Two ways to deploy (pick one)

### A. Re-embed the catalog with the fine-tuned model (BEST — full benefit)
The retrieval quality gain only shows if BOTH the query image AND the catalog are embedded by the
SAME model. So re-embed with the fine-tuned weights:
1. Copy `ckpt/dw_clip_ft.pt` to Kamatera `/root/public-projects/dwphoto/visual-search/dw_clip_ft.pt`.
2. In `embed_catalog.py` AND `search_service.py`, after `create_model_and_transforms(...)`, add:
   ```python
   import torch, os
   _ft = os.path.join(os.path.dirname(__file__), "dw_clip_ft.pt")
   if os.path.exists(_ft): model.load_state_dict(torch.load(_ft, map_location="cpu"), strict=False)
   ```
3. Truncate + re-embed: this is a full re-embed (~255k images, ~4h on Kamatera CPU). Either
   `TRUNCATE image_embeddings` then let `embed-daemon.sh` refill, or embed into a new table and
   swap. **Gated** — it rewrites the whole visual index; do it deliberately, off-peak.
4. `pm2 restart dwphoto-vsearch` + `curl :9914/reload`.

### B. Query-side only (fast, partial benefit)
Load the fine-tuned weights in `search_service.py` ONLY (step 2's snippet). The query image gets the
DW-specialized embedding but the catalog is still base-CLIP → mismatched spaces, retrieval may be
WORSE, not better. **Not recommended** unless you also do A. Mixing embedding models is the classic
foot-gun here — same-model-both-sides is the rule.

## 3. Rollback
Keep the base model one line away: comment out the `load_state_dict` line and restart. The base
embeddings in `image_embeddings` are unaffected unless you did A's re-embed — so before A, snapshot:
`pg_dump ... -t image_embeddings > image_embeddings.bak.sql`.

## TL;DR
- Weights: `visual-search/train/ckpt/dw_clip_ft.pt` (drop-in ViT-B-32).
- Right way: load them in BOTH `embed_catalog.py` + `search_service.py`, re-embed the catalog, restart vsearch.
- Never mix: query and catalog must be embedded by the same model or retrieval degrades.
- Gate the full re-embed (rewrites the index); snapshot `image_embeddings` first.