`SURGERY-PLAN.md`

295 lines
# Designer-Wallcoverings — Git Surgery Plan

**Status:** PREP ONLY. Steve runs the actual surgery. Do not execute the rewrite steps in this file without explicit go.
**Owner:** Steve
**Drafted:** 2026-05-19
**Branch:** `chore/git-surgery-plan` (local-only, NEVER merged to main)

---

## 1. Current state (the key surprise)

The parking-lot description said `Designer-Wallcoverings` has a 20 GB `.git`. That's **only true on Kamatera**. The three clones are wildly divergent:

| Host | Path | `.git` size | HEAD | Branches | Remote | Notes |
|---|---|---|---|---|---|---|
| **Mac2** (this box) | `~/Projects/Designer-Wallcoverings` | **572 MB** | `954f2b1 snapshot — gitify backup 2026-05-19` | 1 (`main`) | none | 20 commits, no stashes, fresh-init local repo. Working tree 5.5 GB. |
| **Mac1** | `~/Projects/Designer-Wallcoverings` | **MISSING** | — | — | — | No clone exists on Mac1 (`StevesMacStudio.lan`). |
| **Kamatera** | `/root/Projects/Designer-Wallcoverings` | **19.22 GB pack + 201 packs = 55 GB working tree** | `183db25 losangelesfabrics: color-family dot filter…` | 15 (`main` + many `ralph/*`) | `origin = https://github.com/Stevemdr/Designer-Wallcoverings.git` | Has 1 stash. Dirty working tree. |

**Critical:** Mac2's HEAD `954f2b1` is **not reachable from** Kamatera's HEAD `183db25`. They are parallel orphan timelines — the Mac2 repo was re-initialized at some point and shares zero history with the canonical Kamatera/GitHub repo. Any "surgery" decision has to pick which timeline is canonical.

**Decision required before surgery starts:** Steve must confirm Kamatera + GitHub is the canonical timeline (it is — it has origin, branches, real history). If yes, Mac2's current 20 commits need to be either (a) cherry-picked onto the Kamatera timeline before surgery, or (b) abandoned. The Mac2 bundle backup preserves them either way.

---

## 2. Top bloat sources (from Mac2 — Kamatera bloat is presumed similar but larger)

Top 25 directories by aggregate blob size (only blobs >100 KB counted):

| Bytes in history | # blobs | Directory |
|---:|---:|---|
| 139,887,734 | 193 | `DW-Websites/GrassclothWallcoverings` |
| 138,950,647 | 778 | `DW-Websites/GrassclothWallpaper.com` |
| 108,511,850 | 201 | `DW-Programming/ImportNewSkufromURL` |
| 85,470,979 | 5 | `DW-Programming/room-setting-app` |
| 60,851,140 | 191 | `DW-Websites/Novasuede` |
| 41,956,706 | 78 | `DW-Programming/DW-Duplicate-Recolor-v1.4` |
| 15,302,405 | 5 | `DW-Websites/GoldLeafWallpaper` |
| 15,077,471 | 13 | `DW-Agents/dw-agents` |
| 13,188,302 | 25 | `shopify/scripts` |
| 9,911,012 | 8 | `DW-Programming/150dpi` |
| 9,342,147 | 9 | `DW-Programming/RemoveDuplicatesInShopify` |
| 5,054,464 | 5 | `OLD_/DW-Programming` |

Single biggest offenders (already excluded by `.gitignore` going forward, but historically present):

- `DW-Programming/room-setting-app/data/gallery/renders.json` — **two 40 MB blobs** (renders.json snapshots)
- `DW-Programming/ImportNewSkufromURL/dwjs-spreadsheet.csv` — 17.9 MB
- `DW-Programming/ImportNewSkufromURL/fmpro-master.csv` — 13.7 MB
- `DW-Websites/Novasuede/images/{left,right}-curtain-d1.png` — 11–13 MB each
- `DW-Websites/GrassclothWallpaper.com/grasscloth-collection-all.json` — 9.7 MB
- `DW-Agents/dw-agents/agent-thibaut-updater/data/thibaut-spreadsheet.csv` — 5.5 MB
- `DW-Programming/ImportNewSkufromURL/plugins/temp-clone/demo.gif` — 11 MB
- `DW-Websites/GrassclothWallpaper.com/downloaded-images/` and `cropped-images/` — hundreds of JPGs in 1–3 MB range
- `DW-Websites/GrassclothWallcoverings/Novasuede/images/colors/` — many 1–3 MB color swatch JPGs
- `OLD_/DW-Programming/ImportNewSkufromURL/data/centralized.db` — 3 MB SQLite

The pattern is clear: **vendor images, downloaded catalog JSON dumps, CSV spreadsheets, and generated room-render PNGs** committed historically. On Kamatera the bloat is presumed to be these same dirs + additional ralph-branch experiments + larger image sets accumulated over time.

---

## 3. Proposed approach

**Tool: `git filter-repo`** (NOT BFG, NOT `git filter-branch`).

Rationale:
- `git filter-repo` is the modern replacement for `git filter-branch` — faster, safer, and explicitly recommended by the git project. BFG is older, requires Java, and only handles file/folder removal + content replacement (no path renaming, no commit-author rewrite, etc.).
- `filter-repo` operates on a fresh clone by default (refuses to run on a repo with a remote) — this guards against accidental destruction of the working repo. Use `--force` only when running on a known-disposable copy.
- Path-based removal is what we need (whole directories of images / generated data), and `filter-repo --path-glob` handles this cleanly.
- All current `.gitignore` exclusions added 2026-05-18 ("Git-surgery additions") tell us exactly what historical paths to strip — the gitignore is effectively the surgery target list.

**Surgery target list (paths to remove from all history):**
```
DW-Programming/room-setting-app/data/gallery/renders.json
DW-Programming/room-setting-app/scripts/*/output/
DW-Programming/ImportNewSkufromURL/dwjs-spreadsheet.csv
DW-Programming/ImportNewSkufromURL/fmpro-master.csv
DW-Programming/ImportNewSkufromURL/plugins/temp-clone/
DW-Programming/ImportNewSkufromURL/public/images/vendors/
DW-Programming/ImportNewSkufromURL/public/vendor-analysis-results.json
DW-Programming/RemoveDuplicatesInShopify/schumacher-*.{json,html}
DW-Programming/150dpi/public/room-*.{jpg,png}
DW-Programming/phillipe-romano-cleanup-log.json
DW-Programming/SearchByColorWheel/public/flockedvelvetwallpaper.png
DW-Programming/DW-Duplicate-Recolor-v1.4/
DW-Websites/GrassclothWallpaper.com/downloaded-images/
DW-Websites/GrassclothWallpaper.com/cropped-images/
DW-Websites/GrassclothWallpaper.com/bottom-cropped/
DW-Websites/GrassclothWallpaper.com/grasscloth-collection-all.json
DW-Websites/GrassclothWallcoverings/Novasuede/images/colors/
DW-Websites/GoldLeafWallpaper/review-shots/
DW-Websites/Novasuede/images/{left,right}-curtain-d1.png
DW-Agents/dw-agents/agent-thibaut-updater/data/
DW-Agents/dw-agents/agent-needs-attention/memory.json.backup-*
shopify/scripts/sandberg-wm-*.json
shopify/scripts/sample-variant-audit-report.json
shopify/scripts/bulk-products-variants.jsonl
OLD_/
vendor-logos/*.jpg     # only if these are duplicated under DW-Websites/
**/debug-screenshots/
**/screenshots/
**/figma-assets/
*.bak
*.backup
*.backup-*
```

Final list to be reviewed by Steve before execution.

---

## 4. Exact commands (run on a FRESH CLONE, not the live repo)

```bash
# === step 0: backup is already taken (see §6). Verify it. ===
ls -lh ~/Projects/_archive/Designer-Wallcoverings-mac2-pre-surgery-2026-05-19.bundle

# === step 1: prepare a disposable clone of the canonical (Kamatera/GitHub) repo ===
# Choose a host with disk + RAM headroom. Mac2 has 34 GB free, which is tight
# for a 55 GB working tree; Kamatera has 25 GB free, also tight. Better:
# clone fresh from GitHub on a box with ≥ 80 GB free, OR run on Kamatera
# after clearing space first.
git clone --mirror https://github.com/Stevemdr/Designer-Wallcoverings.git /tmp/dw-surgery.git
cd /tmp/dw-surgery.git
du -sh .

# === step 2: install filter-repo (one-time, on whichever host runs surgery) ===
brew install git-filter-repo            # macOS
# OR
pipx install git-filter-repo            # any system

# === step 3: DRY RUN — generate the analysis report, do not modify anything ===
git filter-repo --analyze
# Output goes to .git/filter-repo/analysis/ — review path-deleted-sizes.txt
# and directories-deleted-sizes.txt to confirm the surgery target list above.

# === step 4: DRY RUN of the actual rewrite ===
# (filter-repo has no built-in --dry-run, so we run on the mirror clone
#  — which is itself the dry run, because the live repos are untouched.)
# Build a paths-to-delete file from §3:
cat > /tmp/dw-paths-to-delete.txt <<'EOF'
DW-Programming/room-setting-app/data/gallery/renders.json
DW-Programming/ImportNewSkufromURL/dwjs-spreadsheet.csv
DW-Programming/ImportNewSkufromURL/fmpro-master.csv
DW-Programming/ImportNewSkufromURL/plugins/temp-clone/
DW-Programming/ImportNewSkufromURL/public/images/vendors/
DW-Programming/RemoveDuplicatesInShopify/schumacher-full.json
DW-Programming/RemoveDuplicatesInShopify/schumacher-editor.html
DW-Programming/150dpi/public/
DW-Programming/phillipe-romano-cleanup-log.json
DW-Programming/SearchByColorWheel/public/flockedvelvetwallpaper.png
DW-Programming/DW-Duplicate-Recolor-v1.4/
DW-Websites/GrassclothWallpaper.com/downloaded-images/
DW-Websites/GrassclothWallpaper.com/cropped-images/
DW-Websites/GrassclothWallpaper.com/bottom-cropped/
DW-Websites/GrassclothWallpaper.com/grasscloth-collection-all.json
DW-Websites/GrassclothWallcoverings/Novasuede/images/colors/
DW-Websites/GoldLeafWallpaper/review-shots/
DW-Websites/Novasuede/images/left-curtain-d1.png
DW-Websites/Novasuede/images/right-curtain-d1.png
DW-Agents/dw-agents/agent-thibaut-updater/data/
shopify/scripts/sandberg-wm-full-2026-04-20T19-36-05-570Z.json
shopify/scripts/sandberg-wm-reconcile-2026-04-20T19-37-17Z.json
shopify/scripts/sample-variant-audit-report.json
shopify/scripts/bulk-products-variants.jsonl
OLD_/
EOF

git filter-repo --invert-paths --paths-from-file /tmp/dw-paths-to-delete.txt

# === step 5: also strip any blobs > 5 MB regardless of path (safety net) ===
git filter-repo --strip-blobs-bigger-than 5M

# === step 6: aggressive gc to actually reclaim disk ===
git reflog expire --expire=now --all
git gc --prune=now --aggressive
du -sh .

# === step 7: compare before/after sizes ===
# Expected: Kamatera 19.22 GiB pack → ~1-2 GiB pack post-surgery.
# If reduction < 50%, the surgery target list missed something — re-run --analyze.
```

---

## 5. Expected post-surgery `.git` size

Mac2 baseline: 572 MB. After stripping the 25 listed paths + `>5 MB` safety net, expect **~50–100 MB** on the Mac2 timeline.

Kamatera baseline: 19.22 GB pack. Expect **~1.0–2.0 GB** post-surgery (the long history + 15 ralph branches will still produce more pack data than Mac2's 20-commit timeline, even after stripping the same paths).

If post-surgery Kamatera is still >3 GB, that's a sign there are big blobs **outside** the listed paths — re-run `git filter-repo --analyze` and add to the path list.

---

## 6. Backup (DONE)

Already taken — Mac2 repo bundled to:

```
~/Projects/_archive/Designer-Wallcoverings-mac2-pre-surgery-2026-05-19.bundle
```

Size: **570 MB**. Bundle contains all refs (`--all`) and is a complete restore source — a single `git clone <bundle>` reconstructs the Mac2 timeline exactly.

**Kamatera backup is NOT yet taken.** Kamatera has only 25 GB free with a 55 GB working tree — a full `cp -r` won't fit. Before surgery, either:

- (a) `git bundle create /root/Projects/_archive/dw-kamatera-pre-surgery-2026-05-19.bundle --all` on Kamatera (should produce ~19 GB bundle that still fits in 25 GB free with margin); OR
- (b) `git clone --mirror` to a different host with more disk (Mac2 has 34 GB free — tight but possible if we delete Mac2's working tree first, which the bundle already preserves); OR
- (c) `git push --mirror` to a second GitHub repo (free, off-host, durable) — recommended.

**Mac1 has no clone**, so no Mac1 backup is needed.

---

## 7. Per-clone reconciliation after surgery

After Steve runs the rewrite on the surgery copy:

1. **Force-push the rewritten history to GitHub** — `git push --force-with-lease --mirror` on the surgery clone. This destroys all old refs on GitHub and replaces them with the rewritten ones.
   - **Policy exception required:** Steve's standing rule is "Never `git push --force` to main/master." This surgery REQUIRES violating that rule — surface it and get explicit confirmation before pushing.
2. **Mac2 must be re-cloned fresh** from the rewritten GitHub. Its current 20 local-only commits live in the bundle backup; cherry-pick onto the new clone if any are still wanted.
3. **Kamatera must be re-cloned fresh** from the rewritten GitHub. Stash on Kamatera (`stash@{0}: WIP on bulk-wallcovering-update-system: b1b060a`) must be saved as a patch first (`git stash show -p stash@{0} > /root/Projects/_archive/kamatera-stash-2026-05-19.patch`).
4. **Mac1 needs a fresh clone** (currently has none) — straightforward, no reconciliation.
5. **Any pm2 processes referencing `/root/Projects/Designer-Wallcoverings`** on Kamatera must be stopped before re-clone, then `pm2 reload` after. Inventory those processes BEFORE surgery — `pm2 jlist | jq '.[] | select(.pm2_env.pm_cwd | test("Designer-Wallcoverings")) | .name'`.

---

## 8. Pre-surgery checklist (Steve to confirm before running step 1)

- [ ] **Canonical timeline confirmed.** Kamatera + GitHub `Stevemdr/Designer-Wallcoverings` is the source of truth. Mac2's 20-commit orphan timeline is either cherry-picked or abandoned.
- [ ] **Kamatera bundle backup exists** at `/root/Projects/_archive/dw-kamatera-pre-surgery-2026-05-19.bundle` (currently missing — must be created).
- [ ] **GitHub mirror backup exists** at a second `Stevemdr/Designer-Wallcoverings-pre-surgery-backup` repo (recommended belt+suspenders).
- [ ] **No uncommitted work** on Kamatera. Currently dirty (.gitignore + CLAUDE.md + 3 other files modified). Either commit or stash → patch → discard.
- [ ] **Kamatera stash@{0} exported** to a `.patch` file.
- [ ] **No open PRs** on GitHub against `Stevemdr/Designer-Wallcoverings` (force-push will invalidate them). Check `gh pr list -R Stevemdr/Designer-Wallcoverings`.
- [ ] **No in-flight build / scrape / agent run** on Kamatera touching this repo. Audit `pm2 list` for processes with cwd inside `Designer-Wallcoverings`.
- [ ] **15 ralph branches** reviewed — keep all, or merge/abandon some before surgery? (Fewer refs = faster filter-repo, smaller post-surgery pack.)
- [ ] **Force-push exception explicitly approved** by Steve.
- [ ] **Surgery host has disk headroom** — for the 55 GB Kamatera repo, recommend running surgery on a machine with ≥ 80 GB free, OR clear Kamatera disk first.

---

## 9. Rollback procedure

If surgery produces a broken result (missing files we needed, history we needed, etc.):

```bash
# 1. Stop any work on the rewritten repo.
# 2. Restore Mac2 timeline from bundle:
cd /tmp && git clone ~/Projects/_archive/Designer-Wallcoverings-mac2-pre-surgery-2026-05-19.bundle Designer-Wallcoverings-restored

# 3. Restore Kamatera timeline (once that bundle exists):
ssh root@45.61.58.125 'cd /tmp && git clone /root/Projects/_archive/dw-kamatera-pre-surgery-2026-05-19.bundle Designer-Wallcoverings-restored'

# 4. Restore GitHub:
# If we made the belt+suspenders backup repo, force-push from it back to the
# main repo. If we didn't, the bundle on Kamatera is the only source — push
# its mirror back to GitHub:
ssh root@45.61.58.125 'cd /tmp/Designer-Wallcoverings-restored && git push --mirror --force-with-lease https://github.com/Stevemdr/Designer-Wallcoverings.git'

# 5. Validate: every clone host re-clones, checks out main, verifies HEAD
#    matches pre-surgery HEAD on Kamatera (183db25...)
```

Rollback window: the bundles never expire. Recommend keeping them in `_archive/` for at least 90 days post-surgery.

---

## 10. What this plan does NOT cover (out of scope)

- LFS migration (would shrink working-tree pulls but doesn't reduce existing `.git`).
- Splitting the repo into multiple smaller repos (e.g. one per `DW-Websites/<site>/`) — separate project.
- Reconciling Mac2's orphan 20-commit timeline with Kamatera's history — Steve decides cherry-pick vs abandon.
- Cleaning up the 15 ralph branches — separate decision.
- Removing secrets from history — current `.gitignore` blocks `.env*`, but historical commits should be scanned via `git secrets` or `trufflehog` BEFORE surgery (and again AFTER) and any hits added to the filter-repo path list.

---

## 11. Recommended sequence summary

1. Steve reviews & approves this plan.
2. Pre-surgery checklist (§8) — Steve runs.
3. Backup Kamatera (bundle + GitHub-mirror backup repo).
4. Run `git filter-repo --analyze` on a fresh mirror clone, review the report.
5. Adjust `/tmp/dw-paths-to-delete.txt` if §3 missed any big offenders.
6. Run the actual rewrite on the mirror clone.
7. Verify post-surgery `.git` size meets §5 target.
8. Force-push to GitHub (with explicit standing-rule exception).
9. Fresh-clone on Mac2, Mac1, Kamatera.
10. Restore Kamatera's stash patch + cherry-pick any wanted Mac2 commits.
11. Restart any pm2 processes that were stopped in §8.

End of plan.