chore(git-surgery): draft surgery plan for slimming .git (prep only, no rewrite)

ff2dcc5891ec6abb9d18c607032bfbc5ae8d7faa · 2026-05-19 21:18:12 -0700 · Steve
Files touched

A SURGERY-PLAN.md
Diff

commit ff2dcc5891ec6abb9d18c607032bfbc5ae8d7faa
Author: Steve <steve@designerwallcoverings.com>
Date:   Tue May 19 21:18:12 2026 -0700

    chore(git-surgery): draft surgery plan for slimming .git (prep only, no rewrite)
---
 SURGERY-PLAN.md | 294 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 294 insertions(+)

diff --git a/SURGERY-PLAN.md b/SURGERY-PLAN.md
new file mode 100644
index 00000000..b2295f45
--- /dev/null
+++ b/SURGERY-PLAN.md
@@ -0,0 +1,294 @@
+# Designer-Wallcoverings — Git Surgery Plan
+
+**Status:** PREP ONLY. Steve runs the actual surgery. Do not execute the rewrite steps in this file without explicit go.
+**Owner:** Steve
+**Drafted:** 2026-05-19
+**Branch:** `chore/git-surgery-plan` (local-only, NEVER merged to main)
+
+---
+
+## 1. Current state (the key surprise)
+
+The parking-lot description said `Designer-Wallcoverings` has a 20 GB `.git`. That's **only true on Kamatera**. The three clones are wildly divergent:
+
+| Host | Path | `.git` size | HEAD | Branches | Remote | Notes |
+|---|---|---|---|---|---|---|
+| **Mac2** (this box) | `~/Projects/Designer-Wallcoverings` | **572 MB** | `954f2b1 snapshot — gitify backup 2026-05-19` | 1 (`main`) | none | 20 commits, no stashes, fresh-init local repo. Working tree 5.5 GB. |
+| **Mac1** | `~/Projects/Designer-Wallcoverings` | **MISSING** | — | — | — | No clone exists on Mac1 (`StevesMacStudio.lan`). |
+| **Kamatera** | `/root/Projects/Designer-Wallcoverings` | **19.22 GB pack + 201 packs = 55 GB working tree** | `183db25 losangelesfabrics: color-family dot filter…` | 15 (`main` + many `ralph/*`) | `origin = https://github.com/Stevemdr/Designer-Wallcoverings.git` | Has 1 stash. Dirty working tree. |
+
+**Critical:** Mac2's HEAD `954f2b1` is **not reachable from** Kamatera's HEAD `183db25`. They are parallel orphan timelines — the Mac2 repo was re-initialized at some point and shares zero history with the canonical Kamatera/GitHub repo. Any "surgery" decision has to pick which timeline is canonical.
+
+**Decision required before surgery starts:** Steve must confirm Kamatera + GitHub is the canonical timeline (it is — it has origin, branches, real history). If yes, Mac2's current 20 commits need to be either (a) cherry-picked onto the Kamatera timeline before surgery, or (b) abandoned. The Mac2 bundle backup preserves them either way.
+
+---
+
+## 2. Top bloat sources (from Mac2 — Kamatera bloat is presumed similar but larger)
+
+Top 25 directories by aggregate blob size (only blobs >100 KB counted):
+
+| Bytes in history | # blobs | Directory |
+|---:|---:|---|
+| 139,887,734 | 193 | `DW-Websites/GrassclothWallcoverings` |
+| 138,950,647 | 778 | `DW-Websites/GrassclothWallpaper.com` |
+| 108,511,850 | 201 | `DW-Programming/ImportNewSkufromURL` |
+| 85,470,979 | 5 | `DW-Programming/room-setting-app` |
+| 60,851,140 | 191 | `DW-Websites/Novasuede` |
+| 41,956,706 | 78 | `DW-Programming/DW-Duplicate-Recolor-v1.4` |
+| 15,302,405 | 5 | `DW-Websites/GoldLeafWallpaper` |
+| 15,077,471 | 13 | `DW-Agents/dw-agents` |
+| 13,188,302 | 25 | `shopify/scripts` |
+| 9,911,012 | 8 | `DW-Programming/150dpi` |
+| 9,342,147 | 9 | `DW-Programming/RemoveDuplicatesInShopify` |
+| 5,054,464 | 5 | `OLD_/DW-Programming` |
+
+Single biggest offenders (already excluded by `.gitignore` going forward, but historically present):
+
+- `DW-Programming/room-setting-app/data/gallery/renders.json` — **two 40 MB blobs** (renders.json snapshots)
+- `DW-Programming/ImportNewSkufromURL/dwjs-spreadsheet.csv` — 17.9 MB
+- `DW-Programming/ImportNewSkufromURL/fmpro-master.csv` — 13.7 MB
+- `DW-Websites/Novasuede/images/{left,right}-curtain-d1.png` — 11–13 MB each
+- `DW-Websites/GrassclothWallpaper.com/grasscloth-collection-all.json` — 9.7 MB
+- `DW-Agents/dw-agents/agent-thibaut-updater/data/thibaut-spreadsheet.csv` — 5.5 MB
+- `DW-Programming/ImportNewSkufromURL/plugins/temp-clone/demo.gif` — 11 MB
+- `DW-Websites/GrassclothWallpaper.com/downloaded-images/` and `cropped-images/` — hundreds of JPGs in 1–3 MB range
+- `DW-Websites/GrassclothWallcoverings/Novasuede/images/colors/` — many 1–3 MB color swatch JPGs
+- `OLD_/DW-Programming/ImportNewSkufromURL/data/centralized.db` — 3 MB SQLite
+
+The pattern is clear: **vendor images, downloaded catalog JSON dumps, CSV spreadsheets, and generated room-render PNGs** committed historically. On Kamatera the bloat is presumed to be these same dirs + additional ralph-branch experiments + larger image sets accumulated over time.
+
+---
+
+## 3. Proposed approach
+
+**Tool: `git filter-repo`** (NOT BFG, NOT `git filter-branch`).
+
+Rationale:
+- `git filter-repo` is the modern replacement for `git filter-branch` — faster, safer, and explicitly recommended by the git project. BFG is older, requires Java, and only handles file/folder removal + content replacement (no path renaming, no commit-author rewrite, etc.).
+- `filter-repo` operates on a fresh clone by default (refuses to run on a repo with a remote) — this guards against accidental destruction of the working repo. Use `--force` only when running on a known-disposable copy.
+- Path-based removal is what we need (whole directories of images / generated data), and `filter-repo --path-glob` handles this cleanly.
+- All current `.gitignore` exclusions added 2026-05-18 ("Git-surgery additions") tell us exactly what historical paths to strip — the gitignore is effectively the surgery target list.
+
+**Surgery target list (paths to remove from all history):**
+```
+DW-Programming/room-setting-app/data/gallery/renders.json
+DW-Programming/room-setting-app/scripts/*/output/
+DW-Programming/ImportNewSkufromURL/dwjs-spreadsheet.csv
+DW-Programming/ImportNewSkufromURL/fmpro-master.csv
+DW-Programming/ImportNewSkufromURL/plugins/temp-clone/
+DW-Programming/ImportNewSkufromURL/public/images/vendors/
+DW-Programming/ImportNewSkufromURL/public/vendor-analysis-results.json
+DW-Programming/RemoveDuplicatesInShopify/schumacher-*.{json,html}
+DW-Programming/150dpi/public/room-*.{jpg,png}
+DW-Programming/phillipe-romano-cleanup-log.json
+DW-Programming/SearchByColorWheel/public/flockedvelvetwallpaper.png
+DW-Programming/DW-Duplicate-Recolor-v1.4/
+DW-Websites/GrassclothWallpaper.com/downloaded-images/
+DW-Websites/GrassclothWallpaper.com/cropped-images/
+DW-Websites/GrassclothWallpaper.com/bottom-cropped/
+DW-Websites/GrassclothWallpaper.com/grasscloth-collection-all.json
+DW-Websites/GrassclothWallcoverings/Novasuede/images/colors/
+DW-Websites/GoldLeafWallpaper/review-shots/
+DW-Websites/Novasuede/images/{left,right}-curtain-d1.png
+DW-Agents/dw-agents/agent-thibaut-updater/data/
+DW-Agents/dw-agents/agent-needs-attention/memory.json.backup-*
+shopify/scripts/sandberg-wm-*.json
+shopify/scripts/sample-variant-audit-report.json
+shopify/scripts/bulk-products-variants.jsonl
+OLD_/
+vendor-logos/*.jpg     # only if these are duplicated under DW-Websites/
+**/debug-screenshots/
+**/screenshots/
+**/figma-assets/
+*.bak
+*.backup
+*.backup-*
+```
+
+Final list to be reviewed by Steve before execution.
+
+---
+
+## 4. Exact commands (run on a FRESH CLONE, not the live repo)
+
+```bash
+# === step 0: backup is already taken (see §6). Verify it. ===
+ls -lh ~/Projects/_archive/Designer-Wallcoverings-mac2-pre-surgery-2026-05-19.bundle
+
+# === step 1: prepare a disposable clone of the canonical (Kamatera/GitHub) repo ===
+# Choose a host with disk + RAM headroom. Mac2 has 34 GB free, which is tight
+# for a 55 GB working tree; Kamatera has 25 GB free, also tight. Better:
+# clone fresh from GitHub on a box with ≥ 80 GB free, OR run on Kamatera
+# after clearing space first.
+git clone --mirror https://github.com/Stevemdr/Designer-Wallcoverings.git /tmp/dw-surgery.git
+cd /tmp/dw-surgery.git
+du -sh .
+
+# === step 2: install filter-repo (one-time, on whichever host runs surgery) ===
+brew install git-filter-repo            # macOS
+# OR
+pipx install git-filter-repo            # any system
+
+# === step 3: DRY RUN — generate the analysis report, do not modify anything ===
+git filter-repo --analyze
+# Output goes to .git/filter-repo/analysis/ — review path-deleted-sizes.txt
+# and directories-deleted-sizes.txt to confirm the surgery target list above.
+
+# === step 4: DRY RUN of the actual rewrite ===
+# (filter-repo has no built-in --dry-run, so we run on the mirror clone
+#  — which is itself the dry run, because the live repos are untouched.)
+# Build a paths-to-delete file from §3:
+cat > /tmp/dw-paths-to-delete.txt <<'EOF'
+DW-Programming/room-setting-app/data/gallery/renders.json
+DW-Programming/ImportNewSkufromURL/dwjs-spreadsheet.csv
+DW-Programming/ImportNewSkufromURL/fmpro-master.csv
+DW-Programming/ImportNewSkufromURL/plugins/temp-clone/
+DW-Programming/ImportNewSkufromURL/public/images/vendors/
+DW-Programming/RemoveDuplicatesInShopify/schumacher-full.json
+DW-Programming/RemoveDuplicatesInShopify/schumacher-editor.html
+DW-Programming/150dpi/public/
+DW-Programming/phillipe-romano-cleanup-log.json
+DW-Programming/SearchByColorWheel/public/flockedvelvetwallpaper.png
+DW-Programming/DW-Duplicate-Recolor-v1.4/
+DW-Websites/GrassclothWallpaper.com/downloaded-images/
+DW-Websites/GrassclothWallpaper.com/cropped-images/
+DW-Websites/GrassclothWallpaper.com/bottom-cropped/
+DW-Websites/GrassclothWallpaper.com/grasscloth-collection-all.json
+DW-Websites/GrassclothWallcoverings/Novasuede/images/colors/
+DW-Websites/GoldLeafWallpaper/review-shots/
+DW-Websites/Novasuede/images/left-curtain-d1.png
+DW-Websites/Novasuede/images/right-curtain-d1.png
+DW-Agents/dw-agents/agent-thibaut-updater/data/
+shopify/scripts/sandberg-wm-full-2026-04-20T19-36-05-570Z.json
+shopify/scripts/sandberg-wm-reconcile-2026-04-20T19-37-17Z.json
+shopify/scripts/sample-variant-audit-report.json
+shopify/scripts/bulk-products-variants.jsonl
+OLD_/
+EOF
+
+git filter-repo --invert-paths --paths-from-file /tmp/dw-paths-to-delete.txt
+
+# === step 5: also strip any blobs > 5 MB regardless of path (safety net) ===
+git filter-repo --strip-blobs-bigger-than 5M
+
+# === step 6: aggressive gc to actually reclaim disk ===
+git reflog expire --expire=now --all
+git gc --prune=now --aggressive
+du -sh .
+
+# === step 7: compare before/after sizes ===
+# Expected: Kamatera 19.22 GiB pack → ~1-2 GiB pack post-surgery.
+# If reduction < 50%, the surgery target list missed something — re-run --analyze.
+```
+
+---
+
+## 5. Expected post-surgery `.git` size
+
+Mac2 baseline: 572 MB. After stripping the 25 listed paths + `>5 MB` safety net, expect **~50–100 MB** on the Mac2 timeline.
+
+Kamatera baseline: 19.22 GB pack. Expect **~1.0–2.0 GB** post-surgery (the long history + 15 ralph branches will still produce more pack data than Mac2's 20-commit timeline, even after stripping the same paths).
+
+If post-surgery Kamatera is still >3 GB, that's a sign there are big blobs **outside** the listed paths — re-run `git filter-repo --analyze` and add to the path list.
+
+---
+
+## 6. Backup (DONE)
+
+Already taken — Mac2 repo bundled to:
+
+```
+~/Projects/_archive/Designer-Wallcoverings-mac2-pre-surgery-2026-05-19.bundle
+```
+
+Size: **570 MB**. Bundle contains all refs (`--all`) and is a complete restore source — a single `git clone <bundle>` reconstructs the Mac2 timeline exactly.
+
+**Kamatera backup is NOT yet taken.** Kamatera has only 25 GB free with a 55 GB working tree — a full `cp -r` won't fit. Before surgery, either:
+
+- (a) `git bundle create /root/Projects/_archive/dw-kamatera-pre-surgery-2026-05-19.bundle --all` on Kamatera (should produce ~19 GB bundle that still fits in 25 GB free with margin); OR
+- (b) `git clone --mirror` to a different host with more disk (Mac2 has 34 GB free — tight but possible if we delete Mac2's working tree first, which the bundle already preserves); OR
+- (c) `git push --mirror` to a second GitHub repo (free, off-host, durable) — recommended.
+
+**Mac1 has no clone**, so no Mac1 backup is needed.
+
+---
+
+## 7. Per-clone reconciliation after surgery
+
+After Steve runs the rewrite on the surgery copy:
+
+1. **Force-push the rewritten history to GitHub** — `git push --force-with-lease --mirror` on the surgery clone. This destroys all old refs on GitHub and replaces them with the rewritten ones.
+   - **Policy exception required:** Steve's standing rule is "Never `git push --force` to main/master." This surgery REQUIRES violating that rule — surface it and get explicit confirmation before pushing.
+2. **Mac2 must be re-cloned fresh** from the rewritten GitHub. Its current 20 local-only commits live in the bundle backup; cherry-pick onto the new clone if any are still wanted.
+3. **Kamatera must be re-cloned fresh** from the rewritten GitHub. Stash on Kamatera (`stash@{0}: WIP on bulk-wallcovering-update-system: b1b060a`) must be saved as a patch first (`git stash show -p stash@{0} > /root/Projects/_archive/kamatera-stash-2026-05-19.patch`).
+4. **Mac1 needs a fresh clone** (currently has none) — straightforward, no reconciliation.
+5. **Any pm2 processes referencing `/root/Projects/Designer-Wallcoverings`** on Kamatera must be stopped before re-clone, then `pm2 reload` after. Inventory those processes BEFORE surgery — `pm2 jlist | jq '.[] | select(.pm2_env.pm_cwd | test("Designer-Wallcoverings")) | .name'`.
+
+---
+
+## 8. Pre-surgery checklist (Steve to confirm before running step 1)
+
+- [ ] **Canonical timeline confirmed.** Kamatera + GitHub `Stevemdr/Designer-Wallcoverings` is the source of truth. Mac2's 20-commit orphan timeline is either cherry-picked or abandoned.
+- [ ] **Kamatera bundle backup exists** at `/root/Projects/_archive/dw-kamatera-pre-surgery-2026-05-19.bundle` (currently missing — must be created).
+- [ ] **GitHub mirror backup exists** at a second `Stevemdr/Designer-Wallcoverings-pre-surgery-backup` repo (recommended belt+suspenders).
+- [ ] **No uncommitted work** on Kamatera. Currently dirty (.gitignore + CLAUDE.md + 3 other files modified). Either commit or stash → patch → discard.
+- [ ] **Kamatera stash@{0} exported** to a `.patch` file.
+- [ ] **No open PRs** on GitHub against `Stevemdr/Designer-Wallcoverings` (force-push will invalidate them). Check `gh pr list -R Stevemdr/Designer-Wallcoverings`.
+- [ ] **No in-flight build / scrape / agent run** on Kamatera touching this repo. Audit `pm2 list` for processes with cwd inside `Designer-Wallcoverings`.
+- [ ] **15 ralph branches** reviewed — keep all, or merge/abandon some before surgery? (Fewer refs = faster filter-repo, smaller post-surgery pack.)
+- [ ] **Force-push exception explicitly approved** by Steve.
+- [ ] **Surgery host has disk headroom** — for the 55 GB Kamatera repo, recommend running surgery on a machine with ≥ 80 GB free, OR clear Kamatera disk first.
+
+---
+
+## 9. Rollback procedure
+
+If surgery produces a broken result (missing files we needed, history we needed, etc.):
+
+```bash
+# 1. Stop any work on the rewritten repo.
+# 2. Restore Mac2 timeline from bundle:
+cd /tmp && git clone ~/Projects/_archive/Designer-Wallcoverings-mac2-pre-surgery-2026-05-19.bundle Designer-Wallcoverings-restored
+
+# 3. Restore Kamatera timeline (once that bundle exists):
+ssh root@45.61.58.125 'cd /tmp && git clone /root/Projects/_archive/dw-kamatera-pre-surgery-2026-05-19.bundle Designer-Wallcoverings-restored'
+
+# 4. Restore GitHub:
+# If we made the belt+suspenders backup repo, force-push from it back to the
+# main repo. If we didn't, the bundle on Kamatera is the only source — push
+# its mirror back to GitHub:
+ssh root@45.61.58.125 'cd /tmp/Designer-Wallcoverings-restored && git push --mirror --force-with-lease https://github.com/Stevemdr/Designer-Wallcoverings.git'
+
+# 5. Validate: every clone host re-clones, checks out main, verifies HEAD
+#    matches pre-surgery HEAD on Kamatera (183db25...)
+```
+
+Rollback window: the bundles never expire. Recommend keeping them in `_archive/` for at least 90 days post-surgery.
+
+---
+
+## 10. What this plan does NOT cover (out of scope)
+
+- LFS migration (would shrink working-tree pulls but doesn't reduce existing `.git`).
+- Splitting the repo into multiple smaller repos (e.g. one per `DW-Websites/<site>/`) — separate project.
+- Reconciling Mac2's orphan 20-commit timeline with Kamatera's history — Steve decides cherry-pick vs abandon.
+- Cleaning up the 15 ralph branches — separate decision.
+- Removing secrets from history — current `.gitignore` blocks `.env*`, but historical commits should be scanned via `git secrets` or `trufflehog` BEFORE surgery (and again AFTER) and any hits added to the filter-repo path list.
+
+---
+
+## 11. Recommended sequence summary
+
+1. Steve reviews & approves this plan.
+2. Pre-surgery checklist (§8) — Steve runs.
+3. Backup Kamatera (bundle + GitHub-mirror backup repo).
+4. Run `git filter-repo --analyze` on a fresh mirror clone, review the report.
+5. Adjust `/tmp/dw-paths-to-delete.txt` if §3 missed any big offenders.
+6. Run the actual rewrite on the mirror clone.
+7. Verify post-surgery `.git` size meets §5 target.
+8. Force-push to GitHub (with explicit standing-rule exception).
+9. Fresh-clone on Mac2, Mac1, Kamatera.
+10. Restore Kamatera's stash patch + cherry-pick any wanted Mac2 commits.
+11. Restart any pm2 processes that were stopped in §8.
+
+End of plan.
← a52d295a snapshot — gitify backup 2026-05-19 · back to Designer Wallcoverings · Cole & Son: backfill spec metafields (874 products, 11362 mf b8cb44d6 →