← back to Homesonspec
TK-10809: add ops/OFFLOAD-RUNBOOK.md (SAFE prod offload ordering, deletion hard-blocked until legacy-abs rewrite)
915bd9c2b3f99681b97a6989560baab4a13d9e14 · 2026-08-30 09:42:48 -0700 · steve
Files touched
Diff
commit 915bd9c2b3f99681b97a6989560baab4a13d9e14
Author: steve <steve@designerwallcoverings.com>
Date: Sun Aug 30 09:42:48 2026 -0700
TK-10809: add ops/OFFLOAD-RUNBOOK.md (SAFE prod offload ordering, deletion hard-blocked until legacy-abs rewrite)
---
ops/OFFLOAD-RUNBOOK.md | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 74 insertions(+)
diff --git a/ops/OFFLOAD-RUNBOOK.md b/ops/OFFLOAD-RUNBOOK.md
new file mode 100644
index 00000000..7da47b71
--- /dev/null
+++ b/ops/OFFLOAD-RUNBOOK.md
@@ -0,0 +1,74 @@
+# Snapshot-store Offload Runbook (TK-10809)
+
+The raw-snapshot store is ~155G split across two prod dirs. This runbook offloads it
+to a dedicated volume **without ever deleting evidence**. Every prod step below is
+**Steve-gated**. The tooling here (`ops/offload-snapshots.mjs`) is copy-only + verify;
+it NEVER unlinks, prunes, moves, or rewrites the source.
+
+## Facts to confirm ON THE BOX before you start
+
+- Two prod snapshot dirs:
+ - `var/snapshots` — ~98G
+ - `apps/workers/var/snapshots` — ~57G
+- **Confirm the REAL `SNAPSHOT_DIR` the workers actually read.** A prior note found
+ `/root/var/snapshots` empty — do NOT assume the path. Resolve it from the running
+ worker env (`pm2 env <worker-id> | grep SNAPSHOT_DIR`) or the default in
+ `apps/workers/src/snapshot-path.ts` (repo-root `var/snapshots` unless `SNAPSHOT_DIR`
+ is set). Whatever the workers read is the one you rsync + repoint.
+- Pick `<vol>` = the dedicated destination volume (absolute path, disjoint from SRC).
+
+## Ordering (SAFE — do not reorder; old copies stay until a separate approval)
+
+1. **Initial rsync copy-only.** Bulk-copy while ingestion keeps running. This is not
+ the source of truth for completeness — it just moves most bytes ahead of the freeze.
+ ```
+ node ops/offload-snapshots.mjs --dest <vol> --src <REAL_SNAPSHOT_DIR> --apply
+ ```
+ (No `--full` here — this is a first pass; verification happens after the freeze.)
+
+2. **QUIESCE ingestion.** Pause the workers / import loop so no new snapshots are
+ written mid-copy (`pm2 stop <import workers>` or pause the import-sweep launchd job).
+ Nothing should be writing into SRC from this point until step 8.
+
+3. **Final incremental rsync.** Re-run the copy to pick up everything written since
+ step 1. rsync is incremental (`-a`, copy-only, no `--delete`), so this is fast.
+ ```
+ node ops/offload-snapshots.mjs --dest <vol> --src <REAL_SNAPSHOT_DIR> --apply
+ ```
+
+4. **FULL verify — the gate.**
+ ```
+ node ops/offload-snapshots.mjs --dest <vol> --src <REAL_SNAPSHOT_DIR> --apply --full
+ ```
+ - Require the banner **`VERIFY PASS (FULL)`** and **exit 0**.
+ - **Do NOT proceed on exit 3** (SAMPLE-only — completeness unproven).
+ - **Do NOT proceed on exit 1** (dirty — misses / hash mismatches / invalid rows).
+ - Repeat until exit 0. Exit 0 + FULL is the ONLY proof the DEST is complete.
+
+5. **Rewrite legacy-ABSOLUTE `storagePath` rows (REQUIRED before any deletion).**
+ Legacy rows store an absolute `storagePath`; after the volume moves, those absolute
+ paths still point at the OLD location. Before anything is deleted, either:
+ - **(a) canonical dw write — GATED:** rewrite absolute rows under the old prefix to
+ POSIX-relative paths (resolved against `SNAPSHOT_DIR`), matching how new rows are
+ stored (`apps/workers/src/snapshot-path.ts`), **OR**
+ - **(b) symlink** preserving the old absolute prefix so legacy reads keep resolving.
+
+ This step is **HARD-BLOCKED as a prerequisite for deletion** — no old copy may be
+ removed until every legacy-absolute row has a working path at the new location.
+
+6. **Repoint + reload.** Set `SNAPSHOT_DIR=<vol>` in the worker env and `pm2 reload`
+ the workers so new writes AND reads use the new volume.
+
+7. **Smoke-test reads + ingestion.** Confirm a handful of existing snapshots read back
+ correctly from `<vol>` (e.g. `node ops/verify-snapshots.mjs --sample 500`) and that
+ a fresh ingested snapshot lands on `<vol>`.
+
+8. **Resume.** Un-quiesce the workers / import loop (reverse of step 2).
+
+## Deletion (SEPARATE, later, separately-approved)
+
+- **Old copies are RETAINED** through all of the above.
+- Deleting the old store is a **separate, separately-approved** step and is
+ **HARD-BLOCKED until step 5** (legacy-absolute rows rewritten or symlinked) is done
+ and step 4 last returned **`VERIFY PASS (FULL)` exit 0**.
+- This tooling never deletes evidence. The delete, when approved, is a manual gated op.
← 66e0038c TK-10809: extract classifyForDest pure fn + hermetic verify
·
back to Homesonspec
·
TK-10809: snapshot-verify catches DEST symlink false-green b 73a0e94f →