ops: age-based retention prune for snapshot page-cache (7d, live-crawl guard, dry-run default) — TK-10236

eb45f0f3fff8d1fcaf6ab48a51d21ce19c159fbb · 2026-08-05 07:49:25 -0700 · Steve Abrams

Files touched

A ops/prune-snapshot-cache.sh

Diff

commit eb45f0f3fff8d1fcaf6ab48a51d21ce19c159fbb
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Wed Aug 5 07:49:25 2026 -0700

    ops: age-based retention prune for snapshot page-cache (7d, live-crawl guard, dry-run default) — TK-10236
---
 ops/prune-snapshot-cache.sh | 52 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

diff --git a/ops/prune-snapshot-cache.sh b/ops/prune-snapshot-cache.sh
new file mode 100755
index 00000000..d3ce2b1a
--- /dev/null
+++ b/ops/prune-snapshot-cache.sh
@@ -0,0 +1,52 @@
+#!/bin/bash
+# prune-snapshot-cache.sh — age-based retention for the homesonspec content-addressed page cache.
+#
+# var/snapshots/<builder>-site/ holds SHA-256-named .html files (one per fetched page). The cache is
+# read back by backfill-*-from-cache scripts for "$0 cache re-extract" — so a pruned page can only be
+# re-mined via a fresh crawl. This keeps files <= RETAIN_DAYS old and prunes older ones, SKIPPING any
+# builder written to in the last LIVE_MIN minutes (live-crawl guard). Born from the 2026-08-05 disk-full
+# outage (TK-10236): the cache grew unbounded to ~98 GB and crashed prod Postgres with ENOSPC.
+#
+# DRY-RUN by default (lists what it WOULD prune). Pass --apply to actually delete; --apply writes a
+# manifest of every deleted path first (audit trail; deletions are re-crawlable, not restorable-from-map).
+#
+#   RETAIN_DAYS (default 7)  LIVE_MIN (default 15)  SNAP_DIR (default prod path)
+#   ./prune-snapshot-cache.sh            # dry-run
+#   ./prune-snapshot-cache.sh --apply    # prune >RETAIN_DAYS, skipping active-crawl builders
+set -uo pipefail
+SNAP="${SNAP_DIR:-/root/Projects/homesonspec/var/snapshots}"
+RETAIN_DAYS="${RETAIN_DAYS:-7}"
+LIVE_MIN="${LIVE_MIN:-15}"
+APPLY=0; [ "${1:-}" = "--apply" ] && APPLY=1
+gb(){ awk -v b="${1:-0}" 'BEGIN{printf "%.2f", b/1073741824}'; }
+
+[ -d "$SNAP" ] || { echo "ERROR: snapshot dir not found: $SNAP"; exit 1; }
+MANIFEST="$SNAP/../prune-manifest-$(date +%Y%m%d-%H%M%S).log"
+now_mode=$([ $APPLY -eq 1 ] && echo PRUNE || echo DRY-RUN)
+echo "== homesonspec snapshot-cache retention ($now_mode) =="
+echo "   dir=$SNAP  retain<=${RETAIN_DAYS}d  live-skip=${LIVE_MIN}m"
+echo ""
+printf "%-30s %10s %10s  %s\n" "builder" "files" "size" "action"
+total_files=0; total_bytes=0; skipped=""
+for d in "$SNAP"/*/; do
+  [ -d "$d" ] || continue
+  b=$(basename "$d")
+  # live-crawl guard — any write in the last LIVE_MIN minutes => skip the whole builder
+  if [ -n "$(find "$d" -type f -mmin -"$LIVE_MIN" 2>/dev/null | head -1)" ]; then
+    skipped="$skipped $b"; continue
+  fi
+  cnt=$(find "$d" -type f -mtime +"$RETAIN_DAYS" 2>/dev/null | wc -l | tr -d ' ')
+  [ "${cnt:-0}" -eq 0 ] && continue
+  bytes=$(find "$d" -type f -mtime +"$RETAIN_DAYS" -printf '%s\n' 2>/dev/null | awk '{s+=$1} END{print s+0}')
+  printf "%-30s %10d %9sG  %s\n" "$b" "$cnt" "$(gb "$bytes")" "$([ $APPLY -eq 1 ] && echo DELETE || echo would-prune)"
+  total_files=$((total_files+cnt)); total_bytes=$((total_bytes+bytes))
+  if [ $APPLY -eq 1 ]; then
+    find "$d" -type f -mtime +"$RETAIN_DAYS" -printf '%p\n' 2>/dev/null >> "$MANIFEST"
+    find "$d" -type f -mtime +"$RETAIN_DAYS" -delete 2>/dev/null
+  fi
+done
+echo ""
+printf "%s TOTAL: %d files, %sG\n" "$now_mode" "$total_files" "$(gb "$total_bytes")"
+[ -n "$skipped" ] && echo "skipped (active crawl, <${LIVE_MIN}m writes):$skipped"
+[ $APPLY -eq 1 ] && echo "deleted-path manifest: $MANIFEST"
+[ $APPLY -eq 0 ] && echo "(dry-run — nothing deleted. Re-run with --apply to prune.)"

← 059c2688 fix(workers): health classifier — DEGRADED on error RATE (>2 · back to Homesonspec · ops: minute-precision retention (find -mtime +N truncates to 5c1b5c47 →