← back to Nas Setup
NAS setup package: dw_unified dump mirror + canary patch + runbook
eb6346c12bd35ac9e6f6f113be9f9d3bb9c8233b · 2026-06-26 11:49:33 -0700 · Steve
Files touched
A .gitignoreA README.mdA launchd/com.steve.nas-dwdump-mirror.plistA patches/disk-space-canary-add-nas.mdA scripts/pull-dw-dump.sh
Diff
commit eb6346c12bd35ac9e6f6f113be9f9d3bb9c8233b
Author: Steve <steve@designerwallcoverings.com>
Date: Fri Jun 26 11:49:33 2026 -0700
NAS setup package: dw_unified dump mirror + canary patch + runbook
---
.gitignore | 9 +++
README.md | 76 +++++++++++++++++++++++++
launchd/com.steve.nas-dwdump-mirror.plist | 37 ++++++++++++
patches/disk-space-canary-add-nas.md | 39 +++++++++++++
scripts/pull-dw-dump.sh | 95 +++++++++++++++++++++++++++++++
5 files changed, 256 insertions(+)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..8955433
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,9 @@
+node_modules/
+.env*
+tmp/
+*.log
+.DS_Store
+dist/
+build/
+.next/
+data/
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..a5fb703
--- /dev/null
+++ b/README.md
@@ -0,0 +1,76 @@
+# NAS setup — DW fleet backup + archive + offload
+
+Wires a new NAS (2TB + 4TB drives, JBOD = two separate shares) into the existing fleet:
+backup hardening for `dw_unified`, cold archive + Time Machine, and a monitored offload
+target. **Nothing here is live yet** — activate in order once the NAS is racked.
+
+## Layout decided
+- **4TB share → `DW-Archive`** — cold assets: design masters / 150-DPI TIFs, ComfyUI &
+ room-render output archive, plus Time Machine targets for the Macs.
+- **2TB share → `DW-Backups`** — critical backups: nightly `dw_unified` dump mirror +
+ leftover space as cold offload (augments/replaces Henry).
+- **Do NOT RAID** the two drives — mismatched sizes mean RAID1 mirrors only 2TB and
+ wastes the other 2TB. Two independent shares (JBOD/Basic volumes) is correct.
+- NAS = network storage → **backup/archive only, not hot generation work.** Ollama models
+ & active SDXL/ComfyUI scratch stay on local NVMe (see the Ultra 1TB note below).
+
+## Activation order (when the NAS arrives)
+
+### 1. On the NAS (web admin — Synology DSM / QNAP / TrueNAS / etc.)
+1. Create two volumes/shares: `DW-Archive` (4TB) and `DW-Backups` (2TB).
+2. Enable **SMB** (macOS connects over SMB by default; AFP is deprecated).
+3. On `DW-Archive`, enable it as a **Time Machine** target (DSM: Control Panel → File
+ Services → Advanced → enable Time Machine for the shared folder; set a quota).
+4. Create a dedicated NAS user (e.g. `fleet`) with read/write on both shares; save the
+ credential via the `secrets` skill so it's in the registry, not typed ad-hoc.
+
+### 2. On each Mac (mount the shares)
+```sh
+open "smb://<nas-host>/DW-Archive" # Finder → Connect As → save to Keychain
+open "smb://<nas-host>/DW-Backups"
+```
+They appear at `/Volumes/DW-Archive` and `/Volumes/DW-Backups`. To auto-remount on login:
+System Settings → General → Login Items → add both mounted volumes.
+
+### 3. Point Time Machine at the archive share
+System Settings → General → Time Machine → Add Backup Disk → pick `DW-Archive`.
+Do this on Mac2 (and Mac1 / the Ultra later).
+
+### 4. Activate the dw_unified dump mirror (the high-value piece)
+```sh
+# a. set the real share path in the plist if it differs from /Volumes/DW-Backups
+# (edit launchd/com.steve.nas-dwdump-mirror.plist → NAS_BACKUP_DIR)
+
+# b. dry-run the pull once by hand (read-only against Kamatera):
+NAS_BACKUP_DIR=/Volumes/DW-Backups/dw_unified bash scripts/pull-dw-dump.sh
+cat data/latest.json # expect verdict:"PASS"
+
+# c. install the nightly 03:45 job (after Kamatera's 03:00 dump):
+cp launchd/com.steve.nas-dwdump-mirror.plist ~/Library/LaunchAgents/
+launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.steve.nas-dwdump-mirror.plist
+launchctl list | grep nas-dwdump-mirror # confirm registered
+```
+
+### 5. Wire the NAS into the disk canary
+Apply `patches/disk-space-canary-add-nas.md` — two `probe_unit` lines so both shares get
+WARN<12% / CRITICAL<7% monitoring like every other unit.
+
+### 6. (Optional) repoint restore-rehearsal at the local mirror
+`dw-restore-rehearsal` currently `scp`s the dump fresh from Kamatera each run. Once the NAS
+mirror exists you can set `REHEARSE_GLOB=/Volumes/DW-Backups/dw_unified/dw_unified_*.dump`
+and drop the remote `scp` — faster drills against the on-prem copy. (Leave as-is if you'd
+rather keep rehearsing the *cloud* copy end-to-end.)
+
+## The Ultra's 1TB — separate problem, watch it
+The NAS does **not** relieve the Ultra's local disk. 1TB is tight for an AI node: a 70B q4
+model ≈ 40GB, plus vision models + SDXL checkpoints (~6–7GB each). Keep the 1TB lean for
+*hot* resident models + active scratch; auto-archive finished outputs to `DW-Archive`. If
+the local library outgrows 1TB, add a **Thunderbolt external SSD on the Ultra** (local NVMe
+speed) — not the NAS.
+
+## Pieces in this repo
+- `scripts/pull-dw-dump.sh` — nightly read-only pull of newest Kamatera dump → NAS, with
+ size-floor + `pg_restore --list` integrity check; no-ops if the share isn't mounted;
+ retention OFF by default (never autonomous deletion).
+- `launchd/com.steve.nas-dwdump-mirror.plist` — the 03:45 scheduler (staged, not loaded).
+- `patches/disk-space-canary-add-nas.md` — ready-to-apply canary edit.
diff --git a/launchd/com.steve.nas-dwdump-mirror.plist b/launchd/com.steve.nas-dwdump-mirror.plist
new file mode 100644
index 0000000..61c39a8
--- /dev/null
+++ b/launchd/com.steve.nas-dwdump-mirror.plist
@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<!--
+ com.steve.nas-dwdump-mirror — pull the nightly Kamatera dw_unified dump to the NAS.
+ Runs 03:45 (after the 03:00 Kamatera db-backup.sh has finished writing).
+ DO NOT load until the NAS share is mounted; activation steps are in README.md.
+ Adjust NAS_BACKUP_DIR below to your actual 2TB share path before loading.
+-->
+<plist version="1.0">
+<dict>
+ <key>Label</key>
+ <string>com.steve.nas-dwdump-mirror</string>
+ <key>ProgramArguments</key>
+ <array>
+ <string>/bin/bash</string>
+ <string>/Users/stevestudio2/Projects/nas-setup/scripts/pull-dw-dump.sh</string>
+ </array>
+ <key>EnvironmentVariables</key>
+ <dict>
+ <key>NAS_BACKUP_DIR</key>
+ <string>/Volumes/DW-Backups/dw_unified</string>
+ <key>PATH</key>
+ <string>/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
+ </dict>
+ <key>StartCalendarInterval</key>
+ <dict>
+ <key>Hour</key><integer>3</integer>
+ <key>Minute</key><integer>45</integer>
+ </dict>
+ <key>RunAtLoad</key>
+ <false/>
+ <key>StandardOutPath</key>
+ <string>/Users/stevestudio2/Projects/nas-setup/data/launchd.out.log</string>
+ <key>StandardErrorPath</key>
+ <string>/Users/stevestudio2/Projects/nas-setup/data/launchd.err.log</string>
+</dict>
+</plist>
diff --git a/patches/disk-space-canary-add-nas.md b/patches/disk-space-canary-add-nas.md
new file mode 100644
index 0000000..7f6d516
--- /dev/null
+++ b/patches/disk-space-canary-add-nas.md
@@ -0,0 +1,39 @@
+# Patch: add the NAS to disk-space-canary
+
+Apply this **after** the NAS is mounted (so it doesn't log `UNKNOWN` every run).
+
+File: `~/.claude/skills/disk-space-canary/run.sh`
+
+Find the block that probes each unit:
+
+```bash
+MAC2=$(probe_unit "Mac2" df -k /System/Volumes/Data)
+MAC1=$(probe_unit "Mac1" ssh -o ConnectTimeout=8 mac1 df -k /System/Volumes/Data)
+KAM=$( probe_unit "Kamatera" ssh -o ConnectTimeout=8 my-server df -k /)
+HEN=$( probe_unit "Henry" df -k /Volumes/Henry)
+```
+
+Add two NAS lines (one per share — point at the actual mount names):
+
+```bash
+NAS4=$(probe_unit "NAS-Archive" df -k /Volumes/DW-Archive) # 4TB share
+NAS2=$(probe_unit "NAS-Backups" df -k /Volumes/DW-Backups) # 2TB share
+```
+
+Then add them to the RESULTS aggregation a few lines down. Change:
+
+```bash
+RESULTS=$(printf '%s\n%s\n%s\n%s\n' "$MAC2" "$MAC1" "$KAM" "$HEN" | jq -s '.')
+```
+
+to:
+
+```bash
+RESULTS=$(printf '%s\n%s\n%s\n%s\n%s\n%s\n' "$MAC2" "$MAC1" "$KAM" "$HEN" "$NAS4" "$NAS2" | jq -s '.')
+```
+
+That's it — the verdict/transition-alert logic already loops over `RESULTS`, so the NAS
+shares get WARN<12% / CRITICAL<7% treatment automatically with no other changes.
+
+> The canary returns `UNKNOWN` (no alert) for any unreachable mount, so a brief NAS
+> reboot won't spam you — only a worsening *transition* alerts.
diff --git a/scripts/pull-dw-dump.sh b/scripts/pull-dw-dump.sh
new file mode 100755
index 0000000..e8e3f3f
--- /dev/null
+++ b/scripts/pull-dw-dump.sh
@@ -0,0 +1,95 @@
+#!/bin/bash
+# pull-dw-dump.sh — pull the newest Kamatera dw_unified pg_dump down to the NAS 2TB
+# backup share, giving you a SECOND, on-prem copy of the DB. This is the fix for the
+# 12-day silent backup death (Jun 3–15 2026): one cloud copy is a single point of failure.
+#
+# Safe-by-design:
+# * READ-ONLY against Kamatera (ssh + rsync pull only — never writes to the server).
+# * Exits cleanly (code 0, logged) if the NAS share isn't mounted — safe to schedule
+# before the NAS arrives; it just no-ops until /Volumes/<share> exists.
+# * Retention defaults to KEEP-ALL (RETENTION_KEEP=0). Pruning old copies is OFF unless
+# you opt in, honoring Steve's standing rule: NEVER autonomous deletion. When enabled
+# it logs every file it removes.
+# * Verifies the pulled dump with `pg_restore --list` (structure-valid) + a size floor.
+#
+# Env knobs (all optional):
+# NAS_BACKUP_DIR target dir on the 2TB share (default /Volumes/DW-Backups/dw_unified)
+# PGDUMP_HOST ssh target (default root@45.61.58.125)
+# PGDUMP_GLOB remote dump glob (default /root/backups/db/dw_unified_*.dump)
+# PGDUMP_FLOOR_MB reject dumps smaller than this (default 100)
+# RETENTION_KEEP keep N newest local copies, 0 = keep all (default 0)
+# PULL_TO alert email (default steve@designerwallcoverings.com)
+set -uo pipefail
+
+NAS_BACKUP_DIR="${NAS_BACKUP_DIR:-/Volumes/DW-Backups/dw_unified}"
+HOST="${PGDUMP_HOST:-root@45.61.58.125}"
+GLOB="${PGDUMP_GLOB:-/root/backups/db/dw_unified_*.dump}"
+FLOOR_MB="${PGDUMP_FLOOR_MB:-100}"
+RETENTION_KEEP="${RETENTION_KEEP:-0}"
+PGBIN="/opt/homebrew/opt/postgresql@14/bin"; export PATH="$PGBIN:$PATH"
+
+SELF_DIR="$(cd "$(dirname "$0")" && pwd)"
+LOG="$SELF_DIR/../data/pull.log"; mkdir -p "$(dirname "$LOG")"
+LATEST="$SELF_DIR/../data/latest.json"
+FLOOR_BYTES=$(( FLOOR_MB * 1024 * 1024 ))
+log(){ echo "[$(date -Iseconds)] $1" | tee -a "$LOG"; }
+
+alert(){ # CNCP card + George email on FAIL. args: <reason>
+ local reason="$1"; local CNCP="${CNCP_URL:-http://localhost:3333}"
+ local note="[NAS DUMP MIRROR $(date +%F)] dw_unified pull-down FAILED: ${reason}. The on-prem second copy is the safety net for the cloud pg_dump — investigate."
+ curl -sS --max-time 10 "$CNCP/api/parking-lot" -H 'Content-Type: application/json' \
+ -d "$(jq -n --arg u "nas://dw_unified-mirror" --arg note "$note" '{url:$u,note:$note}')" >/dev/null 2>&1 || true
+ if [ -f "$HOME/.claude/skills/_shared/george-send.sh" ]; then
+ . "$HOME/.claude/skills/_shared/george-send.sh"
+ local body="<div style=\"font-family:-apple-system,sans-serif;color:#222\"><h3 style=\"color:#b23b3b\">⚠ NAS dw_unified mirror FAILED</h3><div>${reason}</div><div style=\"color:#666;font-size:13px\">This job pulls the nightly Kamatera dump to the NAS 2TB share so there's a local restore source. Without it you're back to one cloud copy.</div></div>"
+ george_send info "${PULL_TO:-steve@designerwallcoverings.com}" "⚠ NAS dw_unified mirror FAIL — $(date +%F)" "$body" >/dev/null 2>&1 || true
+ fi
+}
+
+# ── 0. NAS share mounted? If not, no-op cleanly (safe to schedule pre-arrival) ──
+NAS_ROOT="$(echo "$NAS_BACKUP_DIR" | awk -F/ '{print "/"$2"/"$3}')" # e.g. /Volumes/DW-Backups
+if ! mount | grep -q " on ${NAS_ROOT} "; then
+ log "NAS share not mounted ($NAS_ROOT) — skipping (no-op until the NAS is online)"
+ exit 0
+fi
+mkdir -p "$NAS_BACKUP_DIR" || { log "cannot create $NAS_BACKUP_DIR on the share"; alert "target dir not writable: $NAS_BACKUP_DIR"; exit 1; }
+
+# ── 1. newest remote dump (read-only) ──
+REMOTE=$(ssh -o ConnectTimeout=15 -o BatchMode=yes "$HOST" "ls -t $GLOB 2>/dev/null | head -1")
+if [ -z "$REMOTE" ]; then log "FAIL: no remote dump found ($GLOB)"; alert "no remote dump matching $GLOB"; exit 1; fi
+BASE="$(basename "$REMOTE")"
+DEST="$NAS_BACKUP_DIR/$BASE"
+log "newest remote dump: $REMOTE"
+
+# ── 2. rsync it down (resumable, skips if already complete & same size) ──
+if rsync -a --partial --append-verify -e "ssh -o ConnectTimeout=15 -o BatchMode=yes" "$HOST:$REMOTE" "$DEST" 2>>"$LOG"; then
+ log "rsync ok → $DEST"
+else
+ log "FAIL: rsync of $BASE failed"; alert "rsync failed for $BASE"; exit 1
+fi
+
+# ── 3. integrity: size floor + pg_restore --list structure check ──
+SIZE=$(stat -f %z "$DEST" 2>/dev/null || echo 0); SIZE_MB=$(( SIZE / 1024 / 1024 ))
+if [ "$SIZE" -lt "$FLOOR_BYTES" ]; then
+ log "FAIL: pulled dump ${SIZE_MB}MB < floor ${FLOOR_MB}MB (truncated?)"; alert "pulled dump ${SIZE_MB}MB < ${FLOOR_MB}MB"; exit 1
+fi
+TOC=$(pg_restore --list "$DEST" 2>>"$LOG" | grep -c ';' || echo 0)
+if [ "$TOC" -lt 100 ]; then
+ log "FAIL: pg_restore --list shows only $TOC entries — dump not structurally valid"; alert "pg_restore --list invalid ($TOC entries)"; exit 1
+fi
+log "PASS: $BASE — ${SIZE_MB}MB, $TOC TOC entries"
+
+# ── 4. retention (OFF by default; logs every deletion when enabled) ──
+if [ "$RETENTION_KEEP" -gt 0 ]; then
+ mapfile -t OLD < <(ls -t "$NAS_BACKUP_DIR"/dw_unified_*.dump 2>/dev/null | tail -n +"$((RETENTION_KEEP+1))")
+ for f in "${OLD[@]}"; do log "retention: removing $(basename "$f")"; rm -f "$f"; done
+ [ "${#OLD[@]}" -gt 0 ] && log "retention: pruned ${#OLD[@]} old copy(ies), kept newest $RETENTION_KEEP"
+else
+ COUNT=$(ls "$NAS_BACKUP_DIR"/dw_unified_*.dump 2>/dev/null | wc -l | tr -d ' ')
+ log "retention OFF (keep-all) — $COUNT local copies on the share"
+fi
+
+jq -n --arg base "$BASE" --argjson size_mb "$SIZE_MB" --argjson toc "$TOC" --arg dir "$NAS_BACKUP_DIR" \
+ '{pulled_at:(now|todate),dump:$base,size_mb:$size_mb,toc_entries:$toc,dir:$dir,verdict:"PASS"}' > "$LATEST"
+log "✓ mirror complete"
+exit 0
(oldest)
·
back to Nas Setup
·
3-2-1: dual-destination dw_unified pull (Henry + NAS), runti ace1e09 →