`README.md`

109 lines
# NAS setup — DW fleet backup + archive + offload

Wires a new NAS (2TB + 4TB drives, JBOD = two separate shares) into the existing fleet:
backup hardening for `dw_unified`, cold archive + Time Machine, and a monitored offload
target.

## 3-2-1 backup topology for `dw_unified` (DTD verdict, 2026-06-26)

The on-prem `dw_unified` backup is **3-2-1**:

| Copy | Where | Status |
|------|-------|--------|
| Cloud | Kamatera `/root/backups/db/dw_unified_*.dump` (nightly 03:00) | exists |
| On-prem #1 | **Henry** local drive `/Volumes/Henry/dw-backups/dw_unified` | **LIVE today** |
| On-prem #2 | **NAS 2TB share** `/Volumes/DW-Backups/dw_unified` | auto-activates on mount |

`scripts/pull-dw-dump.sh` is **dual-destination**: one read-only pull from Kamatera fans
out to BOTH Henry and the NAS. Each leg is **independent** — if one volume is
unmounted/unwritable it's skipped with a logged WARNING but the other still completes, and
the job only FAILs/alerts if *neither* destination got a verified copy. So the Henry leg
runs today; the NAS leg starts the moment `/Volumes/DW-Backups` is mounted, no code change.

> **Per-copy freshness monitoring is part of the verdict** — a second on-prem copy that
> isn't itself monitored fails silently (the exact Jun 3–15 2026 failure mode). Both
> panelists flagged this. See `patches/disk-space-canary-add-nas.md` (space) and the
> proposed `dw-backup-canary` freshness diff drafted to the yolo approval queue (gated).

The NAS pieces below (cold archive, Time Machine, the 4TB share) are **not live yet** —
activate in order once the NAS is racked.

## Layout decided
- **4TB share → `DW-Archive`** — cold assets: design masters / 150-DPI TIFs, ComfyUI &
  room-render output archive, plus Time Machine targets for the Macs.
- **2TB share → `DW-Backups`** — critical backups: nightly `dw_unified` dump mirror +
  leftover space as cold offload (augments/replaces Henry).
- **Do NOT RAID** the two drives — mismatched sizes mean RAID1 mirrors only 2TB and
  wastes the other 2TB. Two independent shares (JBOD/Basic volumes) is correct.
- NAS = network storage → **backup/archive only, not hot generation work.** Ollama models
  & active SDXL/ComfyUI scratch stay on local NVMe (see the Ultra 1TB note below).

## Activation order (when the NAS arrives)

### 1. On the NAS (web admin — Synology DSM / QNAP / TrueNAS / etc.)
1. Create two volumes/shares: `DW-Archive` (4TB) and `DW-Backups` (2TB).
2. Enable **SMB** (macOS connects over SMB by default; AFP is deprecated).
3. On `DW-Archive`, enable it as a **Time Machine** target (DSM: Control Panel → File
   Services → Advanced → enable Time Machine for the shared folder; set a quota).
4. Create a dedicated NAS user (e.g. `fleet`) with read/write on both shares; save the
   credential via the `secrets` skill so it's in the registry, not typed ad-hoc.

### 2. On each Mac (mount the shares)
```sh
open "smb://<nas-host>/DW-Archive"     # Finder → Connect As → save to Keychain
open "smb://<nas-host>/DW-Backups"
```
They appear at `/Volumes/DW-Archive` and `/Volumes/DW-Backups`. To auto-remount on login:
System Settings → General → Login Items → add both mounted volumes.

### 3. Point Time Machine at the archive share
System Settings → General → Time Machine → Add Backup Disk → pick `DW-Archive`.
Do this on Mac2 (and Mac1 / the Ultra later).

### 4. Activate the dw_unified dump mirror (the high-value piece)
The mirror is **already dual-destination and validated** — the Henry leg ran live
2026-06-26 (first on-prem second copy created). The NAS leg auto-activates on mount;
no script change is needed. Remaining steps:
```sh
# a. confirm the NAS path in the plist if it differs from /Volumes/DW-Backups
#    (edit launchd/com.steve.nas-dwdump-mirror.plist → NAS_BACKUP_DIR / HENRY_BACKUP_DIR)

# b. run the pull once by hand (read-only against Kamatera; Henry leg writes locally):
bash scripts/pull-dw-dump.sh
cat data/latest.json    # expect verdict:"PASS" (or "PARTIAL" while the NAS is still unmounted)

# c. install the nightly 03:45 job (after Kamatera's 03:00 dump) — GATED, Steve installs:
#    a new recurring autonomous scheduler is Steve's call (drafted to the yolo approval queue).
cp launchd/com.steve.nas-dwdump-mirror.plist ~/Library/LaunchAgents/
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.steve.nas-dwdump-mirror.plist
launchctl list | grep nas-dwdump-mirror     # confirm registered
```

### 5. Wire the NAS into the disk canary
Apply `patches/disk-space-canary-add-nas.md` — two `probe_unit` lines so both shares get
WARN<12% / CRITICAL<7% monitoring like every other unit.

### 6. (Optional) repoint restore-rehearsal at the local mirror
`dw-restore-rehearsal` currently `scp`s the dump fresh from Kamatera each run. Once the NAS
mirror exists you can set `REHEARSE_GLOB=/Volumes/DW-Backups/dw_unified/dw_unified_*.dump`
and drop the remote `scp` — faster drills against the on-prem copy. (Leave as-is if you'd
rather keep rehearsing the *cloud* copy end-to-end.)

## The Ultra's 1TB — separate problem, watch it
The NAS does **not** relieve the Ultra's local disk. 1TB is tight for an AI node: a 70B q4
model ≈ 40GB, plus vision models + SDXL checkpoints (~6–7GB each). Keep the 1TB lean for
*hot* resident models + active scratch; auto-archive finished outputs to `DW-Archive`. If
the local library outgrows 1TB, add a **Thunderbolt external SSD on the Ultra** (local NVMe
speed) — not the NAS.

## Pieces in this repo
- `scripts/pull-dw-dump.sh` — nightly read-only pull of the newest Kamatera dump →
  **BOTH Henry and the NAS 2TB share** (3-2-1), each destination independent, with a
  per-destination size-floor + `pg_restore --list` integrity check; skips any unmounted/
  unwritable destination with a logged WARNING and FAILs only if neither got a verified
  copy; retention OFF by default (never autonomous deletion). rsync flags are probed at
  runtime so it works on macOS openrsync and Homebrew/Linux rsync 3.x.
- `launchd/com.steve.nas-dwdump-mirror.plist` — the 03:45 scheduler (staged, not loaded;
  install is gated — Steve's call). Env block now sets both HENRY_BACKUP_DIR + NAS_BACKUP_DIR.
- `patches/disk-space-canary-add-nas.md` — ready-to-apply canary edit (space monitoring).