`README.md`

80 lines
# dw-canary-watchdog — the canary that watches the canaries

Officer Council top-5 idea **#1 (2026-06-15)**. Closes the exact blind spot that
let the `dw_unified` nightly pg_dump die **silently for 12 days** (Jun 3–15 2026):
the backup script's *own* alert was the only freshness guard, so when that guard
broke, nobody knew. A scheduled job that stops firing produces **no signal at
all** — silence reads identically to health. This watchdog turns silence into an
alert.

## What it does
Every hour it proves each DW canary actually **ran to completion** recently, and
alerts (CNCP + George email) if any one has gone dark.

Monitored canaries (`manifest.json` — adding one = a single row):

| canary | cadence | warn | fail |
|---|---|---|---|
| dw-uptime-probe | every 15 min | 35 min | 60 min |
| scraper-healthcheck | daily 02:00 | 26 h | 30 h |
| dw-scraper-canary | daily 05:00 | 26 h | 30 h |
| dw-map-auditor | daily 05:30 | 26 h | 30 h |
| dw-backup-canary | weekly Sun 04:00 | 7.5 d | 8.5 d |
| dw-leak-scan | weekly Mon 06:00 | 7.5 d | 8.5 d |

## Detection — DTD-C hybrid (unanimous 2026-06-15)
Per canary, the freshest available signal wins, in this precedence:

1. **explicit stamp** `~/.claude/heartbeats/<label>.stamp` → *authoritative*
   (for future canaries with no result artifact).
2. **result artifact** `<skill>/data/latest.json` mtime → *authoritative*.
   This is only written **after** the canary's real work completes, so it is a
   true completion proof — and importantly it does **not** depend on exit code
   (these canaries `exit 1` when they run fine but *find* a problem).
3. **launchd `/tmp/<label>.log` mtime** → *degraded* confidence. Proves the
   wrapper started, not that the work finished, so thresholds tighten ×0.8 and a
   **recent 0-byte log** is flagged WARN ("ran but produced nothing" — the
   silent-death class itself).
4. **no trace at all** → `FAIL` (never ran / wiped).

READ-ONLY w.r.t. the canaries: it only `stat`s their files, never writes them.

## Self-coverage (don't be an unmonitored SPOF)
Each run writes its own `~/.claude/heartbeats/dw-canary-watchdog.stamp` and posts
a lightweight **"watchdog alive"** beat to CNCP so the monitor itself is
eyeballable. *Follow-up (not yet wired):* an offsite dead-man's-switch ping
(healthchecks.io-style) so even a total-Mac2-down is caught externally.

## Run / schedule
```bash
bash ~/Projects/dw-canary-watchdog/run.sh     # check now (CNCP+George on FAIL)
WATCHDOG_BEAT=0 bash run.sh                    # suppress the healthy CNCP beat
```
Scheduled hourly via `com.steve.dw-canary-watchdog` (see `com.steve.dw-canary-watchdog.plist`).

## Env knobs
`CNCP_URL` (default http://localhost:3333) · `GEORGE_AUTH/GEORGE_HOST/GEORGE_PORT`
· `WATCHDOG_TO` (default steve@designerwallcoverings.com) · `WATCHDOG_BEAT=0`.

## Sibling: `ops-digest.mjs` — unified ops health scorecard (Council #1 fresh, 2026-06-15)
The watchdog proves canaries *ran*; the digest reports what they *found* — one
color-coded scorecard across the whole fleet (uptime / scraper / map-auditor / backup /
leak / scraper-healthcheck + the watchdog itself), 🟢/🟡/🔴 per surface.
```bash
node ~/Projects/dw-canary-watchdog/ops-digest.mjs            # write scorecard; CNCP post only if non-green
node ~/Projects/dw-canary-watchdog/ops-digest.mjs --daily    # force the CNCP card even when all-green
node ~/Projects/dw-canary-watchdog/ops-digest.mjs --email    # also email the digest via George
```
- Always writes `data/digest-latest.json` + `data/digest.md` (the scorecard).
- **Posts to CNCP parking-lot only when the health STATE CHANGES** (a canary flips) or
  `--daily` — persistent known FAILs do NOT re-post every tick (state signature cached in
  `data/.last-post-sig`). Suggested schedule: hourly bare (fires only on a real flip), plus
  one `--daily --email` at 8am for the morning summary.
- Per-canary adapters read each canary's `data/latest.json` (differing shapes).
- *Follow-up (council first-step):* a dedicated CNCP `/api/ops-status` **panel** that shows
  ONE live updating scorecard instead of parking-lot cards — touches cncp-starter, deferred.

## Hard rules
- READ-ONLY on the canaries. Alerts only. Never edits/disables a canary.
- Exit 0 = all running; exit 1 = ≥1 FAIL.