← back to Dw Canary Watchdog
Canary Meta-Watchdog: prove the DW canaries actually ran (Council #1, DTD-C hybrid)
0fb010fb09948d1de2dcd6c76f70c4408c2ca6d7 · 2026-06-15 22:55:56 -0700 · Steve Abrams
Files touched
A .gitignoreA README.mdA com.steve.dw-canary-watchdog.plistA data/.gitkeepA manifest.jsonA run.shA watchdog.mjs
Diff
commit 0fb010fb09948d1de2dcd6c76f70c4408c2ca6d7
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Mon Jun 15 22:55:56 2026 -0700
Canary Meta-Watchdog: prove the DW canaries actually ran (Council #1, DTD-C hybrid)
---
.gitignore | 7 +++
README.md | 61 ++++++++++++++++++++++++
com.steve.dw-canary-watchdog.plist | 22 +++++++++
data/.gitkeep | 0
manifest.json | 60 ++++++++++++++++++++++++
run.sh | 9 ++++
watchdog.mjs | 96 ++++++++++++++++++++++++++++++++++++++
7 files changed, 255 insertions(+)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..a6e4cc2
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,7 @@
+node_modules/
+.env*
+tmp/
+*.log
+.DS_Store
+data/*.json
+!data/.gitkeep
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..5ef32a6
--- /dev/null
+++ b/README.md
@@ -0,0 +1,61 @@
+# dw-canary-watchdog — the canary that watches the canaries
+
+Officer Council top-5 idea **#1 (2026-06-15)**. Closes the exact blind spot that
+let the `dw_unified` nightly pg_dump die **silently for 12 days** (Jun 3–15 2026):
+the backup script's *own* alert was the only freshness guard, so when that guard
+broke, nobody knew. A scheduled job that stops firing produces **no signal at
+all** — silence reads identically to health. This watchdog turns silence into an
+alert.
+
+## What it does
+Every hour it proves each DW canary actually **ran to completion** recently, and
+alerts (CNCP + George email) if any one has gone dark.
+
+Monitored canaries (`manifest.json` — adding one = a single row):
+
+| canary | cadence | warn | fail |
+|---|---|---|---|
+| dw-uptime-probe | every 15 min | 35 min | 60 min |
+| scraper-healthcheck | daily 02:00 | 26 h | 30 h |
+| dw-scraper-canary | daily 05:00 | 26 h | 30 h |
+| dw-map-auditor | daily 05:30 | 26 h | 30 h |
+| dw-backup-canary | weekly Sun 04:00 | 7.5 d | 8.5 d |
+| dw-leak-scan | weekly Mon 06:00 | 7.5 d | 8.5 d |
+
+## Detection — DTD-C hybrid (unanimous 2026-06-15)
+Per canary, the freshest available signal wins, in this precedence:
+
+1. **explicit stamp** `~/.claude/heartbeats/<label>.stamp` → *authoritative*
+ (for future canaries with no result artifact).
+2. **result artifact** `<skill>/data/latest.json` mtime → *authoritative*.
+ This is only written **after** the canary's real work completes, so it is a
+ true completion proof — and importantly it does **not** depend on exit code
+ (these canaries `exit 1` when they run fine but *find* a problem).
+3. **launchd `/tmp/<label>.log` mtime** → *degraded* confidence. Proves the
+ wrapper started, not that the work finished, so thresholds tighten ×0.8 and a
+ **recent 0-byte log** is flagged WARN ("ran but produced nothing" — the
+ silent-death class itself).
+4. **no trace at all** → `FAIL` (never ran / wiped).
+
+READ-ONLY w.r.t. the canaries: it only `stat`s their files, never writes them.
+
+## Self-coverage (don't be an unmonitored SPOF)
+Each run writes its own `~/.claude/heartbeats/dw-canary-watchdog.stamp` and posts
+a lightweight **"watchdog alive"** beat to CNCP so the monitor itself is
+eyeballable. *Follow-up (not yet wired):* an offsite dead-man's-switch ping
+(healthchecks.io-style) so even a total-Mac2-down is caught externally.
+
+## Run / schedule
+```bash
+bash ~/Projects/dw-canary-watchdog/run.sh # check now (CNCP+George on FAIL)
+WATCHDOG_BEAT=0 bash run.sh # suppress the healthy CNCP beat
+```
+Scheduled hourly via `com.steve.dw-canary-watchdog` (see `com.steve.dw-canary-watchdog.plist`).
+
+## Env knobs
+`CNCP_URL` (default http://localhost:3333) · `GEORGE_AUTH/GEORGE_HOST/GEORGE_PORT`
+· `WATCHDOG_TO` (default steve@designerwallcoverings.com) · `WATCHDOG_BEAT=0`.
+
+## Hard rules
+- READ-ONLY on the canaries. Alerts only. Never edits/disables a canary.
+- Exit 0 = all running; exit 1 = ≥1 FAIL.
diff --git a/com.steve.dw-canary-watchdog.plist b/com.steve.dw-canary-watchdog.plist
new file mode 100644
index 0000000..d2b1784
--- /dev/null
+++ b/com.steve.dw-canary-watchdog.plist
@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+ <key>Label</key>
+ <string>com.steve.dw-canary-watchdog</string>
+ <key>ProgramArguments</key>
+ <array>
+ <string>/bin/bash</string>
+ <string>-lc</string>
+ <string>bash "$HOME/Projects/dw-canary-watchdog/run.sh" >> /tmp/dw-canary-watchdog.log 2>&1</string>
+ </array>
+ <key>StartInterval</key>
+ <integer>3600</integer>
+ <key>RunAtLoad</key>
+ <true/>
+ <key>StandardOutPath</key>
+ <string>/tmp/dw-canary-watchdog.launchd.out.log</string>
+ <key>StandardErrorPath</key>
+ <string>/tmp/dw-canary-watchdog.launchd.err.log</string>
+</dict>
+</plist>
diff --git a/data/.gitkeep b/data/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/manifest.json b/manifest.json
new file mode 100644
index 0000000..a445386
--- /dev/null
+++ b/manifest.json
@@ -0,0 +1,60 @@
+{
+ "_comment": "Canary Meta-Watchdog registry. Adding a canary = one row. Detection precedence per row (DTD-C hybrid, 2026-06-15 unanimous): stamp > artifact > launchd_log > never-ran. interval/warn/fail are SECONDS. warn_after/fail_after are wall-clock age of the freshest completion signal.",
+ "checkSelf": true,
+ "canaries": [
+ {
+ "label": "dw-uptime-probe",
+ "cadence": "every 15min (StartInterval 900)",
+ "stamp": "~/.claude/heartbeats/dw-uptime-probe.stamp",
+ "artifact": "~/.claude/skills/dw-uptime-probe/data/latest.json",
+ "launchd_log": "/tmp/dw-uptime-probe.log",
+ "warn_after": 2100,
+ "fail_after": 3600
+ },
+ {
+ "label": "scraper-healthcheck",
+ "cadence": "daily 02:00",
+ "stamp": "~/.claude/heartbeats/scraper-healthcheck.stamp",
+ "artifact": "/tmp/scraper-healthcheck.log",
+ "launchd_log": "/tmp/scraper-healthcheck.log",
+ "warn_after": 93600,
+ "fail_after": 108000
+ },
+ {
+ "label": "dw-scraper-canary",
+ "cadence": "daily 05:00",
+ "stamp": "~/.claude/heartbeats/dw-scraper-canary.stamp",
+ "artifact": "~/.claude/skills/dw-scraper-canary/data/latest.json",
+ "launchd_log": "/tmp/dw-scraper-canary.log",
+ "warn_after": 93600,
+ "fail_after": 108000
+ },
+ {
+ "label": "dw-map-auditor",
+ "cadence": "daily 05:30",
+ "stamp": "~/.claude/heartbeats/dw-map-auditor.stamp",
+ "artifact": "~/.claude/skills/dw-map-auditor/data/latest.json",
+ "launchd_log": "/tmp/dw-map-auditor.log",
+ "warn_after": 93600,
+ "fail_after": 108000
+ },
+ {
+ "label": "dw-backup-canary",
+ "cadence": "weekly Sun 04:00",
+ "stamp": "~/.claude/heartbeats/dw-backup-canary.stamp",
+ "artifact": "~/.claude/skills/dw-backup-canary/data/latest.json",
+ "launchd_log": "/tmp/dw-backup-canary.log",
+ "warn_after": 648000,
+ "fail_after": 734400
+ },
+ {
+ "label": "dw-leak-scan",
+ "cadence": "weekly Mon 06:00",
+ "stamp": "~/.claude/heartbeats/dw-leak-scan.stamp",
+ "artifact": "~/.claude/skills/dw-leak-scanner/data/latest.json",
+ "launchd_log": "/tmp/dw-leak-scan.log",
+ "warn_after": 648000,
+ "fail_after": 734400
+ }
+ ]
+}
diff --git a/run.sh b/run.sh
new file mode 100644
index 0000000..a8f0356
--- /dev/null
+++ b/run.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+# dw-canary-watchdog run — proves the DW canaries actually ran. READ-ONLY w.r.t. canaries.
+set -uo pipefail
+SKILL="$HOME/Projects/dw-canary-watchdog"
+# Load GEORGE_AUTH from secrets if not already in env (mirrors the other canaries).
+if [ -z "${GEORGE_AUTH:-}" ] && [ -f "$HOME/Projects/secrets-manager/.env" ]; then
+ export GEORGE_AUTH="$(grep -E '^GEORGE_AUTH=' "$HOME/Projects/secrets-manager/.env" | head -1 | cut -d= -f2-)"
+fi
+exec node "$SKILL/watchdog.mjs"
diff --git a/watchdog.mjs b/watchdog.mjs
new file mode 100644
index 0000000..ed3cb04
--- /dev/null
+++ b/watchdog.mjs
@@ -0,0 +1,96 @@
+#!/usr/bin/env node
+// Canary Meta-Watchdog — proves the DW canaries actually RAN recently.
+// DTD-C hybrid (2026-06-15 unanimous): detection precedence per canary =
+// explicit stamp > result artifact mtime (authoritative completion)
+// > launchd /tmp log mtime (DEGRADED, stricter) > never-ran (FAIL).
+// READ-ONLY w.r.t. the canaries: only stats their files. On FAIL: CNCP + George.
+// Self-coverage: writes its own stamp + posts a CNCP "watchdog alive" beat each run.
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+
+const SKILL = path.join(os.homedir(), 'Projects/dw-canary-watchdog');
+const DATA = path.join(SKILL, 'data');
+fs.mkdirSync(DATA, { recursive: true });
+const LOG = path.join(DATA, 'run.log');
+const now = Date.now();
+const ENV = process.env;
+const exp = (p) => p ? p.replace(/^~(?=\/)/, os.homedir()) : p;
+const log = (m) => { const l = `[${new Date(now).toISOString()}] ${m}`; console.log(l); try { fs.appendFileSync(LOG, l + '\n'); } catch {} };
+
+const manifest = JSON.parse(fs.readFileSync(path.join(SKILL, 'manifest.json'), 'utf8'));
+
+function freshestSignal(c) {
+ // returns {source, mtime, confidence, emptyRecent} or null
+ const tryStat = (p) => { try { const s = fs.statSync(exp(p)); return s; } catch { return null; } };
+ // 1. explicit stamp — authoritative
+ if (c.stamp) { const s = tryStat(c.stamp); if (s) return { source: 'stamp', mtime: s.mtimeMs, confidence: 'authoritative', size: s.size }; }
+ // 2. result artifact — authoritative completion proof
+ if (c.artifact) { const s = tryStat(c.artifact); if (s) return { source: 'artifact', mtime: s.mtimeMs, confidence: 'authoritative', size: s.size }; }
+ // 3. launchd log — degraded (proves wrapper started, not that work finished)
+ if (c.launchd_log) { const s = tryStat(c.launchd_log); if (s) return { source: 'launchd_log', mtime: s.mtimeMs, confidence: 'degraded', size: s.size }; }
+ return null;
+}
+
+const results = [];
+for (const c of manifest.canaries) {
+ const sig = freshestSignal(c);
+ if (!sig) { results.push({ label: c.label, verdict: 'FAIL', reason: 'no run trace found (stamp/artifact/log all absent) — never ran or wiped', confidence: 'none', age_h: null, cadence: c.cadence }); continue; }
+ const ageS = (now - sig.mtime) / 1000;
+ // DEGRADED → stricter thresholds (×0.8) per Codex refinement
+ const k = sig.confidence === 'degraded' ? 0.8 : 1.0;
+ const warnAfter = c.warn_after * k, failAfter = c.fail_after * k;
+ // "ran but wrote nothing" false-OK guard: recent BUT empty launchd log
+ const emptySuspect = sig.source === 'launchd_log' && sig.size === 0 && ageS < failAfter;
+ let verdict = 'OK', reason = `fresh via ${sig.source}`;
+ if (emptySuspect) { verdict = 'WARN'; reason = `recent launchd log is 0-byte — ran but produced no output (the silent-death class)`; }
+ if (ageS >= failAfter) { verdict = 'FAIL'; reason = `stale ${(ageS / 3600).toFixed(1)}h (fail≥${(failAfter / 3600).toFixed(1)}h) via ${sig.source}`; }
+ else if (ageS >= warnAfter) { verdict = verdict === 'WARN' ? 'WARN' : 'WARN'; reason = `aging ${(ageS / 3600).toFixed(1)}h (warn≥${(warnAfter / 3600).toFixed(1)}h) via ${sig.source}`; }
+ results.push({ label: c.label, verdict, reason, confidence: sig.confidence, source: sig.source, age_h: +(ageS / 3600).toFixed(2), cadence: c.cadence });
+}
+
+const fail = results.filter(r => r.verdict === 'FAIL');
+const warn = results.filter(r => r.verdict === 'WARN');
+const degraded = results.filter(r => r.confidence === 'degraded').map(r => r.label);
+const summary = { ts: new Date(now).toISOString(), total: results.length, ok: results.filter(r => r.verdict === 'OK').length, warn: warn.length, fail: fail.length, degraded_monitoring: degraded, results };
+fs.writeFileSync(path.join(DATA, 'latest.json'), JSON.stringify(summary, null, 2));
+log(`watchdog: ${summary.total} canaries · ${summary.ok} OK · ${warn.length} WARN · ${fail.length} FAIL${degraded.length ? ` · degraded-monitoring: ${degraded.join(',')}` : ''}`);
+for (const r of [...fail, ...warn]) log(` ${r.verdict} ${r.label} — ${r.reason}`);
+
+// self-heartbeat (so the watchdog isn't an unmonitored SPOF)
+try { fs.writeFileSync(exp('~/.claude/heartbeats/dw-canary-watchdog.stamp'), String(now)); } catch {}
+
+async function postCNCP(note, url) {
+ const base = ENV.CNCP_URL || 'http://localhost:3333';
+ try { const ac = new AbortController(); const t = setTimeout(() => ac.abort(), 8000);
+ const r = await fetch(`${base}/api/parking-lot`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ url, note }), signal: ac.signal });
+ clearTimeout(t); return r.ok; } catch { return false; }
+}
+async function emailGeorge(subject, html) {
+ let auth = ENV.GEORGE_AUTH;
+ if (!auth) { try { const env = fs.readFileSync(path.join(os.homedir(), 'Projects/secrets-manager/.env'), 'utf8'); auth = (env.match(/^GEORGE_AUTH=(.*)$/m) || [])[1]; } catch {} }
+ if (!auth) { log('email skipped (no GEORGE_AUTH)'); return false; }
+ const host = ENV.GEORGE_HOST || '100.107.67.67', port = ENV.GEORGE_PORT || '9850';
+ try { const ac = new AbortController(); const t = setTimeout(() => ac.abort(), 15000);
+ const r = await fetch(`http://${host}:${port}/api/send`, { method: 'POST', headers: { 'Authorization': auth, 'Content-Type': 'application/json; charset=utf-8' },
+ body: JSON.stringify({ to: ENV.WATCHDOG_TO || 'steve@designerwallcoverings.com', subject, body: html }), signal: ac.signal });
+ clearTimeout(t); const j = await r.json().catch(() => ({})); return !!j.success; } catch { return false; }
+}
+
+const beat = `[CANARY WATCHDOG ${new Date(now).toISOString().slice(0, 16).replace('T', ' ')}] alive · ${summary.ok}/${summary.total} OK${fail.length ? ` · ${fail.length} FAIL: ${fail.map(f => f.label).join(', ')}` : ''}${warn.length ? ` · ${warn.length} WARN` : ''}`;
+
+if (fail.length) {
+ const note = `[CANARY WATCHDOG ${new Date(now).toISOString().slice(0, 10)}] ${fail.length} canary(ies) NOT running: ${fail.map(f => `${f.label} (${f.reason})`).join('; ')}. A blind canary = a silent failure waiting to happen.`;
+ await postCNCP(note, 'file://' + SKILL);
+ const html = `<div style="font-family:-apple-system,sans-serif;color:#222"><h3 style="color:#b23b3b">⚠ Canary Watchdog — ${fail.length} canary(ies) not running</h3>`
+ + fail.map(f => `<div style="margin:6px 0"><b>${f.label}</b> — ${f.reason} <span style="color:#888">(${f.cadence})</span></div>`).join('')
+ + (warn.length ? `<hr><div style="color:#a07000">WARN: ${warn.map(w => w.label + ' — ' + w.reason).join('; ')}</div>` : '')
+ + `</div>`;
+ const emailed = await emailGeorge(`⚠ Canary Watchdog — ${fail.length} canary down — ${new Date(now).toISOString().slice(0, 10)}`, html);
+ log(emailed ? 'FAIL alert emailed' : 'FAIL alert email not sent');
+ process.exitCode = 1;
+} else {
+ // healthy → post a lightweight alive-beat so Steve can eyeball the watchdog itself on CNCP
+ if (ENV.WATCHDOG_BEAT !== '0') { const ok = await postCNCP(beat, 'file://' + SKILL); log(ok ? 'CNCP alive-beat posted' : 'CNCP alive-beat failed'); }
+ log('✓ all canaries running' + (warn.length ? ` (${warn.length} aging — logged, not alerted)` : ''));
+}
(oldest)
·
back to Dw Canary Watchdog
·
Add ops-digest: unified canary health scorecard + CNCP card 347a2b5 →