← back to Dw Launchd Canary
dw-launchd-canary: hourly read-only launchd fleet health canary (Officer Council #1+#2)
8ab3b3b2c4c342595a2bd8f610b437f359c8b782 · 2026-06-17 14:14:45 -0700 · Steve Abrams
Catches the two silent-death blind spots above the meta-watchdog: jobs that exit
nonzero every run, and active plists silently UNLOADED. Honest by design — renamed
.DISABLED/.done/.RETIRED plists excluded, health-check exit 1 = OK, signal-killed
daemons = KILLED (not failure), expected_unloaded (drain-all-queues) ignored,
debounced 2 runs, one-time baseline so the pre-existing backlog surfaces once for
the #5 batcher without paging daily. Alerts CNCP + George (https MagicDNS host)
on NEW regressions only; writes data/latest.json for the meta-watchdog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Files touched
A .gitignoreA README.mdA canary.mjsA com.steve.dw-launchd-canary.plistA data/baseline.jsonA manifest.jsonA run.sh
Diff
commit 8ab3b3b2c4c342595a2bd8f610b437f359c8b782
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Wed Jun 17 14:14:45 2026 -0700
dw-launchd-canary: hourly read-only launchd fleet health canary (Officer Council #1+#2)
Catches the two silent-death blind spots above the meta-watchdog: jobs that exit
nonzero every run, and active plists silently UNLOADED. Honest by design — renamed
.DISABLED/.done/.RETIRED plists excluded, health-check exit 1 = OK, signal-killed
daemons = KILLED (not failure), expected_unloaded (drain-all-queues) ignored,
debounced 2 runs, one-time baseline so the pre-existing backlog surfaces once for
the #5 batcher without paging daily. Alerts CNCP + George (https MagicDNS host)
on NEW regressions only; writes data/latest.json for the meta-watchdog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
.gitignore | 10 +++
README.md | 52 +++++++++++
canary.mjs | 184 ++++++++++++++++++++++++++++++++++++++
com.steve.dw-launchd-canary.plist | 21 +++++
data/baseline.json | 14 +++
manifest.json | 26 ++++++
run.sh | 7 ++
7 files changed, 314 insertions(+)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..de12622
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,10 @@
+node_modules/
+.env*
+tmp/
+*.log
+.DS_Store
+data/run.log
+data/launchd.out
+data/launchd.err
+data/history.jsonl
+data/latest.json
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..1fdd0e4
--- /dev/null
+++ b/README.md
@@ -0,0 +1,52 @@
+# dw-launchd-canary
+
+Hourly, read-only health canary for the `com.steve.*` launchd fleet — one tier
+above `dw-canary-watchdog`. The watchdog proves the *canaries* ran; this proves
+the broader launchd fleet is actually healthy.
+
+Born from the Officer Idea Council (2026-06-17), where vp-engineering (#1) and
+vp-operations (#2) independently flagged the same live blind spot: jobs sitting
+at `LastExitStatus=1` and plists silently UNLOADED, with zero alerts — the same
+class of silent failure that hid the 12-day pg_dump death.
+
+## What it catches
+
+- **DROPPED** — an active `.plist` is on disk but not in `launchctl` (drift on a
+ reboot/5am fleet restart, a failed bootstrap, or a label mismatch).
+- **FAILING** — a loaded job exits nonzero every run.
+
+## What it deliberately does NOT flag (the honest part)
+
+- Jobs renamed `*.DISABLED-*` / `*.done` / `*.RETIRED` or moved to `disabled-*/`
+ — they don't match the `com.steve.*.plist` glob, so deliberate-offs are invisible
+ to the canary (e.g. `kravet-roll-variants`, disabled for a stale map).
+- `Disabled=true` in the plist → reported as **PAUSED**, never alerted.
+- `health_check_jobs` (see `manifest.json`) exiting **1** — a health check that
+ exits 1 to report "I found a problem" is working as intended (e.g.
+ `enrichment-health` when vendors go dark). Their exit 1 is OK.
+- **Debounce**: a DROPPED/FAILING job must persist `consecutive_fail_threshold`
+ runs (default 2) before it alerts — one transient won't page anyone.
+
+## Output & alerting
+
+- Writes `data/latest.json` (so the meta-watchdog covers this canary too) and
+ appends `data/history.jsonl` (the debounce source).
+- On a *confirmed* regression: CNCP parking-lot card + George email.
+- Drops a `~/.claude/heartbeats/dw-launchd-canary.stamp` each run.
+
+George is reached at the Tailscale MagicDNS host `https://kamatera.tail79cb8e.ts.net:9850`
+(the agent flipped to HTTPS-only on 2026-06-17; the raw `http://<ip>` now 400s).
+
+## Run
+
+```sh
+bash ~/Projects/dw-launchd-canary/run.sh # one shot
+node ~/Projects/dw-launchd-canary/canary.mjs # direct
+```
+
+## Schedule (gated — Steve installs)
+
+```sh
+launchctl bootstrap gui/$(id -u) ~/Projects/dw-launchd-canary/com.steve.dw-launchd-canary.plist
+launchctl kickstart -k gui/$(id -u)/com.steve.dw-launchd-canary
+```
diff --git a/canary.mjs b/canary.mjs
new file mode 100644
index 0000000..b6eaedd
--- /dev/null
+++ b/canary.mjs
@@ -0,0 +1,184 @@
+#!/usr/bin/env node
+// dw-launchd-canary — one tier above the meta-watchdog.
+//
+// The meta-watchdog proves the DW *canaries* ran. This proves the broader
+// launchd fleet is HEALTHY: it catches the two blind spots that hid the
+// 12-day pg_dump death and that the Officer Council flagged 2026-06-17 —
+// (1) a job that exits nonzero every run, and
+// (2) a plist that should be scheduled but isn't even LOADED (drift on a
+// reboot/fleet restart, failed bootstrap, or label mismatch).
+//
+// HONEST-by-design (mirrors scripts/launchd-audit/audit.sh's hard-won rules):
+// * Auto-discovers ~/Library/LaunchAgents/com.steve.*.plist. Jobs renamed
+// .DISABLED / .done / .RETIRED, or moved to disabled-*/ , do NOT match the
+// glob — so deliberate-offs never false-alarm.
+// * health_check_jobs (manifest) exit 1 = "reporting a problem", NOT broken.
+// * Debounced: a job must be bad on >= consecutive_fail_threshold runs before
+// it FAILs — one transient (a 5am restart mid-run) won't page anyone.
+// READ-ONLY: never loads/unloads/edits a job. On a confirmed regression: CNCP
+// card + George email. Writes data/latest.json so the meta-watchdog covers it.
+import fs from 'node:fs';
+import os from 'node:os';
+import path from 'node:path';
+import { execSync } from 'node:child_process';
+
+const HOME = os.homedir();
+const SKILL = path.join(HOME, 'Projects/dw-launchd-canary');
+const DATA = path.join(SKILL, 'data');
+const LA = path.join(HOME, 'Library/LaunchAgents');
+fs.mkdirSync(DATA, { recursive: true });
+const now = Date.now();
+const ENV = process.env;
+const LOG = path.join(DATA, 'run.log');
+const HIST = path.join(DATA, 'history.jsonl');
+const log = (m) => { const l = `[${new Date(now).toISOString()}] ${m}`; console.log(l); try { fs.appendFileSync(LOG, l + '\n'); } catch {} };
+
+const manifest = JSON.parse(fs.readFileSync(path.join(SKILL, 'manifest.json'), 'utf8'));
+const healthCheck = new Set(manifest.health_check_jobs || []);
+const expectedUnloaded = new Set(manifest.expected_unloaded || []);
+const notes = manifest.notes || {};
+const THRESH = manifest.consecutive_fail_threshold || 2;
+
+// --- discover the jobs that SHOULD be scheduled (active .plist files only) ---
+const plists = fs.readdirSync(LA).filter(f => /^com\.steve\..+\.plist$/.test(f));
+
+function launchctlInfo(label) {
+ // returns { loaded, exit } — exit is the macOS-raw LastExitStatus or null
+ let line = '';
+ try { line = execSync(`launchctl list ${label} 2>/dev/null`, { encoding: 'utf8' }); } catch { line = ''; }
+ if (!line.trim()) {
+ // fall back to the column form (some labels only show in the table)
+ try {
+ const tbl = execSync(`launchctl list 2>/dev/null | grep -F ${label}`, { encoding: 'utf8' }).trim();
+ if (!tbl) return { loaded: false, exit: null };
+ const cols = tbl.split(/\s+/); // PID STATUS LABEL
+ const ex = cols[1] === '-' ? null : parseInt(cols[1], 10);
+ return { loaded: true, exit: Number.isNaN(ex) ? null : ex };
+ } catch { return { loaded: false, exit: null }; }
+ }
+ const m = line.match(/"LastExitStatus"\s*=\s*(\d+)/);
+ return { loaded: true, exit: m ? parseInt(m[1], 10) : null };
+}
+
+const results = [];
+for (const f of plists) {
+ const label = f.replace(/\.plist$/, '');
+ const short = label.replace(/^com\.steve\./, '');
+ // is it Disabled=true in the plist itself? (deliberate pause, keeps the .plist name)
+ let disabled = false;
+ try { disabled = /<key>Disabled<\/key>\s*<true\/>/.test(fs.readFileSync(path.join(LA, f), 'utf8')); } catch {}
+ const { loaded, exit } = launchctlInfo(label);
+
+ let verdict = 'OK', reason = 'loaded, exit 0';
+ if (disabled) { verdict = 'PAUSED'; reason = 'Disabled=true in plist (deliberate)'; }
+ else if (expectedUnloaded.has(short)) { verdict = 'EXPECTED_OFF'; reason = 'intentionally unloaded (manifest expected_unloaded)'; }
+ else if (!loaded) { verdict = 'DROPPED'; reason = 'plist on disk but NOT in launchctl — silently unloaded (drift/reboot/failed bootstrap)'; }
+ else if (exit === null) { verdict = 'OK'; reason = 'loaded, no exit status yet (not run since load)'; }
+ else if (exit === 0) { verdict = 'OK'; reason = 'loaded, exit 0'; }
+ else if (exit % 256 !== 0) {
+ // raw status whose low byte is nonzero = terminated by a signal (9=KILL,15=TERM).
+ // For KeepAlive daemons that's restart noise, not a job failure — surface, don't page.
+ verdict = 'KILLED'; reason = `terminated by signal ${exit & 0x7f} (daemon restart / kill — not a job failure)`;
+ } else {
+ const code = exit / 256; // clean exit code: 256→1, 512→2 …
+ if (healthCheck.has(short) && code === 1) { verdict = 'OK'; reason = 'exit 1 = health-check reporting a finding (intended)'; }
+ else { verdict = 'FAILING'; reason = `nonzero exit (raw ${exit}, code ${code})`; }
+ }
+ results.push({ short, verdict, reason, note: notes[short] || null });
+}
+
+// --- debounce against history: a regression must persist THRESH runs ---
+let hist = [];
+try { hist = fs.readFileSync(HIST, 'utf8').trim().split('\n').filter(Boolean).map(JSON.parse); } catch {}
+const recent = hist.slice(-(THRESH - 1)); // prior runs to combine with this one
+const bad = new Set(['DROPPED', 'FAILING']);
+function confirmed(short, thisVerdict) {
+ if (!bad.has(thisVerdict)) return false;
+ // bad now AND bad in each of the prior (THRESH-1) runs
+ if (recent.length < THRESH - 1) return false; // not enough history yet → hold
+ return recent.every(run => {
+ const r = (run.results || []).find(x => x.short === short);
+ return r && bad.has(r.verdict);
+ });
+}
+
+// One-time baseline: the set of jobs already bad when this canary was installed.
+// They're acknowledged (the existing bootstrap backlog → fed to the #5 plist
+// batcher), so we surface them but DON'T page about them every hour. Only a job
+// that goes bad AFTER the baseline is a true regression worth an alert.
+const BASE = path.join(DATA, 'baseline.json');
+let baseline;
+if (fs.existsSync(BASE)) { try { baseline = new Set(JSON.parse(fs.readFileSync(BASE, 'utf8')).known || []); } catch { baseline = new Set(); } }
+else {
+ baseline = new Set(results.filter(r => bad.has(r.verdict)).map(r => r.short));
+ fs.writeFileSync(BASE, JSON.stringify({ seeded_at: new Date(now).toISOString(), known: [...baseline] }, null, 2));
+ log(`seeded baseline with ${baseline.size} pre-existing bad job(s): ${[...baseline].join(', ') || '(none)'}`);
+}
+
+const confirmedBad = results.filter(r => bad.has(r.verdict) && confirmed(r.short, r.verdict));
+const regressions = confirmedBad.filter(r => !baseline.has(r.short)); // NEW breakage → page
+const acknowledged = confirmedBad.filter(r => baseline.has(r.short)); // pre-existing backlog → surface only
+const pending = results.filter(r => bad.has(r.verdict) && !confirmed(r.short, r.verdict)); // bad but not yet THRESH runs
+
+const summary = {
+ ts: new Date(now).toISOString(),
+ total: results.length,
+ ok: results.filter(r => r.verdict === 'OK').length,
+ paused: results.filter(r => r.verdict === 'PAUSED').length,
+ expected_off: results.filter(r => r.verdict === 'EXPECTED_OFF').length,
+ killed: results.filter(r => r.verdict === 'KILLED').length,
+ dropped: results.filter(r => r.verdict === 'DROPPED').length,
+ failing: results.filter(r => r.verdict === 'FAILING').length,
+ confirmed_regressions: regressions.map(r => r.short),
+ acknowledged_backlog: acknowledged.map(r => r.short),
+ pending_confirmation: pending.map(r => r.short),
+ results
+};
+fs.writeFileSync(path.join(DATA, 'latest.json'), JSON.stringify(summary, null, 2));
+fs.appendFileSync(HIST, JSON.stringify({ ts: summary.ts, results: results.map(r => ({ short: r.short, verdict: r.verdict })) }) + '\n');
+// keep history bounded
+try { const lines = fs.readFileSync(HIST, 'utf8').trim().split('\n'); if (lines.length > 500) fs.writeFileSync(HIST, lines.slice(-500).join('\n') + '\n'); } catch {}
+
+log(`launchd-canary: ${summary.total} jobs · ${summary.ok} OK · ${summary.paused} paused · ${summary.expected_off} expected-off · ${summary.killed} killed · ${summary.dropped} dropped · ${summary.failing} failing`);
+for (const r of regressions) log(` REGRESSION ${r.short} — ${r.reason}`);
+for (const r of acknowledged) log(` acknowledged-backlog ${r.short} — ${r.reason}`);
+for (const r of pending) log(` pending ${r.short} — ${r.reason}`);
+
+// self-heartbeat so the meta-watchdog can prove THIS canary ran
+try { fs.mkdirSync(path.join(HOME, '.claude/heartbeats'), { recursive: true }); fs.writeFileSync(path.join(HOME, '.claude/heartbeats/dw-launchd-canary.stamp'), String(now)); } catch {}
+
+// --- alerting (George flipped to HTTPS-only 2026-06-17 — use the MagicDNS host) ---
+const GEORGE = ENV.GEORGE_URL || 'https://kamatera.tail79cb8e.ts.net:9850';
+async function postCNCP(note) {
+ const base = ENV.CNCP_URL || 'http://localhost:3333';
+ try { const ac = new AbortController(); const t = setTimeout(() => ac.abort(), 8000);
+ const r = await fetch(`${base}/api/parking-lot`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ url: 'file://' + SKILL, note }), signal: ac.signal });
+ clearTimeout(t); return r.ok; } catch { return false; }
+}
+async function emailGeorge(subject, html) {
+ let auth = ENV.GEORGE_AUTH;
+ if (!auth) { try { auth = (fs.readFileSync(path.join(HOME, 'Projects/secrets-manager/.env'), 'utf8').match(/^GEORGE_AUTH=(.*)$/m) || [])[1]; } catch {} }
+ if (!auth) { log('email skipped (no GEORGE_AUTH)'); return false; }
+ try { const ac = new AbortController(); const t = setTimeout(() => ac.abort(), 15000);
+ const r = await fetch(`${GEORGE}/api/send`, { method: 'POST', headers: { 'Authorization': auth, 'Content-Type': 'application/json; charset=utf-8' },
+ body: JSON.stringify({ to: ENV.CANARY_TO || 'steve@designerwallcoverings.com', subject, body: html }), signal: ac.signal });
+ clearTimeout(t); const j = await r.json().catch(() => ({})); return !!(j.success || j.messageId); } catch { return false; }
+}
+
+if (regressions.length) {
+ const note = `[LAUNCHD CANARY ${summary.ts.slice(0, 10)}] ${regressions.length} confirmed launchd regression(s) (bad ${THRESH}+ runs): `
+ + regressions.map(r => `${r.short} — ${r.verdict} (${r.reason})`).join('; ')
+ + `. A job that exits nonzero every run or fell out of launchctl emits zero alerts on its own — that's the silent-death class.`;
+ await postCNCP(note);
+ const html = `<div style="font-family:-apple-system,sans-serif;color:#222"><h3 style="color:#b23b3b">⚠ launchd canary — ${regressions.length} confirmed regression(s)</h3>`
+ + regressions.map(r => `<div style="margin:6px 0"><b>${r.short}</b> — <span style="color:#b23b3b">${r.verdict}</span> ${r.reason}${r.note ? ` <span style="color:#888">(${r.note})</span>` : ''}</div>`).join('')
+ + (pending.length ? `<hr><div style="color:#a07000">Watching (not yet ${THRESH} runs): ${pending.map(p => p.short + ' — ' + p.verdict).join('; ')}</div>` : '')
+ + `<hr><div style="color:#888;font-size:12px">${summary.ok} OK · ${summary.paused} paused (deliberate) · ${summary.dropped} dropped · ${summary.failing} failing</div></div>`;
+ const emailed = await emailGeorge(`⚠ launchd canary — ${regressions.length} regression — ${summary.ts.slice(0, 10)}`, html);
+ log(emailed ? 'regression alert emailed' : 'regression alert email NOT sent');
+ process.exitCode = 1;
+} else {
+ log('✓ no NEW launchd regressions'
+ + (acknowledged.length ? ` (${acknowledged.length} pre-existing backlog, not paged: ${acknowledged.map(a => a.short).join(', ')})` : '')
+ + (pending.length ? ` (${pending.length} watching: ${pending.map(p => p.short).join(', ')})` : ''));
+}
diff --git a/com.steve.dw-launchd-canary.plist b/com.steve.dw-launchd-canary.plist
new file mode 100644
index 0000000..4eb966b
--- /dev/null
+++ b/com.steve.dw-launchd-canary.plist
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+ <key>Label</key>
+ <string>com.steve.dw-launchd-canary</string>
+ <key>ProgramArguments</key>
+ <array>
+ <string>/bin/zsh</string>
+ <string>/Users/stevestudio2/Projects/dw-launchd-canary/run.sh</string>
+ </array>
+ <key>StartInterval</key>
+ <integer>3600</integer>
+ <key>RunAtLoad</key>
+ <true/>
+ <key>StandardOutPath</key>
+ <string>/Users/stevestudio2/Projects/dw-launchd-canary/data/launchd.out</string>
+ <key>StandardErrorPath</key>
+ <string>/Users/stevestudio2/Projects/dw-launchd-canary/data/launchd.err</string>
+</dict>
+</plist>
diff --git a/data/baseline.json b/data/baseline.json
new file mode 100644
index 0000000..35f47a6
--- /dev/null
+++ b/data/baseline.json
@@ -0,0 +1,14 @@
+{
+ "seeded_at": "2026-06-17T21:14:30.276Z",
+ "known": [
+ "claude-auto-resume",
+ "color-cleaner-nightly",
+ "daily-overnight-skus",
+ "gmc-optout-reminder",
+ "henry-probe",
+ "idea-loop-digest",
+ "ollama-henry",
+ "ollama-rewarm",
+ "wallco-variants"
+ ]
+}
\ No newline at end of file
diff --git a/manifest.json b/manifest.json
new file mode 100644
index 0000000..76f0b00
--- /dev/null
+++ b/manifest.json
@@ -0,0 +1,26 @@
+{
+ "description": "launchd health canary — config for the dw-launchd-canary. Auto-discovers every ~/Library/LaunchAgents/com.steve.*.plist (deliberate-off jobs renamed .DISABLED/.done/.RETIRED or moved to disabled-*/ are naturally excluded, so they never false-alarm). Only the exceptions below need hand-maintaining.",
+ "consecutive_fail_threshold": 2,
+ "_comment_health_check_jobs": "Jobs whose nonzero exit is a SIGNAL, not a failure — a health-check that exits 1 to report 'I found a problem' is working as intended. enrichment-health exits 1 when active vendors go dark. Do NOT flag these on exit 1.",
+ "health_check_jobs": [
+ "enrichment-health",
+ "dw-canary-watchdog",
+ "dw-launchd-canary",
+ "dw-leak-guard",
+ "dw-map-auditor",
+ "dw-uptime-probe",
+ "dw-scraper-canary",
+ "dw-leak-scanner",
+ "dw-backup-canary"
+ ],
+ "_comment_expected_unloaded": "Jobs whose .plist is intentionally KEPT on disk but left UNLOADED — re-arming them is a known hazard, so 'not in launchctl' is the desired state, not a drop. Don't flag.",
+ "expected_unloaded": [
+ "drain-all-queues"
+ ],
+ "_comment_notes": "Free-text context shown alongside a job in the report; does not affect verdicts.",
+ "notes": {
+ "romo-roll-pricing": "one-shot self-unloader: renames itself .done when all rolls land. Absence is success, not a drop.",
+ "kravet-roll-variants": "deliberately DISABLED 2026-06-16 (stale product_map). Off by design.",
+ "drain-all-queues": "deliberately unloaded — re-arming re-inserts SKU-less RW rows + the $4.25 drain. Keep OFF."
+ }
+}
diff --git a/run.sh b/run.sh
new file mode 100644
index 0000000..53c6e5e
--- /dev/null
+++ b/run.sh
@@ -0,0 +1,7 @@
+#!/bin/zsh
+# dw-launchd-canary launchd entrypoint — hourly, read-only.
+export PATH="/usr/local/bin:/opt/homebrew/bin:/usr/bin:/bin"
+SKILL="$HOME/Projects/dw-launchd-canary"
+# Load GEORGE_AUTH (+ any overrides) for the alert path.
+set -a; . "$HOME/Projects/secrets-manager/.env" 2>/dev/null; set +a
+exec /usr/local/bin/node "$SKILL/canary.mjs" >> "$SKILL/data/launchd.out" 2>&1
(oldest)
·
back to Dw Launchd Canary
·
Add gated-plist-install batcher (Officer Council #5) 0a4edf7 →