`README.md`

53 lines
# dw-launchd-canary

Hourly, read-only health canary for the `com.steve.*` launchd fleet — one tier
above `dw-canary-watchdog`. The watchdog proves the *canaries* ran; this proves
the broader launchd fleet is actually healthy.

Born from the Officer Idea Council (2026-06-17), where vp-engineering (#1) and
vp-operations (#2) independently flagged the same live blind spot: jobs sitting
at `LastExitStatus=1` and plists silently UNLOADED, with zero alerts — the same
class of silent failure that hid the 12-day pg_dump death.

## What it catches

- **DROPPED** — an active `.plist` is on disk but not in `launchctl` (drift on a
  reboot/5am fleet restart, a failed bootstrap, or a label mismatch).
- **FAILING** — a loaded job exits nonzero every run.

## What it deliberately does NOT flag (the honest part)

- Jobs renamed `*.DISABLED-*` / `*.done` / `*.RETIRED` or moved to `disabled-*/`
  — they don't match the `com.steve.*.plist` glob, so deliberate-offs are invisible
  to the canary (e.g. `kravet-roll-variants`, disabled for a stale map).
- `Disabled=true` in the plist → reported as **PAUSED**, never alerted.
- `health_check_jobs` (see `manifest.json`) exiting **1** — a health check that
  exits 1 to report "I found a problem" is working as intended (e.g.
  `enrichment-health` when vendors go dark). Their exit 1 is OK.
- **Debounce**: a DROPPED/FAILING job must persist `consecutive_fail_threshold`
  runs (default 2) before it alerts — one transient won't page anyone.

## Output & alerting

- Writes `data/latest.json` (so the meta-watchdog covers this canary too) and
  appends `data/history.jsonl` (the debounce source).
- On a *confirmed* regression: CNCP parking-lot card + George email.
- Drops a `~/.claude/heartbeats/dw-launchd-canary.stamp` each run.

George is reached at the Tailscale MagicDNS host `https://kamatera.tail79cb8e.ts.net:9850`
(the agent flipped to HTTPS-only on 2026-06-17; the raw `http://<ip>` now 400s).

## Run

```sh
bash ~/Projects/dw-launchd-canary/run.sh          # one shot
node ~/Projects/dw-launchd-canary/canary.mjs       # direct
```

## Schedule (gated — Steve installs)

```sh
launchctl bootstrap gui/$(id -u) ~/Projects/dw-launchd-canary/com.steve.dw-launchd-canary.plist
launchctl kickstart -k gui/$(id -u)/com.steve.dw-launchd-canary
```