← back to Cncp Failure Collector

README.md

111 lines

# cncp-failure-collector

Tails pm2 + launchd for failures and posts them as CNCP failure-ledger rows. Local-only, pm2-managed, fail-soft.

Decided 2026-05-11 via best-practices reviewer + 4-LLM debate-team consensus on architecture option B. Six iterative ticks shipped 2026-05-11.

## How it works

Every `POLL_INTERVAL_MS` (default 30 s) the daemon:

1. **pm2 scan** — runs `pm2 jlist`, diffs against `state.json`. Any process where `restart_time` bumped or status flipped away from `online` is posted. Bootstrap-silent (first run seeds baseline, no historical-restart spam). Skips itself.
2. **launchd plist tailing** — scans `~/Library/LaunchAgents/com.steve.*.plist`, plutil-parses each for `StandardErrorPath` (or `StandardOutPath` fallback), builds a watchlist of ~97 jobs. Refreshed every 10 min. Reads only new bytes since last cycle. Caps at 200 watched files; skips files larger than 100 MB.
3. **POST to CNCP** at `http://127.0.0.1:3333/api/failures`. CNCP applies sticky dedup (collapse repeat events into the same open row; only `resolved`/`dismissed` rows fork a new row on next event).
4. **Auto-classify (fire-and-forget)** — on a NEW row response (`deduped:false`), the collector POSTs `/api/failures/<id>/classify` with no body. Per-process 5-min rate limit (in-memory `Map`). Server-side: single classifier in flight, FIFO queue (cap 10) drains on completion. Hermes 3 on **localhost** (Steve override 2026-05-11) returns severity / category / suggestedAction in ~8 s.
5. **Heartbeat** every 10 quiet cycles (~5 min): logs `heartbeat — N quiet cycles, X pm2 tracked, Y logs tracked`.

State (last restart counts, log byte offsets, plist scan timestamp) lives in `state.json` in this dir — gitignored, per-machine runtime data.

## Run

```bash
pm2 start ecosystem.config.js
pm2 save
```

One-shot dev test:

```bash
npm run once
```

## Environment

| Var | Default | Purpose |
|-----|---------|---------|
| `CNCP_URL` | `http://127.0.0.1:3333` | Where to POST failures + trigger classify |
| `POLL_INTERVAL_MS` | `30000` | Poll cadence (pm2 + launchd) |
| `CLASSIFIER_ENABLED` | `1` | Set `0` to disable fire-and-forget classify (manual classify-by-UI still works) |

CNCP-side env (in CNCP's own start, not this collector):

| Var | Default | Purpose |
|-----|---------|---------|
| `HERMES_URL` | `http://127.0.0.1:11434` | Local Ollama for the classifier (Steve override of MS1 default) |
| `HERMES_MODEL` | `hermes3:8b` | Pulled on MS2 + MS1 |
| `GEORGE_ALERTS_ENABLED` | `0` (DRY-RUN) | Set `1` to actually email Steve on severity=high (1-hour throttle per processName via `cncp-alert-throttle.json`) |

## CNCP endpoints

- `GET /api/failures[?status=...&source=...]` — list, optionally filtered.
- `POST /api/failures` — append (with sticky dedup); body `{source, processName, exitCode?, lastLines?, restartCount?}`.
- `PATCH /api/failures/:id` — update status / severity / category / suggestedAction.
- `DELETE /api/failures/:id` — remove.
- `POST /api/failures/:id/classify` — synchronous (15 s timeout, single-flight queue). Hermes 3 returns `{category, severity, suggestedAction}`. Fail-soft: returns 200 with `classified:false` on any error.
- `POST /api/failures/:id/chat` — refinement chat. Body `{message}`. Hermes can emit a trailing `{refine: {severity, category, suggestedAction}}` JSON line which auto-PATCHes the row.
- `POST /api/idea-chat` — global "suggest an idea to build out" prompt; body `{prompt, savePin?}`. SavePin drops the outline into the Parking Lot.
- `POST /api/failures/cleanup-transients` — manual trigger; the same logic runs 5 s after CNCP startup and every 30 min automatically. Dismisses `triage`/`low|medium`/`transient` rows quiet for 24 h.

## CNCP UI

`http://localhost:3333/` → **Failures** nav button (with open-count badge).

- Card grid, sort + density slider (Steve standing-rule pattern).
- Per-card: source + severity + category badges, processName, status dropdown (triage / investigating / resolved / dismissed), event count, last-output snippet, italic suggestedAction line.
- Per-card **Refine** button — opens an inline Hermes chat thread. Persisted in `task.messages[]`.
- Cross-top **idea-chat bar** — free-form "suggest an idea to build out" input. "Save to Lot" routes to Parking Lot.

## Sidebar

Failures shell shows `N open / M total` with top-6 source-badged process names; click to open the main view.

## Shopify cached-page price canary

`canary-shopify-prices.js` — separate daily script (NOT the live pm2 daemon). Compares each Shopify product's variant price (from `/products/<handle>.json`) against what's rendered in the *cached* product page HTML. Files a `source: 'shopify-cache'` row to `/api/failures` for any product where the cache is missing/wrong relative to the data.

Diagnosed first 2026-05-11 — `flore-batik-pillow-square-22x22-indigo-scalamandre` had `$504.94` in the variant but the cached HTML rendered with no price block. Re-saving the product in Shopify Admin rotates the cache. Audit found 46 confirmed-stale Scalamandre products created that day; the canary now watches for new ones nightly.

### Run

Ad-hoc:
```bash
VENDOR_FILTER=scalamandre LOOKBACK_HOURS=24 node canary-shopify-prices.js
```

Nightly (launchd at 4:17 AM PT):
```bash
launchctl load ~/Library/LaunchAgents/com.steve.shopify-price-canary.plist
```

### Env

| Var | Default | Notes |
|-----|---------|-------|
| `STORE` | `https://www.designerwallcoverings.com` | Public Shopify storefront URL |
| `CNCP_URL` | `http://127.0.0.1:3333` | Where to POST failure rows |
| `LOOKBACK_HOURS` | `24` | Products created/updated in this window |
| `REQUESTS_PER_SECOND` | `2` | Throttle to stay under Shopify edge rate limits |
| `MAX_PAGES` | `20` | Hard cap on /products.json paging |
| `VENDOR_FILTER` | (none) | Regex; only check products whose vendor or tags match |

### What gets posted to /api/failures

`source: 'shopify-cache'`, `processName: <SKU>`, `lastLines` containing handle / product ID / data price / cached price / fresh price / admin URL. The Hermes classifier will pick these up via the existing fire-and-forget path and surface them in the Failures UI alongside pm2/launchd failures.

## What's NOT in here

- **Cloud agents** — local-only by Steve's standing rule.
- **Anthropic API** — never. All inference via local Ollama.
- **Auto-restart** — that stays with `process-hawk` + per-project hawks. This is the observability ledger.
- **Email sending in production** — `GEORGE_ALERTS_ENABLED` defaults `0`. Flip when comfortable.