`src/content/posts/2026-05-30-teaching-my-monitoring-to-stop-crying-wolf.md`

61 lines

---
title: "Teaching My Monitoring to Stop Crying Wolf"
description: "A day spent making alerts trustworthy again — because a monitor that fires 291 times a day is a monitor you've already learned to ignore."
date: 2026-05-30
tags: ["monitoring", "reliability", "automation", "observability", "founder-log"]
---

Two problems today, one theme: the systems that are supposed to tell you when something's wrong will happily lie to you if you let them. A monitor that alerts constantly is functionally the same as no monitor at all — because you stop reading it.

## Problem One: The Security Alert That Was Nothing

My command center kept surfacing a high-severity "bad key share" alert. High severity gets your attention. So I dug in.

The root cause was completely benign: a VPN scanner out on the internet was sending a malformed TLS handshake — a client-side reject that nginx correctly bounced. It's background noise. The internet is full of scanners poking at every open port. This one just happened to trip a log pattern that got classified as scary.

The fix had two layers. There was already a "benign filter" written to catch exactly this kind of thing — it just was never actually wired into the log collector. So I wired it in. Then I added a durable suppression rule at the ingestion layer, so future TLS-probe noise auto-dismisses the moment it arrives, no manual triage. I tested it by feeding a synthetic copy of the alert through and confirming it got dismissed automatically.

## Problem Two: The Monitor That Failed 291 Times a Day

On a different box, a service-monitor was flooding the ledger with **291 events**. The monitor was checking whether services were alive — and it was the monitor *itself* that kept failing.

Here's the trap: the monitor detected a defunct entry (a port with nothing listening and no process behind it) and *exited non-zero* to signal "something's wrong." But the supervisor interpreted a non-zero exit as "the monitor unit failed" — and logged an event. Every two minutes. Forever.

So a *correctly detected* problem was being recorded as a *monitor failure*, 291 times over. The signal and the noise had swapped places.

The fix path: harden the monitor so that detecting an outage no longer makes the monitor mark *itself* as failed, plus remove the stale defunct entry it was tripping on. (I also caught a plaintext token leaking in one of the service unit files while I was in there — a separate finding, but that's how it goes when you actually read the logs instead of skimming them.)

## The Real Lesson

**Alert fatigue is a bug, not a personality trait.** When your monitoring cries wolf 291 times, the failure isn't that you started ignoring it — the failure is the monitoring.

Two principles I keep relearning:

1. **A monitor's own failure mode must be distinct from the failures it detects.** If "I found a problem" and "I broke" produce the same signal, you've built a machine that can't tell you anything.
2. **Suppression has to be durable and self-verified.** Dismissing one noisy alert by hand is a chore you'll be doing again tomorrow. Wiring the suppression into ingestion — and proving it works with a synthetic test — means the noise is gone for good.

## Bonus: Hands-Free Recovery

Same day, I also built a small thing I'd wanted for a while: a watcher that detects when a session hits a rate limit and automatically resumes it once the limit clears. It reads the pane, matches the throttle message (even when the terminal hard-wraps it mid-word), respects a cooldown, and has a kill-switch. Small quality-of-life win, but it means stalled work recovers itself instead of waiting for me to notice.

## Why This Matters for Business Owners

The whole point of monitoring is to earn your trust so you can *stop* watching. A noisy monitor does the opposite — it demands attention while providing none. If your dashboards are a wall of red you've learned to scroll past, they're not protecting you. They're theater. Tuning them so that a red light *actually means something* is some of the highest-leverage work you can do.

## Let's Connect

I build AI-powered automation for real businesses — not demos, not prototypes, production systems that run 24/7.

If you're a **founder, entrepreneur, or small business owner** looking to automate operations with AI, let's talk:

- [**@agentabrams on YouTube**](https://youtube.com/@AgentAbrams) — walkthroughs and demos
- [**@agentabrams on X**](https://x.com/agentabrams) — DMs open
- [**@agentabrams on Bluesky**](https://bsky.app/profile/agentabrams.bsky.social) — follow along
- [**goodquestion.ai**](https://goodquestion.ai) — you're here

**Advisory & Board Opportunities:** I'm actively looking to join boards where AI automation can drive real business value. If your company is exploring AI-driven operations, data pipelines, or autonomous agent systems — I'd love to contribute as a board member or advisor. Reach out on any platform above.

---

*Built with [Claude Code](https://claude.ai). Shipped in production. Every day. This is what one founder + AI looks like.*