← back to Dw Monitoring

init: dw-monitoring + daily broken-image regression guard (baseline 4989)

5be39bd6c1e0ad5cccf4625fa43e83777ef8afde · 2026-05-19 21:17:34 -0700 · Steve Abrams

- scripts/broken-image-check.js — reads dw_unified read-only, joins
  shopify_products → image_hashes, alerts if count grows >=5% vs baseline.
- data/broken-image-baseline.json — locked at {count:4989, asOf:2026-05-19}.
- data/broken-image-history.jsonl — appends per-run snapshot.
- LaunchAgent com.steve.broken-image-check daily at 09:00 local
  (plist installed at ~/Library/LaunchAgents/; non-interactive session can't
  bootstrap, must be loaded once from an interactive Terminal — README has
  the launchctl load -w command).
- SMTP degraded to file-only alerts: secrets-manager/.env has no GMAIL_USER
  / GMAIL_APP_PASSWORD pair yet, so alerts land in alerts/<ISO>.txt until
  Steve wires creds. nodemailer require is lazy + try/catch so the absence
  doesn't break the check.
- First run logged baseline=count=4989 ratio=1.0000 (no breach).

Files touched

• A .gitignore
• A README.md
• A data/broken-image-baseline.json
• A package-lock.json
• A package.json
• A scripts/broken-image-check.js

Diff

commit 5be39bd6c1e0ad5cccf4625fa43e83777ef8afde
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Tue May 19 21:17:34 2026 -0700

    init: dw-monitoring + daily broken-image regression guard (baseline 4989)
    
    - scripts/broken-image-check.js — reads dw_unified read-only, joins
      shopify_products → image_hashes, alerts if count grows >=5% vs baseline.
    - data/broken-image-baseline.json — locked at {count:4989, asOf:2026-05-19}.
    - data/broken-image-history.jsonl — appends per-run snapshot.
    - LaunchAgent com.steve.broken-image-check daily at 09:00 local
      (plist installed at ~/Library/LaunchAgents/; non-interactive session can't
      bootstrap, must be loaded once from an interactive Terminal — README has
      the launchctl load -w command).
    - SMTP degraded to file-only alerts: secrets-manager/.env has no GMAIL_USER
      / GMAIL_APP_PASSWORD pair yet, so alerts land in alerts/<ISO>.txt until
      Steve wires creds. nodemailer require is lazy + try/catch so the absence
      doesn't break the check.
    - First run logged baseline=count=4989 ratio=1.0000 (no breach).
---
 .gitignore                      |   7 ++
 README.md                       |  46 +++++++++++
 data/broken-image-baseline.json |   1 +
 package-lock.json               | 175 ++++++++++++++++++++++++++++++++++++++++
 package.json                    |  16 ++++
 scripts/broken-image-check.js   | 155 +++++++++++++++++++++++++++++++++++
 6 files changed, 400 insertions(+)

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..3ad7820
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,7 @@
+node_modules/
+.env
+.env.*
+data/*.jsonl
+alerts/
+*.log
+.DS_Store
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..064daad
--- /dev/null
+++ b/README.md
@@ -0,0 +1,46 @@
+# dw-monitoring
+
+Daily regression guards for Designer Wallcoverings infrastructure.
+
+## broken-image-check
+
+Counts `shopify_products` rows whose `image_url` joins to an `image_hashes`
+row with `status != 'success'`. Baseline: **4,989** (2026-05-19, 4,989 products on
+`designer-laboratory-sandbox.myshopify.com`).
+
+If today's count grows **≥ 5%** above the stored baseline, writes an alert
+file to `alerts/<ISO>.txt` and attempts an SMTP send if Gmail creds are
+present in `~/Projects/secrets-manager/.env` (`GMAIL_USER` + `GMAIL_APP_PASSWORD`).
+Currently degrades to **file-only alerts** — SMTP creds not yet wired.
+
+Run manually:
+
+```bash
+node scripts/broken-image-check.js
+```
+
+### Schedule (launchd)
+
+Plist at `~/Library/LaunchAgents/com.steve.broken-image-check.plist` runs
+daily at **09:00 local**. Log: `/tmp/broken-image-check.log`.
+
+Load (run from an interactive Terminal session — non-Aqua sessions cannot
+bootstrap LaunchAgents):
+
+```bash
+launchctl load -w ~/Library/LaunchAgents/com.steve.broken-image-check.plist
+launchctl list | grep broken-image
+```
+
+Trigger an immediate run for verification:
+
+```bash
+launchctl kickstart -k "gui/$(id -u)/com.steve.broken-image-check"
+```
+
+## Hard constraints
+
+- **Read-only** against `dw_unified` — never modifies `shopify_products`
+  or `image_hashes`.
+- Alert fires **only** on growth. Baseline is canonical; rebaseline by
+  editing `data/broken-image-baseline.json`.
diff --git a/data/broken-image-baseline.json b/data/broken-image-baseline.json
new file mode 100644
index 0000000..05600d9
--- /dev/null
+++ b/data/broken-image-baseline.json
@@ -0,0 +1 @@
+{"count":4989,"asOf":"2026-05-19"}
diff --git a/package-lock.json b/package-lock.json
new file mode 100644
index 0000000..eb48f89
--- /dev/null
+++ b/package-lock.json
@@ -0,0 +1,175 @@
+{
+  "name": "dw-monitoring",
+  "version": "1.0.0",
+  "lockfileVersion": 3,
+  "requires": true,
+  "packages": {
+    "": {
+      "name": "dw-monitoring",
+      "version": "1.0.0",
+      "license": "ISC",
+      "dependencies": {
+        "dotenv": "^17.4.2",
+        "pg": "^8.21.0"
+      }
+    },
+    "node_modules/dotenv": {
+      "version": "17.4.2",
+      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.2.tgz",
+      "integrity": "sha512-nI4U3TottKAcAD9LLud4Cb7b2QztQMUEfHbvhTH09bqXTxnSie8WnjPALV/WMCrJZ6UV/qHJ6L03OqO3LcdYZw==",
+      "license": "BSD-2-Clause",
+      "engines": {
+        "node": ">=12"
+      },
+      "funding": {
+        "url": "https://dotenvx.com"
+      }
+    },
+    "node_modules/pg": {
+      "version": "8.21.0",
+      "resolved": "https://registry.npmjs.org/pg/-/pg-8.21.0.tgz",
+      "integrity": "sha512-AUP1EYJuHraQGsVoCQVIcM7TEJVGtDzxWtGFZd8rds9d+CCXlU5Js1rYgfLNvxy9iJrpHjGrRjoi/3BT9fRyiA==",
+      "license": "MIT",
+      "dependencies": {
+        "pg-connection-string": "^2.13.0",
+        "pg-pool": "^3.14.0",
+        "pg-protocol": "^1.14.0",
+        "pg-types": "2.2.0",
+        "pgpass": "1.0.5"
+      },
+      "engines": {
+        "node": ">= 16.0.0"
+      },
+      "optionalDependencies": {
+        "pg-cloudflare": "^1.4.0"
+      },
+      "peerDependencies": {
+        "pg-native": ">=3.0.1"
+      },
+      "peerDependenciesMeta": {
+        "pg-native": {
+          "optional": true
+        }
+      }
+    },
+    "node_modules/pg-cloudflare": {
+      "version": "1.4.0",
+      "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",
+      "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",
+      "license": "MIT",
+      "optional": true
+    },
+    "node_modules/pg-connection-string": {
+      "version": "2.13.0",
+      "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.13.0.tgz",
+      "integrity": "sha512-EMnU9E2fSULdsbErBbMaXJvFeD9B4+nPcM3f+4lsiCR0BHLPrLVjv3DbyM2hgQQviKJaTWIRRTjKjWlHg3p2ig==",
+      "license": "MIT"
+    },
+    "node_modules/pg-int8": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
+      "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
+      "license": "ISC",
+      "engines": {
+        "node": ">=4.0.0"
+      }
+    },
+    "node_modules/pg-pool": {
+      "version": "3.14.0",
+      "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
+      "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
+      "license": "MIT",
+      "peerDependencies": {
+        "pg": ">=8.0"
+      }
+    },
+    "node_modules/pg-protocol": {
+      "version": "1.14.0",
+      "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.14.0.tgz",
+      "integrity": "sha512-n5taZ1kO3s9ngDTVxsEznOqCyToTgz0FLuPq0B33COy5pPpuWJpY3/2oRBVETuOgzdqRXfWpM9HIhp2LBBT1BA==",
+      "license": "MIT"
+    },
+    "node_modules/pg-types": {
+      "version": "2.2.0",
+      "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
+      "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
+      "license": "MIT",
+      "dependencies": {
+        "pg-int8": "1.0.1",
+        "postgres-array": "~2.0.0",
+        "postgres-bytea": "~1.0.0",
+        "postgres-date": "~1.0.4",
+        "postgres-interval": "^1.1.0"
+      },
+      "engines": {
+        "node": ">=4"
+      }
+    },
+    "node_modules/pgpass": {
+      "version": "1.0.5",
+      "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
+      "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
+      "license": "MIT",
+      "dependencies": {
+        "split2": "^4.1.0"
+      }
+    },
+    "node_modules/postgres-array": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
+      "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
+      "license": "MIT",
+      "engines": {
+        "node": ">=4"
+      }
+    },
+    "node_modules/postgres-bytea": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
+      "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
+      "license": "MIT",
+      "engines": {
+        "node": ">=0.10.0"
+      }
+    },
+    "node_modules/postgres-date": {
+      "version": "1.0.7",
+      "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
+      "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
+      "license": "MIT",
+      "engines": {
+        "node": ">=0.10.0"
+      }
+    },
+    "node_modules/postgres-interval": {
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
+      "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
+      "license": "MIT",
+      "dependencies": {
+        "xtend": "^4.0.0"
+      },
+      "engines": {
+        "node": ">=0.10.0"
+      }
+    },
+    "node_modules/split2": {
+      "version": "4.2.0",
+      "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
+      "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
+      "license": "ISC",
+      "engines": {
+        "node": ">= 10.x"
+      }
+    },
+    "node_modules/xtend": {
+      "version": "4.0.2",
+      "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
+      "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
+      "license": "MIT",
+      "engines": {
+        "node": ">=0.4"
+      }
+    }
+  }
+}
diff --git a/package.json b/package.json
new file mode 100644
index 0000000..e381d7e
--- /dev/null
+++ b/package.json
@@ -0,0 +1,16 @@
+{
+  "name": "dw-monitoring",
+  "version": "1.0.0",
+  "main": "index.js",
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "keywords": [],
+  "author": "",
+  "license": "ISC",
+  "description": "",
+  "dependencies": {
+    "dotenv": "^17.4.2",
+    "pg": "^8.21.0"
+  }
+}
diff --git a/scripts/broken-image-check.js b/scripts/broken-image-check.js
new file mode 100644
index 0000000..6362905
--- /dev/null
+++ b/scripts/broken-image-check.js
@@ -0,0 +1,155 @@
+#!/usr/bin/env node
+/**
+ * dw-monitoring · daily broken-image regression guard
+ *
+ * Counts shopify_products whose image_url maps to an image_hashes row with
+ * status != 'success'. If today's count grows >=5% above the stored baseline,
+ * writes an alert file (and attempts SMTP if creds are present).
+ *
+ * Read-only against dw_unified. Never modifies shopify_products / image_hashes.
+ */
+
+'use strict';
+
+const fs = require('fs');
+const path = require('path');
+
+// Load secrets-manager .env (canonical) without overriding existing process env.
+require('dotenv').config({
+  path: path.join(process.env.HOME, 'Projects/secrets-manager/.env'),
+});
+
+const { Client } = require('pg');
+
+const ROOT = path.resolve(__dirname, '..');
+const DATA_DIR = path.join(ROOT, 'data');
+const ALERTS_DIR = path.join(ROOT, 'alerts');
+const BASELINE_FILE = path.join(DATA_DIR, 'broken-image-baseline.json');
+const HISTORY_FILE = path.join(DATA_DIR, 'broken-image-history.jsonl');
+
+const GROWTH_THRESHOLD = 1.05; // 5% above baseline triggers alert
+
+const QUERY = `
+  SELECT COUNT(*)::int AS broken
+  FROM shopify_products sp
+  JOIN image_hashes ih ON sp.image_url = ih.url
+  WHERE ih.status != 'success'
+`;
+
+function ensureDirs() {
+  for (const d of [DATA_DIR, ALERTS_DIR]) {
+    if (!fs.existsSync(d)) fs.mkdirSync(d, { recursive: true });
+  }
+}
+
+function readBaseline() {
+  if (!fs.existsSync(BASELINE_FILE)) {
+    const init = { count: 4989, asOf: '2026-05-19' };
+    fs.writeFileSync(BASELINE_FILE, JSON.stringify(init, null, 2) + '\n');
+    return init;
+  }
+  return JSON.parse(fs.readFileSync(BASELINE_FILE, 'utf8'));
+}
+
+function appendHistory(entry) {
+  fs.appendFileSync(HISTORY_FILE, JSON.stringify(entry) + '\n');
+}
+
+function writeAlert(payload) {
+  const ts = new Date().toISOString().replace(/[:.]/g, '-');
+  const file = path.join(ALERTS_DIR, `${ts}.txt`);
+  const body = [
+    'DW-MONITORING · BROKEN-IMAGE REGRESSION ALERT',
+    '='.repeat(60),
+    `Detected at:   ${payload.ts}`,
+    `Current count: ${payload.count}`,
+    `Baseline:      ${payload.baseline.count} (asOf ${payload.baseline.asOf})`,
+    `Growth:        ${(payload.ratio * 100 - 100).toFixed(2)}% above baseline`,
+    `Threshold:     +${((GROWTH_THRESHOLD - 1) * 100).toFixed(0)}%`,
+    '',
+    'Read-only monitoring. Investigate via:',
+    `  psql -d dw_unified -c "SELECT COUNT(*) FROM shopify_products sp`,
+    `    JOIN image_hashes ih ON sp.image_url = ih.url`,
+    `    WHERE ih.status != 'success';"`,
+    '',
+  ].join('\n');
+  fs.writeFileSync(file, body);
+  return { file, body };
+}
+
+async function trySmtpSend(subject, body) {
+  // Look for optional Gmail SMTP creds. If absent, return false (file-only mode).
+  const user = process.env.GMAIL_USER || process.env.GEORGE_GMAIL_USER;
+  const pass = process.env.GMAIL_APP_PASSWORD || process.env.GEORGE_PASSWORD;
+  if (!user || !pass) return { sent: false, reason: 'no SMTP creds' };
+
+  let nodemailer;
+  try {
+    nodemailer = require('nodemailer');
+  } catch (e) {
+    return { sent: false, reason: 'nodemailer not installed' };
+  }
+  const to = process.env.MONITORING_ALERT_TO || 'info@designerwallcoverings.com';
+  const transporter = nodemailer.createTransport({
+    host: 'smtp.gmail.com',
+    port: 465,
+    secure: true,
+    auth: { user, pass },
+  });
+  try {
+    await transporter.sendMail({ from: user, to, subject, text: body });
+    return { sent: true };
+  } catch (err) {
+    return { sent: false, reason: err.message };
+  }
+}
+
+async function main() {
+  ensureDirs();
+  const baseline = readBaseline();
+
+  // Explicit conn config so the pool ignores stray env from secrets-manager .env.
+  // Local pg uses peer auth as the OS user — no password needed.
+  const client = new Client({
+    host: process.env.DW_PGHOST || '/tmp',
+    user: process.env.DW_PGUSER || process.env.USER,
+    database: 'dw_unified',
+    password: undefined,
+  });
+  await client.connect();
+  let count;
+  try {
+    const res = await client.query(QUERY);
+    count = res.rows[0].broken;
+  } finally {
+    await client.end();
+  }
+
+  const ts = new Date().toISOString();
+  const ratio = count / baseline.count;
+  const entry = { ts, count, baseline: baseline.count, ratio: +ratio.toFixed(4) };
+  appendHistory(entry);
+
+  const breach = count >= baseline.count * GROWTH_THRESHOLD;
+  if (breach) {
+    const alert = writeAlert({ ts, count, baseline, ratio });
+    const smtp = await trySmtpSend(
+      `[dw-monitoring] broken-image count up ${(ratio * 100 - 100).toFixed(2)}% (now ${count})`,
+      alert.body,
+    );
+    console.log(
+      `[ALERT] count=${count} baseline=${baseline.count} ratio=${ratio.toFixed(4)} ` +
+        `alert=${alert.file} smtp=${smtp.sent ? 'sent' : 'skipped: ' + smtp.reason}`,
+    );
+  } else {
+    console.log(
+      `[ok] count=${count} baseline=${baseline.count} ratio=${ratio.toFixed(4)} ` +
+        `(threshold ${GROWTH_THRESHOLD.toFixed(2)})`,
+    );
+  }
+}
+
+main().catch((err) => {
+  console.error('[ERROR]', err.message);
+  process.exit(1);
+});

(oldest)  ·  back to Dw Monitoring  ·  rescrape-todo stubs for romo/arte/rlf/glitter (dw-stale-urls 8ef7977 →