← back to Domain Sniper
ct-source-certspotter-brands.js + free-CT-source landscape doc
083e982f628168d258b6e2c59f5926d3bfe7f037 · 2026-05-12 16:04:43 -0700 · Steve Abrams
Per-brand polling blocked by free-tier FQDN-only restriction (403 not_allowed_by_plan
on bare keyword 'callr'). Module retained — drops in once CERTSPOTTER_API_KEY routed
via secrets-manager. README now maps all 6 free CT-source options with their failure
modes; near-term path is crt.sh polling (currently 502 transient, retry next tick).
Files touched
M README.mdA ct-source-certspotter-brands.js
Diff
commit 083e982f628168d258b6e2c59f5926d3bfe7f037
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Tue May 12 16:04:43 2026 -0700
ct-source-certspotter-brands.js + free-CT-source landscape doc
Per-brand polling blocked by free-tier FQDN-only restriction (403 not_allowed_by_plan
on bare keyword 'callr'). Module retained — drops in once CERTSPOTTER_API_KEY routed
via secrets-manager. README now maps all 6 free CT-source options with their failure
modes; near-term path is crt.sh polling (currently 502 transient, retry next tick).
---
README.md | 22 +++++-
ct-source-certspotter-brands.js | 171 ++++++++++++++++++++++++++++++++++++++++
2 files changed, 191 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 1d0f4e3..1a2dac0 100644
--- a/README.md
+++ b/README.md
@@ -55,9 +55,27 @@ Spot-checks via DoH at increasing intervals to find where the snipe window close
| ~24h | full batch | — |
| ~48h | full batch (final) | — |
-## Known issues
+## Known issues — public CT-log infrastructure is fragile
-- **2026-05-12 22:54 UTC** — `wss://certstream.calidog.io/` connects but closes with code 1000 after ~60s without sending any messages. Both `watch-ct.js`, `brand-typo-watcher.js`, and `dashboard.js` are affected. The public Calidog CertStream server has been documented as flaky for 18+ months. **Fallback to ship next:** poll `https://api.certspotter.com/v1/issuances?after=<last_id>&expand=dns_names&expand=issuer` every 30s (100 req/hr free tier — well within budget at 120 req/hr × 1 endpoint). Cursor state in `data/certspotter-cursor.json`. Affected scripts already have reconnect-with-backoff loops, so a unified fallback can drop in as a sibling `ct-source-certspotter.js` module that both consumers import.
+State of the world for **free** real-time CT-log access, as of 2026-05-12 ~23:05 UTC:
+
+| Source | Status | Why it doesn't (currently) work |
+| --- | --- | --- |
+| `wss://certstream.calidog.io/` | Connects, closes after ~60s with code 1000, **0 messages**. Known flaky for 18+ months. | Public Calidog server depopulated; the official replacement is "host your own". |
+| Cert Spotter free `/v1/issuances` (anonymous firehose) | 400 — requires `domain=` per request | No anonymous firehose tier. |
+| Cert Spotter free `/v1/issuances?domain=<keyword>` (per-brand) | 403 `not_allowed_by_plan` — domain must be FQDN | Bare keywords (`callr`) rejected; FQDNs (`callr.com`) only match issuances *for* that exact domain, useless for typo-squat defense. |
+| crt.sh `?q=<kw>&output=json` | 502 nginx (transient) | crt.sh frequently overloaded; not reliable as a primary source. |
+| Cert Spotter paid (`Authorization: Bearer`) | Untested — would need API key via secrets-manager | Cheapest plan ~$/mo. |
+| Run own CertStream server | Heavy (Go + indexed Postgres of CT logs) | Last resort. |
+
+**Code shipped covering this landscape:**
+- `ct-source-certspotter.js` — anonymous-firehose attempt (blocked by free-tier policy; ready for API key)
+- `ct-source-certspotter-brands.js` — per-brand polling attempt (blocked by FQDN-only restriction)
+- Both modules emit the CertStream payload shape, so once a working source comes online they drop in to `watch-ct.js` / `brand-typo-watcher.js` / `dashboard.js` via a `CT_SOURCE` env switch.
+
+**Best near-term path:** retry crt.sh with backoff (it works most of the time, just not right now), and build a `ct-source-crtsh.js` polling module. This is the only free option that supports the keyword/substring matching we actually need for brand defense.
+
+Meanwhile the **honeypot + DoH spot-check** path (the actual aggregator-attribution experiment) is unaffected — that runs on Cloudflare 1.1.1.1 DoH and Node `child_process whois`, neither of which depends on CT logs.
## YOLO loop notes
diff --git a/ct-source-certspotter-brands.js b/ct-source-certspotter-brands.js
new file mode 100644
index 0000000..7d14465
--- /dev/null
+++ b/ct-source-certspotter-brands.js
@@ -0,0 +1,171 @@
+// ct-source-certspotter-brands.js — per-brand polling wrapper around the
+// Cert Spotter free tier. Cycles through Steve's brand keywords, hitting
+// /v1/issuances?domain=<kw> for each. Emits the same cert_update events as
+// ct-source-certspotter.js so existing watchers (brand-typo-watcher, etc.)
+// consume without changes.
+//
+// Why: Cert Spotter free tier requires domain= per request — no firehose.
+// 100 req/hr budget. 18 brands × 1 poll/12min = 90 req/hr → safe.
+//
+// Cursor is per-brand, persisted to data/certspotter-brand-cursors.json.
+//
+// Usage as library:
+// const src = require('./ct-source-certspotter-brands');
+// const s = src.start({ keywords: [...], cycleMs: 12*60_000, onEvent });
+//
+// Standalone:
+// node ct-source-certspotter-brands.js
+
+const https = require('https');
+const fs = require('fs');
+const path = require('path');
+const { EventEmitter } = require('events');
+
+const DATA_DIR = path.join(__dirname, 'data');
+const CURSOR_FILE = path.join(DATA_DIR, 'certspotter-brand-cursors.json');
+const API_HOST = 'api.certspotter.com';
+const API_PATH = '/v1/issuances';
+const UA = 'domain-sniper/0.1 (defensive brand monitoring)';
+
+const DEFAULT_BRAND_KEYWORDS = [
+ 'designerwallcoverings', 'philipperomano', 'venturacorridor', 'venturaclaw',
+ 'starsofdesign', 'bubbesblock', 'wholivedthere', 'agentabrams',
+ 'nationalpaperhangers', 'novasuede', 'callr', 'butlr',
+ 'flockedwallpaper', 'grasscloth', 'glassbeaded',
+ 'lacountyeats', 'venturablvd', 'nosarabeachfrontrentals',
+];
+
+function loadCursors() {
+ try { return JSON.parse(fs.readFileSync(CURSOR_FILE, 'utf8')); }
+ catch { return {}; }
+}
+function saveCursors(c) {
+ fs.mkdirSync(DATA_DIR, { recursive: true });
+ fs.writeFileSync(CURSOR_FILE, JSON.stringify(c, null, 2));
+}
+
+function fetchBrand(domain, after, apiKey) {
+ return new Promise((resolve, reject) => {
+ const qs = new URLSearchParams();
+ qs.set('domain', domain);
+ qs.set('include_subdomains', 'true');
+ if (after) qs.set('after', after);
+ qs.append('expand', 'dns_names');
+ qs.append('expand', 'issuer');
+ const headers = { 'User-Agent': UA, 'Accept': 'application/json' };
+ if (apiKey) headers['Authorization'] = 'Bearer ' + apiKey;
+ const req = https.request({
+ host: API_HOST, path: API_PATH + '?' + qs.toString(), method: 'GET', headers, timeout: 15_000,
+ }, (res) => {
+ let buf = '';
+ res.on('data', (c) => buf += c);
+ res.on('end', () => {
+ if (res.statusCode === 200) {
+ try { resolve({ status: 200, body: JSON.parse(buf) }); }
+ catch (e) { reject(new Error('parse: ' + e.message)); }
+ } else {
+ resolve({ status: res.statusCode, body: buf, retryAfter: res.headers['retry-after'] });
+ }
+ });
+ });
+ req.on('error', reject);
+ req.on('timeout', () => { req.destroy(new Error('timeout')); });
+ req.end();
+ });
+}
+
+function start({
+ keywords = DEFAULT_BRAND_KEYWORDS,
+ cycleMs = 12 * 60 * 1000, // full cycle: each keyword polled once per 12min
+ apiKey = process.env.CERTSPOTTER_API_KEY,
+ onEvent,
+ onStatus,
+} = {}) {
+ const ee = new EventEmitter();
+ if (onEvent) ee.on('certificate_update', onEvent);
+ if (onStatus) ee.on('status', onStatus);
+
+ const cursors = loadCursors();
+ const slotMs = Math.max(2000, Math.floor(cycleMs / Math.max(1, keywords.length)));
+ let stopped = false;
+ let idx = 0;
+ let consecutiveErrs = 0;
+
+ ee.emit('status', { type: 'started', keywords: keywords.length, cycleMs, slotMs });
+
+ async function tick() {
+ if (stopped) return;
+ const kw = keywords[idx % keywords.length];
+ idx++;
+ const after = cursors[kw] || null;
+ try {
+ const r = await fetchBrand(kw, after, apiKey);
+ if (r.status === 429 || r.status === 503) {
+ const waitMs = parseInt(r.retryAfter, 10) * 1000 || Math.min(120_000, 5000 * Math.pow(2, consecutiveErrs));
+ consecutiveErrs++;
+ ee.emit('status', { type: 'throttled', kw, status: r.status, retryInMs: waitMs });
+ setTimeout(tick, waitMs);
+ return;
+ }
+ if (r.status !== 200) {
+ consecutiveErrs++;
+ ee.emit('status', { type: 'http_error', kw, status: r.status, snippet: String(r.body).slice(0, 160) });
+ setTimeout(tick, Math.min(120_000, 5000 * Math.pow(2, Math.min(consecutiveErrs, 4))));
+ return;
+ }
+ consecutiveErrs = 0;
+ const items = Array.isArray(r.body) ? r.body : [];
+ let newCursor = after;
+ let emitted = 0;
+ for (const it of items) {
+ const domains = (it.dns_names || []).filter(Boolean);
+ if (!domains.length) continue;
+ ee.emit('certificate_update', {
+ message_type: 'certificate_update',
+ data: {
+ cert_index: it.id,
+ brand_match: kw,
+ leaf_cert: {
+ all_domains: domains,
+ issuer: it.issuer && (it.issuer.name || it.issuer.friendly_name || null),
+ not_before: it.not_before || null,
+ not_after: it.not_after || null,
+ },
+ },
+ });
+ emitted++;
+ if (!newCursor || it.id > newCursor) newCursor = it.id;
+ }
+ if (newCursor && newCursor !== after) {
+ cursors[kw] = newCursor;
+ saveCursors(cursors);
+ }
+ ee.emit('status', { type: 'tick', kw, count: items.length, emitted, cursor: cursors[kw] || null });
+ } catch (e) {
+ consecutiveErrs++;
+ ee.emit('status', { type: 'error', kw, msg: e.message });
+ }
+ if (!stopped) setTimeout(tick, slotMs);
+ }
+ tick();
+
+ return {
+ on: (...a) => ee.on(...a),
+ stop: () => { stopped = true; ee.emit('status', { type: 'stopped' }); },
+ cursors: () => ({ ...cursors }),
+ };
+}
+
+module.exports = { start, DEFAULT_BRAND_KEYWORDS };
+
+if (require.main === module) {
+ console.log(`[${new Date().toISOString()}] ct-source-certspotter-brands starting`);
+ const s = start({
+ onEvent: (e) => {
+ const d = e.data.leaf_cert.all_domains;
+ console.log(` [${e.data.brand_match}] ${d.slice(0, 3).join(', ')}${d.length > 3 ? ` …+${d.length - 3}` : ''}`);
+ },
+ onStatus: (st) => console.log(` [status] ${JSON.stringify(st)}`),
+ });
+ process.on('SIGINT', () => { s.stop(); process.exit(0); });
+}
← 7a91f94 ct-source-certspotter.js: skeleton module + standalone smoke
·
back to Domain Sniper
·
ct-source-ctlog.js: WORKING direct CT-log polling — bypasses b00757f →