← back to Ventura Corridor
feat(enrich): website-discovery resilience + category prioritization
828ab33fca7d61685382bbe76bc71cc7599d7763 · 2026-05-07 11:26:56 -0700 · SteveStudio2
Brave free tier hit its $15/mo cap mid-tick. Built two pieces of
resilience for when the quota resets and for in-the-meantime work:
1. website_via_brave.ts — added --high_value category filter. Default
queue now skips "office: company" / "office: estate_agent" /
"office: lawyer" shells (which are 70 percent of the corridor table
and ~0 percent of the websites) and prioritizes restaurants, cafes,
bars, retail, beauty, fitness, dentists, hotels, pet shops. Big
hit-rate boost when Brave comes back online.
2. website_via_ddg.ts — new DuckDuckGo HTML fallback. No API key, no
monthly quota. Same domain-credibility logic as Brave/Exa.
Smoke-tested 1/15 hits before DDG started returning 202 anti-abuse
challenges; useful for bursts of <30 queries when nothing else
works. Wired as `enrich:websites:ddg` npm script.
3. STATUS_NEWS_PIPELINE.md — frozen snapshot of what shipped and what
unblocks next. Highest-leverage move is Steve provisioning a
FOURSQUARE_API_KEY (free 100K/mo, returns websites explicitly) so
`npm run ingest:foursquare` can land thousands of new rows with URLs.
No data corruption: failed Brave attempts left the rows untouched
(the script only UPDATEs on a winner).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Files touched
A STATUS_NEWS_PIPELINE.mdM package.jsonM src/enrich/website_via_brave.tsA src/enrich/website_via_ddg.ts
Diff
commit 828ab33fca7d61685382bbe76bc71cc7599d7763
Author: SteveStudio2 <stevestudio2@SteveStacStudio.lan>
Date: Thu May 7 11:26:56 2026 -0700
feat(enrich): website-discovery resilience + category prioritization
Brave free tier hit its $15/mo cap mid-tick. Built two pieces of
resilience for when the quota resets and for in-the-meantime work:
1. website_via_brave.ts — added --high_value category filter. Default
queue now skips "office: company" / "office: estate_agent" /
"office: lawyer" shells (which are 70 percent of the corridor table
and ~0 percent of the websites) and prioritizes restaurants, cafes,
bars, retail, beauty, fitness, dentists, hotels, pet shops. Big
hit-rate boost when Brave comes back online.
2. website_via_ddg.ts — new DuckDuckGo HTML fallback. No API key, no
monthly quota. Same domain-credibility logic as Brave/Exa.
Smoke-tested 1/15 hits before DDG started returning 202 anti-abuse
challenges; useful for bursts of <30 queries when nothing else
works. Wired as `enrich:websites:ddg` npm script.
3. STATUS_NEWS_PIPELINE.md — frozen snapshot of what shipped and what
unblocks next. Highest-leverage move is Steve provisioning a
FOURSQUARE_API_KEY (free 100K/mo, returns websites explicitly) so
`npm run ingest:foursquare` can land thousands of new rows with URLs.
No data corruption: failed Brave attempts left the rows untouched
(the script only UPDATEs on a winner).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
STATUS_NEWS_PIPELINE.md | 35 ++++++++
package.json | 3 +-
src/enrich/website_via_brave.ts | 46 +++++++++-
src/enrich/website_via_ddg.ts | 183 ++++++++++++++++++++++++++++++++++++++++
4 files changed, 263 insertions(+), 4 deletions(-)
diff --git a/STATUS_NEWS_PIPELINE.md b/STATUS_NEWS_PIPELINE.md
new file mode 100644
index 0000000..4a19b76
--- /dev/null
+++ b/STATUS_NEWS_PIPELINE.md
@@ -0,0 +1,35 @@
+# News pipeline status — 2026-05-07
+
+## Current data
+- 17,462 businesses · 153 with websites (0.9 %)
+- 76 news articles scraped across 46 distinct businesses
+- 0 summarized (Mac1 saturated by `magazine:gen 2000`; queued for 5:30am launchd drain)
+- /news.html magazine surface live behind basic-auth at `:9780/news.html`
+
+## Bottleneck — website coverage
+
+153 of 17,462 businesses have a website. **All 153 came from OSM Overpass.** The 17,238 LA BTRC rows have zero websites because BTRC is a tax-record dump, not a storefront list (most BTRC entries are LLCs, individuals, or PO boxes that won't ever have websites).
+
+Without more websites the news scraper has nothing to scrape.
+
+## Discovery sources audit
+
+| source | state | next step |
+|---|---|---|
+| **Brave Search API** | Free tier exhausted (USAGE_LIMIT_EXCEEDED, $15/mo cap hit) | Resets 1st of next month. `enrich:websites` already wired w/ `--high_value` category filter. |
+| **DuckDuckGo HTML** (no key) | Returns 202 anti-abuse challenge on bulk queries — `enrich:websites:ddg` works for ~1 query/min before rate-limit kicks in | Useful only for one-offs; unsuitable for the 17k backlog |
+| **Exa** | No `EXA_API_KEY` in any env file | Steve to provision via /secrets if desired |
+| **HERE Discover** | `HERE_API_KEY=` empty in .env (free tier 30K req/mo available) | Steve to register at platform.here.com |
+| **Foursquare Places** | `FOURSQUARE_API_KEY` not set (free tier 100K req/mo available) | Steve to register at location.foursquare.com |
+| **OSM Overpass** | Already ingested (224 rows · 154 with sites) | Could re-run with looser bbox / category filter |
+
+## What I shipped tonight (tick 39)
+
+1. **`enrich:websites --high_value`** — Brave script now filters to consumer-facing OSM categories (restaurants, retail, beauty, fitness, dentists, hotels, etc.) instead of churning through `office:` shells. Will materially boost hit-rate when Brave quota resets.
+2. **`enrich:websites:ddg`** — new no-key DuckDuckGo HTML fallback. 1/15 hit-rate in smoke (Sephora) before DDG started returning 202 challenges. Code is sound; bottleneck is DDG's anti-abuse, not our parsing.
+
+## What unblocks next
+
+If Steve provisions HERE or Foursquare keys (free tiers cover 17k queries with room to spare), `npm run ingest:here` and `npm run ingest:foursquare` will land thousands of new rows with websites + categories — wiring Foursquare in particular is the highest-leverage move because their `places/search` returns the business website explicitly.
+
+Until then: news pipeline is functioning on the 153 OSM-sourced sites + whatever the nightly scrape finds in their blogs.
diff --git a/package.json b/package.json
index 4e7acb9..88a45b3 100644
--- a/package.json
+++ b/package.json
@@ -37,7 +37,8 @@
"magazine:auto-dedupe": "tsx src/jobs/auto_dedupe_headlines.ts",
"magazine:corpus-summary": "tsx src/jobs/morning_corpus_summary.ts",
"crawl:news": "tsx src/crawl/news_scraper.ts",
- "news:summarize": "tsx src/jobs/summarize_news.ts"
+ "news:summarize": "tsx src/jobs/summarize_news.ts",
+ "enrich:websites:ddg": "tsx src/enrich/website_via_ddg.ts"
},
"dependencies": {
"@types/multer": "^2.1.0",
diff --git a/src/enrich/website_via_brave.ts b/src/enrich/website_via_brave.ts
index 1a34be8..470cb8a 100644
--- a/src/enrich/website_via_brave.ts
+++ b/src/enrich/website_via_brave.ts
@@ -117,7 +117,44 @@ async function brave(q: string): Promise<string[]> {
interface Row { id: number; name: string; city: string | null; }
-async function pick(limit: number): Promise<Row[]> {
+// Categories most likely to have public-facing websites we can index +
+// ingest for the news magazine. Excludes "office: company" / "office:
+// estate_agent" / "office: lawyer" which are dominated by single-person
+// LLCs and back-office shells where the hit rate is near zero.
+const HIGH_VALUE_CATEGORY_PREFIXES = [
+ 'amenity: restaurant', 'amenity: fast_food', 'amenity: cafe',
+ 'amenity: bar', 'amenity: pub', 'amenity: nightclub',
+ 'shop: hairdresser', 'shop: clothes', 'shop: shoes', 'shop: jewelry',
+ 'shop: bakery', 'shop: butcher', 'shop: coffee', 'shop: tea',
+ 'shop: cosmetics', 'shop: beauty', 'shop: tattoo', 'shop: pet',
+ 'shop: bicycle', 'shop: bookstore', 'shop: convenience',
+ 'shop: department_store', 'shop: gift', 'shop: hardware',
+ 'shop: hifi', 'shop: optician', 'shop: photo', 'shop: stationery',
+ 'shop: supermarket', 'shop: variety_store',
+ 'leisure: fitness_centre', 'leisure: dance', 'leisure: bowling_alley',
+ 'amenity: dentist', 'amenity: pharmacy', 'amenity: veterinary',
+ 'tourism: hotel', 'tourism: motel', 'tourism: apartment',
+ 'amenity: car_wash', 'shop: car_parts',
+];
+
+async function pick(limit: number, categoryFilter: 'high_value' | 'all'): Promise<Row[]> {
+ if (categoryFilter === 'high_value') {
+ // ANY-prefix-match against the curated whitelist. Skips "office: ..."
+ // shells which are 70% of the corridor and ~0% of the websites.
+ const conds = HIGH_VALUE_CATEGORY_PREFIXES.map((_, i) => `category ILIKE $${i + 2}`).join(' OR ');
+ const params: any[] = [limit, ...HIGH_VALUE_CATEGORY_PREFIXES.map(p => p + '%')];
+ const r = await query<Row>(`
+ SELECT id, name, city FROM businesses
+ WHERE on_corridor AND website IS NULL
+ AND length(name) >= 5
+ AND name NOT LIKE '% LLC'
+ AND name NOT ILIKE 'POST OFFICE BOX%'
+ AND (${conds})
+ ORDER BY id
+ LIMIT $1
+ `, params);
+ return r.rows;
+ }
const r = await query<Row>(`
SELECT id, name, city FROM businesses
WHERE on_corridor AND source = 'la_btrc' AND website IS NULL
@@ -137,8 +174,11 @@ async function main() {
}
const argLimit = process.argv.find((a) => a.startsWith('--limit='));
const limit = argLimit ? parseInt(argLimit.split('=')[1], 10) : 20;
- const queue = await pick(limit);
- console.log(`[brave] enriching ${queue.length} rows · throttling 1.1s/query`);
+ // Default to high-value categories (consumer-facing) — restaurants, retail,
+ // beauty, fitness. Pass --all to fall back to every BTRC row.
+ const categoryFilter = process.argv.includes('--all') ? 'all' : 'high_value';
+ const queue = await pick(limit, categoryFilter);
+ console.log(`[brave] enriching ${queue.length} rows · category=${categoryFilter} · throttling 1.1s/query`);
let ok = 0, miss = 0, err = 0;
for (const row of queue) {
const q = `"${row.name}" Ventura Blvd ${row.city || 'Sherman Oaks'}`;
diff --git a/src/enrich/website_via_ddg.ts b/src/enrich/website_via_ddg.ts
new file mode 100644
index 0000000..bed7c44
--- /dev/null
+++ b/src/enrich/website_via_ddg.ts
@@ -0,0 +1,183 @@
+/**
+ * Website discovery via DuckDuckGo HTML endpoint (html.duckduckgo.com/html).
+ *
+ * No API key, no monthly quota — polite alternative when Brave's free
+ * tier is exhausted. Lower precision than Brave's API (parses raw HTML),
+ * but we apply the same domain-credibility filter so false positives
+ * collapse to "no winner found".
+ *
+ * Run:
+ * npm run enrich:websites:ddg # default 30, high-value categories
+ * tsx src/enrich/website_via_ddg.ts -- --limit=100
+ * tsx src/enrich/website_via_ddg.ts -- --limit=50 --all # any BTRC row
+ *
+ * Polite: 2.5s gap between queries, 8s timeout, custom UA.
+ */
+import * as cheerio from 'cheerio';
+import { query, pool } from '../../db/pool.ts';
+
+const UA = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 ventura-corridor/0.1 (website-discovery; +https://leads.venturaclaw.com)';
+const GAP_MS = parseInt(process.env.DDG_GAP_MS || '2500', 10);
+const TIMEOUT_MS = parseInt(process.env.DDG_TIMEOUT_MS || '8000', 10);
+
+const BLOCK = new Set([
+ // Social
+ 'yelp.com', 'facebook.com', 'instagram.com', 'twitter.com', 'x.com',
+ 'linkedin.com', 'tiktok.com', 'youtube.com', 'pinterest.com', 'reddit.com',
+ // Maps / aggregators / dirs
+ 'mapquest.com', 'google.com', 'maps.google.com', 'foursquare.com',
+ 'tripadvisor.com', 'opentable.com', 'doordash.com', 'ubereats.com',
+ 'grubhub.com', 'wikipedia.org', 'wikidata.org',
+ 'angi.com', 'angieslist.com', 'thumbtack.com', 'alignable.com',
+ 'yellowpages.com', 'whitepages.com', 'apollo.io', 'zoominfo.com',
+ 'crunchbase.com', 'spokeo.com', 'fastpeoplesearch.com',
+ // Real estate / business records
+ 'zillow.com', 'realtor.com', 'redfin.com', 'apartments.com',
+ 'bbb.org', 'manta.com', 'bizapedia.com', 'opencorporates.com',
+ 'dnb.com', 'corporationwiki.com',
+ // Reviews / health / legal
+ 'avvo.com', 'lawyers.com', 'martindale.com', 'findlaw.com',
+ 'healthgrades.com', 'zocdoc.com', 'vitals.com', 'webmd.com',
+ // Jobs / news (we want the biz site, not a press hit)
+ 'glassdoor.com', 'indeed.com', 'duckduckgo.com',
+ // Gov
+ 'lacity.org', 'data.lacity.org', 'lavote.gov', 'lacounty.gov', 'dca.lacity.org'
+]);
+
+function host(u: string): string | null {
+ try { return new URL(u).hostname.replace(/^www\./, '').toLowerCase(); } catch { return null; }
+}
+
+function credible(u: string): boolean {
+ const h = host(u);
+ if (!h) return false;
+ if (BLOCK.has(h)) return false;
+ // Block any subdomain of a blocked root.
+ for (const root of BLOCK) if (h === root || h.endsWith('.' + root)) return false;
+ if (h.endsWith('.gov') || h.endsWith('.gov.us')) return false;
+ return true;
+}
+
+function nameTokens(name: string): string[] {
+ return name.toLowerCase().replace(/[^a-z0-9 ]/g, ' ').split(/\s+/).filter(t => t.length >= 3 && !['the','and','for','inc','llc','corp','of'].includes(t));
+}
+
+function domainMatchesName(host: string, name: string): boolean {
+ const stem = host.split('.').slice(0, -1).join('').replace(/[^a-z0-9]/g, '');
+ const tokens = nameTokens(name);
+ if (!tokens.length) return false;
+ // Strict: at least one name-token of >=4 chars appears in the domain stem
+ return tokens.some(t => t.length >= 4 && stem.includes(t));
+}
+
+const HIGH_VALUE_CATEGORY_PREFIXES = [
+ 'amenity: restaurant', 'amenity: fast_food', 'amenity: cafe',
+ 'amenity: bar', 'amenity: pub', 'amenity: nightclub',
+ 'shop: hairdresser', 'shop: clothes', 'shop: shoes', 'shop: jewelry',
+ 'shop: bakery', 'shop: butcher', 'shop: coffee', 'shop: tea',
+ 'shop: cosmetics', 'shop: beauty', 'shop: tattoo', 'shop: pet',
+ 'shop: bicycle', 'shop: bookstore', 'shop: convenience',
+ 'shop: department_store', 'shop: gift', 'shop: hardware',
+ 'shop: hifi', 'shop: optician', 'shop: photo', 'shop: stationery',
+ 'shop: supermarket', 'shop: variety_store',
+ 'leisure: fitness_centre', 'leisure: dance', 'leisure: bowling_alley',
+ 'amenity: dentist', 'amenity: pharmacy', 'amenity: veterinary',
+ 'tourism: hotel', 'tourism: motel', 'tourism: apartment',
+ 'amenity: car_wash', 'shop: car_parts',
+];
+
+interface Row { id: number; name: string; city: string | null }
+
+async function pick(limit: number, mode: 'high_value' | 'all'): Promise<Row[]> {
+ if (mode === 'high_value') {
+ const conds = HIGH_VALUE_CATEGORY_PREFIXES.map((_, i) => `category ILIKE $${i + 2}`).join(' OR ');
+ const params: any[] = [limit, ...HIGH_VALUE_CATEGORY_PREFIXES.map(p => p + '%')];
+ const r = await query<Row>(`
+ SELECT id, name, city FROM businesses
+ WHERE on_corridor AND website IS NULL
+ AND length(name) >= 5
+ AND name NOT LIKE '% LLC'
+ AND name NOT ILIKE 'POST OFFICE BOX%'
+ AND (${conds})
+ ORDER BY id
+ LIMIT $1
+ `, params);
+ return r.rows;
+ }
+ const r = await query<Row>(`
+ SELECT id, name, city FROM businesses
+ WHERE on_corridor AND website IS NULL
+ AND length(name) >= 5
+ AND name NOT LIKE '% LLC'
+ AND name NOT ILIKE 'POST OFFICE BOX%'
+ ORDER BY id LIMIT $1
+ `, [limit]);
+ return r.rows;
+}
+
+async function ddgSearch(q: string): Promise<string[]> {
+ // html.duckduckgo.com renders <a class="result__a"> with the target URL in
+ // the href (as a redirect through /l/?uddg=...). We unwrap that.
+ const url = 'https://html.duckduckgo.com/html/?q=' + encodeURIComponent(q);
+ const res = await fetch(url, {
+ headers: { 'User-Agent': UA, 'Accept': 'text/html', 'Accept-Language': 'en-US,en;q=0.9' },
+ redirect: 'follow',
+ signal: AbortSignal.timeout(TIMEOUT_MS)
+ });
+ if (!res.ok) throw new Error(`ddg ${res.status}`);
+ const html = await res.text();
+ const $ = cheerio.load(html);
+ const out: string[] = [];
+ $('a.result__a').each((_, el) => {
+ const href = String($(el).attr('href') || '');
+ if (!href) return;
+ // DDG sometimes returns absolute URLs, sometimes wraps them in /l/?uddg=...
+ let abs = href;
+ try {
+ const u = new URL(href, 'https://duckduckgo.com');
+ const uddg = u.searchParams.get('uddg');
+ if (uddg) abs = decodeURIComponent(uddg);
+ } catch {}
+ if (/^https?:\/\//i.test(abs)) out.push(abs);
+ });
+ return out.slice(0, 12);
+}
+
+async function main() {
+ const argLimit = process.argv.find(a => a.startsWith('--limit='));
+ const limit = argLimit ? parseInt(argLimit.split('=')[1], 10) : 30;
+ const mode = process.argv.includes('--all') ? 'all' : 'high_value';
+ const queue = await pick(limit, mode);
+ console.log(`[ddg] enriching ${queue.length} rows · category=${mode} · throttling ${GAP_MS}ms/query`);
+ let ok = 0, miss = 0, err = 0;
+ for (let i = 0; i < queue.length; i++) {
+ const row = queue[i];
+ const q = `"${row.name}" Ventura Blvd ${row.city || 'Sherman Oaks'}`;
+ try {
+ const urls = await ddgSearch(q);
+ const winner = urls.find(u => {
+ if (!credible(u)) return false;
+ const h = host(u);
+ return h && domainMatchesName(h, row.name);
+ });
+ if (winner) {
+ const h = host(winner);
+ const website = 'https://' + h;
+ await query(`UPDATE businesses SET website = $1, last_seen_at = now() WHERE id = $2`, [website, row.id]);
+ ok++;
+ console.log(` [${String(i + 1).padStart(3)}/${queue.length}] ✓ ${row.name.padEnd(40).slice(0, 40)} → ${website}`);
+ } else {
+ miss++;
+ console.log(` [${String(i + 1).padStart(3)}/${queue.length}] · ${row.name.padEnd(40).slice(0, 40)} (no credible match)`);
+ }
+ } catch (e: any) {
+ err++;
+ console.log(` [${String(i + 1).padStart(3)}/${queue.length}] ⚠ ${row.name.slice(0, 40)} ${e.message?.slice(0, 60)}`);
+ }
+ await new Promise(r => setTimeout(r, GAP_MS));
+ }
+ console.log(`[ddg] done · ${ok} found · ${miss} no-result · ${err} errors`);
+ await pool.end();
+}
+
+main().catch(e => { console.error(e); pool.end(); process.exit(1); });
← 0a78f9e feat(news): nightly launchd jobs + patient summarizer backof
·
back to Ventura Corridor
·
feat(news): /this-week.html editorial digest + /api/news/dig e45334e →