`STATUS_NEWS_PIPELINE.md`

36 lines
# News pipeline status — 2026-05-07

## Current data
- 17,462 businesses · 153 with websites (0.9 %)
- 76 news articles scraped across 46 distinct businesses
- 0 summarized (Mac1 saturated by `magazine:gen 2000`; queued for 5:30am launchd drain)
- /news.html magazine surface live behind basic-auth at `:9780/news.html`

## Bottleneck — website coverage

153 of 17,462 businesses have a website. **All 153 came from OSM Overpass.** The 17,238 LA BTRC rows have zero websites because BTRC is a tax-record dump, not a storefront list (most BTRC entries are LLCs, individuals, or PO boxes that won't ever have websites).

Without more websites the news scraper has nothing to scrape.

## Discovery sources audit

| source | state | next step |
|---|---|---|
| **Brave Search API** | Free tier exhausted (USAGE_LIMIT_EXCEEDED, $15/mo cap hit) | Resets 1st of next month. `enrich:websites` already wired w/ `--high_value` category filter. |
| **DuckDuckGo HTML** (no key) | Returns 202 anti-abuse challenge on bulk queries — `enrich:websites:ddg` works for ~1 query/min before rate-limit kicks in | Useful only for one-offs; unsuitable for the 17k backlog |
| **Exa** | No `EXA_API_KEY` in any env file | Steve to provision via /secrets if desired |
| **HERE Discover** | `HERE_API_KEY=` empty in .env (free tier 30K req/mo available) | Steve to register at platform.here.com |
| **Foursquare Places** | `FOURSQUARE_API_KEY` not set (free tier 100K req/mo available) | Steve to register at location.foursquare.com |
| **OSM Overpass** | Already ingested (224 rows · 154 with sites) | Could re-run with looser bbox / category filter |

## What I shipped tonight (tick 39)

1. **`enrich:websites --high_value`** — Brave script now filters to consumer-facing OSM categories (restaurants, retail, beauty, fitness, dentists, hotels, etc.) instead of churning through `office:` shells. Will materially boost hit-rate when Brave quota resets.
2. **`enrich:websites:ddg`** — new no-key DuckDuckGo HTML fallback. 1/15 hit-rate in smoke (Sephora) before DDG started returning 202 challenges. Code is sound; bottleneck is DDG's anti-abuse, not our parsing.

## What unblocks next

If Steve provisions HERE or Foursquare keys (free tiers cover 17k queries with room to spare), `npm run ingest:here` and `npm run ingest:foursquare` will land thousands of new rows with websites + categories — wiring Foursquare in particular is the highest-leverage move because their `places/search` returns the business website explicitly.

Until then: news pipeline is functioning on the 153 OSM-sourced sites + whatever the nightly scrape finds in their blogs.