`docs/blocked_sources.md`

64 lines
# Blocked sources — require Playwright / browser session

Two sources we wanted but cannot reach with plain HTTP under our compliance policy
(no captcha bypass, respect site terms).

## CA Secretary of State bizfile API

- Endpoint: `POST https://bizfileonline.sos.ca.gov/api/Records/businesssearch`
- Probe result: **403 Forbidden** with empty body on plain POST.
- Reason: their SPA injects a Google reCAPTCHA v3 token in the `x-recaptcha` header
  for every API call. Without a real browser executing their reCAPTCHA challenge,
  the gateway rejects the request.
- Path forward: a Playwright session that (a) loads the search page so reCAPTCHA
  initializes, (b) submits the form, (c) extracts the token from the live request,
  (d) replays via fetch. This is **inside compliance** because we are not bypassing
  reCAPTCHA — we are letting it run normally and just persisting the result.

## CA State Bar — Attorney Licensee Search

- Form page: `https://apps.calbar.ca.gov/attorney/LicenseeSearch/AdvancedSearch`
- Detail page: `https://apps.calbar.ca.gov/attorney/Licensee/Detail/{barNumber}`
- Probe result: every URL returns the **same 624 KB SPA shell** with an empty
  `#moduleAttorneySearchList` div.  Server-side rendering doesn't expose result data;
  the page loads a Telerik Kendo grid that fetches results client-side via an
  endpoint we couldn't locate from the static HTML/JS.
- Path forward: same Playwright pattern as bizfile — load `Detail/{n}` for each
  Bar # of interest, wait for the data-bound elements, scrape the rendered DOM.

## Why not now

Both imply adding Playwright + headless Chromium + DOM extractors + per-page rate
limits much stricter than for plain HTTP. That's a project unto itself; tonight's
run sticks with the sources that work cleanly via curl-grade requests:

- OpenStreetMap Overpass (already wired)
- Wikidata SPARQL (already wired)

When ready to do Playwright pass, see `src/ingest/calbar_playwright.ts` (TODO).

## Adjacent-city open-data portals

LA County firm coverage would benefit from the same kind of "Active Business Licenses"
dataset for cities outside LA City proper:

- Beverly Hills (~35K pop) — `beverlyhills-cityofbh.opendata.arcgis.com`
- Santa Monica (~92K) — `data-smgov.opendata.arcgis.com`
- Pasadena (~140K) — `data-cityofpasadena.opendata.arcgis.com`
- Long Beach (~462K) — `data-longbeach.opendata.arcgis.com`

All four serve their portal as a client-rendered SPA with no public Socrata/JSON API
and no DCAT feed at the standard `/api/feed/dcat-us/1.1` path. The dataset listings
load via JavaScript that calls Esri's internal Hub API with auth tokens generated
client-side.

Path forward (not tonight):
- per-portal investigation to find each city's "businesses" or "active business licenses"
  dataset, then call its FeatureServer query endpoint directly:
  `https://services.arcgis.com/<orgId>/ArcGIS/rest/services/<dataset>/FeatureServer/0/query?where=...`
- OR a Playwright pass that visits each portal, extracts dataset links from the rendered
  catalog, and discovers the FeatureServer URLs.

Beverly Hills + Santa Monica are likely low-yield (small cities, may not even publish
business license data). Long Beach + Pasadena are higher-value targets.