`docs/CRE_PR_DATA_SOURCES.md`

70 lines
# CRE PR Intelligence — Data Sources

Source hierarchy (highest trust first), what is implemented, and the exact terms notes
recorded per source. Every stored fact carries a `pr_sources` row (URL, limited excerpt,
retrieval date, usage note, primary/secondary flag) — full page copies are never stored.

## 1. Organization websites (`website` adapter — PRIMARY)
Newsroom / press / media-contact / team / leadership / contact pages, press-release
contact blocks, author bios. Live-fetched with a declared User-Agent, robots.txt honored
(cached per host, 1h), throttled ≥2s per host, ≤5 pages per profile pass.
**Terms note stored:** "Public organization website; facts extracted, limited excerpts stored."

## 2. Official government / regulatory datasets (`registry`, `edgar` — PRIMARY)
- **FDIC BankFind Suite API** (banks): open federal API, key-free.
  Endpoint (verified 2026-07-30): `https://api.fdic.gov/banks/institutions`
  (the old `banks.data.fdic.gov` host now 301s here). Returns 116 active CA banks with
  name, city, county, website. Public-domain reuse.
- **SEC EDGAR** (`edgar` adapter): public-domain; requires a declared User-Agent
  (`PR_EDGAR_UA`, verified — anonymous requests 403) and ≤10 req/s per the SEC
  fair-access policy. `company_tickers.json` + `data.sec.gov/submissions/CIK*.json`.
- **NCUA credit unions:** the mapping API endpoint we probed returns the SPA HTML, not
  JSON — the adapter reports this and the supported path is the **NCUA directory CSV
  download → CSV import** (ncua.gov → Analysis → Credit Union & Corporate Call Report
  Data). Status: awaiting the CSV (or a corrected endpoint).
- **California licensing** (DRE brokers, DFPI lenders/escrow, CDI title insurers) and
  **Arizona licensing** (ADRE, DIFI): these publish lookup tools and periodic data files
  whose export/reuse terms vary per dataset. Policy: **verify the specific dataset's
  terms, download manually, load via CSV import with the terms recorded in the usage
  note.** No scraping of licensing portals.

## 3. Association / chapter pages (`website` adapter on association orgs)
Leadership, sponsor, committee pages of AIR CRE, NAIOP, ULI, CREW, BOMA, CBPA, etc.
Treated as org-website sources on those organizations' records.

## 4. Public press releases (`rss` adapter — PRIMARY for spokesperson facts)
Organization newsroom RSS/Atom feeds; headlines + ≤300-char summaries stored.

## 5. Public news articles naming spokespeople (secondary)
Stored as `news_article` sources with excerpt + publisher; used for corroboration.

## 6. Authorized search APIs (`search` adapter — secondary, low confidence)
Exa (`PR_EXA_API_KEY`) or Brave (`PR_BRAVE_API_KEY`). Used for the query matrix and for
locating **publicly indexed LinkedIn URLs**. Result snippets are stored at confidence
~40 and are **never** promoted to verified titles without corroboration from source
classes 1–5. We never scrape search-result pages.

## 7. User-provided exports / licensed data (`import` paths)
CSV/JSON/LinkedIn-URL/contact-list imports with column mapping, preview, validation,
dry-run, duplicate detection, and a reversible batch id. The uploader's asserted right
to use the data is recorded in the batch's usage note.

## LinkedIn rules (hard)
Permitted: authorized-search locate of public URLs; manual entry; user-supplied exports.
Stored: URL + indexed title + indexed snippet + retrieval date, labeled
`found_uncorroborated` until a class-1/2/4 source confirms the role
(`found_corroborated`) or an admin verifies (`manually_verified`).
Never: logged-in scraping, browser automation, cookie reuse, rate-limit evasion,
connection automation. CoStar/LoopNet/Reonomy/ZoomInfo/Apollo and similar restricted
databases are **not** used (no licensed integration is configured).

## Adding another source adapter
1. Create `src/pr/adapters/<name>.js` exporting `{name, kind, configured, missingConfig,
   sourceMeta, …methods}` and `register(...)` it (see `adapters/index.js`).
2. Add the require to the bottom of `adapters/index.js`.
3. Every method returns the standard AdapterReport (provider, query, timestamp,
   result_count, rate-limit state, cursor, errors, terms note, primary/secondary,
   confidence recommendation).
4. Verify the source's terms BEFORE coding the fetch; record them in `sourceMeta()`.
5. Wire it into a job in `src/pr/jobs/index.js` if it should run in the pipeline.