`README.md`

59 lines
# re-flyer-aggregator (TK-10708)

Locate + aggregate **commercial real-estate flyers / offering memoranda / one-sheets**
for the RE builds (**usre** data engine, **CRCP** for Frank, **RENTV** news + 50k list).
Codex-reviewed 2026-08-19.

## What "flyers" means here
Commercial property marketing one-sheets: offering memoranda (OMs), property-spotlight
flyers, brokerage brochures, deal recaps. **Not** residential listing flyers, agent-recruiting
flyers, or ad inventory (confirm with Steve before expanding scope).

## Compliance model (the important part)
Three-gate: **copyright ∧ contract(ToS) ∧ privacy**. Corrected posture (codex):
**"publicly accessible" ≠ "authorized for automated harvesting."** Automated access is
governed by each source's ToS + robots.txt, independent of human viewability.

| Tier | What | Automated OK? |
|---|---|---|
| **1** | RENTV first-party + **broker-owned** public property landing pages + owner/REIT press rooms + public-record docs | Yes — discovery of the URL only, robots-respecting, rate-limited. Store URL + provenance, never the file. |
| **2** | **Self-generated closed-sale recap** from public-record deal data + our own brand marks | Yes — public-record facts + our template only. No third-party photos/maps/logos. |
| **3** | **CRE marketplaces** (Crexi, LoopNet/CityFeet-CoStar, Brevitas, Biproxi, CommercialCafe) + any **re-host** of a third-party PDF | **GATED** — ToS prohibits automated access; draft to pending-approval, never auto-run. |

## Data model
`db/001_external_marketing_asset.sql` (DRAFT — not applied to prod):
- `external_marketing_asset` — property/listing-scoped; link-only fields (source_landing_url,
  document_url, discovery_method, **rights_basis**, access_status, robots_ok, last_verified_at).
  No content_hash / local_path unless a Tier-2 generate or gated Tier-3 download happened.
- `asset_subject_link` — asset ↔ subject (parcel `(county_fips,ain)` or deal `(county_fips,doc_number)`),
  deal key **nullable**, `match_confidence`.
- Staging lives HERE; only **validated** rows promote into usre as the canonical dataset.
  CRCP/RENTV consume a **view/API**, never these staging tables.

## Scripts (both run against live usre, $0 local)
- `scripts/generate-spotlight.mjs` — Tier-2 closed-sale recap → `out/spotlight-*.html` (print → PDF).
  `node scripts/generate-spotlight.mjs --n 5` or `--doc <doc_number>`.
- `scripts/provenance-probe.mjs` — codex's #1 first move: N recent deals → `out/provenance-report.md`
  with a per-deal broker-scoped discovery query + confidence columns. **DRY** (fetches nothing).
  `node scripts/provenance-probe.mjs --n 25 --county 06037`.

## Staging store (built + populated — local `reflyers` Postgres)
`createdb reflyers` + `db/001` + `db/002`. Populated from live discovery on the top Miami closed deals:
**15 assets / 5 deals** — 7 first-party (owner/dev/press), 2 broker-owned (Blanca, CBRE), 5 self-generated
recaps, 1 GATED (LoopNet, hidden from the consumer view). Reversible: `dropdb reflyers` (ledgered).
- `scripts/ingest-assets.mjs <findings.json> | --recaps | --report` — load classified findings + register recaps.
- `db/002_deal_assets_view.sql` — `deal_assets_v`, the read view CRCP/RENTV consume (GATED tier-3 excluded).
- `scripts/render-asset-index.mjs` — static consumer preview → `out/deal-assets.html`.
- `data/findings-2026-08-19.json` — the real classified hits · `out/provenance-findings.md` — the verdict.

## Roadmap
1. **Done:** schema, tiered source registry, Tier-2 generator, provenance probe w/ real firm-domain join,
   live discovery on top Miami deals, staging DB + ingest + read view + consumer index.
2. **Next (safe, local):** widen discovery to the rest of the Miami-Dade feed; add a scheduled recap batch
   for RENTV; add a small sort+density viewer over `deal_assets_v`.
3. **Gated (needs Steve):** promote validated rows into **usre**; create `deal_assets_v` there; wire CRCP
   deal rows + RENTV articles to show the recap/first-party links; any Kamatera deploy.
4. **Never without Steve:** marketplace automated access, any PDF re-host, prod deploy, DNS, send-to-list.
5. **Upgrade path:** populate `broker_of_record_history.county_fips/ain` → state-level firm-domain join
   becomes exact per-property broker resolution.