`ingest/socrata/README.md`

52 lines
# LA Socrata CRE-prospect ingester

Pulls two FREE, keyless LA City open-data (SODA) datasets into a de-duplicated
CRE advertiser-prospect table, surfacing only firms NOT already in the curated
LABJ set (`data/labj_master.json`).

| Source | Dataset | Yields | Role |
|---|---|---|---|
| Listing of Active Businesses | `6rrh-rzua` (current) | ~65k CRE-NAICS firms | Broker / Property Manager / Architect-Engineer / Developer |
| LADBS Building Permits | `d9aa-v8bm` (**historical 2013–2023**) | contractors on Commercial/Apartment permits | Contractor |

Output: Postgres `realestate.socrata_cre_prospects` (upsert on `dedup_key`,
`first_seen`/`last_seen` tracked) + a `socrata_prospects.jsonl` staging file.

## Run
```
python3 ingest.py                       # 15k businesses + 24mo permits -> PG
python3 ingest.py --dry-run             # no DB write; prints role breakdown
python3 ingest.py --businesses-only --max-businesses 40000
python3 ingest.py --include-lessors     # also pull the 43k "Lessors of real estate" (property owners)
```
`SOCRATA_APP_TOKEN` (optional) avoids throttling on large pulls. Cost: $0 (keyless, local PG).

## Permit-dataset landscape (verified 2026-08-10)
Two LADBS permit datasets exist, with a real trade-off:

| Dataset | Recency | Contractor/firm names? | Use for |
|---|---|---|---|
| `d9aa-v8bm` (current `PERMITS_ID`) | historical, ends 2023-05 | YES — contractors_business_name + applicant | firm prospects (Contractor role) |
| `pi9x-tg5x` "Building Permits Issued 2020→Present" | LIVE (to 2026-08-08) | NO — address/valuation/use/apn only | recent PROJECT SITES, not firms |

**The current LADBS feed dropped contractor/owner names.** Swapping `PERMITS_ID` to
`pi9x-tg5x` gains recency but LOSES the firm-name signal that made permits useful for
prospecting. The permit window is computed relative to the dataset's own max
`issue_date`, so `d9aa-v8bm` still yields its 2013–2023 contractors.

**APN→owner recovery — VERIFIED DEAD END for free (la-research-agent, 2026-08-10).**
`pi9x-tg5x` carries `apn`, but there is NO free, programmatic APN→owner-name source for
LA County. Owner name is deliberately withheld from every free dataset — Assessor
FeatureServer `70d93266…` (51 fields, no owner), the PAIS API
`portal.assessor.lacounty.gov/api/parceldetail|parcel_ownershiphistory` (owner-less;
ownership-history returns only RR/CC document numbers), the RR/CC grantor-grantee index
(no online AIN search), NavigateLA layer 397, and ZIMAS — all confirmed owner-less.
Owner name is gated behind the Assessor's PAID data-sale program
(datasales@assessor.lacounty.gov). Do NOT build a free owner-enrichment pipeline.
Compliance if the paid extract is bought: YELLOW (facts uncopyrightable + commercial
entities out of CCPA scope; the gate is the paid extract's data-license review).
Meanwhile the
**Active Businesses** dataset stays the primary current firm-prospect engine.

Scheduled refresh is a Steve-gated launchd install, drafted to pending-approval.