`README.md`

120 lines
# LA Permits — free data puller (school project)

A $0, no-API-key tool that pulls **City of Los Angeles building permits** from the
public LA Open Data Portal and writes a clean CSV — including **contractor,
applicant, and principal names**, project **valuation**, address, **APN**, and a
ready-made **California Secretary of State** lookup URL per contractor.

## ⚡ LIVE vs. historical (read this first)
The City **stopped publishing contractor/applicant NAMES in the open feed after
2023-05-19** (privacy/policy change). So you pick one:

- **Want CURRENT permits (updated ~daily)?** → `pull_permits_live.py` (feed `pi9x-tg5x`,
  refreshed to within ~2 days). Has valuation / type / address / **APN** / council
  district / status — but **no contractor names**. Enrich the **owner/developer** live
  via the LA County Assessor (by APN) and the contractor via an LADBS permit lookup.
  ```bash
  python3 pull_permits_live.py --min-valuation 5000000 --days 120   # -> permits_live.csv
  ```
- **Want contractor/applicant/principal NAMES?** → `pull_permits.py` (feed `d9aa-v8bm`),
  but that data is a **frozen 2013–2023 snapshot** — the last free named permit data.

**UPDATE — live names ARE recoverable per-permit.** LADBS's public "Permit & Inspection
Report" detail page is a plain GET keyed by the permit's three segments
(`.../PermitReport/PcisPermitDetail?id1=&id2=&id3=`) and renders
`Contractor <name>; Lic. No.: <license>`. So `enrich_live_contractor.py` recovers the
live contractor + CSLB license for each current permit — free, no browser, no login,
cached + rate-limited. The bulk open feed is still nameless; you just enrich per permit.

### Self-running loop
`run_pipeline.sh` = one full cycle (refresh live permits → recover contractors → merge
into the accumulating `master_live.csv` → regenerate `LIVE-BOARD.md` → commit). It has an
overlap lockfile and the LADBS fetch is cached, so cycles after the first are near-instant.
```bash
./run_pipeline.sh          # run one cycle by hand anytime
```
A launchd job (`~/Library/LaunchAgents/com.steve.la-permits-live-refresh.plist`, every 6h)
is provided but must be loaded by the user (the agent's harness blocks installing a
standing daemon):
```bash
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.steve.la-permits-live-refresh.plist
```

Two free public data sources, no signup, no cost:

| Source | What it gives | Access |
|---|---|---|
| **LADBS building permits** (`data.lacity.org`, dataset `d9aa-v8bm`) | Permit type, valuation, address, APN, **contractor / applicant / principal names**, license, stories, dwelling units | Socrata API — free, no key |
| **CA Secretary of State** (`bizfileonline.sos.ca.gov`) | Legal entity + agent for service of process behind a business name | Free web lookup (the tool generates the URL per contractor) |

## Quick start
No install needed — Python 3 standard library only.

```bash
python3 pull_permits.py                          # new builds + additions > $1M -> permits.csv
python3 pull_permits.py --min-valuation 5000000  # only projects over $5M
python3 pull_permits.py --permit-types "Bldg-New" --since 2020-01-01
python3 pull_permits.py --limit 500 --out sample.csv
```

### Options
- `--min-valuation N` — minimum project $ (default `1000000`; `0` = no filter)
- `--permit-types "A,B"` — e.g. `Bldg-New`, `Bldg-Addition`, `Bldg-Alter/Repair`,
  `Bldg-Demolition`, `Sign` (`""` = all types)
- `--since YYYY-MM-DD` — only permits issued on/after this date
- `--limit N` — cap rows written (`0` = all matches)
- `--out FILE.csv` — output path
- `--app-token TOKEN` — optional free Socrata token (only if you hit rate limits)

## The pipeline
1. **`pull_permits.py`** → bulk CSV of significant permits with the party names.
2. **`enrich_contractors.py`** → dedupe those permits to **unique contractors**,
   ranked by total project value, each with ready-to-click **CA SOS** (by name) and
   **CSLB** (by license #) lookup links:
   ```bash
   python3 enrich_contractors.py                 # permits.csv -> contractors.csv
   python3 enrich_contractors.py --min-permits 2 # firms with 2+ permits only
   ```
   Example: 1,000 permits → ~290 unique firms; top row = Hensel Phelps ($1.6B / 10
   permits). This deduped list IS your "top LA contractors" ranking.
3. **`summary.py`** → class-ready descriptive stats printed to the terminal
   (headline totals, permits & $ by year, top contractors, build $ by council
   district, permit-type mix, biggest single projects):
   ```bash
   python3 summary.py --top 8
   ```
4. **`make_chart.py`** → a slide-ready "value by year" bar chart (pure-stdlib SVG,
   2023 hatched as partial). Rasterize to PNG with a local renderer — no pip:
   ```bash
   python3 make_chart.py                         # -> value_by_year.svg
   # then PNG (either works, both free/local):
   "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --headless=new \
     --screenshot="$PWD/value_by_year.png" --window-size=960,540 "file://$PWD/value_by_year.svg"
   # or: qlmanage -t -s 1600 -o . value_by_year.svg   (square crop; Chrome keeps 16:9)
   ```
5. **CA SOS + CSLB** (free lookups) → open the URLs on the firms you care about to
   resolve the legal entity + agent (SOS) and license status/classification (CSLB).

### Why the enrichment is a worklist, not an auto-scrape
There is no reliable *free* HTTP endpoint for entity data: CA SOS bizfile sits behind
Imperva/Incapsula bot protection, OpenCorporates now requires a paid token, and CSLB
session-gates its license page. Rather than fight three anti-bot walls (or drive a real
browser, a ToS gray area), the tool **dedupes to the handful of unique firms** so a
guided manual lookup is trivial — and ranks them so you check the important ones first.

## Data notes
- `d9aa-v8bm` is a **stable historical snapshot: 2013-01 → 2023-05, ~317k permits**.
  Great for a reproducible course project. (The *live* daily permit feed exists but
  the name-bearing fields are login-gated; the open live feed `pi9x-tg5x` omits
  contractor/applicant names.)
- `apn` is the full LA County parcel id (book-page-parcel), e.g. `4129-027-902`.
- Real examples in a default run: the **$554M LAX addition** (Hensel Phelps) and
  the **$476M tower at 100 S Grand Ave** (Tishman).

## Compliance
Both sources are **public / open data**, free to reuse. This project uses only
public records — it does **not** touch paid or license-restricted databases
(e.g. Data Axle / Reference Solutions library edition, which is academic-use-only
and must not be redistributed). If this work is ever used beyond coursework,
re-check terms before republishing compiled lists.