← back to La Socrata Ingester
LA Socrata/ArcGIS public-data ingester → realestate DB (TK-10450)
899b4fca7f42c032382f7ac33aa18df6172b0cd7 · 2026-08-10 21:58:19 -0700 · steve
4 categories (building permits, assessor parcels, code enforcement, business/film)
across 10 datasets. Socrata SODA + LA County ArcGIS FeatureServer adapters behind
one page-generator; generic chunked ON CONFLICT upsert; incremental high-water-mark
state. Smoke-tested live all 5 sources; idempotency + permit↔parcel join verified. $0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Files touched
A .env.exampleA .gitignoreA README.mdA db/schema.sqlA package-lock.jsonA package.jsonA src/adapters/arcgis.jsA src/adapters/http.jsA src/adapters/socrata.jsA src/cli.jsA src/db.jsA src/ingest.jsA src/sources.js
Diff
commit 899b4fca7f42c032382f7ac33aa18df6172b0cd7
Author: steve <steve@designerwallcoverings.com>
Date: Mon Aug 10 21:58:19 2026 -0700
LA Socrata/ArcGIS public-data ingester → realestate DB (TK-10450)
4 categories (building permits, assessor parcels, code enforcement, business/film)
across 10 datasets. Socrata SODA + LA County ArcGIS FeatureServer adapters behind
one page-generator; generic chunked ON CONFLICT upsert; incremental high-water-mark
state. Smoke-tested live all 5 sources; idempotency + permit↔parcel join verified. $0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
.env.example | 11 +++
.gitignore | 9 +++
README.md | 103 ++++++++++++++++++++++++++
db/schema.sql | 155 +++++++++++++++++++++++++++++++++++++++
package-lock.json | 165 ++++++++++++++++++++++++++++++++++++++++++
package.json | 19 +++++
src/adapters/arcgis.js | 44 ++++++++++++
src/adapters/http.js | 23 ++++++
src/adapters/socrata.js | 38 ++++++++++
src/cli.js | 79 ++++++++++++++++++++
src/db.js | 74 +++++++++++++++++++
src/ingest.js | 71 ++++++++++++++++++
src/sources.js | 187 ++++++++++++++++++++++++++++++++++++++++++++++++
13 files changed, 978 insertions(+)
diff --git a/.env.example b/.env.example
new file mode 100644
index 0000000..9d1a05a
--- /dev/null
+++ b/.env.example
@@ -0,0 +1,11 @@
+# realestate Postgres — defaults to the local `realestate` DB over the unix socket.
+# Override any standard libpq var (PGHOST/PGUSER/PGPASSWORD/PGPORT) or pass a full URL.
+REALESTATE_DB=realestate
+# DATABASE_URL=postgres://user:pass@host:5432/realestate
+
+# Socrata app token (OPTIONAL). Anonymous works, but a free token lifts the
+# throttle from ~1 req/s to ~32 req/s. Register at https://data.lacity.org/profile/edit/developer_settings
+SOCRATA_APP_TOKEN=
+
+# Politeness delay between paged requests, ms (default 120)
+INGEST_PAGE_DELAY_MS=120
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..b07f084
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,9 @@
+node_modules/
+.env
+.env.*
+!.env.example
+tmp/
+*.log
+.DS_Store
+dist/
+build/
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..9c8dd27
--- /dev/null
+++ b/README.md
@@ -0,0 +1,103 @@
+# LA Socrata / ArcGIS Public-Data Ingester
+
+Programmatic ingester for Los Angeles public data → the local `realestate` Postgres DB.
+**$0** — pulls from free public government APIs, writes to local Postgres. No scraping,
+no captcha, no login. Feeds the WhoLivedThere / ClaimMyAddress / BubbesBlock home-history stack.
+
+Built TK-10450 (2026-08-10). All endpoints live-probed 2026-08-10.
+
+## Why Socrata (+ one ArcGIS adapter)
+
+LA City and LA County publish open data on **Socrata**, which exposes the **SODA API** —
+real REST/JSON endpoints with SoQL (`$where`/`$select`/`$order`/`$limit`/`$offset`). That's
+the "feed-first" doctrine applied to gov data: no selector-rot, no headless browser.
+
+The one exception: **LA County assessor parcels live on ArcGIS FeatureServer**, not Socrata,
+so this ingester carries two adapters behind one common page-generator interface.
+
+## Sources
+
+| Source (`--list` name) | Platform | Dataset | Rows | Cadence | Dedup key | Cursor |
+|---------------------------------|----------|--------------------|---------|---------|------------------------|-----------------------|
+| `building_permits` | Socrata | `pi9x-tg5x` | 405k | daily | (dataset_id,permit_nbr)| `issue_date` |
+| `building_permits_2010_2019` | Socrata | `dyxf-7hc4` | 533k | static | (dataset_id,permit_nbr)| — |
+| `building_permits_pre2010` | Socrata | `e67z-kt2n` | 640k | static | (dataset_id,permit_nbr)| — |
+| `building_permits_electrical` | Socrata | `ysqd-apz7` | 354k | daily | (dataset_id,permit_nbr)| `issue_date` |
+| `building_permits_mech_plumb` | Socrata | `67is-svtd` | 293k | daily | (dataset_id,permit_nbr)| `issue_date` |
+| `assessor_parcels` | ArcGIS | Parcel_Data_2021 | 12.1M | annual | (ain,roll_year) | `RollYear` |
+| `code_enforcement_open` | Socrata | `u82d-eh7z` | 29k | daily | apno | `adddttm` |
+| `code_enforcement_closed` | Socrata | `rken-a55j` | 822k | daily | apno | `resdttm` |
+| `business_registrations` | Socrata | `6rrh-rzua` | 632k | weekly | location_account | `location_start_date` |
+| `film_permits_weho` | Socrata | `xd5y-dzvf` | 1.9k | live | permit_id | — |
+
+**Film permits — LA City has NO live public dataset** (both prior datasets deprecated).
+FilmLA holds the consolidated LA City/County permit and only releases bulk via CPRA.
+WeHo is the live free feed; a business-registration NAICS `512100` filter is a weak proxy
+for LA City production companies.
+
+## Setup
+
+```bash
+cd ~/Projects/la-socrata-ingester
+npm install # one dep: pg
+cp .env.example .env # optional: add a free SOCRATA_APP_TOKEN to raise throttle
+npm run schema # creates la_*_raw + la_ingest_state in realestate DB
+```
+
+## Usage
+
+```bash
+node src/cli.js --list # show all sources
+node src/cli.js building_permits --max=25 # smoke test (25 rows)
+node src/cli.js code_enforcement_open # incremental refresh (since last cursor)
+node src/cli.js "building permits" # whole group
+node src/cli.js all # every non-static source, incremental
+node src/cli.js assessor_parcels --full # 12.1M-row backfill (ALL roll years)
+```
+
+- No args / `--full`: full crawl. Otherwise **incremental** from the stored high-water mark
+ in `la_ingest_state` (Socrata `cursor > last`, so daily re-runs are cheap).
+- `assessor_parcels` defaults to the **latest roll year only** (~2.4M); `--full` pulls all 12.1M.
+- `--max=N` caps rows (testing); `--page=N` overrides page size.
+- Static historical permit buckets are excluded from `all` — run them once explicitly.
+
+### One-time backfill (run in order; each is resumable — cursor persists on failure)
+
+```bash
+node src/cli.js building_permits --full
+node src/cli.js building_permits_2010_2019
+node src/cli.js building_permits_pre2010
+node src/cli.js code_enforcement_open --full
+node src/cli.js code_enforcement_closed --full
+node src/cli.js business_registrations --full
+node src/cli.js film_permits_weho --full
+node src/cli.js assessor_parcels # latest roll year first (sane); --full later
+```
+
+### Scheduled incremental (suggested launchd/cron, daily)
+
+```bash
+cd ~/Projects/la-socrata-ingester && node src/cli.js all >> tmp/ingest.log 2>&1
+```
+
+## Schema & joins
+
+Each `la_*_raw` table keeps full source fidelity in `raw jsonb` **plus** typed columns for
+the fields we query/join, **plus** provenance (`source_url`, `fetched_at`; `dataset_id` where
+multiple datasets share a table). Upserts are `ON CONFLICT (<key>) DO UPDATE`.
+
+- **Permits ↔ parcels**: `la_building_permits_raw.apn = la_assessor_parcels_raw.ain`
+ (both 10-digit APN/AIN). Clean join.
+- **Code enforcement ↔ parcels**: code cases carry LADBS's internal `prclid`, **not** the AIN —
+ join by geocoded address / lat-lon, not parcel key.
+- **Business ↔ address**: `zip_code` + `street_address` (+ lat/lon from the source GeoJSON).
+
+## Design notes / gotchas
+
+- Socrata pages via `$order=:id` (always-unique system field) so rows arriving mid-crawl
+ never shift the window. ArcGIS pages via `resultOffset`/`resultRecordCount` (2k cap) and
+ honors `exceededTransferLimit`.
+- Rows with a null primary-key component are skipped (logged as "keyless"), never crash.
+- HTTP layer retries with exponential backoff on 429/5xx.
+- Deprecated datasets to avoid (verified dead): permits `xnhu-aczu`; film `tg4x-b46p`, `c2az-nhru`.
+- Assessor is a **tabular** FeatureServer (no geometry); parcel polygons are a separate service.
diff --git a/db/schema.sql b/db/schema.sql
new file mode 100644
index 0000000..f95d6ee
--- /dev/null
+++ b/db/schema.sql
@@ -0,0 +1,155 @@
+-- LA Socrata/ArcGIS ingester — landing schema for the `realestate` DB.
+-- Convention follows the existing *_raw tables (cslb_raw, azdre_raw): raw source
+-- fidelity kept in `raw jsonb`, plus typed columns for the fields we query/join on,
+-- plus provenance (dataset_id, source_url, fetched_at). Every table has a real
+-- dedup key so incremental refreshes upsert cleanly (ON CONFLICT).
+-- Idempotent: safe to run repeatedly.
+
+-- ---------------------------------------------------------------------------
+-- 1. Building permits (LA City LADBS) — Socrata.
+-- One table holds all permit sub-datasets (bldg 2020+/2010-19/pre-2010,
+-- electrical, mechanical/plumbing); dataset_id distinguishes them, so the
+-- dedup key is (dataset_id, permit_nbr) to avoid cross-dataset collisions.
+-- ---------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS la_building_permits_raw (
+ dataset_id text NOT NULL,
+ permit_nbr text NOT NULL,
+ primary_address text,
+ apn text, -- 10-digit APN → joins la_assessor_parcels_raw.ain
+ zip_code text,
+ council_district text,
+ permit_group text,
+ permit_type text,
+ permit_sub_type text,
+ use_desc text,
+ issue_date timestamptz,
+ status_desc text,
+ valuation numeric,
+ lat double precision,
+ lon double precision,
+ work_desc text,
+ raw jsonb,
+ source_url text,
+ fetched_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (dataset_id, permit_nbr)
+);
+CREATE INDEX IF NOT EXISTS idx_la_permits_apn ON la_building_permits_raw (apn);
+CREATE INDEX IF NOT EXISTS idx_la_permits_issue_date ON la_building_permits_raw (issue_date);
+CREATE INDEX IF NOT EXISTS idx_la_permits_zip ON la_building_permits_raw (zip_code);
+
+-- ---------------------------------------------------------------------------
+-- 2. Assessor parcels (LA County) — ArcGIS FeatureServer (NOT Socrata).
+-- Annual snapshot; composite key (ain, roll_year) keeps the time-series.
+-- ---------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS la_assessor_parcels_raw (
+ ain text NOT NULL, -- 10-digit AIN → joins la_building_permits_raw.apn
+ roll_year text NOT NULL,
+ assessor_id text, -- formatted APN e.g. 2038-020-084
+ property_location text,
+ situs_house_no text,
+ situs_street text,
+ situs_city text,
+ situs_zip5 text,
+ use_type text,
+ use_code text,
+ year_built text,
+ sqft_main numeric,
+ bedrooms text,
+ bathrooms text,
+ units text,
+ land_value numeric,
+ imp_value numeric,
+ total_value numeric,
+ recording_date bigint, -- unix ms epoch of last deed recording
+ center_lat double precision,
+ center_lon double precision,
+ raw jsonb,
+ source_url text,
+ fetched_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (ain, roll_year)
+);
+CREATE INDEX IF NOT EXISTS idx_la_parcels_ain ON la_assessor_parcels_raw (ain);
+CREATE INDEX IF NOT EXISTS idx_la_parcels_zip ON la_assessor_parcels_raw (situs_zip5);
+CREATE INDEX IF NOT EXISTS idx_la_parcels_year ON la_assessor_parcels_raw (roll_year);
+
+-- ---------------------------------------------------------------------------
+-- 3. Code enforcement cases (LA City LADBS) — Socrata (open + closed).
+-- apno is unique across both datasets; a case moves open→closed by updating
+-- the same row, so the key is apno alone.
+-- ---------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS la_code_enforcement_raw (
+ apno text NOT NULL,
+ apname text,
+ address text, -- concatenated from stno/predir/stname/suffix/postdir
+ zip text,
+ add_dttm timestamptz, -- case opened
+ res_dttm timestamptz, -- case resolved (null if open)
+ prclid text, -- LADBS internal parcel id (NOT the AIN)
+ ap_type text,
+ apc text, -- area planning commission
+ stat text, -- 'O' open / 'C' closed
+ dataset_id text,
+ raw jsonb,
+ source_url text,
+ fetched_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (apno)
+);
+CREATE INDEX IF NOT EXISTS idx_la_code_zip ON la_code_enforcement_raw (zip);
+CREATE INDEX IF NOT EXISTS idx_la_code_stat ON la_code_enforcement_raw (stat);
+CREATE INDEX IF NOT EXISTS idx_la_code_add ON la_code_enforcement_raw (add_dttm);
+
+-- ---------------------------------------------------------------------------
+-- 4. Active business registrations (LA City) — Socrata.
+-- ---------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS la_business_registrations_raw (
+ location_account text NOT NULL,
+ business_name text,
+ dba_name text,
+ street_address text,
+ city text,
+ zip_code text,
+ naics text,
+ primary_naics_description text,
+ council_district text,
+ location_start_date date,
+ location_end_date date, -- null if active
+ lat double precision,
+ lon double precision,
+ raw jsonb,
+ source_url text,
+ fetched_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (location_account)
+);
+CREATE INDEX IF NOT EXISTS idx_la_biz_zip ON la_business_registrations_raw (zip_code);
+CREATE INDEX IF NOT EXISTS idx_la_biz_naics ON la_business_registrations_raw (naics);
+CREATE INDEX IF NOT EXISTS idx_la_biz_start ON la_business_registrations_raw (location_start_date);
+
+-- ---------------------------------------------------------------------------
+-- 5. Film permits (WeHo) — Socrata. LA City proper has NO live public dataset
+-- (both deprecated); FilmLA bulk is CPRA-only. WeHo is the live free feed.
+-- Schema kept tolerant (raw jsonb + best-effort typed fields).
+-- ---------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS la_film_permits_raw (
+ permit_id text NOT NULL,
+ title text,
+ permit_type text,
+ address text,
+ start_date timestamptz,
+ end_date timestamptz,
+ raw jsonb,
+ source_url text,
+ fetched_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (permit_id)
+);
+
+-- ---------------------------------------------------------------------------
+-- Incremental high-water-mark state (one row per source).
+-- ---------------------------------------------------------------------------
+CREATE TABLE IF NOT EXISTS la_ingest_state (
+ source text NOT NULL PRIMARY KEY,
+ dataset_id text,
+ last_cursor text, -- max value of the source's cursor field seen so far
+ last_run timestamptz,
+ rows_upserted bigint NOT NULL DEFAULT 0,
+ last_status text
+);
diff --git a/package-lock.json b/package-lock.json
new file mode 100644
index 0000000..a3d82dd
--- /dev/null
+++ b/package-lock.json
@@ -0,0 +1,165 @@
+{
+ "name": "la-socrata-ingester",
+ "version": "0.1.0",
+ "lockfileVersion": 3,
+ "requires": true,
+ "packages": {
+ "": {
+ "name": "la-socrata-ingester",
+ "version": "0.1.0",
+ "license": "UNLICENSED",
+ "dependencies": {
+ "pg": "^8.13.1"
+ },
+ "bin": {
+ "la-ingest": "src/cli.js"
+ }
+ },
+ "node_modules/pg": {
+ "version": "8.23.0",
+ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",
+ "integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-connection-string": "^2.14.0",
+ "pg-pool": "^3.14.0",
+ "pg-protocol": "^1.16.0",
+ "pg-types": "2.2.0",
+ "pgpass": "1.0.5"
+ },
+ "engines": {
+ "node": ">= 16.0.0"
+ },
+ "optionalDependencies": {
+ "pg-cloudflare": "^1.4.0"
+ },
+ "peerDependencies": {
+ "pg-native": ">=3.0.1"
+ },
+ "peerDependenciesMeta": {
+ "pg-native": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/pg-cloudflare": {
+ "version": "1.4.0",
+ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",
+ "integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",
+ "license": "MIT",
+ "optional": true
+ },
+ "node_modules/pg-connection-string": {
+ "version": "2.14.0",
+ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",
+ "integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-int8": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
+ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
+ "license": "ISC",
+ "engines": {
+ "node": ">=4.0.0"
+ }
+ },
+ "node_modules/pg-pool": {
+ "version": "3.14.0",
+ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
+ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
+ "license": "MIT",
+ "peerDependencies": {
+ "pg": ">=8.0"
+ }
+ },
+ "node_modules/pg-protocol": {
+ "version": "1.16.0",
+ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",
+ "integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-types": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
+ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-int8": "1.0.1",
+ "postgres-array": "~2.0.0",
+ "postgres-bytea": "~1.0.0",
+ "postgres-date": "~1.0.4",
+ "postgres-interval": "^1.1.0"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/pgpass": {
+ "version": "1.0.5",
+ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
+ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
+ "license": "MIT",
+ "dependencies": {
+ "split2": "^4.1.0"
+ }
+ },
+ "node_modules/postgres-array": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
+ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/postgres-bytea": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
+ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-date": {
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
+ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-interval": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
+ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
+ "license": "MIT",
+ "dependencies": {
+ "xtend": "^4.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/split2": {
+ "version": "4.2.0",
+ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
+ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
+ "license": "ISC",
+ "engines": {
+ "node": ">= 10.x"
+ }
+ },
+ "node_modules/xtend": {
+ "version": "4.0.2",
+ "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
+ "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.4"
+ }
+ }
+ }
+}
diff --git a/package.json b/package.json
new file mode 100644
index 0000000..f427243
--- /dev/null
+++ b/package.json
@@ -0,0 +1,19 @@
+{
+ "name": "la-socrata-ingester",
+ "version": "0.1.0",
+ "description": "Programmatic ingester for LA public data (Socrata SODA + LA County ArcGIS FeatureServer) -> realestate Postgres. $0, free public APIs.",
+ "type": "module",
+ "bin": {
+ "la-ingest": "src/cli.js"
+ },
+ "scripts": {
+ "schema": "psql \"${REALESTATE_DB:-realestate}\" -f db/schema.sql",
+ "ingest": "node src/cli.js",
+ "list": "node src/cli.js --list"
+ },
+ "dependencies": {
+ "pg": "^8.13.1"
+ },
+ "author": "steve@designerwallcoverings.com",
+ "license": "UNLICENSED"
+}
diff --git a/src/adapters/arcgis.js b/src/adapters/arcgis.js
new file mode 100644
index 0000000..9fba835
--- /dev/null
+++ b/src/adapters/arcgis.js
@@ -0,0 +1,44 @@
+import { fetchJson } from './http.js';
+
+// ArcGIS FeatureServer paginator. Yields { rows, url } per page of `.attributes`.
+// Pages via resultOffset/resultRecordCount, ordered by a stable field. Honors
+// exceededTransferLimit to know when more pages remain.
+//
+// src fields used: endpoint (…/FeatureServer/0), orderBy, cursorField?, defaultWhere?
+// opts: since (string | null), pageSize (default 2000), maxRows
+export async function* arcgisPages(src, opts = {}) {
+ const { since = null, pageSize = 2000, maxRows = Infinity } = opts;
+ const base = `${src.endpoint}/query`;
+
+ const clauses = [];
+ if (since && src.cursorField) clauses.push(`${src.cursorField} >= '${since}'`);
+ else if (src.defaultWhere) clauses.push(src.defaultWhere);
+ const where = clauses.length ? clauses.join(' AND ') : '1=1';
+
+ let offset = 0;
+ for (;;) {
+ const count = Math.min(pageSize, maxRows - offset);
+ if (count <= 0) break;
+
+ const p = new URLSearchParams({
+ where,
+ outFields: '*',
+ returnGeometry: 'false',
+ orderByFields: src.orderBy || 'AIN',
+ resultOffset: String(offset),
+ resultRecordCount: String(count),
+ f: 'json',
+ });
+ const url = `${base}?${p}`;
+ const data = await fetchJson(url);
+ if (data.error) throw new Error(`ArcGIS error: ${JSON.stringify(data.error).slice(0, 300)}`);
+
+ const rows = (data.features || []).map((f) => f.attributes);
+ if (!rows.length) break;
+
+ yield { rows, url };
+
+ offset += rows.length;
+ if (!data.exceededTransferLimit && rows.length < count) break;
+ }
+}
diff --git a/src/adapters/http.js b/src/adapters/http.js
new file mode 100644
index 0000000..fe5ea96
--- /dev/null
+++ b/src/adapters/http.js
@@ -0,0 +1,23 @@
+export const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+
+// GET JSON with retry + exponential backoff on 429 / 5xx / network errors.
+export async function fetchJson(url, { headers = {}, tries = 5 } = {}) {
+ let attempt = 0;
+ for (;;) {
+ attempt++;
+ try {
+ const res = await fetch(url, { headers });
+ if (res.status === 429 || res.status >= 500) {
+ if (attempt >= tries) throw new Error(`HTTP ${res.status} after ${tries} tries: ${url}`);
+ const wait = Math.min(30000, 500 * 2 ** attempt);
+ await sleep(wait);
+ continue;
+ }
+ if (!res.ok) throw new Error(`HTTP ${res.status}: ${url}\n${(await res.text()).slice(0, 300)}`);
+ return await res.json();
+ } catch (err) {
+ if (attempt >= tries) throw err;
+ await sleep(Math.min(30000, 500 * 2 ** attempt));
+ }
+ }
+}
diff --git a/src/adapters/socrata.js b/src/adapters/socrata.js
new file mode 100644
index 0000000..67c80dd
--- /dev/null
+++ b/src/adapters/socrata.js
@@ -0,0 +1,38 @@
+import { fetchJson } from './http.js';
+
+// Socrata SODA paginator. Yields { rows, url } per page.
+// Stable pagination via $order=:id (the always-unique system field), so new rows
+// arriving mid-crawl never shift the window. Incremental via $where cursor filter.
+//
+// src fields used: domain, datasetId, cursorField?, staticFilter?
+// opts: since (ISO string | null), pageSize (default 50000), appToken, maxRows
+export async function* socrataPages(src, opts = {}) {
+ const { since = null, pageSize = 50000, appToken, maxRows = Infinity } = opts;
+ const base = `https://${src.domain}/resource/${src.datasetId}.json`;
+ const headers = appToken ? { 'X-App-Token': appToken } : {};
+ let offset = 0;
+
+ for (;;) {
+ const limit = Math.min(pageSize, maxRows - offset);
+ if (limit <= 0) break;
+
+ const p = new URLSearchParams();
+ p.set('$limit', String(limit));
+ p.set('$offset', String(offset));
+ p.set('$order', ':id');
+
+ const where = [];
+ if (src.staticFilter) where.push(src.staticFilter);
+ if (since && src.cursorField) where.push(`${src.cursorField} > '${since}'`);
+ if (where.length) p.set('$where', where.join(' AND '));
+
+ const url = `${base}?${p}`;
+ const rows = await fetchJson(url, { headers });
+ if (!rows.length) break;
+
+ yield { rows, url };
+
+ offset += rows.length;
+ if (rows.length < limit) break; // last page
+ }
+}
diff --git a/src/cli.js b/src/cli.js
new file mode 100644
index 0000000..ddae529
--- /dev/null
+++ b/src/cli.js
@@ -0,0 +1,79 @@
+#!/usr/bin/env node
+import { SOURCES, GROUPS } from './sources.js';
+import { ingestSource, closePool } from './ingest.js';
+
+const argv = process.argv.slice(2);
+const flags = new Set(argv.filter((a) => a.startsWith('--') && !a.includes('=')));
+const kv = Object.fromEntries(
+ argv.filter((a) => a.startsWith('--') && a.includes('=')).map((a) => a.slice(2).split('='))
+);
+const positional = argv.filter((a) => !a.startsWith('--'));
+
+function printList() {
+ console.log('\nLA Socrata/ArcGIS ingester — sources:\n');
+ for (const [group, names] of Object.entries(GROUPS)) {
+ console.log(` ${group}`);
+ for (const n of names) {
+ const s = SOURCES[n];
+ const id = s.datasetId || (s.endpoint ? 'arcgis' : '');
+ console.log(` • ${n.padEnd(30)} ${s.platform.padEnd(8)} ${id}${s.static ? ' [static]' : ''}`);
+ }
+ }
+ console.log(`\nUsage:
+ node src/cli.js --list
+ node src/cli.js <source|group|all> [--full] [--max=N] [--page=N]
+
+Examples:
+ node src/cli.js building_permits --max=25 # smoke test: 25 rows
+ node src/cli.js code_enforcement_open # incremental refresh
+ node src/cli.js "building permits" # whole group
+ node src/cli.js assessor_parcels --full # 12.1M-row backfill (all roll years)
+ node src/cli.js all # every non-static source, incremental
+`);
+}
+
+function resolveTargets(token) {
+ if (!token || token === 'all') {
+ // everything except static historical bulk (run those explicitly)
+ return Object.keys(SOURCES).filter((n) => !SOURCES[n].static);
+ }
+ if (GROUPS[token]) return GROUPS[token];
+ if (SOURCES[token]) return [token];
+ return null;
+}
+
+async function main() {
+ if (flags.has('--list') || argv.length === 0) return printList();
+
+ const token = positional[0];
+ const targets = resolveTargets(token);
+ if (!targets) {
+ console.error(`Unknown source/group: "${token}". Run --list to see options.`);
+ process.exitCode = 1;
+ return;
+ }
+
+ const opts = {
+ full: flags.has('--full'),
+ maxRows: kv.max ? Number(kv.max) : undefined,
+ pageSize: kv.page ? Number(kv.page) : undefined,
+ };
+
+ const results = [];
+ for (const name of targets) {
+ try {
+ results.push(await ingestSource(name, opts));
+ } catch (err) {
+ results.push({ name, error: err.message });
+ }
+ }
+
+ console.log('\n=== summary ===');
+ for (const r of results) {
+ console.log(r.error ? ` ✖ ${r.name}: ${r.error}` : ` ✔ ${r.name}: ${r.total} rows`);
+ }
+ const failed = results.filter((r) => r.error).length;
+ if (failed) process.exitCode = 1;
+}
+
+main().finally(closePool);
diff --git a/src/db.js b/src/db.js
new file mode 100644
index 0000000..529903c
--- /dev/null
+++ b/src/db.js
@@ -0,0 +1,74 @@
+import pg from 'pg';
+
+// Connect to the realestate DB. Prefer DATABASE_URL; otherwise let libpq env vars
+// (PGHOST/PGUSER/...) apply and just set the database name to REALESTATE_DB.
+const { Pool } = pg;
+export const pool = process.env.DATABASE_URL
+ ? new Pool({ connectionString: process.env.DATABASE_URL })
+ : new Pool({ database: process.env.REALESTATE_DB || 'realestate' });
+
+export async function q(text, params) {
+ const client = await pool.connect();
+ try {
+ return await client.query(text, params);
+ } finally {
+ client.release();
+ }
+}
+
+// Generic chunked upsert.
+// table — target table name
+// conflict — array of PK column names for ON CONFLICT
+// rows — array of plain objects; keys are column names, one key may be `raw`
+// (stringified to jsonb). Columns are taken from the first row.
+// Returns the number of rows sent.
+export async function upsert(table, conflict, rows) {
+ if (!rows.length) return 0;
+ const cols = Object.keys(rows[0]);
+ const updatable = cols.filter((c) => !conflict.includes(c));
+ const setClause = updatable.length
+ ? 'DO UPDATE SET ' + updatable.map((c) => `"${c}" = EXCLUDED."${c}"`).join(', ')
+ : 'DO NOTHING';
+
+ const CHUNK = 500;
+ let sent = 0;
+ for (let i = 0; i < rows.length; i += CHUNK) {
+ const slice = rows.slice(i, i + CHUNK);
+ const values = [];
+ const tuples = slice.map((row) => {
+ const ph = cols.map((c) => {
+ let v = row[c];
+ if (c === 'raw' && v != null && typeof v !== 'string') v = JSON.stringify(v);
+ values.push(v === undefined ? null : v);
+ return `$${values.length}${c === 'raw' ? '::jsonb' : ''}`;
+ });
+ return `(${ph.join(',')})`;
+ });
+ const sql =
+ `INSERT INTO ${table} (${cols.map((c) => `"${c}"`).join(',')}) ` +
+ `VALUES ${tuples.join(',')} ` +
+ `ON CONFLICT (${conflict.map((c) => `"${c}"`).join(',')}) ${setClause}`;
+ await q(sql, values);
+ sent += slice.length;
+ }
+ return sent;
+}
+
+export async function getState(source) {
+ const { rows } = await q('SELECT * FROM la_ingest_state WHERE source = $1', [source]);
+ return rows[0] || null;
+}
+
+export async function setState(source, { dataset_id, last_cursor, rows_upserted, last_status }) {
+ await q(
+ `INSERT INTO la_ingest_state (source, dataset_id, last_cursor, last_run, rows_upserted, last_status)
+ VALUES ($1,$2,$3, now(), $4, $5)
+ ON CONFLICT (source) DO UPDATE SET
+ dataset_id = EXCLUDED.dataset_id,
+ last_cursor = COALESCE(EXCLUDED.last_cursor, la_ingest_state.last_cursor),
+ last_run = EXCLUDED.last_run,
+ rows_upserted = la_ingest_state.rows_upserted + EXCLUDED.rows_upserted,
+ last_status = EXCLUDED.last_status`,
+ [source, dataset_id, last_cursor, rows_upserted, last_status]
+ );
+}
diff --git a/src/ingest.js b/src/ingest.js
new file mode 100644
index 0000000..0567d3a
--- /dev/null
+++ b/src/ingest.js
@@ -0,0 +1,71 @@
+import { upsert, getState, setState, pool } from './db.js';
+import { socrataPages } from './adapters/socrata.js';
+import { arcgisPages } from './adapters/arcgis.js';
+import { sleep } from './adapters/http.js';
+import { SOURCES, resolveMap } from './sources.js';
+
+// Tables that carry a dataset_id column (so we stamp it onto each row).
+const HAS_DATASET_ID = new Set(['la_building_permits_raw', 'la_code_enforcement_raw']);
+
+const log = (...a) => console.log(new Date().toISOString(), ...a);
+
+// Run one source end-to-end.
+// opts: { full, maxRows, pageSize }
+export async function ingestSource(name, opts = {}) {
+ const src = SOURCES[name];
+ if (!src) throw new Error(`unknown source: ${name}`);
+ const map = resolveMap(name);
+ const datasetLabel = src.datasetId || name;
+ const appToken = process.env.SOCRATA_APP_TOKEN || undefined;
+ const pageDelay = Number(process.env.INGEST_PAGE_DELAY_MS || 120);
+
+ const prev = await getState(name);
+ const since = !opts.full && src.cursorField ? prev?.last_cursor || null : null;
+
+ const pagerOpts = { since, appToken, maxRows: opts.maxRows ?? Infinity };
+ if (opts.pageSize) pagerOpts.pageSize = opts.pageSize;
+ const pager = src.platform === 'socrata' ? socrataPages(src, pagerOpts) : arcgisPages(src, pagerOpts);
+
+ log(`▶ ${name} [${src.platform} ${datasetLabel}] ${opts.full ? 'FULL' : since ? `since ${since}` : 'initial'}${opts.maxRows ? ` (max ${opts.maxRows})` : ''}`);
+
+ let total = 0;
+ let maxCursor = prev?.last_cursor || null;
+ let status = 'ok';
+ try {
+ for await (const { rows, url } of pager) {
+ const mapped = rows.map((r) => {
+ const row = { ...map(r), raw: r, source_url: url };
+ if (HAS_DATASET_ID.has(src.table)) row.dataset_id = datasetLabel;
+ return row;
+ });
+ // drop rows with a null primary key component (can't upsert those)
+ const clean = mapped.filter((row) => src.conflict.every((k) => row[k] != null));
+ const skipped = mapped.length - clean.length;
+ await upsert(src.table, src.conflict, clean);
+ total += clean.length;
+
+ // advance high-water mark
+ if (src.cursorField) {
+ for (const r of rows) {
+ const c = r[src.cursorField];
+ if (c != null && (maxCursor == null || String(c) > maxCursor)) maxCursor = String(c);
+ }
+ }
+ log(` +${clean.length}${skipped ? ` (skipped ${skipped} keyless)` : ''} total=${total}`);
+ if (pageDelay) await sleep(pageDelay);
+ }
+ } catch (err) {
+ status = 'error: ' + err.message;
+ log(`✖ ${name} failed: ${err.message}`);
+ await setState(name, { dataset_id: datasetLabel, last_cursor: maxCursor, rows_upserted: total, last_status: status });
+ throw err;
+ }
+
+ await setState(name, { dataset_id: datasetLabel, last_cursor: maxCursor, rows_upserted: total, last_status: status });
+ log(`✔ ${name} done — ${total} rows upserted this run${maxCursor ? `, cursor→${maxCursor}` : ''}`);
+ return { name, total, maxCursor };
+}
+
+export async function closePool() {
+ await pool.end();
+}
diff --git a/src/sources.js b/src/sources.js
new file mode 100644
index 0000000..2a4fbce
--- /dev/null
+++ b/src/sources.js
@@ -0,0 +1,187 @@
+// Source registry. Every endpoint here was live-probed by the la-research-agent
+// on 2026-08-10. `map(row)` returns the typed columns; the full source record is
+// stored separately as `raw` jsonb by ingest.js, so upstream schema drift never
+// loses data.
+
+// --- small typed-coercion helpers ---
+const str = (v) => (v === undefined || v === null || v === '' ? null : String(v));
+const num = (v) => {
+ if (v === undefined || v === null || v === '') return null;
+ const n = Number(v);
+ return Number.isFinite(n) ? n : null;
+};
+const iso = (v) => str(v); // Socrata/ArcGIS return ISO strings; let pg cast to date/timestamptz
+
+// LA City code-enforcement addresses are split across 6 columns with no combined field.
+const codeAddress = (r) =>
+ [r.stno, r.predir, r.stname, r.suffix, r.postdir]
+ .map((x) => (x == null ? '' : String(x).trim()))
+ .filter(Boolean)
+ .join(' ') || null;
+
+// Business dataset carries lat/lon inside a GeoJSON `location_1` point ([lon,lat]).
+const bizLatLon = (r) => {
+ const c = r.location_1 && r.location_1.coordinates;
+ return Array.isArray(c) && c.length === 2 ? { lon: num(c[0]), lat: num(c[1]) } : { lon: null, lat: null };
+};
+
+export const SOURCES = {
+ // ========================= BUILDING PERMITS (Socrata) =====================
+ building_permits: {
+ platform: 'socrata',
+ domain: 'data.lacity.org',
+ datasetId: 'pi9x-tg5x', // Issued 2020–present, ~405k, daily
+ table: 'la_building_permits_raw',
+ conflict: ['dataset_id', 'permit_nbr'],
+ cursorField: 'issue_date',
+ map: (r) => ({
+ permit_nbr: str(r.permit_nbr),
+ primary_address: str(r.primary_address),
+ apn: str(r.apn),
+ zip_code: str(r.zip_code),
+ council_district: str(r.cd),
+ permit_group: str(r.permit_group),
+ permit_type: str(r.permit_type),
+ permit_sub_type: str(r.permit_sub_type),
+ use_desc: str(r.use_desc),
+ issue_date: iso(r.issue_date),
+ status_desc: str(r.status_desc),
+ valuation: num(r.valuation),
+ lat: num(r.lat),
+ lon: num(r.lon),
+ work_desc: str(r.work_desc),
+ }),
+ },
+ // Static historical + related-trade permits share the same table/mapper.
+ building_permits_2010_2019: {
+ platform: 'socrata', domain: 'data.lacity.org', datasetId: 'dyxf-7hc4',
+ table: 'la_building_permits_raw', conflict: ['dataset_id', 'permit_nbr'],
+ static: true, mapFrom: 'building_permits',
+ },
+ building_permits_pre2010: {
+ platform: 'socrata', domain: 'data.lacity.org', datasetId: 'e67z-kt2n',
+ table: 'la_building_permits_raw', conflict: ['dataset_id', 'permit_nbr'],
+ static: true, mapFrom: 'building_permits',
+ },
+ building_permits_electrical: {
+ platform: 'socrata', domain: 'data.lacity.org', datasetId: 'ysqd-apz7',
+ table: 'la_building_permits_raw', conflict: ['dataset_id', 'permit_nbr'],
+ cursorField: 'issue_date', mapFrom: 'building_permits',
+ },
+ building_permits_mech_plumb: {
+ platform: 'socrata', domain: 'data.lacity.org', datasetId: '67is-svtd',
+ table: 'la_building_permits_raw', conflict: ['dataset_id', 'permit_nbr'],
+ cursorField: 'issue_date', mapFrom: 'building_permits',
+ },
+
+ // ========================= ASSESSOR PARCELS (ArcGIS) =====================
+ assessor_parcels: {
+ platform: 'arcgis',
+ endpoint:
+ 'https://services.arcgis.com/RmCCgQtiZLDCtblq/arcgis/rest/services/Parcel_Data_2021_Table/FeatureServer/0',
+ table: 'la_assessor_parcels_raw',
+ conflict: ['ain', 'roll_year'],
+ orderBy: 'AIN',
+ cursorField: 'RollYear',
+ // Default (non-full) refresh: newest roll year only (~2.4M vs 12.1M total).
+ defaultWhere: "RollYear = '2025'",
+ map: (r) => ({
+ ain: str(r.AIN),
+ roll_year: str(r.RollYear),
+ assessor_id: str(r.AssessorID),
+ property_location: str(r.PropertyLocation),
+ situs_house_no: str(r.SitusHouseNo),
+ situs_street: str(r.SitusStreet),
+ situs_city: str(r.SitusCity),
+ situs_zip5: str(r.SitusZIP5),
+ use_type: str(r.UseType),
+ use_code: str(r.UseCode),
+ year_built: str(r.YearBuilt),
+ sqft_main: num(r.SQFTmain),
+ bedrooms: str(r.Bedrooms),
+ bathrooms: str(r.Bathrooms),
+ units: str(r.Units),
+ land_value: num(r.Roll_LandValue),
+ imp_value: num(r.Roll_ImpValue),
+ total_value: num(r.Roll_TotalValue),
+ recording_date: num(r.RecordingDate),
+ center_lat: num(r.CENTER_LAT),
+ center_lon: num(r.CENTER_LON),
+ }),
+ },
+
+ // ========================= CODE ENFORCEMENT (Socrata) ====================
+ code_enforcement_open: {
+ platform: 'socrata', domain: 'data.lacity.org', datasetId: 'u82d-eh7z',
+ table: 'la_code_enforcement_raw', conflict: ['apno'], cursorField: 'adddttm',
+ map: (r) => ({
+ apno: str(r.apno), apname: str(r.apname), address: codeAddress(r),
+ zip: str(r.zip), add_dttm: iso(r.adddttm), res_dttm: iso(r.resdttm),
+ prclid: str(r.prclid), ap_type: str(r.aptype), apc: str(r.apc), stat: str(r.stat) || 'O',
+ }),
+ },
+ code_enforcement_closed: {
+ platform: 'socrata', domain: 'data.lacity.org', datasetId: 'rken-a55j',
+ table: 'la_code_enforcement_raw', conflict: ['apno'], cursorField: 'resdttm',
+ mapFrom: 'code_enforcement_open',
+ mapOverride: (r) => ({ stat: 'C' }),
+ },
+
+ // ========================= BUSINESS REGISTRATIONS (Socrata) ==============
+ business_registrations: {
+ platform: 'socrata', domain: 'data.lacity.org', datasetId: '6rrh-rzua',
+ table: 'la_business_registrations_raw', conflict: ['location_account'],
+ cursorField: 'location_start_date',
+ map: (r) => {
+ const { lat, lon } = bizLatLon(r);
+ return {
+ location_account: str(r.location_account),
+ business_name: str(r.business_name),
+ dba_name: str(r.dba_name),
+ street_address: str(r.street_address),
+ city: str(r.city),
+ zip_code: str(r.zip_code),
+ naics: str(r.naics),
+ primary_naics_description: str(r.primary_naics_description),
+ council_district: str(r.council_district),
+ location_start_date: iso(r.location_start_date),
+ location_end_date: iso(r.location_end_date),
+ lat, lon,
+ };
+ },
+ },
+
+ // ========================= FILM PERMITS (WeHo Socrata) ===================
+ // LA City proper has no live public dataset (both deprecated; FilmLA = CPRA only).
+ // WeHo is the live free feed. Schema fields probed defensively.
+ film_permits_weho: {
+ platform: 'socrata', domain: 'data.weho.org', datasetId: 'xd5y-dzvf',
+ table: 'la_film_permits_raw', conflict: ['permit_id'],
+ // WeHo schema (probed): permit_no, production, production_company, category, address, date.
+ map: (r) => ({
+ permit_id: str(r.permit_no || r.permit_number || r.id),
+ title: str(r.production || r.production_company),
+ permit_type: str(r.category),
+ address: str(r.address || r.location),
+ start_date: iso(r.date),
+ end_date: iso(r.date),
+ }),
+ },
+};
+
+// Resolve a source's effective mapper (supports mapFrom inheritance + mapOverride).
+export function resolveMap(name) {
+ const src = SOURCES[name];
+ const base = src.mapFrom ? SOURCES[src.mapFrom].map : src.map;
+ if (!base) throw new Error(`source ${name} has no map()`);
+ if (src.mapOverride) return (r) => ({ ...base(r), ...src.mapOverride(r) });
+ return base;
+}
+
+// Grouped for --list output.
+export const GROUPS = {
+ 'building permits': ['building_permits', 'building_permits_2010_2019', 'building_permits_pre2010', 'building_permits_electrical', 'building_permits_mech_plumb'],
+ 'assessor parcels': ['assessor_parcels'],
+ 'code enforcement': ['code_enforcement_open', 'code_enforcement_closed'],
+ 'business / film': ['business_registrations', 'film_permits_weho'],
+};
(oldest)
·
back to La Socrata Ingester
·
Fix: dedupe rows by conflict key within upsert batch (last-w 4c680aa →