← back to La Socrata Ingester
fix(la-data): repoint 9 NavigateLA layers after upstream +1 re-index; pin layers by NAME
88e40a8a1bc6aa9cc5c90e7915068d17a4635d2f · 2026-09-10 07:40:43 -0700 · Steve
On 2026-09-05 LA inserted a layer into the NavigateLA MapServer and every layer id
shifted +1. Sources pinned bare numeric ids, so all 9 GIS layers silently re-pointed
at a different dataset. Only 2 failed loudly (gis_fault_zones hit a Group Layer,
gis_flood hit a layer with no FLD_AR_ID) — those 400s are what has been failing the
nightly run's exit code since 09-05. The other 7 landed on queryable layers and
ingested a stranger's rows under our labels for 6 days with no error at all
(council_district 15->3, neighborhood_council 99->26, hpoz 35->7843,
community_plan_area 36->7, liquefaction 474->14181, fire_vhfhsz 14->247).
- repoint 71->72, 75->76, 119->120, 124->125, 254->255, 358->359, 414->415,
418->419, 439->440; verified read-only that all 6 countable layers return their
exact historical row counts and that 120/255/72 carry the fields our mappers need.
- add assertLayerIdentity(): every gisSrc now pins the expected upstream layer NAME
and the adapter verifies it before paging, so the next re-index fails LOUD instead
of silently ingesting the wrong layer. An ArcGIS layer id is a positional index,
not an identifier — this is the second id-drift incident (cf. 559875d).
- test/layer-identity.test.mjs: zero-network/zero-DB proof, incl. that all 9 sources
are pinned to the post-re-index ids.
Does not touch the 2026-08-30 allowFailure exit policy (ece3ea5): run-policy and
arcgis-retry suites still pass.
TK-10955
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019LHiEXMR2FCqRqUmoK4kQY
Files touched
M src/adapters/arcgis.jsM src/sources.jsA test/layer-identity.test.mjs
Diff
commit 88e40a8a1bc6aa9cc5c90e7915068d17a4635d2f
Author: Steve <steve@designerwallcoverings.com>
Date: Thu Sep 10 07:40:43 2026 -0700
fix(la-data): repoint 9 NavigateLA layers after upstream +1 re-index; pin layers by NAME
On 2026-09-05 LA inserted a layer into the NavigateLA MapServer and every layer id
shifted +1. Sources pinned bare numeric ids, so all 9 GIS layers silently re-pointed
at a different dataset. Only 2 failed loudly (gis_fault_zones hit a Group Layer,
gis_flood hit a layer with no FLD_AR_ID) — those 400s are what has been failing the
nightly run's exit code since 09-05. The other 7 landed on queryable layers and
ingested a stranger's rows under our labels for 6 days with no error at all
(council_district 15->3, neighborhood_council 99->26, hpoz 35->7843,
community_plan_area 36->7, liquefaction 474->14181, fire_vhfhsz 14->247).
- repoint 71->72, 75->76, 119->120, 124->125, 254->255, 358->359, 414->415,
418->419, 439->440; verified read-only that all 6 countable layers return their
exact historical row counts and that 120/255/72 carry the fields our mappers need.
- add assertLayerIdentity(): every gisSrc now pins the expected upstream layer NAME
and the adapter verifies it before paging, so the next re-index fails LOUD instead
of silently ingesting the wrong layer. An ArcGIS layer id is a positional index,
not an identifier — this is the second id-drift incident (cf. 559875d).
- test/layer-identity.test.mjs: zero-network/zero-DB proof, incl. that all 9 sources
are pinned to the post-re-index ids.
Does not touch the 2026-08-30 allowFailure exit policy (ece3ea5): run-policy and
arcgis-retry suites still pass.
TK-10955
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019LHiEXMR2FCqRqUmoK4kQY
---
src/adapters/arcgis.js | 26 +++++++++++++++++
src/sources.js | 31 +++++++++++++-------
test/layer-identity.test.mjs | 69 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 116 insertions(+), 10 deletions(-)
diff --git a/src/adapters/arcgis.js b/src/adapters/arcgis.js
index 3654e6a..bd7b9fa 100644
--- a/src/adapters/arcgis.js
+++ b/src/adapters/arcgis.js
@@ -28,6 +28,30 @@ async function fetchArcgis(url, { tries = 5 } = {}) {
}
}
+// ── Layer-identity preflight ────────────────────────────────────────────────
+// An ArcGIS layer id is a POSITIONAL INDEX, not a stable identifier. On 2026-09-05
+// LA re-indexed the NavigateLA MapServer and EVERY layer id shifted +1, silently
+// re-pointing all 9 of our layers at a different dataset. Only 2 failed loudly (a
+// Group Layer and a layer missing our oidField); the other 7 landed on queryable
+// layers and ingested a stranger's rows under our labels for 6 days with no error.
+// So: when a source declares expectName, verify the LIVE layer name before paging
+// and fail LOUD on drift. Ingesting the wrong layer is strictly worse than not
+// ingesting at all — one metadata GET per source per run ($0, public API).
+const normName = (s) => String(s ?? '').trim().toLowerCase().replace(/\s+/g, ' ');
+
+export async function assertLayerIdentity(src, fetcher = fetchArcgis) {
+ if (!src.expectName) return; // opt-in: unpinned sources keep prior behaviour
+ const meta = await fetcher(`${src.endpoint}?f=json`);
+ const actual = meta?.name;
+ if (normName(actual) !== normName(src.expectName)) {
+ throw new Error(
+ `ArcGIS layer identity drift: ${src.endpoint} is now "${actual ?? '(unnamed)'}" ` +
+ `but this source expects "${src.expectName}". Upstream re-indexed its layer ids — ` +
+ `repoint the layer id (verify by NAME), do NOT ingest.`
+ );
+ }
+}
+
// ArcGIS FeatureServer/MapServer paginator. Yields { rows, url } per page, where
// each row is the feature's attributes (with __geometry attached when src.geometry).
//
@@ -46,6 +70,8 @@ export async function* arcgisPages(src, opts = {}) {
const oidField = src.oidField || 'OBJECTID';
const geom = !!src.geometry;
+ await assertLayerIdentity(src); // fail loud if the layer id no longer means what we think
+
// base filters (everything except the oid cursor)
const filters = [];
if (whereOverride) filters.push(whereOverride);
diff --git a/src/sources.js b/src/sources.js
index 0a683ac..22bd085 100644
--- a/src/sources.js
+++ b/src/sources.js
@@ -172,6 +172,13 @@ export const SOURCES = {
};
// ========================= GIS LAYERS (LA City NavigateLA ArcGIS) ==========
+// NavigateLA MapServer. HISTORY — 2026-09-05: LA inserted a layer and EVERY layer id
+// shifted +1 (71->72, 75->76, 119->120, 124->125, 254->255, 358->359, 414->415,
+// 418->419, 439->440). We pinned bare ids, so all 9 sources silently re-pointed:
+// gis_fault_zones + gis_flood 400'd loudly, the other 7 ingested the WRONG layer's rows
+// under our labels for 6 days (council_district 15->3, neighborhood_council 99->26,
+// hpoz 35->7843, community_plan_area 36->7, liquefaction 474->14181, fire_vhfhsz 14->247).
+// Fixed by repointing +1 AND pinning expectName so the next re-index fails loud instead.
const NAV = 'https://maps.lacity.org/arcgis/rest/services/Mapping/NavigateLA/MapServer';
// generic feature -> la_gis_features row
const gisFeat = (layer, nameFields = []) => (r) => ({
@@ -179,9 +186,13 @@ const gisFeat = (layer, nameFields = []) => (r) => ({
name: nameFields.map((f) => r[f]).find((v) => v != null && v !== '') ?? str(r.TOOLTIP),
geom: r.__geometry,
});
-const gisSrc = (layer, layerId, nameFields, extra = {}) => ({
+// NavigateLA layer ids are POSITIONAL and DO drift (see the 2026-09-05 +1 re-index
+// below), so every gisSrc pins the expected upstream layer NAME. The arcgis adapter
+// verifies it before paging and fails loud rather than ingesting the wrong dataset.
+const gisSrc = (layer, layerId, nameFields, expectName, extra = {}) => ({
platform: 'arcgis', endpoint: `${NAV}/${layerId}`, geometry: true,
table: 'la_gis_features', conflict: ['layer', 'oid'],
+ expectName,
map: gisFeat(layer, nameFields), ...extra,
});
@@ -201,7 +212,7 @@ Object.assign(SOURCES, {
},
// Zoning + land use (~50–59k). NavigateLA gateway 502s on sorted geometry -> use
// offset paging with NO orderByFields (noOrder) + small pages.
- gis_zoning: gisSrc('zoning', 71, ['ZONE_CMPLT', 'ZONE_CLASS'], {
+ gis_zoning: gisSrc('zoning', 72, ['ZONE_CMPLT', 'ZONE_CLASS'], 'Generalized Zoning', {
pageSize: 2000,
noOrder: true,
// Audited exception: keep attempting this source so an upstream recovery is
@@ -223,15 +234,15 @@ Object.assign(SOURCES, {
map: (r) => ({ layer: 'landuse', oid: r.objectid, name: str(r.gplu_desc || r.gplu), geom: r.__geometry }),
},
// Boundaries (small).
- gis_council_districts: gisSrc('council_district', 418, ['District_Name', 'NAME', 'District'], { pageSize: 20000 }),
- gis_neighborhood_councils: gisSrc('neighborhood_council', 439, ['NAME'], { pageSize: 20000 }),
- gis_hpoz: gisSrc('hpoz', 75, ['NAME'], { pageSize: 20000 }),
- gis_community_plan_areas: gisSrc('community_plan_area', 414, ['NAME'], { pageSize: 20000 }),
+ gis_council_districts: gisSrc('council_district', 419, ['District_Name', 'NAME', 'District'], 'Council Districts', { pageSize: 20000 }),
+ gis_neighborhood_councils: gisSrc('neighborhood_council', 440, ['NAME'], 'Neighborhood Councils (Certified)', { pageSize: 20000 }),
+ gis_hpoz: gisSrc('hpoz', 76, ['NAME'], 'Historic Preservation Overlay Zone District',{ pageSize: 20000 }),
+ gis_community_plan_areas: gisSrc('community_plan_area', 415, ['NAME'], 'Community Plan Areas', { pageSize: 20000 }),
// Hazards (small).
- gis_fault_zones: gisSrc('fault_zone', 119, ['HAZ_TYPE'], { pageSize: 20000 }),
- gis_liquefaction: gisSrc('liquefaction', 124, [], { pageSize: 20000 }),
- gis_flood: gisSrc('flood', 254, ['FLD_ZONE'], { pageSize: 20000, oidField: 'FLD_AR_ID' }),
- gis_fire_vhfhsz: gisSrc('fire_vhfhsz', 358, [], { pageSize: 20000 }),
+ gis_fault_zones: gisSrc('fault_zone', 120, ['HAZ_TYPE'], 'Alquist Priolo Earthquake Fault Zones', { pageSize: 20000 }),
+ gis_liquefaction: gisSrc('liquefaction', 125, [], 'Liquefaction', { pageSize: 20000 }),
+ gis_flood: gisSrc('flood', 255, ['FLD_ZONE'], 'Special Flood Hazard Areas (S_FLD_HAZ_AR) (OPCS) (Eff. 4/21/2021)', { pageSize: 20000, oidField: 'FLD_AR_ID' }),
+ gis_fire_vhfhsz: gisSrc('fire_vhfhsz', 359, [], 'Very High Fire Hazard Severity Zones', { pageSize: 20000 }),
// NOTE: methane (layer 354) has NO pagination support — needs spatial tiling; deferred.
});
diff --git a/test/layer-identity.test.mjs b/test/layer-identity.test.mjs
new file mode 100644
index 0000000..044cd93
--- /dev/null
+++ b/test/layer-identity.test.mjs
@@ -0,0 +1,69 @@
+// TK-10955: zero-network, zero-database proof of the ArcGIS layer-identity guard.
+//
+// Regression under test: on 2026-09-05 LA re-indexed the NavigateLA MapServer and every
+// layer id shifted +1. Because sources pinned bare numeric ids, 7 of 9 layers silently
+// ingested a DIFFERENT dataset under our labels for 6 days with no error at all. The
+// guard must make that class loud, and must never ingest a mismatched layer.
+import assert from 'node:assert/strict';
+import { assertLayerIdentity } from '../src/adapters/arcgis.js';
+import { SOURCES } from '../src/sources.js';
+
+const fakeFetch = (name) => async () => ({ name });
+
+// 1. Matching name passes.
+await assertLayerIdentity(
+ { endpoint: 'x/419', expectName: 'Council Districts' },
+ fakeFetch('Council Districts')
+);
+
+// 2. Whitespace/case differences are not drift.
+await assertLayerIdentity(
+ { endpoint: 'x/419', expectName: 'Council Districts' },
+ fakeFetch(' council districts ')
+);
+
+// 3. The exact 2026-09-05 shift must THROW, not silently ingest.
+await assert.rejects(
+ () => assertLayerIdentity(
+ { endpoint: 'x/418', expectName: 'Council Districts' },
+ fakeFetch('Contract Administration Inspection Districts')
+ ),
+ /layer identity drift/,
+ 'a shifted layer id must fail loud'
+);
+
+// 4. A missing/unnamed layer (e.g. the Group Layer at old id 119) must THROW.
+await assert.rejects(
+ () => assertLayerIdentity(
+ { endpoint: 'x/119', expectName: 'Alquist Priolo Earthquake Fault Zones' },
+ async () => ({})
+ ),
+ /layer identity drift/,
+ 'an unnamed/group layer must fail loud'
+);
+
+// 5. Sources without expectName keep prior behaviour (opt-in guard, no network call).
+let called = false;
+await assertLayerIdentity({ endpoint: 'x/0' }, async () => { called = true; return {}; });
+assert.equal(called, false, 'unpinned source must not preflight');
+
+// 6. Every NavigateLA source is pinned, and pinned to the post-2026-09-05 ids.
+const EXPECTED = {
+ gis_zoning: [72, 'Generalized Zoning'],
+ gis_council_districts: [419, 'Council Districts'],
+ gis_neighborhood_councils: [440, 'Neighborhood Councils (Certified)'],
+ gis_hpoz: [76, 'Historic Preservation Overlay Zone District'],
+ gis_community_plan_areas: [415, 'Community Plan Areas'],
+ gis_fault_zones: [120, 'Alquist Priolo Earthquake Fault Zones'],
+ gis_liquefaction: [125, 'Liquefaction'],
+ gis_fire_vhfhsz: [359, 'Very High Fire Hazard Severity Zones'],
+ gis_flood: [255, 'Special Flood Hazard Areas (S_FLD_HAZ_AR) (OPCS) (Eff. 4/21/2021)'],
+};
+for (const [name, [id, expectName]] of Object.entries(EXPECTED)) {
+ const src = SOURCES[name];
+ assert.ok(src, `${name} must exist`);
+ assert.equal(src.expectName, expectName, `${name} must pin its upstream layer name`);
+ assert.ok(src.endpoint.endsWith(`/${id}`), `${name} must point at layer ${id}, got ${src.endpoint}`);
+}
+
+console.log('✔ layer-identity guard: 6/6 checks pass (no network, no database)');
← f5caca4 auto-data-snapshot: 2026-09-08T08:24:00 (1 data files) — dat
·
back to La Socrata Ingester
·
fix(la-data): scope allowFailure to the audited failure, not f6042cd →