← back to La Socrata Ingester
fix(la-data): repoint 9 NavigateLA layers after a SECOND upstream +1 insertion
314f1dca5e526b2b1f6fdaafb4d76e75b0477bbd · 2026-09-11 11:24:41 -0700 · Steve Abrams
LA inserted the Feature Layer "Special Event Permits" at position 52 of the
NavigateLA MapServer (layer count 452 -> 453), shifting every layer at id >= 52
by +1 — the same mechanism as 2026-09-05, six days later.
This time the expectName guard from 88e40a8 caught it: the 2026-09-11 04:30 run
failed all 8 pinned NavigateLA sources loud at preflight and ingested NOTHING,
so there was no corruption to clean up. la_gis_features was verified to still
hold exactly the correct upstream counts. The exit=1 was a TRUE POSITIVE.
Repointed +1 and verified 9/9 on BOTH planes that matter — exact upstream layer
NAME and historical ROW COUNT: zoning 72->73, council_districts 419->420,
neighborhood_councils 440->441, hpoz 76->77, community_plan_areas 415->416,
fault_zones 120->121, liquefaction 125->126, flood 255->256, fire_vhfhsz
359->360. Live: 9/9 preflight pass, 9/9 negative control (the now-stale ids are
all rejected). Re-ingest of the 8 affected sources: 8/8 exit 0 with exactly the
historical counts, content spot-checked semantically (real council members, real
community plan areas, real HPOZ districts) — not just matching counts.
test/layer-identity.test.mjs check 6 rewritten. It asserted specific numeric ids,
which is a restatement of config: it proves nothing about correctness and turns
every legitimate repoint into a test edit. It now asserts the durable invariants —
every NavigateLA-served source is ARMED (pins an expectName, or the guard is a
no-op for it) and no two sources expect the same upstream name. Both are proven
to go red under injected faults.
Documented the trap that nearly shaped the fix wrongly: /MapServer?f=json is
cacheable and was observed serving a STALE body (452 layers, old index) while the
per-layer endpoints already served the new one. Resolving a layer by name off the
catalog alone would have returned the WRONG id. The authoritative plane is
/MapServer/<id>?f=json, because that is what /query follows.
TK-10955. $0 local (free public ArcGIS GETs + local Postgres).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CocwcUw6gFt3tr4CZJmra6
Files touched
M src/sources.jsM test/layer-identity.test.mjs
Diff
commit 314f1dca5e526b2b1f6fdaafb4d76e75b0477bbd
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Fri Sep 11 11:24:41 2026 -0700
fix(la-data): repoint 9 NavigateLA layers after a SECOND upstream +1 insertion
LA inserted the Feature Layer "Special Event Permits" at position 52 of the
NavigateLA MapServer (layer count 452 -> 453), shifting every layer at id >= 52
by +1 — the same mechanism as 2026-09-05, six days later.
This time the expectName guard from 88e40a8 caught it: the 2026-09-11 04:30 run
failed all 8 pinned NavigateLA sources loud at preflight and ingested NOTHING,
so there was no corruption to clean up. la_gis_features was verified to still
hold exactly the correct upstream counts. The exit=1 was a TRUE POSITIVE.
Repointed +1 and verified 9/9 on BOTH planes that matter — exact upstream layer
NAME and historical ROW COUNT: zoning 72->73, council_districts 419->420,
neighborhood_councils 440->441, hpoz 76->77, community_plan_areas 415->416,
fault_zones 120->121, liquefaction 125->126, flood 255->256, fire_vhfhsz
359->360. Live: 9/9 preflight pass, 9/9 negative control (the now-stale ids are
all rejected). Re-ingest of the 8 affected sources: 8/8 exit 0 with exactly the
historical counts, content spot-checked semantically (real council members, real
community plan areas, real HPOZ districts) — not just matching counts.
test/layer-identity.test.mjs check 6 rewritten. It asserted specific numeric ids,
which is a restatement of config: it proves nothing about correctness and turns
every legitimate repoint into a test edit. It now asserts the durable invariants —
every NavigateLA-served source is ARMED (pins an expectName, or the guard is a
no-op for it) and no two sources expect the same upstream name. Both are proven
to go red under injected faults.
Documented the trap that nearly shaped the fix wrongly: /MapServer?f=json is
cacheable and was observed serving a STALE body (452 layers, old index) while the
per-layer endpoints already served the new one. Resolving a layer by name off the
catalog alone would have returned the WRONG id. The authoritative plane is
/MapServer/<id>?f=json, because that is what /query follows.
TK-10955. $0 local (free public ArcGIS GETs + local Postgres).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CocwcUw6gFt3tr4CZJmra6
---
src/sources.js | 48 ++++++++++++++++++++++++++++--------------
test/layer-identity.test.mjs | 50 ++++++++++++++++++++++++++++----------------
2 files changed, 64 insertions(+), 34 deletions(-)
diff --git a/src/sources.js b/src/sources.js
index 5c59794..2a2b8d2 100644
--- a/src/sources.js
+++ b/src/sources.js
@@ -172,13 +172,29 @@ export const SOURCES = {
};
// ========================= GIS LAYERS (LA City NavigateLA ArcGIS) ==========
-// NavigateLA MapServer. HISTORY — 2026-09-05: LA inserted a layer and EVERY layer id
-// shifted +1 (71->72, 75->76, 119->120, 124->125, 254->255, 358->359, 414->415,
-// 418->419, 439->440). We pinned bare ids, so all 9 sources silently re-pointed:
-// gis_fault_zones + gis_flood 400'd loudly, the other 7 ingested the WRONG layer's rows
-// under our labels for 6 days (council_district 15->3, neighborhood_council 99->26,
-// hpoz 35->7843, community_plan_area 36->7, liquefaction 474->14181, fire_vhfhsz 14->247).
-// Fixed by repointing +1 AND pinning expectName so the next re-index fails loud instead.
+// NavigateLA layer ids are POSITIONAL: they are an index into a list LA edits, so
+// inserting ONE layer renumbers every layer below it. This has now happened twice.
+//
+// 2026-09-05 ids shifted +1 (71->72, 75->76, 119->120, 124->125, 254->255,
+// 358->359, 414->415, 418->419, 439->440). Sources pinned bare ids, so
+// all 9 silently re-pointed: fault_zones + flood 400'd loudly, the other
+// 7 ingested a DIFFERENT dataset under our labels for 6 days with no
+// error (council_district 15->3, neighborhood_council 99->26, hpoz
+// 35->7843, community_plan_area 36->7, liquefaction 474->14181,
+// fire_vhfhsz 14->247). Fixed by repointing +1 and adding expectName.
+// 2026-09-11 ids shifted +1 AGAIN — LA inserted the Feature Layer 'Special Event
+// Permits' at position 52 (layer count 452 -> 453), shifting everything
+// at id >= 52. This time the expectName guard caught it: all 8 pinned
+// NavigateLA sources failed loud at preflight and ingested NOTHING, so
+// there was no corruption to clean up. Repointed +1 again (see ids below).
+//
+// TRAP, learned 2026-09-11 — do NOT resolve an id by name from the SERVICE CATALOG
+// (/MapServer?f=json) alone. That response is cacheable and was observed serving a
+// STALE body (452 layers, old index) while the per-layer endpoints already served the
+// new index; resolving 'Council Districts' off it would have returned 419, which by
+// then was 'Contract Administration Inspection Districts'. The authoritative plane is
+// the per-layer endpoint /MapServer/<id>?f=json, because that is what /query follows —
+// and that is the plane assertLayerIdentity reads.
const NAV = 'https://maps.lacity.org/arcgis/rest/services/Mapping/NavigateLA/MapServer';
// generic feature -> la_gis_features row
const gisFeat = (layer, nameFields = []) => (r) => ({
@@ -212,7 +228,7 @@ Object.assign(SOURCES, {
},
// Zoning + land use (~50–59k). NavigateLA gateway 502s on sorted geometry -> use
// offset paging with NO orderByFields (noOrder) + small pages.
- gis_zoning: gisSrc('zoning', 72, ['ZONE_CMPLT', 'ZONE_CLASS'], 'Generalized Zoning', {
+ gis_zoning: gisSrc('zoning', 73, ['ZONE_CMPLT', 'ZONE_CLASS'], 'Generalized Zoning', {
pageSize: 2000,
noOrder: true,
// Audited exception: keep attempting this source so an upstream recovery is
@@ -240,15 +256,15 @@ Object.assign(SOURCES, {
map: (r) => ({ layer: 'landuse', oid: r.objectid, name: str(r.gplu_desc || r.gplu), geom: r.__geometry }),
},
// Boundaries (small).
- gis_council_districts: gisSrc('council_district', 419, ['District_Name', 'NAME', 'District'], 'Council Districts', { pageSize: 20000 }),
- gis_neighborhood_councils: gisSrc('neighborhood_council', 440, ['NAME'], 'Neighborhood Councils (Certified)', { pageSize: 20000 }),
- gis_hpoz: gisSrc('hpoz', 76, ['NAME'], 'Historic Preservation Overlay Zone District',{ pageSize: 20000 }),
- gis_community_plan_areas: gisSrc('community_plan_area', 415, ['NAME'], 'Community Plan Areas', { pageSize: 20000 }),
+ gis_council_districts: gisSrc('council_district', 420, ['District_Name', 'NAME', 'District'], 'Council Districts', { pageSize: 20000 }),
+ gis_neighborhood_councils: gisSrc('neighborhood_council', 441, ['NAME'], 'Neighborhood Councils (Certified)', { pageSize: 20000 }),
+ gis_hpoz: gisSrc('hpoz', 77, ['NAME'], 'Historic Preservation Overlay Zone District',{ pageSize: 20000 }),
+ gis_community_plan_areas: gisSrc('community_plan_area', 416, ['NAME'], 'Community Plan Areas', { pageSize: 20000 }),
// Hazards (small).
- gis_fault_zones: gisSrc('fault_zone', 120, ['HAZ_TYPE'], 'Alquist Priolo Earthquake Fault Zones', { pageSize: 20000 }),
- gis_liquefaction: gisSrc('liquefaction', 125, [], 'Liquefaction', { pageSize: 20000 }),
- gis_flood: gisSrc('flood', 255, ['FLD_ZONE'], 'Special Flood Hazard Areas (S_FLD_HAZ_AR) (OPCS) (Eff. 4/21/2021)', { pageSize: 20000, oidField: 'FLD_AR_ID' }),
- gis_fire_vhfhsz: gisSrc('fire_vhfhsz', 359, [], 'Very High Fire Hazard Severity Zones', { pageSize: 20000 }),
+ gis_fault_zones: gisSrc('fault_zone', 121, ['HAZ_TYPE'], 'Alquist Priolo Earthquake Fault Zones', { pageSize: 20000 }),
+ gis_liquefaction: gisSrc('liquefaction', 126, [], 'Liquefaction', { pageSize: 20000 }),
+ gis_flood: gisSrc('flood', 256, ['FLD_ZONE'], 'Special Flood Hazard Areas (S_FLD_HAZ_AR) (OPCS) (Eff. 4/21/2021)', { pageSize: 20000, oidField: 'FLD_AR_ID' }),
+ gis_fire_vhfhsz: gisSrc('fire_vhfhsz', 360, [], 'Very High Fire Hazard Severity Zones', { pageSize: 20000 }),
// NOTE: methane (layer 354) has NO pagination support — needs spatial tiling; deferred.
});
diff --git a/test/layer-identity.test.mjs b/test/layer-identity.test.mjs
index 044cd93..97108a7 100644
--- a/test/layer-identity.test.mjs
+++ b/test/layer-identity.test.mjs
@@ -47,23 +47,37 @@ let called = false;
await assertLayerIdentity({ endpoint: 'x/0' }, async () => { called = true; return {}; });
assert.equal(called, false, 'unpinned source must not preflight');
-// 6. Every NavigateLA source is pinned, and pinned to the post-2026-09-05 ids.
-const EXPECTED = {
- gis_zoning: [72, 'Generalized Zoning'],
- gis_council_districts: [419, 'Council Districts'],
- gis_neighborhood_councils: [440, 'Neighborhood Councils (Certified)'],
- gis_hpoz: [76, 'Historic Preservation Overlay Zone District'],
- gis_community_plan_areas: [415, 'Community Plan Areas'],
- gis_fault_zones: [120, 'Alquist Priolo Earthquake Fault Zones'],
- gis_liquefaction: [125, 'Liquefaction'],
- gis_fire_vhfhsz: [359, 'Very High Fire Hazard Severity Zones'],
- gis_flood: [255, 'Special Flood Hazard Areas (S_FLD_HAZ_AR) (OPCS) (Eff. 4/21/2021)'],
-};
-for (const [name, [id, expectName]] of Object.entries(EXPECTED)) {
- const src = SOURCES[name];
- assert.ok(src, `${name} must exist`);
- assert.equal(src.expectName, expectName, `${name} must pin its upstream layer name`);
- assert.ok(src.endpoint.endsWith(`/${id}`), `${name} must point at layer ${id}, got ${src.endpoint}`);
+// 6. INVARIANT (not a config restatement): every source served off the NavigateLA
+// MapServer must be ARMED — it must pin an expectName — or the identity guard is a
+// no-op for it (assertLayerIdentity returns early when expectName is absent, check 5).
+// Deliberately NOT asserting specific numeric ids: upstream layer ids are POSITIONAL
+// and shift +1 whenever LA inserts a layer (twice in six days: 2026-09-05 and
+// 2026-09-11, the latter caused by inserting 'Special Event Permits' at position 52).
+// Asserting them would make every legitimate repoint a test edit while proving nothing
+// about correctness — the id is upstream state, not our invariant. Ids as of
+// 2026-09-11 for the record: zoning 73, council_districts 420, neighborhood_councils 441,
+// hpoz 77, community_plan_areas 416, fault_zones 121, liquefaction 126, flood 256,
+// fire_vhfhsz 360.
+const NAV_PREFIX = 'https://maps.lacity.org/arcgis/rest/services/Mapping/NavigateLA/MapServer/';
+const navSources = Object.entries(SOURCES).filter(
+ ([, src]) => typeof src.endpoint === 'string' && src.endpoint.startsWith(NAV_PREFIX)
+);
+assert.ok(navSources.length >= 9, `expected >=9 NavigateLA sources, got ${navSources.length}`);
+for (const [name, src] of navSources) {
+ assert.ok(
+ typeof src.expectName === 'string' && src.expectName.trim() !== '',
+ `${name} is served off NavigateLA but pins no expectName — the identity guard is DISARMED for it`
+ );
+ assert.match(src.endpoint.slice(NAV_PREFIX.length), /^\d+$/, `${name} must end in a numeric layer id`);
+}
+
+// 7. Two sources must never expect the SAME upstream layer name — that would mean one
+// of them is mislabelled, and a name-based repoint could not tell them apart.
+const seen = new Map();
+for (const [name, src] of navSources) {
+ const key = src.expectName.trim().toLowerCase().replace(/\s+/g, ' ');
+ assert.ok(!seen.has(key), `${name} and ${seen.get(key)} both expect "${src.expectName}"`);
+ seen.set(key, name);
}
-console.log('✔ layer-identity guard: 6/6 checks pass (no network, no database)');
+console.log('✔ layer-identity guard: 7/7 checks pass (no network, no database)');
← f6042cd fix(la-data): scope allowFailure to the audited failure, not
·
back to La Socrata Ingester
·
feat(la-data): verified auto-repoint for drifting NavigateLA 482ed58 →