← back to La Socrata Ingester
fix(la-data): adaptive page-size backoff for oversized-geometry gateway 502s (TK-11500)
cbc32be36c23253529f549a454d871d342afbc12 · 2026-09-12 20:22:20 -0700 · Steve Abrams
Zoning stalled at 75% (43,957 of 58,953 rows) because the NavigateLA gateway 502s
on a GEOMETRY page whose response exceeds ~30s to serialize. Root cause (pre-measured
by claude-run-10955, re-confirmed live 2026-09-12): the OBJECTID 42000-56000 band
carries ~22KB of geometry per feature, so a 2000-row page there is ~44MB. It is a
fixed OBJECTID band, not a fixed depth, so neither offset- nor oid-cursor paging
avoids it — the constraint is response SIZE.
Fix: arcgisPages now halves resultRecordCount on a gateway size-failure and retries
the SAME cursor position (down to a floor, LA_ADAPTIVE_MIN_PAGE default 50), then
grows back x2 on success so the rest of the layer still pages fast (AIMD). General to
both offset and oid modes. fetchArcgis gains an httpTries knob so the descent fails
fast while shrinking and spends the full retry budget only at the floor — a page that
502s even at the floor is a real outage and is rethrown (fail-loud; the aggregate
run's allowFailure exemption still catches a genuinely stuck source). Only geometry
pages shrink; a non-geometry transient is a real upstream outage and is never masked.
Proven three ways: negative test test/arcgis-adaptive-page.test.mjs (red on the pre-fix
baseline, green with the fix; covers offset backoff, all-band completion, non-geometry
fail-loud, oid mode); full existing suite unchanged; and a bounded read-only live run
against the real OBJECTID>42000 band — real 502 at 2000 caught, halved to 1000,
completed 2200 rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkjvkxHU1B2a7gfVQFKzxX
Files touched
M src/adapters/arcgis.jsA test/arcgis-adaptive-page.test.mjs
Diff
commit cbc32be36c23253529f549a454d871d342afbc12
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Sat Sep 12 20:22:20 2026 -0700
fix(la-data): adaptive page-size backoff for oversized-geometry gateway 502s (TK-11500)
Zoning stalled at 75% (43,957 of 58,953 rows) because the NavigateLA gateway 502s
on a GEOMETRY page whose response exceeds ~30s to serialize. Root cause (pre-measured
by claude-run-10955, re-confirmed live 2026-09-12): the OBJECTID 42000-56000 band
carries ~22KB of geometry per feature, so a 2000-row page there is ~44MB. It is a
fixed OBJECTID band, not a fixed depth, so neither offset- nor oid-cursor paging
avoids it — the constraint is response SIZE.
Fix: arcgisPages now halves resultRecordCount on a gateway size-failure and retries
the SAME cursor position (down to a floor, LA_ADAPTIVE_MIN_PAGE default 50), then
grows back x2 on success so the rest of the layer still pages fast (AIMD). General to
both offset and oid modes. fetchArcgis gains an httpTries knob so the descent fails
fast while shrinking and spends the full retry budget only at the floor — a page that
502s even at the floor is a real outage and is rethrown (fail-loud; the aggregate
run's allowFailure exemption still catches a genuinely stuck source). Only geometry
pages shrink; a non-geometry transient is a real upstream outage and is never masked.
Proven three ways: negative test test/arcgis-adaptive-page.test.mjs (red on the pre-fix
baseline, green with the fix; covers offset backoff, all-band completion, non-geometry
fail-loud, oid mode); full existing suite unchanged; and a bounded read-only live run
against the real OBJECTID>42000 band — real 502 at 2000 caught, halved to 1000,
completed 2200 rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkjvkxHU1B2a7gfVQFKzxX
---
src/adapters/arcgis.js | 103 ++++++++++++++++++++++------
test/arcgis-adaptive-page.test.mjs | 134 +++++++++++++++++++++++++++++++++++++
2 files changed, 218 insertions(+), 19 deletions(-)
diff --git a/src/adapters/arcgis.js b/src/adapters/arcgis.js
index 1a4d1df..37e6acb 100644
--- a/src/adapters/arcgis.js
+++ b/src/adapters/arcgis.js
@@ -11,11 +11,16 @@ const isTransientArcgisError = (e) =>
!!e && (TRANSIENT_ARCGIS.has(e.code) || e.messageCode === 'CONT_0001');
// fetchJson + ArcGIS-payload-aware retry with capped exponential backoff.
-async function fetchArcgis(url, { tries = 5 } = {}) {
+// httpTries is the HTTP-level retry budget inside fetchJson (default 5). The adaptive
+// page-size backoff (below) passes a SMALL httpTries while it still has room to shrink a
+// page, so an oversized-geometry band does not pay the full 5x/~30s backoff storm on every
+// halving step — and it restores the full budget at the floor, so a genuine gateway outage
+// (a page that 502s even at the floor) still surfaces loudly rather than being cut short.
+async function fetchArcgis(url, { tries = 5, httpTries = 5 } = {}) {
let attempt = 0;
for (;;) {
attempt++;
- const data = await fetchJson(url); // HTTP-level (429/5xx/network) retry already inside
+ const data = await fetchJson(url, { tries: httpTries }); // HTTP-level (429/5xx/network) retry already inside
if (!data.error) return data;
if (!isTransientArcgisError(data.error) || attempt >= tries) {
throw new Error(`ArcGIS error: ${JSON.stringify(data.error).slice(0, 200)}`);
@@ -28,6 +33,61 @@ async function fetchArcgis(url, { tries = 5 } = {}) {
}
}
+// ── Adaptive page-size backoff (TK-11500) ───────────────────────────────────
+// The NavigateLA gateway 502s on a GEOMETRY page whose response takes >30s to serialize.
+// MEASURED on the Generalized Zoning layer: the OBJECTID 42000..56000 band carries ~22KB of
+// geometry per feature (vs ~0.2KB elsewhere), so a 2000-row page there is ~44MB and the
+// gateway times out. Because it is a fixed OBJECTID band and not a fixed depth, NEITHER
+// offset- nor oid-cursor paging avoids it (both were live-tested and both 502 at the same
+// band) — the real constraint is response SIZE. So on a gateway size-failure, HALVE the page
+// and retry the SAME cursor position until the response fits (down to a floor), then grow the
+// size back so the rest of the layer still pages fast (AIMD: multiplicative-decrease on
+// failure, x2 grow-back on success). General — any oversized-geometry source benefits, not
+// zoning-specific. A page that still fails at the floor is a real outage, not a size problem,
+// so it is rethrown (fail-loud; the aggregate run's allowFailure exemption still catches a
+// genuinely stuck source, so this degrades exactly as before rather than looping forever).
+// Read at CALL time, not module-load time — same lesson the repoint kill switches learned
+// (a module-level const bakes the value in at import and ignores a later env override).
+const adaptiveMinPage = () => Math.max(1, Number(process.env.LA_ADAPTIVE_MIN_PAGE ?? 50));
+// A gateway 502/503/504 in either shape fetchArcgis can throw it: the HTTP-level form from
+// fetchJson ("HTTP 502 after N tries") and the ArcGIS body-level form ('ArcGIS error:
+// {"code":502,...}'). This is the class we shrink for.
+const isGatewaySizeFailure = (err) =>
+ /HTTP 50[234]\b|"code":\s*50[234]\b/.test(String(err?.message || ''));
+// After a shrunk page succeeds, grow back toward pageSize (x2) so throughput recovers once
+// past a heavy band; a page that ran at full size stays there (no-op).
+const nextPageSize = (usedCount, pageSize) =>
+ usedCount < pageSize ? Math.min(pageSize, usedCount * 2) : pageSize;
+
+// Fetch one page at `count`, shrinking on a gateway size-failure and retrying the SAME
+// cursor. buildUrl(count) produces the query URL for a given resultRecordCount. Only shrinks
+// for GEOMETRY pages — a non-geometry page is already tiny, so a transient there is a real
+// upstream outage, not a size problem, and must not be masked. Returns { data, usedCount }
+// where usedCount is the size that actually succeeded (feeds the end-of-layer test + grow-back).
+async function fetchPageAdaptive(buildUrl, count, geom) {
+ const floor = adaptiveMinPage();
+ for (;;) {
+ const atFloor = count <= floor;
+ try {
+ // Fail fast (httpTries:2) while there is still room to shrink; spend the full retry
+ // budget only at the floor, so a real outage (fails even at the floor) still throws.
+ const data = await fetchArcgis(buildUrl(count), { httpTries: atFloor ? 5 : 2 });
+ return { data, usedCount: count };
+ } catch (err) {
+ if (geom && !atFloor && isGatewaySizeFailure(err)) {
+ const next = Math.max(floor, Math.floor(count / 2));
+ console.error(
+ ` ⤵ ArcGIS geometry page too large — halving resultRecordCount ${count} → ${next} at the same cursor ` +
+ `(${String(err.message).slice(0, 70)})`
+ );
+ count = next;
+ continue;
+ }
+ throw err;
+ }
+ }
+}
+
// ── Layer-identity preflight ────────────────────────────────────────────────
// An ArcGIS layer id is a POSITIONAL INDEX, not a stable identifier. On 2026-09-05
// LA re-indexed the NavigateLA MapServer and EVERY layer id shifted +1, silently
@@ -251,41 +311,46 @@ export async function* arcgisPages(src, opts = {}) {
feats.map((ft) => (geom ? { ...ft.attributes, __geometry: ft.geometry } : ft.attributes));
if (src.paginate === 'oid') {
- let lastOid = -1, fetched = 0;
+ let lastOid = -1, fetched = 0, curCount = pageSize;
for (;;) {
if (fetched >= maxRows) break;
const where = [...filters, `${oidField} > ${lastOid}`].join(' AND ');
- const count = Math.min(pageSize, maxRows - fetched);
- const p = new URLSearchParams({ ...common, where, orderByFields: oidField, resultRecordCount: String(count) });
- const url = `${base}?${p}`;
- const data = await fetchArcgis(url);
+ const remaining = maxRows - fetched;
+ const buildUrl = (n) =>
+ `${base}?${new URLSearchParams({ ...common, where, orderByFields: oidField, resultRecordCount: String(n) })}`;
+ // Adaptive page size: shrink on an oversized-geometry gateway 502, grow back on success.
+ const { data, usedCount } = await fetchPageAdaptive(buildUrl, Math.min(curCount, remaining), geom);
const feats = data.features || [];
if (!feats.length) break;
- yield { rows: mapRows(feats), url };
+ yield { rows: mapRows(feats), url: buildUrl(usedCount) };
lastOid = Math.max(...feats.map((ft) => ft.attributes[oidField]));
fetched += feats.length;
- if (feats.length < count) break;
+ curCount = nextPageSize(usedCount, pageSize);
+ if (feats.length < usedCount) break;
}
return;
}
// offset mode
- let offset = 0;
+ let offset = 0, curCount = pageSize;
const where = filters.length ? filters.join(' AND ') : '1=1';
- for (;;) {
- const count = Math.min(pageSize, maxRows - offset);
- if (count <= 0) break;
- const params = { ...common, where, resultOffset: String(offset), resultRecordCount: String(count) };
+ const buildUrl = (n) => {
+ const params = { ...common, where, resultOffset: String(offset), resultRecordCount: String(n) };
// src.noOrder: omit orderByFields — the NavigateLA gateway 502s on sorted
// geometry queries; MapServer returns natural OBJECTID order anyway.
if (!src.noOrder) params.orderByFields = src.orderBy || oidField;
- const p = new URLSearchParams(params);
- const url = `${base}?${p}`;
- const data = await fetchArcgis(url);
+ return `${base}?${new URLSearchParams(params)}`;
+ };
+ for (;;) {
+ const remaining = maxRows - offset;
+ if (remaining <= 0) break;
+ // Adaptive page size: shrink on an oversized-geometry gateway 502, grow back on success.
+ const { data, usedCount } = await fetchPageAdaptive(buildUrl, Math.min(curCount, remaining), geom);
const feats = data.features || [];
if (!feats.length) break;
- yield { rows: mapRows(feats), url };
+ yield { rows: mapRows(feats), url: buildUrl(usedCount) };
offset += feats.length;
- if (!data.exceededTransferLimit && feats.length < count) break;
+ curCount = nextPageSize(usedCount, pageSize);
+ if (!data.exceededTransferLimit && feats.length < usedCount) break;
}
}
diff --git a/test/arcgis-adaptive-page.test.mjs b/test/arcgis-adaptive-page.test.mjs
new file mode 100644
index 0000000..7506df3
--- /dev/null
+++ b/test/arcgis-adaptive-page.test.mjs
@@ -0,0 +1,134 @@
+// Regression test for TK-11500: the NavigateLA gateway 502s on a GEOMETRY page whose
+// response is too large (the Generalized Zoning OBJECTID 42000-56000 band — ~44MB/2000-row
+// page). arcgisPages must HALVE the page and retry the SAME cursor until it fits, then GROW
+// the size back once past the heavy band, and stay FAIL-LOUD when a page fails for a
+// non-size reason (non-geometry) so a real outage is never masked as a shrink.
+// Run: node test/arcgis-adaptive-page.test.mjs (zero deps; a few s due to real backoff)
+// Small floor so the test's small page sizes exercise real shrinking (read at call time).
+process.env.LA_ADAPTIVE_MIN_PAGE = '4';
+import { arcgisPages } from '../src/adapters/arcgis.js';
+
+let fail = 0;
+const feat = (i) => ({ attributes: { OBJECTID: i, ZONE_CMPLT: `Z${i}` }, geometry: { rings: [[[i, i]]] } });
+
+// Build a mock `fetch` over a finite OFFSET-mode dataset of `total` features, where a
+// gateway 502 is returned whenever the requested resultRecordCount exceeds `okAtOrBelow`
+// AND the current resultOffset is inside [0, bandEnd) — i.e. an oversized-geometry BAND.
+// Records every requested page size and each 502 so the test can assert shrink + recovery.
+function offsetMock({ total, okAtOrBelow, bandEnd }) {
+ const log = { sizes: [], gateway502: 0 };
+ globalThis.fetch = async (url) => {
+ const u = new URL(url);
+ const rc = Number(u.searchParams.get('resultRecordCount'));
+ const off = Number(u.searchParams.get('resultOffset'));
+ log.sizes.push(rc);
+ if (rc > okAtOrBelow && off < bandEnd) {
+ log.gateway502++;
+ return { ok: false, status: 502, json: async () => ({}), text: async () => 'gateway timeout' };
+ }
+ const slice = [];
+ for (let i = off; i < Math.min(off + rc, total); i++) slice.push(feat(i));
+ const body = { features: slice, exceededTransferLimit: off + rc < total };
+ return { ok: true, status: 200, json: async () => body, text: async () => JSON.stringify(body) };
+ };
+ return log;
+}
+
+const collect = async (src, opts) => {
+ const out = [];
+ for await (const page of arcgisPages(src, opts)) out.push(...page.rows);
+ return out;
+};
+
+// ── 1) OFFSET geometry: descends through an oversized band, then GROWS back ──────────────
+// total 25, oversized band = offsets [0,20) (502 if page>10 there), floor 5, start page 20.
+{
+ const log = offsetMock({ total: 25, okAtOrBelow: 10, bandEnd: 20 });
+ const src = { endpoint: 'http://fake/MapServer/73', geometry: true, noOrder: true };
+ try {
+ const rows = await collect(src, { pageSize: 20 });
+ const complete = rows.length === 25 && rows.every((r, i) => r.OBJECTID === i && r.__geometry);
+ const shrank = log.gateway502 >= 1 && log.sizes.includes(10); // it halved 20->10 in the band
+ const recovered = log.sizes.some((s, k) => s === 20 && k === log.sizes.length - 1); // last page ran full-size past the band
+ const ok = complete && shrank && recovered;
+ console.log(`OFFSET-BACKOFF: ${ok ? 'PASS (descended past the band, completed 25/25, grew back to full page past it)' :
+ `FAIL (rows=${rows.length}, 502s=${log.gateway502}, sizes=${log.sizes.join(',')})`}`);
+ if (!ok) fail++;
+ } catch (e) {
+ console.log(`OFFSET-BACKOFF: FAIL (threw: ${e.message}; sizes=${log.sizes.join(',')})`);
+ fail++;
+ }
+}
+
+// ── 2) NEGATIVE: without the fix this class is unrecoverable — prove the FIX is load-bearing.
+// Same oversized band but page size can NEVER be honored at 20; a naive paginator (no
+// shrink) would 502 out. With the fix it must still complete. We assert it did NOT simply
+// error, AND that it actually used a smaller page (proving the completion came from shrinking,
+// not from the band happening to allow 20).
+{
+ const log = offsetMock({ total: 12, okAtOrBelow: 5, bandEnd: 999 }); // band = whole layer
+ const src = { endpoint: 'http://fake/MapServer/73', geometry: true, noOrder: true };
+ try {
+ const rows = await collect(src, { pageSize: 20 });
+ const ok = rows.length === 12 && log.gateway502 >= 1 && log.sizes.some((s) => s <= 5);
+ console.log(`ALL-BAND-COMPLETE: ${ok ? 'PASS (a layer that 502s at every full page still completes 12/12 via shrink)' :
+ `FAIL (rows=${rows.length}, 502s=${log.gateway502}, minPage=${Math.min(...log.sizes)})`}`);
+ if (!ok) fail++;
+ } catch (e) {
+ console.log(`ALL-BAND-COMPLETE: FAIL (threw instead of shrinking to complete: ${e.message})`);
+ fail++;
+ }
+}
+
+// ── 3) SAFETY: a NON-geometry 502 is a real outage, NOT a size problem — must NOT be masked
+// by shrinking; it must throw (fail-loud). geometry:false, 502 on every page.
+{
+ const log = offsetMock({ total: 10, okAtOrBelow: -1, bandEnd: 999 }); // always 502
+ const src = { endpoint: 'http://fake/MapServer/73', geometry: false, noOrder: true };
+ const before = log.sizes.length;
+ try {
+ await collect(src, { pageSize: 20 });
+ console.log('NONGEOM-FAILLOUD: FAIL (did not throw on a non-geometry gateway outage)');
+ fail++;
+ } catch (e) {
+ // it must have thrown WITHOUT ever halving the page (no size below the starting 20)
+ const neverShrank = log.sizes.every((s) => s === 20);
+ const ok = /HTTP 502/.test(e.message) && neverShrank;
+ console.log(`NONGEOM-FAILLOUD: ${ok ? 'PASS (non-geometry 502 threw loud, never masked by a page shrink)' :
+ `FAIL (msg=${e.message}, sizes=${log.sizes.slice(before).join(',')})`}`);
+ if (!ok) fail++;
+ }
+}
+
+// ── 4) OID mode is wired too (zoning is offset, but the fix is general) ───────────────────
+// oid-cursor mock: 502 if page>8; success returns features with OBJECTID > lastOid.
+{
+ const log = { sizes: [], gateway502: 0 };
+ const total = 20;
+ globalThis.fetch = async (url) => {
+ const u = new URL(url);
+ const rc = Number(u.searchParams.get('resultRecordCount'));
+ const m = /OBJECTID%20%3E%20(-?\d+)|OBJECTID > (-?\d+)/.exec(decodeURIComponent(u.searchParams.get('where') || ''));
+ const lastOid = Number((m && (m[1] ?? m[2])) ?? -1);
+ log.sizes.push(rc);
+ if (rc > 8) { log.gateway502++; return { ok: false, status: 502, json: async () => ({}), text: async () => 'to' }; }
+ const slice = [];
+ for (let i = lastOid + 1; i < total && slice.length < rc; i++) slice.push(feat(i));
+ const body = { features: slice };
+ return { ok: true, status: 200, json: async () => body, text: async () => JSON.stringify(body) };
+ };
+ const src = { endpoint: 'http://fake/FeatureServer/0', paginate: 'oid', geometry: true, oidField: 'OBJECTID' };
+ try {
+ const rows = await collect(src, { pageSize: 16 });
+ const ok = rows.length === total && log.gateway502 >= 1 && log.sizes.includes(8);
+ console.log(`OID-BACKOFF: ${ok ? 'PASS (oid-cursor mode also shrinks + completes 20/20)' :
+ `FAIL (rows=${rows.length}, 502s=${log.gateway502}, sizes=${log.sizes.join(',')})`}`);
+ if (!ok) fail++;
+ } catch (e) {
+ console.log(`OID-BACKOFF: FAIL (threw: ${e.message})`);
+ fail++;
+ }
+}
+
+console.log(fail ? `\n${fail} FAILED` : '\nALL PASS');
+process.exit(fail);
← 1879ff0 chore: v0.2.4 (session close)
·
back to La Socrata Ingester
·
auto-data-snapshot: 2026-09-15T08:12:49 (1 data files) — dat d0c1307 →