[object Object]

← back to La Socrata Ingester

fix(la-data): tolerate assessor_parcels' audited page-1 400 (TK-10955)

93a7ac5b5c28a18e099214303f8df5a443b70fd3 · 2026-09-12 13:21:28 -0700 · Steve Abrams

FeatureServer has twice returned the same generic HTTP 400 ("Cannot perform
query. Invalid query parameters.") on page 1 before any row is fetched
(2026-09-10, 2026-09-12), always ~55.3s after the request to within 50ms
across both occurrences -- consistent with an upstream execution-time
ceiling under load, not a bad query. Manual replay of the byte-identical
runtime query succeeded live both times. Table is upsert-only (conflict on
ain+roll_year, never deleted) and the OID-cursor rescans the full current
roll year every run, so a skipped day just leaves yesterday's 2.4M-row
snapshot in place one extra day -- no data loss, no corruption.

Added an allowFailure exemption scoped to the exact audited message,
mirroring gis_zoning's existing precedent exactly: a genuinely different
assessor_parcels error still fails the aggregate run loud, and a direct
(non-aggregate) run stays strict. Extended test/run-policy.test.mjs with a
positive control, a negative control, and a direct-run-stays-strict check.

Verified: 4/4 suites pass (18 checks); live smoke test (--max=200) confirms
normal ingestion is unaffected; full 'cli.js all' aggregate re-run today
exits 0 via the real classifyRunResults against real SOURCES (gis_zoning's
known 502 tolerated, assessor_parcels succeeded outright this run).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MffyeUFB1BstaK2xS7ufYd

Files touched

Diff

commit 93a7ac5b5c28a18e099214303f8df5a443b70fd3
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Sat Sep 12 13:21:28 2026 -0700

    fix(la-data): tolerate assessor_parcels' audited page-1 400 (TK-10955)
    
    FeatureServer has twice returned the same generic HTTP 400 ("Cannot perform
    query. Invalid query parameters.") on page 1 before any row is fetched
    (2026-09-10, 2026-09-12), always ~55.3s after the request to within 50ms
    across both occurrences -- consistent with an upstream execution-time
    ceiling under load, not a bad query. Manual replay of the byte-identical
    runtime query succeeded live both times. Table is upsert-only (conflict on
    ain+roll_year, never deleted) and the OID-cursor rescans the full current
    roll year every run, so a skipped day just leaves yesterday's 2.4M-row
    snapshot in place one extra day -- no data loss, no corruption.
    
    Added an allowFailure exemption scoped to the exact audited message,
    mirroring gis_zoning's existing precedent exactly: a genuinely different
    assessor_parcels error still fails the aggregate run loud, and a direct
    (non-aggregate) run stays strict. Extended test/run-policy.test.mjs with a
    positive control, a negative control, and a direct-run-stays-strict check.
    
    Verified: 4/4 suites pass (18 checks); live smoke test (--max=200) confirms
    normal ingestion is unaffected; full 'cli.js all' aggregate re-run today
    exits 0 via the real classifyRunResults against real SOURCES (gis_zoning's
    known 502 tolerated, assessor_parcels succeeded outright this run).
    
    Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01MffyeUFB1BstaK2xS7ufYd
---
 src/sources.js           | 15 +++++++++++++++
 test/run-policy.test.mjs | 29 +++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+)

diff --git a/src/sources.js b/src/sources.js
index 436aa82..95bcbef 100644
--- a/src/sources.js
+++ b/src/sources.js
@@ -87,6 +87,21 @@ export const SOURCES = {
     cursorField: 'RollYear',
     // Default (non-full) refresh: newest roll year only (~2.4M vs 12.1M total).
     defaultWhere: "RollYear = '2025'",
+    // Audited exception (TK-10955): this FeatureServer has twice returned this exact
+    // generic 400 on page 1 — before any row is fetched — always ~55s after the request
+    // (2026-09-10, 2026-09-12). Both times a manual replay of the byte-identical query
+    // succeeded seconds to minutes later, so this is upstream load, not a bad query.
+    // Table is upsert-only (never deleted) and the OID-cursor rescans the whole current
+    // roll year every run, so a skipped day just leaves yesterday's snapshot in place one
+    // day longer — no row loss, no corruption. Scoped to the EXACT audited message so a
+    // genuinely different assessor_parcels failure (a real bad query) still fails the
+    // aggregate run loud, same discipline as gis_zoning's exemption above.
+    allowFailure: {
+      since: '2026-09-10',
+      scope: 'aggregate-all-only',
+      reason: 'FeatureServer intermittently 400s on page 1 under load (~55s in); resolves on manual retry; upsert-only table so a skipped day is not data loss',
+      match: ['Cannot perform query. Invalid query parameters.'],
+    },
     map: (r) => ({
       ain: str(r.AIN),
       roll_year: str(r.RollYear),
diff --git a/test/run-policy.test.mjs b/test/run-policy.test.mjs
index 867414d..246222c 100644
--- a/test/run-policy.test.mjs
+++ b/test/run-policy.test.mjs
@@ -90,3 +90,32 @@ assert.equal(
 );
 
 console.log('PASS allowFailure is scoped to the audited failure; identity drift still exits 1');
+
+// ── TK-10955 (2026-09-12 recurrence): assessor_parcels' audited page-1 400 must be
+// tolerated in aggregate `all`, but a genuinely different assessor_parcels failure (a
+// real bad query, or an upstream schema change) must still fail loud — same discipline
+// as gis_zoning's exemption above, now proven on a second source.
+const auditedAssessor400 = classifyRunResults(
+  [{ name: 'assessor_parcels', error: 'ArcGIS error: {"code":400,"message":"Cannot perform query. Invalid query parameters.","details":["Unable to perform query. Please check your parameters."]}' }],
+  SOURCES, 'all'
+);
+assert.equal(auditedAssessor400.exitCode, 0, 'the audited assessor_parcels page-1 400 must stay tolerated in aggregate all');
+assert.deepEqual(auditedAssessor400.tolerated.map((r) => r.name), ['assessor_parcels']);
+
+const unrelatedAssessorError = classifyRunResults(
+  [{ name: 'assessor_parcels', error: 'ArcGIS error: {"code":400,"message":"Invalid field: RollYear2"}' }],
+  SOURCES, 'all'
+);
+assert.equal(unrelatedAssessorError.exitCode, 1, 'a different assessor_parcels 400 must NOT be swallowed by the audited exemption');
+assert.deepEqual(unrelatedAssessorError.unexpected.map((r) => r.name), ['assessor_parcels']);
+assert.equal(unrelatedAssessorError.tolerated.length, 0);
+
+const directAssessor400 = classifyRunResults(
+  [{ name: 'assessor_parcels', error: 'ArcGIS error: {"code":400,"message":"Cannot perform query. Invalid query parameters.","details":["Unable to perform query. Please check your parameters."]}' }],
+  SOURCES, 'assessor_parcels'
+);
+assert.equal(directAssessor400.exitCode, 1, 'a direct assessor_parcels run must remain strict (aggregate-all-only scope)');
+
+assert.ok(Array.isArray(SOURCES.assessor_parcels.allowFailure.match), 'assessor_parcels must scope its exemption');
+
+console.log('PASS assessor_parcels audited page-1 400 tolerated in aggregate all; unrelated/direct failures still exit 1');

← 28fcd49 fix(la-data): don't cache a failed catalog fetch; report pro  ·  back to La Socrata Ingester  ·  chore: v0.2.4 (session close) 1879ff0 →