cycle 25 docs: YOLO_NOTES ledger — ingest fetch timeout + per-record isolation

86389afd47b3b046d11e50e9f2825bfb500f3a40 · 2026-09-24 04:40:39 -0700 · Steve

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TouFkmUGKHtwqpZwgReVic

Files touched

M YOLO_NOTES.md

Diff

commit 86389afd47b3b046d11e50e9f2825bfb500f3a40
Author: Steve <steve@designerwallcoverings.com>
Date:   Thu Sep 24 04:40:39 2026 -0700

    cycle 25 docs: YOLO_NOTES ledger — ingest fetch timeout + per-record isolation
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01TouFkmUGKHtwqpZwgReVic
---
 YOLO_NOTES.md | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/YOLO_NOTES.md b/YOLO_NOTES.md
index 6dceaf0..2455014 100644
--- a/YOLO_NOTES.md
+++ b/YOLO_NOTES.md
@@ -542,3 +542,23 @@ Canonical apply order now: `004_marketplace → 005_apple → 006_contacts → 0
 **Queued (Cody's 3 lower-sev ingest items, no live exploit path — next cycles, reversible/no-gate):** (1) slug-collision dedup in meic-pymes.js silently MERGES two different businesses (missing-cedula fallback slugs on name+canton — common Spanish trade names collide → ON CONFLICT grafts B's fields onto A); (2) one poison record aborts the WHOLE run in meic-pymes.js + ict-cst.js (no per-row try/catch — google-places.js + local-portals.js got it right); (3) no fetch timeout in scripts/ingest/_lib.js (a hung gov site stalls run-all.js forever; cr-portal-crawl.js has a correct 20s AbortController to copy).
 
 **Backlog:** FOUR gated items on Steve's desk — host-claim approval, reconciler cron, payout-visibility decision, and now the XSS deploy (security-urgent). Next cycle: ingest hardening items (2)+(3) above (per-record isolation + fetch timeout — cheapest, reversible), then (1) the dedup key.
+
+---
+
+## Cycle 25 — ingest layer resilience: fetch timeout + per-record isolation (TK-10346)
+
+**Surface:** continuing cycle 24's ingest-layer audit — the two lower-severity, no-live-exploit-path findings (fetch timeout, poison-record batch abort), done as reversible/no-gate fixes.
+
+**Fix 1 — fetch timeout (`scripts/ingest/_lib.js`):** `fetchText`/`fetchJson`/`fetchBuffer` accepted `opts.signal` but no caller ever passed one, and Node's `fetch` has no default timeout — a hung gov site (CKAN, ICT WordPress) could stall `run-all.js` forever (it runs modules sequentially). Added a default `AbortSignal.timeout()`: 30s for HTML/JSON (`SCRAPER_TIMEOUT_MS`), 60s for file downloads (a separate `SCRAPER_BUFFER_TIMEOUT_MS` — see the gate finding below for why it needed its OWN knob). A caller-supplied `opts.signal` is honored as-is (none currently exist — verified all 8 `_lib` callers).
+
+**Fix 2 — per-record isolation (`meic-pymes.js` + `ict-cst.js`):** the per-row/per-listing body sat inside ONE outer try; a single bad row aborted the whole run (`meic-pymes` can process up to 15,000 rows — a poison row at #500 drops the other ~14,500). Wrapped each in its own try/catch (`errors++`, warn capped at 20, continue); `finishRun` downgrades to `status='partial'` instead of `'error'`/exit 1 when errors>0 — mirrors the pattern `google-places.js`/`local-portals.js` already used correctly. Verified `status='partial'` has no schema constraint and only 2 read-only consumers (no branching risk).
+
+**Cody gate caught a real bug in my own fix:** `fetchBuffer`'s timeout fell through `Number(opts.timeoutMs) || fallbackMs || DEFAULT_TIMEOUT_MS` — an absent `opts.timeoutMs` → `NaN` (falsy) → the hardcoded `60000` `fallbackMs` literal always won, silently making `SCRAPER_TIMEOUT_MS` a no-op for the one fetch (the multi-MB MEIC XLSX) that most needs a real override on a slow link. Confirmed by direct reproduction (env=999999 → still resolved to 60000). Fixed by giving `fetchBuffer` its **own** dedicated `SCRAPER_BUFFER_TIMEOUT_MS` env var instead of a fallback chain, then verified with a **live-timing** test (swap env vars, measure actual elapsed ms — not just code inspection) proving 35ms vs. an irrelevant 5000ms setting.
+
+**Tests:** `test/ingest-resilience.test.js` — behavioral (fake never-resolving fetch → `TimeoutError`; caller-signal honored; fetchBuffer's own env knob proven live) + structural (per-record catch + partial-status regression guards on both scripts). Suite **209 → 215**, serial green. Commit `f10bf96`.
+
+**Deprioritized correctly (Cody, no action taken):** `osm-fetch.js` has a raw unguarded `fetch()` but is NOT in `run-all.js`'s `ORDER` array — can't stall the pipeline, manual-run-only. `cr-osm-match.js` has the same no-per-record-isolation pattern but on a DB-only loop (no fetch) — queued for a later cycle, lower priority.
+
+**Cost:** $0 (local + one Cody gate pass that found a real bug in the fix's env-override claim, verified independently before and after the fix with live timing, not just code review).
+
+**Backlog:** still FOUR gated items on Steve's desk (host-claim approval, reconciler cron, payout-visibility, place-xss-DEPLOY — security-urgent). Concrete-deferred code backlog: `cr-osm-match.js` per-record isolation (cheap, same pattern) + the meic-pymes slug-collision dedup (Cody's item #1 — likely warrants its own DECISION memo since it touches ingest identity semantics, not a pure mechanical fix).

← f10bf96 costa-rica: ingest layer resilience — fetch timeout + per-re · back to Costa Rica · costa-rica: cr-osm-match.js — per-record isolation + guarant f12347a →