TODO: log generator batch-timeout diagnosis + local fix (deploy Steve-gated)

9d134cd4e6f10210b341d4882d8791338bbd4ca1 · 2026-06-01 14:31:29 -0700 · Steve Abrams

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

M TODO.md

Diff

commit 9d134cd4e6f10210b341d4882d8791338bbd4ca1
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Mon Jun 1 14:31:29 2026 -0700

    TODO: log generator batch-timeout diagnosis + local fix (deploy Steve-gated)
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 TODO.md | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/TODO.md b/TODO.md
index 5378667..7be4987 100644
--- a/TODO.md
+++ b/TODO.md
@@ -4,6 +4,12 @@ Surfaced to CNCP at http://localhost:3333 (via /api/tasks reading this file).
 Format — `## Section` headers + `- [ ] task` / `- [x] done`. CNCP picks
 up unchecked rows automatically.
 
+## Generator-health regression — batch child timeout (2026-06-01, DIAGNOSED + LOCAL FIX, deploy Steve-gated)
+- [x] Root cause: `0/N survived the quality gate (code null)` was NOT a gate rejection. The batch child SIGTERM-timed-out mid-flight. Batch route budgeted 150s/render (`Math.max(180000, n*150000)`) but comfy polls up to 15min/render (commit 86d053ad). Every multi-render comfy batch was killed before finishing; runs 27/28/30 hit their exact timeouts (3/10/30 min) and died at code=null. 5 designs (10251-10255) DID land in prod PG, correctly unpublished — the run rows lied.
+- [x] Fix (commit 3eeffee3, server.js + lib/{ghost,composition,subject}-detector.js): (1) per-render budget 150s→18min +5min slack, cap 4h (env `WALLCO_BATCH_PER_RENDER_MS`); (2) code=null signal-kill distinguished from real gate-reject(code 1), reported accurately; (3) reconciler falls back to harvesting `✓ #<id>` lines when killed-child never prints the `Created…IDs:` summary; (4) boot reaper closes runs stuck in `running` >30min (8 prod orphans incl pm2-wipe #29); (5) retired gemini-2.0-flash (404 since 2026-06-01) → gemini-2.5-flash in the 3 in-gen detectors (matches b25dd485).
+- [ ] DEPLOY Steve-gated: `./deploy-kamatera.sh` from ~/Projects/wallco-ai. On boot the reaper closes the 8 stuck `running` rows (#1,2,21-25,29). node -c clean. No publish/override touched.
+- [ ] Not in batch path, but still on retired 2.0-flash (curation/audit tools — fix when convenient): scripts/bleed_guard.js, scripts/rate_design.js, scripts/audit-{dogs-breed,stoned-animals-mood,cohort-generic,drunk-animals-abstraction}.js.
+
 ## Prompt-vs-image rating (2026-06-01, Steve-approved both gated steps)
 - [x] Migration: user_prompt_match (smallint) + user_prompt_verdict (text) + user_prompt_rated_at (timestamptz). LOCAL via CREATE OR REPLACE VIEW (spoon_all_designs is a view over all_designs locally); PROD via ALTER TABLE spoon_all_designs (it's a real table on prod — topology drift). Both applied + verified.
 - [x] prompt-rate API: PG is source of truth (list reads cols; POST writes PG-first strict, merge-from-PG, then JSONL audit append). is_published never touched (verified live before=after).

← 929ffc6 fix(generator): batch child timeout + orphan-run reaper + re · back to Wallco Ai · migrate remaining retired gemini-2.0-flash vision refs to ge efb671c →