← back to Wallco Ai
fix(generator): batch child timeout + orphan-run reaper + retired-gemini gate model
929ffc66d97e0e83a778e2e5e5384827bcba1c80 · 2026-06-01 14:30:49 -0700 · Steve Abrams
Root cause of the '0/N survived the quality gate (code null)' regression:
the batch child SIGTERM-timed-out mid-flight, NOT a gate rejection.
- batch timeout was 150s/render but comfy polls up to 15min/render (29ac6e71),
so any n>2 batch was killed before finishing; budget now 18min/render +5min
slack, capped 4h (WALLCO_BATCH_PER_RENDER_MS override).
- code=null (signal kill) was mislabeled survived-the-quality-gate; now
distinguished from real gate rejection (code 1) and reported accurately.
- reconciler only parsed the final Created-IDs line a killed child never
prints; now falls back to harvesting per-render checkmark id lines so
survivors (5 designs DID land in prod PG for runs 27/28/30) are credited.
- new boot reaper closes generator runs stuck in running >30min — orphans
from pm2 wipes / OOM / hard restarts (8 stuck rows on prod, incl pm2-wipe #29).
- ghost/composition/subject detectors defaulted to retired gemini-2.0-flash
(404 since 2026-06-01); migrated to gemini-2.5-flash, matching 71535517.
Local-only; node -c clean. Deploy is Steve-gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Files touched
M lib/composition-detector.jsM lib/ghost-detector.jsM lib/subject-detector.jsM server.js
Diff
commit 929ffc66d97e0e83a778e2e5e5384827bcba1c80
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Mon Jun 1 14:30:49 2026 -0700
fix(generator): batch child timeout + orphan-run reaper + retired-gemini gate model
Root cause of the '0/N survived the quality gate (code null)' regression:
the batch child SIGTERM-timed-out mid-flight, NOT a gate rejection.
- batch timeout was 150s/render but comfy polls up to 15min/render (29ac6e71),
so any n>2 batch was killed before finishing; budget now 18min/render +5min
slack, capped 4h (WALLCO_BATCH_PER_RENDER_MS override).
- code=null (signal kill) was mislabeled survived-the-quality-gate; now
distinguished from real gate rejection (code 1) and reported accurately.
- reconciler only parsed the final Created-IDs line a killed child never
prints; now falls back to harvesting per-render checkmark id lines so
survivors (5 designs DID land in prod PG for runs 27/28/30) are credited.
- new boot reaper closes generator runs stuck in running >30min — orphans
from pm2 wipes / OOM / hard restarts (8 stuck rows on prod, incl pm2-wipe #29).
- ghost/composition/subject detectors defaulted to retired gemini-2.0-flash
(404 since 2026-06-01); migrated to gemini-2.5-flash, matching 71535517.
Local-only; node -c clean. Deploy is Steve-gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
lib/composition-detector.js | 4 +-
lib/ghost-detector.js | 7 +++-
lib/subject-detector.js | 4 +-
server.js | 92 +++++++++++++++++++++++++++++++++++++++------
4 files changed, 93 insertions(+), 14 deletions(-)
diff --git a/lib/composition-detector.js b/lib/composition-detector.js
index f91bb8a..c89c7ad 100644
--- a/lib/composition-detector.js
+++ b/lib/composition-detector.js
@@ -20,7 +20,9 @@
const fs = require('fs');
const path = require('path');
-const GEMINI_MODEL = process.env.COMPOSITION_DETECTOR_MODEL || 'gemini-2.0-flash';
+// 2026-06-01: gemini-2.0-flash retired by Google (404) — migrate to 2.5-flash
+// (matches settlement-post-gen-vision fix b25dd485). Override via COMPOSITION_DETECTOR_MODEL.
+const GEMINI_MODEL = process.env.COMPOSITION_DETECTOR_MODEL || 'gemini-2.5-flash';
const GEMINI_BASE = 'https://generativelanguage.googleapis.com/v1beta/models';
// Scenic categories — these are MURAL designs (single panoramic scenes spanning
diff --git a/lib/ghost-detector.js b/lib/ghost-detector.js
index 9502cf1..eafb9f8 100644
--- a/lib/ghost-detector.js
+++ b/lib/ghost-detector.js
@@ -21,7 +21,12 @@
const fs = require('fs');
const path = require('path');
-const GEMINI_MODEL = process.env.GHOST_DETECTOR_MODEL || 'gemini-2.0-flash';
+// 2026-06-01: gemini-2.0-flash was retired by Google and now 404s ("model no
+// longer available"), making every ghost-vote error out (non-fatal in the batch
+// path but pure latency + log noise, and fail-CLOSED anywhere this gate is
+// mandatory). Same migration commit b25dd485 already moved settlement-post-gen-
+// vision to 2.5-flash. Override via GHOST_DETECTOR_MODEL if needed.
+const GEMINI_MODEL = process.env.GHOST_DETECTOR_MODEL || 'gemini-2.5-flash';
const GEMINI_BASE = 'https://generativelanguage.googleapis.com/v1beta/models';
const PROMPT_TILE = `You are inspecting a TILED WALLPAPER REPEAT-PATTERN image for a manufacturing defect called a "ghost layer."
diff --git a/lib/subject-detector.js b/lib/subject-detector.js
index 1629703..223bf70 100644
--- a/lib/subject-detector.js
+++ b/lib/subject-detector.js
@@ -25,7 +25,9 @@
const fs = require('fs');
const path = require('path');
-const GEMINI_MODEL = process.env.SUBJECT_DETECTOR_MODEL || 'gemini-2.0-flash';
+// 2026-06-01: gemini-2.0-flash retired by Google (404) — migrate to 2.5-flash
+// (matches settlement-post-gen-vision fix b25dd485). Override via SUBJECT_DETECTOR_MODEL.
+const GEMINI_MODEL = process.env.SUBJECT_DETECTOR_MODEL || 'gemini-2.5-flash';
const GEMINI_BASE = 'https://generativelanguage.googleapis.com/v1beta/models';
// Anti-prompt qualifiers we strip from extracted subject phrases — these
diff --git a/server.js b/server.js
index 60fc6d1..466e44f 100644
--- a/server.js
+++ b/server.js
@@ -4706,6 +4706,33 @@ function ensureUserRemovedColumn() {
}
ensureUserRemovedColumn();
+// Reap orphaned generator runs (2026-06-01). A batch/recipe child whose
+// 'exit' handler never fired — killed by a pm2 wipe (`pm2 kill`/`pm2 update`),
+// an OOM, or a hard server restart — leaves its wallco_generator_runs row
+// stuck at status='running' forever, never flipping to done/error. On every
+// boot, any run still 'running' from BEFORE this process started is by
+// definition an orphan (a live in-flight child can only belong to a process
+// that's still up — and this is a fresh boot), so close it as 'error' with a
+// clear cause. The 30-min floor is belt-and-suspenders for a run that started
+// seconds before a crash. This is a status-bookkeeping UPDATE only; it never
+// touches designs, never publishes, never deletes.
+function reapOrphanedGeneratorRuns() {
+ try {
+ const n = psqlQuery(`WITH reaped AS (
+ UPDATE wallco_generator_runs
+ SET status='error',
+ error='orphaned: server restarted while run was in-flight (child exit handler never fired — pm2 wipe / OOM / hard restart). See data/logs/batch-<id>-*.log for any designs that landed.',
+ finished_at=now()
+ WHERE status='running'
+ AND started_at < now() - interval '30 minutes'
+ RETURNING id)
+ SELECT count(*) FROM reaped;`);
+ const cnt = parseInt(n, 10) || 0;
+ if (cnt > 0) console.log(`[migrate] reaped ${cnt} orphaned generator run(s) stuck in 'running'`);
+ } catch (e) { console.error('[migrate] reapOrphanedGeneratorRuns:', e.message); }
+}
+reapOrphanedGeneratorRuns();
+
// Audit-log — every user + admin action lands here. Append-only. Steve's call:
// trade users see only their own at /me/activity; admin sees everything at /admin/activity.
function ensureAuditLogTable() {
@@ -21605,16 +21632,26 @@ app.post('/api/generator/batch', (req, res) => {
// Non-detached spawn: the child is owned by the server process, not the
// HTTP request, so returning the response now does NOT kill it and the
- // browser tab closing is irrelevant. Generous timeout: ~2 min/render.
+ // browser tab closing is irrelevant.
+ //
+ // Timeout budget (2026-06-01 fix): the comfy backend polls up to 15 min
+ // PER render (scripts/generate_designs.js, commit 86d053ad), so the old
+ // 150 s/render budget SIGTERM-killed every multi-render batch mid-flight —
+ // the child died at code=null after only ~2-3 renders landed, and we then
+ // mislabeled it "0/N survived the quality gate". Budget 18 min/render +
+ // 5 min slack so a full batch can actually finish, capped at 4 h so a
+ // wedged comfy can't leak a child forever.
+ const perRenderMs = parseInt(process.env.WALLCO_BATCH_PER_RENDER_MS || '1080000', 10); // 18 min
+ const childTimeout = Math.min(4 * 60 * 60 * 1000, n * perRenderMs + 5 * 60 * 1000);
const child = spawn('node', [
path.join(__dirname, 'scripts', 'generate_designs.js'),
'--n', String(n), '--kind', 'seamless_tile',
'--category', category,
'--prompts', promptsArg,
], { cwd: __dirname, env: { ...process.env, GEN_BACKEND: process.env.GEN_BACKEND || 'replicate' },
- stdio: ['ignore', out, out], timeout: Math.max(180000, n * 150000) });
+ stdio: ['ignore', out, out], timeout: childTimeout, killSignal: 'SIGTERM' });
- child.on('exit', (code) => {
+ child.on('exit', (code, signal) => {
// Reconcile from the LOG, not the global max id. generate_designs.js
// prints "Created X/N designs · IDs: a,b,c" — parse that so we record
// THIS batch's actual output, not whatever an unrelated concurrent
@@ -21625,20 +21662,53 @@ app.post('/api/generator/batch', (req, res) => {
const logText = fs.readFileSync(logFile, 'utf8');
const m = logText.match(/Created\s+\d+\/\d+\s+designs\s*·?\s*IDs?:\s*([0-9,\s]+)/i);
if (m) createdIds = m[1].split(',').map(s => parseInt(s.trim(), 10)).filter(Number.isFinite);
+ // FALLBACK (2026-06-01): a child killed by signal (timeout / pm2 wipe /
+ // OOM) exits BEFORE printing the "Created…IDs:" summary, so the regex
+ // above misses real persisted designs. The per-render "✓ #<id>" lines
+ // are printed as each design lands in PG — harvest those so a killed
+ // batch is credited with the work that actually survived instead of
+ // being falsely reported as "0 survived the quality gate".
+ if (createdIds.length === 0) {
+ const idRe = /✓\s*#(\d+)\b/g;
+ let mm; const seen = new Set();
+ while ((mm = idRe.exec(logText)) !== null) {
+ const id = parseInt(mm[1], 10);
+ if (Number.isFinite(id)) seen.add(id);
+ }
+ createdIds = [...seen];
+ }
} catch {}
- // "done" iff the child exited clean AND at least one design landed.
- // code 1 from generate_designs.js means 0 created (every render was
- // rejected by the ghost/frame/aesthetic gate) — that's a real failure
- // worth surfacing, not a silent success.
- const ok = code === 0 && createdIds.length > 0;
+ // A null exit code means the child was terminated by a SIGNAL (timeout,
+ // SIGKILL/SIGTERM from a pm2 wipe, OOM), NOT that the gate rejected
+ // everything. Distinguish the two so the 8% "pass rate" reflects real
+ // gate verdicts, not killed batches.
+ const killed = code === null;
+ // "done" iff at least one design landed (clean exit OR a signal-kill that
+ // still produced surviving renders). code 1 from generate_designs.js means
+ // 0 created (every render was rejected by the seam/aesthetic gate) — a
+ // real failure worth surfacing, not a silent success.
+ const ok = createdIds.length > 0 && (code === 0 || killed);
const firstId = createdIds[0] || null;
+ let errMsg = null;
+ if (!ok) {
+ if (killed) {
+ errMsg = `batch child killed by ${signal || 'signal'} before any design landed `
+ + `(timeout/pm2-wipe/OOM — NOT a quality-gate rejection)`;
+ } else if (createdIds.length === 0) {
+ errMsg = `0/${n} designs survived the quality gate (code ${code})`;
+ } else {
+ errMsg = `batch exited code ${code}, ${createdIds.length}/${n} created`;
+ }
+ } else if (killed) {
+ // partial success — surface it in the error column for visibility but
+ // keep status=done since real designs landed.
+ errMsg = `partial: ${createdIds.length}/${n} landed before ${signal || 'signal'} kill`;
+ }
try {
psqlQuery(`UPDATE wallco_generator_runs SET
status='${ok ? 'done' : 'error'}',
design_id=${firstId || 'NULL'},
- error=${ok ? 'NULL' : `'${(createdIds.length === 0
- ? `0/${n} designs survived the quality gate (code ${code})`
- : `batch exited code ${code}, ${createdIds.length}/${n} created`).replace(/'/g,"''")}'`},
+ error=${errMsg === null ? 'NULL' : `'${errMsg.replace(/'/g,"''")}'`},
finished_at=now() WHERE id=${runId};`);
} catch {}
});
← b3db6de SECURITY: SQL-escape interpolated paths/category in elements
·
back to Wallco Ai
·
TODO: log generator batch-timeout diagnosis + local fix (dep 9d134cd →