← back to Designerwallcoverings

TK-10933: add --reset-breaker, the explicit local repair for the latching 429 breaker

6838cedac805db341406228b8697b2692fb96bab · 2026-09-10 07:45:10 -0700 · Steve Abrams

hard_quota_open latched permanently: the guard in main() returns exit 42 before
either defaultBreaker() reset path (one requires cooldown_until, a soft-429 field;
the other sits inside the per-item loop that is never reached), and there was no
reset flag. Proven: live breaker state + a success mock => exit 42 / 0 verdicts;
same run with the state cleared => exit 0 / 1 verdict. So restoring Gemini credits
alone could never bring the job back — repair meant hand-deleting a 0600 JSON file.

Cody named this a residual ('repair UX is intentionally manual', 'explicit repair
runbook is outside this bounded increment'). This closes it without weakening
fail-closed-by-default:

  - standalone only: --reset-breaker with --apply is REFUSED (exit 46), so
    clearing the breaker and spending money cannot happen in one invocation
  - attempts zero items, exits 0
  - refuses under a LIVE lock owner (exit 45) so a running batch is never yanked
  - repairs a quarantined/corrupt state too (clears the .invalid sentinel), which
    is precisely when a repair is needed
  - records prior hard_quota_open / reason / cooldown_until / invalid-sentinel to
    the attempt ledger, so every repair is auditable

Dead-PID lock reclamation deliberately NOT touched — Cody rejected auto-reclaim.

Regression: 4 new cases (latched->reset->work resumes; --apply combination
refused; corrupt-state repair; live-lock refusal). Full suite PASS and every
pre-existing assertion still matches Cody's recorded baseline exactly:
hard=42, hard-resume=42, soft=43/24 attempts, soft-resume=43, corrupt=[44,44],
concurrent=[0,45], stale-lock=[45,45], resets=0. Diagnostics 4/4 PASS,
watchdog PASS. No network, no spend, no Shopify/dw_unified, no launchd action.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QzGkUdf9QjmDmeKrt6raHi

Files touched

• M scripts/stroheim-onboard/settlement-gate.mjs
• M scripts/stroheim-onboard/test-settlement-429-breaker.mjs

Diff

commit 6838cedac805db341406228b8697b2692fb96bab
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Thu Sep 10 07:45:10 2026 -0700

    TK-10933: add --reset-breaker, the explicit local repair for the latching 429 breaker
    
    hard_quota_open latched permanently: the guard in main() returns exit 42 before
    either defaultBreaker() reset path (one requires cooldown_until, a soft-429 field;
    the other sits inside the per-item loop that is never reached), and there was no
    reset flag. Proven: live breaker state + a success mock => exit 42 / 0 verdicts;
    same run with the state cleared => exit 0 / 1 verdict. So restoring Gemini credits
    alone could never bring the job back — repair meant hand-deleting a 0600 JSON file.
    
    Cody named this a residual ('repair UX is intentionally manual', 'explicit repair
    runbook is outside this bounded increment'). This closes it without weakening
    fail-closed-by-default:
    
      - standalone only: --reset-breaker with --apply is REFUSED (exit 46), so
        clearing the breaker and spending money cannot happen in one invocation
      - attempts zero items, exits 0
      - refuses under a LIVE lock owner (exit 45) so a running batch is never yanked
      - repairs a quarantined/corrupt state too (clears the .invalid sentinel), which
        is precisely when a repair is needed
      - records prior hard_quota_open / reason / cooldown_until / invalid-sentinel to
        the attempt ledger, so every repair is auditable
    
    Dead-PID lock reclamation deliberately NOT touched — Cody rejected auto-reclaim.
    
    Regression: 4 new cases (latched->reset->work resumes; --apply combination
    refused; corrupt-state repair; live-lock refusal). Full suite PASS and every
    pre-existing assertion still matches Cody's recorded baseline exactly:
    hard=42, hard-resume=42, soft=43/24 attempts, soft-resume=43, corrupt=[44,44],
    concurrent=[0,45], stale-lock=[45,45], resets=0. Diagnostics 4/4 PASS,
    watchdog PASS. No network, no spend, no Shopify/dw_unified, no launchd action.
    
    Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01QzGkUdf9QjmDmeKrt6raHi
---
 scripts/stroheim-onboard/settlement-gate.mjs       | 35 ++++++++++++
 .../test-settlement-429-breaker.mjs                | 62 +++++++++++++++++++++-
 2 files changed, 95 insertions(+), 2 deletions(-)

diff --git a/scripts/stroheim-onboard/settlement-gate.mjs b/scripts/stroheim-onboard/settlement-gate.mjs
index 2e63f6e..b5f6ce6 100644
--- a/scripts/stroheim-onboard/settlement-gate.mjs
+++ b/scripts/stroheim-onboard/settlement-gate.mjs
@@ -26,6 +26,7 @@ import { fileURLToPath } from 'node:url';
 const HERE = path.dirname(fileURLToPath(import.meta.url));
 const OUT = process.env.STROHEIM_SETTLEMENT_OUT || path.join(HERE, 'out');
 const APPLY = process.argv.includes('--apply');
+const RESET_BREAKER = process.argv.includes('--reset-breaker');
 const LIMIT = parseInt((process.argv.find(a => a.startsWith('--limit=')) || '').split('=')[1] || '0', 10);
 const ONLY = (process.argv.find(a => a.startsWith('--sku=')) || '').split('=')[1];
 const ONLY_SET = ONLY ? new Set(ONLY.split(',').map(s => s.trim().toUpperCase())) : null;
@@ -308,6 +309,40 @@ async function processItemBounded(p) {
 }
 
 async function main() {
+  // --reset-breaker (TK-10933): the explicit local repair for the fail-closed 429 breaker.
+  // hard_quota_open latches permanently — the guard below returns before either reset path —
+  // so without this flag a quota incident is a one-way outage requiring a hand-edited 0600
+  // state file. Deliberately STANDALONE: it refuses to combine with --apply, so clearing the
+  // breaker and spending money can never happen in a single invocation, and it attempts
+  // zero items. Works even when the state is quarantined, because that is exactly when a
+  // repair is needed.
+  if (RESET_BREAKER) {
+    if (APPLY) {
+      console.error('--reset-breaker cannot be combined with --apply; run the reset alone, then re-run with --apply');
+      process.exitCode = 46;
+      return;
+    }
+    if (fs.existsSync(RUN_LOCK) && lockOwnerState().owner_state === 'live') {
+      console.error('settlement gate lock held by a live owner; refusing to reset the breaker under a running batch');
+      process.exitCode = 45;
+      return;
+    }
+    let prior = null;
+    try { prior = JSON.parse(fs.readFileSync(BREAKER_STATE, 'utf8')); } catch { /* absent or unreadable */ }
+    const wasInvalid = fs.existsSync(BREAKER_INVALID);
+    appendAttempt({
+      status: 'breaker_manual_reset',
+      prior_hard_quota_open: typeof prior?.hard_quota_open === 'boolean' ? prior.hard_quota_open : null,
+      prior_reason: prior?.reason ?? null,
+      prior_cooldown_until: prior?.cooldown_until ?? null,
+      prior_state_invalid: wasInvalid
+    });
+    fs.rmSync(BREAKER_INVALID, { force: true });
+    saveBreaker(defaultBreaker());
+    console.log(`breaker reset · prior hard_quota_open=${prior?.hard_quota_open ?? 'none'} · prior reason=${prior?.reason ?? 'none'}${wasInvalid ? ' · cleared invalid sentinel' : ''} · 0 items attempted · $0`);
+    return;
+  }
+
   const payloads = loadJsonl('payloads.jsonl');
   const done = new Map(loadJsonl('settlement-verdicts.jsonl').map(r => [r.sku.toUpperCase(), r]));
   let todo = payloads.filter(p => !done.has(p.sku.toUpperCase()));
diff --git a/scripts/stroheim-onboard/test-settlement-429-breaker.mjs b/scripts/stroheim-onboard/test-settlement-429-breaker.mjs
index 21f7483..3b9ad28 100644
--- a/scripts/stroheim-onboard/test-settlement-429-breaker.mjs
+++ b/scripts/stroheim-onboard/test-settlement-429-breaker.mjs
@@ -134,7 +134,62 @@ assert.equal(staleRefusals.length, 2);
 assert(staleRefusals.every(e => e.status === 'lock_refused' && e.owner_state === 'dead' && e.owner_pid === 2147483647));
 assert.equal(events(staleOut).filter(e => e.status === 'item_started' || e.status === 'provider_attempt').length, 0);
 
-for (const out of [hardOut, softOut, resetOut, non429Out, corruptOut, concurrentOut, staleOut]) {
+// ---- TK-10933: --reset-breaker, the explicit local repair for the latching hard breaker ----
+// Without this, hard_quota_open is a one-way outage: the guard returns exit 42 before either
+// reset path, so restoring Gemini credits alone can never bring the job back.
+const runArgs = (out, mode, args, extra = {}) => spawnSync(process.execPath, [gate, ...args], {
+  encoding: 'utf8', timeout: 3000,
+  env: { ...process.env, STROHEIM_SETTLEMENT_OUT: out, STROHEIM_SETTLEMENT_TEST_MODE: mode,
+    STROHEIM_SETTLEMENT_429_COOLDOWN_MS: '60000', ...extra }
+});
+
+// 1. latched hard breaker -> reset -> real work resumes
+const manualOut = makeOut('manual', 1);
+assert.equal(run(manualOut, 'hard-quota-429').status, 42);
+assert.equal(JSON.parse(fs.readFileSync(path.join(manualOut, 'settlement-429-breaker.json'), 'utf8')).hard_quota_open, true);
+const attemptsBeforeReset = events(manualOut).filter(e => e.status === 'provider_attempt').length;
+const manualReset = runArgs(manualOut, 'success', ['--reset-breaker']);
+assert.equal(manualReset.status, 0, manualReset.stderr);
+assert.equal(JSON.parse(fs.readFileSync(path.join(manualOut, 'settlement-429-breaker.json'), 'utf8')).hard_quota_open, false);
+// the reset itself must attempt nothing and must record what it cleared
+assert.equal(events(manualOut).filter(e => e.status === 'provider_attempt').length, attemptsBeforeReset);
+const resetEvent = events(manualOut).at(-1);
+assert.equal(resetEvent.status, 'breaker_manual_reset');
+assert.equal(resetEvent.prior_hard_quota_open, true);
+assert.equal(resetEvent.prior_reason, 'hard_quota_exhausted');
+// and the gate must actually work again afterwards
+const manualResume = runArgs(manualOut, 'success', ['--apply']);
+assert.equal(manualResume.status, 0, manualResume.stderr);
+assert.equal(fs.readFileSync(path.join(manualOut, 'settlement-verdicts.jsonl'), 'utf8').trim().split('\n').filter(Boolean).length, 1);
+
+// 2. reset must NEVER combine with --apply (no clear-then-spend in one invocation)
+const comboOut = makeOut('combo', 1);
+assert.equal(run(comboOut, 'hard-quota-429').status, 42);
+const combo = runArgs(comboOut, 'success', ['--reset-breaker', '--apply']);
+assert.equal(combo.status, 46, combo.stderr);
+assert.equal(JSON.parse(fs.readFileSync(path.join(comboOut, 'settlement-429-breaker.json'), 'utf8')).hard_quota_open, true);
+assert.equal(events(comboOut).filter(e => e.status === 'breaker_manual_reset').length, 0);
+
+// 3. reset repairs a quarantined/corrupt state too (that is when repair is most needed)
+const repairOut = makeOut('repair', 1);
+fs.writeFileSync(path.join(repairOut, 'settlement-429-breaker.json'), '{broken\n');
+assert.equal(run(repairOut, 'success').status, 44);
+assert.equal(fs.existsSync(path.join(repairOut, 'settlement-429-breaker.invalid')), true);
+const repairReset = runArgs(repairOut, 'success', ['--reset-breaker']);
+assert.equal(repairReset.status, 0, repairReset.stderr);
+assert.equal(fs.existsSync(path.join(repairOut, 'settlement-429-breaker.invalid')), false);
+assert.equal(events(repairOut).at(-1).prior_state_invalid, true);
+assert.equal(runArgs(repairOut, 'success', ['--apply']).status, 0);
+
+// 4. reset refuses under a LIVE lock owner (never yank the breaker out from a running batch)
+const lockedOut = makeOut('resetlock', 1);
+fs.mkdirSync(path.join(lockedOut, 'settlement-gate.lock'));
+fs.writeFileSync(path.join(lockedOut, 'settlement-gate.lock', 'owner.json'), JSON.stringify({ pid: process.pid }) + '\n');
+const lockedReset = runArgs(lockedOut, 'success', ['--reset-breaker']);
+assert.equal(lockedReset.status, 45, lockedReset.stderr);
+assert.equal(events(lockedOut).filter(e => e.status === 'breaker_manual_reset').length, 0);
+
+for (const out of [hardOut, softOut, resetOut, non429Out, corruptOut, concurrentOut, staleOut, manualOut, comboOut, repairOut, lockedOut]) {
   const attemptFile = path.join(out, 'settlement-attempts.jsonl');
   const raw = fs.existsSync(attemptFile) ? fs.readFileSync(attemptFile, 'utf8') : '';
   assert(!/SECRET_SHOULD_NOT_APPEAR|Requests per minute|Credits depleted|billing quota exhausted/.test(raw));
@@ -143,4 +198,7 @@ console.log(JSON.stringify({ verdict: 'PASS', hard_exit: hard.status, soft_exit:
   hard_resume_exit: hardResume.status, soft_provider_attempts: 24, resume_exit: resume.status,
   corrupt_exits: [corrupt.status, corruptResume.status], concurrent_exits: concurrentCodes,
   stale_lock_exits: [stale1.status, stale2.status],
-  success_reset_exit: reset.status, non_429_reset_exit: non429.status }, null, 2));
+  success_reset_exit: reset.status, non_429_reset_exit: non429.status,
+  manual_reset_exit: manualReset.status, manual_resume_exit: manualResume.status,
+  reset_with_apply_refused_exit: combo.status, reset_repairs_invalid_exit: repairReset.status,
+  reset_under_live_lock_exit: lockedReset.status }, null, 2));

← 1b39a0e TK-10933: rotate stale stroheim launchd.err (31 pre-fcffc7f  ·  back to Designerwallcoverings  ·  auto-data-snapshot: 2026-09-10T08:12:39 (5 data files) — scr 2f5e49f →