TK-11372: fault-injection A/B discriminates the fix, plus two discarded runs

299731eaddd5d69829f28af6de50ab02dfa2faa3 · 2026-09-11 12:01:22 -0700 · Steve Abrams

Injected a slow lsof on PATH (the seam both arms use, since each invokes lsof
by name) to exceed the code own 4000ms timeout without waiting for load 70.
Baseline stalled a request 3.512s behind execSync; the patched arm stayed
sub-millisecond across 12 probes. Discriminated.

Records two runs that did NOT count, rather than quietly dropping them:
- a first A/B was void because the baseline arm ran outside the repo and died
  on MODULE_NOT_FOUND, so its 14/14 failures were connection-refused from a
  dead process; a listening guard now gates every probe
- a second A/B at load 35 was non-discriminating, both arms 16/16 clean, which
  is the honest reading: the defect is load-dependent

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PkwT8hDyw7SCCkYBviZTbq

Files touched

A verification/tk11372/ab-fault-injection.json

Diff

commit 299731eaddd5d69829f28af6de50ab02dfa2faa3
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Fri Sep 11 12:01:22 2026 -0700

    TK-11372: fault-injection A/B discriminates the fix, plus two discarded runs
    
    Injected a slow lsof on PATH (the seam both arms use, since each invokes lsof
    by name) to exceed the code own 4000ms timeout without waiting for load 70.
    Baseline stalled a request 3.512s behind execSync; the patched arm stayed
    sub-millisecond across 12 probes. Discriminated.
    
    Records two runs that did NOT count, rather than quietly dropping them:
    - a first A/B was void because the baseline arm ran outside the repo and died
      on MODULE_NOT_FOUND, so its 14/14 failures were connection-refused from a
      dead process; a listening guard now gates every probe
    - a second A/B at load 35 was non-discriminating, both arms 16/16 clean, which
      is the honest reading: the defect is load-dependent
    
    Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PkwT8hDyw7SCCkYBviZTbq
---
 verification/tk11372/ab-fault-injection.json | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/verification/tk11372/ab-fault-injection.json b/verification/tk11372/ab-fault-injection.json
new file mode 100644
index 00000000..8917033d
--- /dev/null
+++ b/verification/tk11372/ab-fault-injection.json
@@ -0,0 +1,22 @@
+{
+  "ticket": "TK-11372",
+  "purpose": "Break it on purpose, watch it go red, put it back (CLAUDE.md TK-11431 amendment 3). Discriminate the patched server from the baseline on the SAME box, without waiting for load 70.",
+  "testability_seam": "both arms invoke lsof BY NAME (baseline via /bin/sh, patched via execFile), so PATH injection needs no code change and no flag a plist could pass",
+  "fault": "a shim `lsof` on PATH that sleeps 10s then execs the real /usr/sbin/lsof — deliberately exceeding the code's own 4000ms timeout",
+  "arms": {
+    "baseline": {"port": 19795, "source": "server.pre-activation-backup.js (eaff7476)", "listening_confirmed": true},
+    "patched": {"port": 19794, "source": "server.js (57baae6e)", "listening_confirmed": true}
+  },
+  "result": {
+    "baseline": "12/12 answered 200 but probe 8 took 3.512707s — the request sat blocked behind execSync; 1/12 degraded",
+    "patched": "12/12 answered 200, slowest 0.003188s, all others sub-millisecond; 0/12 degraded",
+    "discriminated": true
+  },
+  "hit_rate_explained": "the baseline only spawns when its 5s cache TTL has expired, and the 4500ms interval fires just inside that TTL, so it blocks ~4s out of every ~9s (~44%). A ~13s probe run therefore crosses only 1-2 block windows, which matches 1 hit. On the real box at load 70 BOTH lsof AND ps overrun their 4s timeouts, giving ~8s of blocking per refresh — consistent with the 74% blocked measured by sample(1) on the live wedged process.",
+  "invalid_runs_discarded": [
+    "A first A/B was VOID: the baseline arm was copied outside the repo so it died instantly on MODULE_NOT_FOUND (require './lib.js'), and its 14/14 'failures' were connection-refused from a dead process in 0.3ms, not the bug. Added an explicit is-it-actually-LISTENING guard before any probe is allowed to count.",
+    "A second A/B at load ~35 was NON-DISCRIMINATING: both arms served 16/16 sub-millisecond. Reported as such rather than as a pass — the defect is load-dependent, which is exactly why the fault had to be injected."
+  ],
+  "cleanup": "both shadow instances killed, tmp-baseline-server.js removed, PATH shim deleted, git tree clean apart from an unrelated ledger file",
+  "cost_usd": 0
+}

← a5e9b14e TK-11372: prove the root cause and correct the pipe-EOF hypo · back to Ticket System · TK-11372: note the defect is load-dependent in the runbook 037b8e20 →