TK-11372: ACTIVATED and live-verified; residual symptom split to TK-11506

97bd9fd643680f251f5a281dafa8f98590273a82 · 2026-09-11 12:21:47 -0700 · Steve Abrams

Steve authorized in-session. Activation PASS, and step 1 captured the defect
at the moment of the fix: pre-restart healthz HTTP 000 at 8.006s, post-restart
200 at 0.0008s, 10/10 concurrent, unauth 401 intact, single God daemon, pm2
CLI lock released, old UNs process replaced by a normal one.

Original cause CONFIRMED fixed by stack proof: SyncProcessRunner and
uv__run_timers no longer appear on the main thread of the new pid, where they
previously held 2810/3794 samples. Orphan lsof leak closed too.

Not claiming total success. A 90s soak from load 75 still saw 6x 10s timeouts
on the patched board while control services answered in milliseconds, so it is
this service. The cause is independent and now dominant: /api/tickets is
uncached and synchronous, re-parsing a 32MB 191k-event log and rematerializing
1640 tickets per request then stringifying 14MB -- measured 223ms of loop
blocking per call, 142ms on an immediate second call, so definitively uncached.
Filed as TK-11506 rather than folded in, keeping this activation byte-identical
to the reviewed candidate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PkwT8hDyw7SCCkYBviZTbq

Files touched

A verification/tk11372/activation-result.json
A verification/tk11372/post-activation-soak.json

Diff

commit 97bd9fd643680f251f5a281dafa8f98590273a82
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Fri Sep 11 12:21:47 2026 -0700

    TK-11372: ACTIVATED and live-verified; residual symptom split to TK-11506
    
    Steve authorized in-session. Activation PASS, and step 1 captured the defect
    at the moment of the fix: pre-restart healthz HTTP 000 at 8.006s, post-restart
    200 at 0.0008s, 10/10 concurrent, unauth 401 intact, single God daemon, pm2
    CLI lock released, old UNs process replaced by a normal one.
    
    Original cause CONFIRMED fixed by stack proof: SyncProcessRunner and
    uv__run_timers no longer appear on the main thread of the new pid, where they
    previously held 2810/3794 samples. Orphan lsof leak closed too.
    
    Not claiming total success. A 90s soak from load 75 still saw 6x 10s timeouts
    on the patched board while control services answered in milliseconds, so it is
    this service. The cause is independent and now dominant: /api/tickets is
    uncached and synchronous, re-parsing a 32MB 191k-event log and rematerializing
    1640 tickets per request then stringifying 14MB -- measured 223ms of loop
    blocking per call, 142ms on an immediate second call, so definitively uncached.
    Filed as TK-11506 rather than folded in, keeping this activation byte-identical
    to the reviewed candidate.
    
    Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PkwT8hDyw7SCCkYBviZTbq
---
 verification/tk11372/activation-result.json    |  1 +
 verification/tk11372/post-activation-soak.json | 34 ++++++++++++++++++++++++++
 2 files changed, 35 insertions(+)

diff --git a/verification/tk11372/activation-result.json b/verification/tk11372/activation-result.json
new file mode 100644
index 00000000..635368e5
--- /dev/null
+++ b/verification/tk11372/activation-result.json
@@ -0,0 +1 @@
+{"ticket":"TK-11372","verdict":"PASS","at":"2026-09-11T19:16:30Z","pm2_id":57,"old_pid":"36347","new_pid":"60904","checks_pass":12,"checks_fail":0,"concurrent_200_of_10":10,"server_sha256":"57baae6e06577798fa003588c37611c529f3a0bb14a0e6ce1a4fdff40b7bfe2b","cost_usd":0}
diff --git a/verification/tk11372/post-activation-soak.json b/verification/tk11372/post-activation-soak.json
new file mode 100644
index 00000000..b5c923be
--- /dev/null
+++ b/verification/tk11372/post-activation-soak.json
@@ -0,0 +1,34 @@
+{
+  "ticket": "TK-11372",
+  "activation": {
+    "ran_at_local": "2026-09-11T12:0x-07:00",
+    "authorized_by": "Steve, in-session: 'unblock and run now'",
+    "pre_restart_healthz": "HTTP 000 / curl28 at 8.005591s — the defect captured live at the moment of activation",
+    "post_restart_healthz": "200 in 0.000812s, 0.000761s, 0.000717s; repeated past the 5s cache TTL; 10/10 concurrent",
+    "unauth_api_tickets": "401 (auth gate intact)",
+    "auth_api_tickets": "200, 14.86 MB body",
+    "api_running": "200",
+    "old_pid": 36347, "new_pid": 60904, "pm2_id": 57,
+    "god_daemons_after": 1,
+    "pm2_cli_lock": "released cleanly despite the wrapper reporting >20000ms",
+    "process_state_change": "old 36347 was UNs (uninterruptible); new 60904 is SNs/RNs (normal) — the wedge signature is gone",
+    "script_verdict": "PASS"
+  },
+  "original_defect_FIXED": {
+    "claim": "the execSync-in-the-4500ms-timer event-loop starvation is gone",
+    "proof": "sample(1) of the NEW pid shows the main thread in uv__io_poll+1252 running request JS (Builtins_ArrayMap / CallApiCallback). SyncProcessRunner and uv__run_timers no longer appear anywhere on the main thread. Before the fix they held 2810/3794 samples.",
+    "orphan_lsof_leak": "closed — no new ppid=1 lsof accumulating",
+    "verdict": "CONFIRMED FIXED"
+  },
+  "but_the_SYMPTOM_still_reproduces": {
+    "observation": "a 90s soak begun at load 75.34 saw 6 consecutive healthz timeouts (HTTP 000 at 10s) then a 200 at 7.664s on the PATCHED process",
+    "not_box_wide": "control probes at the same moment: look-alike :9795 200 in 0.003s, cabinet :9766 200 in 0.008s, CNCP :3333 answered. So it is this service, not the machine.",
+    "newly_dominant_cause": "GET /api/tickets is UNCACHED and fully synchronous: it re-parses ~/.claude/tickets/events.jsonl (32 MB, 191356 events) and re-materializes 1640 tickets on EVERY request, then stringifies 14.0 MB",
+    "measured": {"tickets_materialize_ms": 192.4, "json_stringify_ms": 30.8, "total_blocking_per_request_ms": 223.2, "second_call_ms": 142.7, "cached": false},
+    "why_it_produces_10s_timeouts": "223ms of loop blocking per request serializes across every concurrent board poller and agent; at 28 cores oversubscribed to load 75 those blocks stack into multi-second and 10s waits",
+    "relationship": "SAME FAMILY as the fixed defect (synchronous work on the board's single main thread) but a DIFFERENT root cause, and it is now the top one. This is exactly the 'synchronous reducer remains an unproven limit' that the prior codex session flagged and could not prove — now proven and quantified.",
+    "successor_ticket": "TK-11500"
+  },
+  "honest_verdict": "TK-11372's proven root cause is fixed and live-verified. The ticket's headline symptom can still occur under heavy load from an independent cause, which is tracked separately rather than folded in or papered over.",
+  "cost_usd": 0
+}

← e8d4f1be TK-11372: post-patch blocking audit; split the skills scan t · back to Ticket System · board: move ticket nav bar to a full-width strip across the ae20b9e2 →