← back to Terminal Status
TK-11831: cache discover()'s two ps enrichment calls (the per-turn ps storm)
28cf4f63ec8d2a25f2496e47981b61b017d117f1 · 2026-09-16 14:06:49 -0700 · Steve Abrams
discover() shelled out to `ps -p <all live pids> -o args=` AND `ps eww` on
EVERY self-paint -- i.e. every UserPromptSubmit and every Stop hook -- across
~68 concurrent sessions. Measured 2.46s per paint at 18 live owners and ~4.4s
under overnight load: the `ps` storm the watchtower attributed 245% aggregate
CPU to, above node (112%) and iTerm2 (57%) at the 09:46 peak.
TK-11835 already removed this from the CROSS-TTY path via discover's
argv={}/env={} skip-seam but deliberately left the SELF path alone, and that
was right: measured here, full enrichment binds 13 tickets and the skip path
binds 0. Skipping would silently unbind every tab's ticket. So the fix is to
CACHE, not to skip.
Both values are IMMUTABLE for the life of a process -- argv is fixed at exec
and `ps eww` reports the env the process started with -- so a cache keyed on
pid AND start time is exact rather than approximate, and the start time makes
it safe against pid reuse (a recycled pid misses and is re-read). Only pids
MISSING from the cache are ps'd, so a steady fleet makes ZERO enrichment ps
calls and a newly-spawned session ps's a one-entry pid list, not all ~68.
Shared via an atomically-replaced /tmp file, so 68 sessions share one answer.
Fails OPEN at every step: any cache fault degrades to the uncached behaviour,
exactly as a slow ps already degrades the label but never the paint.
MEASURED A/B against HEAD, alternating under live load 78-101 (n=8):
ORIG mean 3.08s worst 9.92s -> NEW mean 0.68s worst 1.00s
Bindings IDENTICAL on every iteration (23==23) cold and warm -- a pure caching
win, not a behaviour change. The tail is now bounded because there is no ps
call left to spiral (an earlier ORIG sample hit 40.9s).
TESTS: 45 pass (was 41; +4). MUTATION-VERIFIED per CLAUDE.md TK-11431
amendment 3 -- all three injected faults go RED on throwaway copies:
M1 drop start-time from the key (pid-reuse hole) -> RED
M2 cache read always empty (caching bypassed) -> RED
M3 never store the TERM_SESSION_ID (lossy cache) -> RED
Control GREEN. A positive-only test on a cache proves nothing.
Reversible: git revert, or TERMINAL_STATUS_PS_CACHE_TTL=-1 to disable hits.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BD2ty1QgqWc2QVUVtYPRQ
Files touched
M test_terminal_status.pyM ticket_binding.py
Diff
commit 28cf4f63ec8d2a25f2496e47981b61b017d117f1
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Wed Sep 16 14:06:49 2026 -0700
TK-11831: cache discover()'s two ps enrichment calls (the per-turn ps storm)
discover() shelled out to `ps -p <all live pids> -o args=` AND `ps eww` on
EVERY self-paint -- i.e. every UserPromptSubmit and every Stop hook -- across
~68 concurrent sessions. Measured 2.46s per paint at 18 live owners and ~4.4s
under overnight load: the `ps` storm the watchtower attributed 245% aggregate
CPU to, above node (112%) and iTerm2 (57%) at the 09:46 peak.
TK-11835 already removed this from the CROSS-TTY path via discover's
argv={}/env={} skip-seam but deliberately left the SELF path alone, and that
was right: measured here, full enrichment binds 13 tickets and the skip path
binds 0. Skipping would silently unbind every tab's ticket. So the fix is to
CACHE, not to skip.
Both values are IMMUTABLE for the life of a process -- argv is fixed at exec
and `ps eww` reports the env the process started with -- so a cache keyed on
pid AND start time is exact rather than approximate, and the start time makes
it safe against pid reuse (a recycled pid misses and is re-read). Only pids
MISSING from the cache are ps'd, so a steady fleet makes ZERO enrichment ps
calls and a newly-spawned session ps's a one-entry pid list, not all ~68.
Shared via an atomically-replaced /tmp file, so 68 sessions share one answer.
Fails OPEN at every step: any cache fault degrades to the uncached behaviour,
exactly as a slow ps already degrades the label but never the paint.
MEASURED A/B against HEAD, alternating under live load 78-101 (n=8):
ORIG mean 3.08s worst 9.92s -> NEW mean 0.68s worst 1.00s
Bindings IDENTICAL on every iteration (23==23) cold and warm -- a pure caching
win, not a behaviour change. The tail is now bounded because there is no ps
call left to spiral (an earlier ORIG sample hit 40.9s).
TESTS: 45 pass (was 41; +4). MUTATION-VERIFIED per CLAUDE.md TK-11431
amendment 3 -- all three injected faults go RED on throwaway copies:
M1 drop start-time from the key (pid-reuse hole) -> RED
M2 cache read always empty (caching bypassed) -> RED
M3 never store the TERM_SESSION_ID (lossy cache) -> RED
Control GREEN. A positive-only test on a cache proves nothing.
Reversible: git revert, or TERMINAL_STATUS_PS_CACHE_TTL=-1 to disable hits.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BD2ty1QgqWc2QVUVtYPRQ
---
test_terminal_status.py | 93 ++++++++++++++++++++++++++++++
ticket_binding.py | 150 ++++++++++++++++++++++++++++++++++++++++++------
2 files changed, 226 insertions(+), 17 deletions(-)
diff --git a/test_terminal_status.py b/test_terminal_status.py
index 7361ffa..0edeb3f 100644
--- a/test_terminal_status.py
+++ b/test_terminal_status.py
@@ -11,6 +11,7 @@ import unittest
from unittest.mock import patch
import terminal_status as ts
+import ticket_binding as tb
class StatusTests(unittest.TestCase):
@@ -634,5 +635,97 @@ class StatusTests(unittest.TestCase):
self.assertEqual(record["state"], "purple")
+class EnrichmentCacheTests(unittest.TestCase):
+ """TK-11831: the shared pid-keyed cache for discover()'s two `ps` enrichment calls.
+
+ These ran on EVERY self-paint (every UserPromptSubmit AND every Stop hook)
+ across ~68 concurrent sessions -- measured 2.46s each at 18 live owners and
+ ~4.4s under overnight load, which is the `ps` storm behind this ticket.
+ """
+
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temp.cleanup)
+ self.root = Path(self.temp.name)
+ events = self.root / ".claude/tickets/events.jsonl"
+ events.parent.mkdir(parents=True, exist_ok=True)
+ events.write_text("") # discover() returns early without this
+ cache = self.root / "psenrich.json"
+ patcher = patch.object(tb, "_PS_CACHE_PATH", str(cache))
+ patcher.start()
+ self.addCleanup(patcher.stop)
+ self.cache = cache
+ # A real, live pid so `ps` genuinely answers: this process.
+ started = dt.datetime.fromtimestamp(
+ os.path.getmtime(__file__)).strftime("%a %b %d %H:%M:%S %Y")
+ self.proc = ts.Process(os.getpid(), 1, "ttys010", started, "/bin/claude")
+ self.rows = {self.proc.pid: self.proc}
+ self.live = {self.proc.tty: self.proc.owner()}
+
+ def discover(self):
+ return tb.discover(self.root, self.rows, self.live, ts.ancestors)
+
+ def counted(self):
+ """Count only the enrichment `ps` calls discover() shells out to."""
+ real, calls = tb.subprocess.run, []
+
+ def spy(cmd, *a, **kw):
+ if isinstance(cmd, (list, tuple)) and cmd and cmd[0] == "ps":
+ calls.append(list(cmd))
+ return real(cmd, *a, **kw)
+ return patch.object(tb.subprocess, "run", spy), calls
+
+ def test_warm_cache_is_exact_and_shells_out_zero_times(self):
+ # The whole claim in one test: a second pass must return the IDENTICAL
+ # answer while making NO `ps` call at all. Goes RED if the cache is
+ # bypassed (calls reappear) or if it is lossy (answers diverge).
+ spy, calls = self.counted()
+ with spy:
+ cold, cold_known = self.discover()
+ self.assertEqual(len(calls), 2, "cold pass makes both enrichment reads")
+ calls.clear()
+ warm, warm_known = self.discover()
+ self.assertEqual(calls, [], "warm pass must not touch ps at all")
+ self.assertEqual(cold, warm)
+ self.assertEqual(cold_known, warm_known)
+ self.assertTrue(self.cache.exists(), "the warm pass had something to hit")
+
+ def test_cache_key_is_pid_reuse_safe(self):
+ # NEGATIVE TEST (CLAUDE.md TK-11431 amendment 3). argv/env are immutable
+ # per process, which is what makes caching EXACT -- but only while the
+ # pid means the same process. A recycled pid carries a DIFFERENT start
+ # time, so it must MISS and be re-read; serving the dead process's argv
+ # would bind a tab to a ticket that is not running. Mutation-verified:
+ # dropping `started` from _enrich_key() makes this go RED.
+ self.discover() # populate
+ recycled = dataclasses.replace(self.proc, started="Wed Sep 9 08:35:08 2026")
+ self.rows = {recycled.pid: recycled}
+ self.live = {recycled.tty: recycled.owner()}
+ spy, calls = self.counted()
+ with spy:
+ self.discover()
+ self.assertEqual(len(calls), 2,
+ "a recycled pid must re-read ps, never serve the dead pid's argv")
+
+ def test_unreadable_cache_degrades_to_the_uncached_answer(self):
+ # Fail-open: a torn or hostile cache file must cost correctness nothing.
+ # Every cache fault degrades to today's behaviour, exactly as a slow ps
+ # already degrades the label but never the paint.
+ expected, _ = self.discover()
+ for junk in ("", "{", "null", '{"entries": "not-a-dict"}', '{"entries": {"x": 7}}'):
+ self.cache.write_text(junk)
+ self.assertEqual(self.discover()[0], expected, "corrupt cache: " + repr(junk))
+
+ def test_stale_entries_age_out(self):
+ # A pid-keyed cache on a box up for days must not grow without bound or
+ # serve an entry whose process is long gone; the TTL is the backstop.
+ self.discover()
+ with patch.object(tb, "_PS_CACHE_TTL", -1):
+ spy, calls = self.counted()
+ with spy:
+ self.discover()
+ self.assertEqual(len(calls), 2, "an expired entry must be re-read")
+
+
if __name__ == "__main__":
unittest.main(verbosity=2)
diff --git a/ticket_binding.py b/ticket_binding.py
index 4b38908..c48299a 100644
--- a/ticket_binding.py
+++ b/ticket_binding.py
@@ -4,12 +4,80 @@ import json
import os
import re
import subprocess
+import tempfile
+import time
# TK-11505: shared timeout constant so ticket_binding uses the same configurable
# ceiling as terminal_status._scan_processes(). Set TERMINAL_STATUS_PS_TIMEOUT
# to override (default 90s).
_PS_TIMEOUT = float(os.environ.get("TERMINAL_STATUS_PS_TIMEOUT", "90"))
+# --- TK-11831: shared, pid-keyed cache for the two `ps` enrichment calls ------
+# Both enrichment reads in discover() are IMMUTABLE for the life of a process:
+# argv is fixed at exec, and `ps eww` reports the environment the process was
+# started with. So a cache keyed on the pid AND its start time is EXACT rather
+# than approximate, and including the start time makes it safe against pid
+# reuse -- a recycled pid simply misses and is re-read. Only pids MISSING from
+# the cache are ps'd, so a steady fleet pays ZERO ps calls per paint and a
+# newly-spawned session ps's a one-entry pid list instead of all ~68.
+#
+# Why this matters (measured, TK-11831 / TK-11832): the pair cost 2.46s at 18
+# live owners and ~4.4s under overnight load, and it ran on EVERY self-paint --
+# every UserPromptSubmit and every Stop hook -- across ~68 concurrent sessions.
+# That is the `ps` storm the watchtower attributed 245% aggregate CPU to, above
+# both node (112%) and iTerm2 (57%) at the 09:46 peak.
+#
+# Why CACHE and not SKIP: TK-11835 removed these calls from the CROSS-TTY path
+# via discover's argv={}/env={} skip-seam but deliberately left the SELF path
+# alone, and that was right -- measured on this box, full enrichment binds 13
+# tickets and the skip path binds 0, so skipping here would silently unbind
+# every tab's ticket. Caching keeps the answer byte-identical and only stops us
+# recomputing it ~68 times a turn.
+#
+# Every step FAILS OPEN: any cache fault degrades to the uncached behaviour,
+# exactly as a slow ps already degrades the label but never the paint.
+_PS_CACHE_PATH = os.environ.get(
+ "TERMINAL_STATUS_PS_CACHE",
+ os.path.join(tempfile.gettempdir(), "terminal-status-psenrich-%d.json" % os.getuid()))
+_PS_CACHE_TTL = float(os.environ.get("TERMINAL_STATUS_PS_CACHE_TTL", "3600"))
+_PS_CACHE_MAX = 512
+
+
+def _enrich_key(owner):
+ """pid + start time: a recycled pid misses instead of reading a stale argv."""
+ return "%d:%s" % (owner.pid, owner.started)
+
+
+def _enrich_cache_read():
+ """Best-effort read of the shared cache. Any fault reads as 'empty'."""
+ try:
+ with open(_PS_CACHE_PATH) as handle:
+ blob = json.load(handle)
+ entries = blob.get("entries") if isinstance(blob, dict) else None
+ if not isinstance(entries, dict):
+ return {}
+ floor = time.time() - _PS_CACHE_TTL
+ return {k: v for k, v in entries.items()
+ if isinstance(v, dict) and float(v.get("t", 0)) > floor}
+ except (OSError, ValueError, TypeError, AttributeError):
+ return {}
+
+
+def _enrich_cache_write(entries):
+ """Atomic replace, so ~68 concurrent writers can never read a torn file."""
+ if len(entries) > _PS_CACHE_MAX: # newest-first; a long-lived box can't grow forever
+ entries = dict(sorted(entries.items(), key=lambda kv: -float(kv[1].get("t", 0)))[:_PS_CACHE_MAX])
+ tmp = "%s.%d.tmp" % (_PS_CACHE_PATH, os.getpid())
+ try:
+ with open(tmp, "w") as handle:
+ json.dump({"entries": entries}, handle)
+ os.replace(tmp, _PS_CACHE_PATH)
+ except (OSError, ValueError, TypeError):
+ try:
+ os.unlink(tmp)
+ except OSError:
+ pass
+
SHORT = re.compile(r"^TK-\d+(?=$|-)", re.I)
CID = re.compile(r"^(?:assign|create|action)-[a-z0-9]+-(\d+)-[a-z0-9]+$")
@@ -115,6 +183,11 @@ def discover(root, rows, live, chain, argv=None, env=None):
target[owner.tty] = {"id": ticket, "at": at,
"source": "ticket_ledger", "correlation_id": match[0]}
result = {tty: claims.get(tty, actions.get(tty)) for tty in live}
+ # TK-11831: read the shared ps-enrichment cache once for both blocks below.
+ # Only consulted when a block would actually shell out (argv/env is None) --
+ # the cross-tty skip-seam still short-circuits to zero work, as TK-11835 left it.
+ _cache = _enrich_cache_read() if (argv is None or env is None) and live else {}
+ _learned = {}
# A unique ticket in the main process launch command is a conservative fallback.
if argv is None and live:
# TK-11369: was timeout=8 and FATAL. With ~49 live sessions this pid list
@@ -125,15 +198,29 @@ def discover(root, rows, live, chain, argv=None, env=None):
# degrade the label, never the paint. Raised to match the sibling
# process-table read (_PS_TIMEOUT, now configurable via
# TERMINAL_STATUS_PS_TIMEOUT, default 90s -- TK-11505), and made non-fatal.
+ #
+ # TK-11831: serve what the shared cache already knows and ps ONLY the
+ # pids it is missing. On a steady fleet `missing` is empty and this
+ # block makes no subprocess call at all.
argv = {}
+ missing = {}
+ for owner in live.values():
+ hit = _cache.get(_enrich_key(owner))
+ if hit is not None and "a" in hit:
+ argv[owner.pid] = hit["a"]
+ else:
+ missing[owner.pid] = owner
try:
- output = subprocess.run(["ps", "-p", ",".join(str(o.pid) for o in live.values()),
+ if missing:
+ output = subprocess.run(["ps", "-p", ",".join(str(pid) for pid in missing),
"-o", "pid=,args="], capture_output=True, text=True,
timeout=_PS_TIMEOUT)
- for line in output.stdout.splitlines():
- parts = line.strip().split(None, 1)
- if len(parts) == 2:
- argv[int(parts[0])] = parts[1]
+ for line in output.stdout.splitlines():
+ parts = line.strip().split(None, 1)
+ if len(parts) == 2:
+ argv[int(parts[0])] = parts[1]
+ for pid, owner in missing.items():
+ _learned.setdefault(_enrich_key(owner), {})["a"] = argv.get(pid, "")
except (OSError, subprocess.TimeoutExpired) as exc:
# Label enrichment unavailable; the dot itself still paints.
try:
@@ -150,27 +237,56 @@ def discover(root, rows, live, chain, argv=None, env=None):
# we skip that pid rather than risk the trap. Optional enrichment like the argv
# read: a slow/absent ps degrades the label, never the paint.
if env is None and live:
+ #
+ # TK-11831: same shared cache. The parsed TERM_SESSION_ID is stored, not
+ # the raw `ps eww` line, so a cache hit never re-runs the prefix-strip
+ # (and never re-exposes the raw env to the measurement trap above).
env = {}
+ missing = {}
+ for owner in live.values():
+ hit = _cache.get(_enrich_key(owner))
+ if hit is not None and "s" in hit:
+ if hit["s"]:
+ env[owner.pid] = hit["s"]
+ else:
+ missing[owner.pid] = owner
try:
- output = subprocess.run(["ps", "eww", "-o", "pid=,command=",
- "-p", ",".join(str(o.pid) for o in live.values())],
+ if missing:
+ output = subprocess.run(["ps", "eww", "-o", "pid=,command=",
+ "-p", ",".join(str(pid) for pid in missing)],
capture_output=True, text=True, timeout=_PS_TIMEOUT)
- for line in output.stdout.splitlines():
- parts = line.strip().split(None, 1)
- if len(parts) != 2:
- continue
- pid, full = int(parts[0]), parts[1]
- args = (argv or {}).get(pid, "")
- remainder = full[len(args):] if args and full.startswith(args) else ""
- m = re.search(r"TERM_SESSION_ID=(\S+)", remainder)
- if m:
- env[pid] = m.group(1)
+ for line in output.stdout.splitlines():
+ parts = line.strip().split(None, 1)
+ if len(parts) != 2:
+ continue
+ pid, full = int(parts[0]), parts[1]
+ args = (argv or {}).get(pid, "")
+ remainder = full[len(args):] if args and full.startswith(args) else ""
+ m = re.search(r"TERM_SESSION_ID=(\S+)", remainder)
+ if m:
+ env[pid] = m.group(1)
+ for pid, owner in missing.items():
+ _learned.setdefault(_enrich_key(owner), {})["s"] = env.get(pid, "")
except (OSError, subprocess.TimeoutExpired) as exc:
try:
from terminal_status import _record_enum_blind
_record_enum_blind("ticket_binding ps eww: " + type(exc).__name__)
except Exception:
pass
+ # TK-11831: persist whatever this pass had to learn, MERGED over the entry
+ # already on disk -- a pass that only ran the argv block must not drop a
+ # sibling's cached TERM_SESSION_ID (and vice versa). Refreshing "t" keeps a
+ # still-live session from ageing out mid-run. Write-back is best-effort: if
+ # it fails, the next pass simply re-reads ps, which is today's behaviour.
+ if _learned:
+ now = time.time()
+ merged = dict(_cache)
+ for key, value in _learned.items():
+ entry = dict(merged.get(key, {}))
+ entry.update(value)
+ entry["t"] = now
+ merged[key] = entry
+ _enrich_cache_write(merged)
for tty, owner in live.items():
if result[tty]:
continue
← c3c5843 dot.sh: auto-force retry so bridge/sub sessions paint their
·
back to Terminal Status
·
TK-11831: harden the ps-enrichment cache against the codex-c c6b317d →