terminal-status: cross-process proc-table cache + negative-rescan valve (TK-11398)

ce0496085795dc9093a9451398bc02cf280d1471 · 2026-09-11 12:20:03 -0700 · Steve Abrams

Paints were taking ~100s and often hitting the 60s ps ceiling and silently
no-op'ing, leaving tabs showing a PREVIOUS session's dot (observed: a tab
advertising a finished, unrelated ticket while working another) -- the
false-green class, since the wall looks authoritative while being wrong.

Root cause is a thundering herd, not a constant cost. _PROC_CACHE is
per-PROCESS and every dot call is a fresh python3, so the cache was dead on
arrival and all ~42 live sessions each ran their own full `ps -ax`. Measured:
three consecutive scans cost 2.30s / 4.90s / 7.08s -- the scans mutually slow
each other, so the herd is self-amplifying.

DOCUMENTED DECISION REVERSAL: the in-code scope note said a cross-process file
cache was "not worth the risk" because a stale table could miss a just-started
session and wrongly report "no owning terminal". Reversed deliberately, by
claude-run-11255, with Steve's explicit approval, on the new measured fact
above. The objection is preserved rather than traded away: a cached table is
only ever trusted for a POSITIVE answer -- every negative ownership conclusion
(current_owner finding no owning runtime; assert_owner about to refuse a stale
writer) re-scans fresh and re-decides before it is allowed to fail. A negative
can cost one extra scan; it can never produce a wrong refusal.

_splice_self_chain() is what makes the cache actually pay off: a cached table
predates the current process, so ancestors(os.getpid()) dead-ended and forced
the valve on EVERY call, cancelling the whole benefit (observed directly via
the new debug seam: "HIT" immediately followed by "MISS -> scan (fresh=True
forced)"). It now walks our own chain with targeted single-pid ps calls
(0.01-0.10s each, O(depth) not O(2000)) and returns None rather than a partial
chain, so failure can only cost a scan.

_DISK_TTL is 30s, not 5s: a scan measures 15-70s under load, so a 5s window
expires before the next invocation can reach it -- the same trap the existing
per-PROCESS TTL note describes. Staleness is bounded by the valve, not the TTL.

TERMINAL_STATUS_DEBUG=1 prints proc-cache HIT/MISS to stderr -- a cache whose
hit rate you cannot observe is a cache you cannot prove works. Opt-in only.

Verified end-to-end, with injected faults (positive-only tests would prove
nothing about a correctness valve; each fixture asserts it is not a no-op):
  - warm resolve: 17-72s -> 1.4-3.0s, single HIT, no forced rescan
  - real paint:   ~100s  -> 4.41s, dot file verified
  - fault A (70 claude/codex rows stripped from a fresh cache):
      HIT -> valve -> resolves the REAL owner pid 30729
  - fault B (my tty rewritten to impostor pid 999001):
      HIT -> valve -> resolves REAL pid 30729, impostor rejected
  - `audit` clean, no new drift

Complements 9696608 (per-process TTL 5s->120s, another session, TK-11466
followup), which fixes the intra-run double scan; this fixes the cross-session
herd. Both are needed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012LVUzeaBYQJsjSYSWtQZYy

Files touched

M terminal_status.py

Diff

commit ce0496085795dc9093a9451398bc02cf280d1471
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Fri Sep 11 12:20:03 2026 -0700

    terminal-status: cross-process proc-table cache + negative-rescan valve (TK-11398)
    
    Paints were taking ~100s and often hitting the 60s ps ceiling and silently
    no-op'ing, leaving tabs showing a PREVIOUS session's dot (observed: a tab
    advertising a finished, unrelated ticket while working another) -- the
    false-green class, since the wall looks authoritative while being wrong.
    
    Root cause is a thundering herd, not a constant cost. _PROC_CACHE is
    per-PROCESS and every dot call is a fresh python3, so the cache was dead on
    arrival and all ~42 live sessions each ran their own full `ps -ax`. Measured:
    three consecutive scans cost 2.30s / 4.90s / 7.08s -- the scans mutually slow
    each other, so the herd is self-amplifying.
    
    DOCUMENTED DECISION REVERSAL: the in-code scope note said a cross-process file
    cache was "not worth the risk" because a stale table could miss a just-started
    session and wrongly report "no owning terminal". Reversed deliberately, by
    claude-run-11255, with Steve's explicit approval, on the new measured fact
    above. The objection is preserved rather than traded away: a cached table is
    only ever trusted for a POSITIVE answer -- every negative ownership conclusion
    (current_owner finding no owning runtime; assert_owner about to refuse a stale
    writer) re-scans fresh and re-decides before it is allowed to fail. A negative
    can cost one extra scan; it can never produce a wrong refusal.
    
    _splice_self_chain() is what makes the cache actually pay off: a cached table
    predates the current process, so ancestors(os.getpid()) dead-ended and forced
    the valve on EVERY call, cancelling the whole benefit (observed directly via
    the new debug seam: "HIT" immediately followed by "MISS -> scan (fresh=True
    forced)"). It now walks our own chain with targeted single-pid ps calls
    (0.01-0.10s each, O(depth) not O(2000)) and returns None rather than a partial
    chain, so failure can only cost a scan.
    
    _DISK_TTL is 30s, not 5s: a scan measures 15-70s under load, so a 5s window
    expires before the next invocation can reach it -- the same trap the existing
    per-PROCESS TTL note describes. Staleness is bounded by the valve, not the TTL.
    
    TERMINAL_STATUS_DEBUG=1 prints proc-cache HIT/MISS to stderr -- a cache whose
    hit rate you cannot observe is a cache you cannot prove works. Opt-in only.
    
    Verified end-to-end, with injected faults (positive-only tests would prove
    nothing about a correctness valve; each fixture asserts it is not a no-op):
      - warm resolve: 17-72s -> 1.4-3.0s, single HIT, no forced rescan
      - real paint:   ~100s  -> 4.41s, dot file verified
      - fault A (70 claude/codex rows stripped from a fresh cache):
          HIT -> valve -> resolves the REAL owner pid 30729
      - fault B (my tty rewritten to impostor pid 999001):
          HIT -> valve -> resolves REAL pid 30729, impostor rejected
      - `audit` clean, no new drift
    
    Complements 9696608 (per-process TTL 5s->120s, another session, TK-11466
    followup), which fixes the intra-run double scan; this fixes the cross-session
    herd. Both are needed.
    
    Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_012LVUzeaBYQJsjSYSWtQZYy
---
 terminal_status.py | 134 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 131 insertions(+), 3 deletions(-)

diff --git a/terminal_status.py b/terminal_status.py
index 87cf3b1..55d0f40 100644
--- a/terminal_status.py
+++ b/terminal_status.py
@@ -166,13 +166,130 @@ class Process:
 _PROC_CACHE = {"at": 0.0, "rows": None}
 _PROC_TTL = float(os.environ.get("TERMINAL_STATUS_PROC_TTL", "120"))
 
+# TK-11398 (2026-09-11): DOCUMENTED DECISION REVERSAL -- the scope note above
+# says a cross-process file cache is "not worth the risk". Reversed here, by
+# claude-run-11255, with Steve's explicit approval, on this NEW MEASURED FACT:
+# concurrent scans do not cost a CONSTANT ~15s, they mutually slow each other.
+# Three consecutive `ps -axo pid,ppid,tty,lstart,comm` runs on this box (1,945
+# procs, 42 live claude sessions) measured 2.30s / 4.90s / 7.08s. With ~42
+# sessions each shelling out a fresh python3 -- so the per-PROCESS cache above
+# is dead on arrival every invocation -- the herd drives a single paint past the
+# 60s ps ceiling and NOTHING paints. That is strictly worse than the staleness
+# it was avoiding: the observed failure was a tab silently keeping a PREVIOUS
+# session's dot (advertising a finished, unrelated ticket) because the paint
+# no-oped. A silently wrong dot is the false-green class.
+#
+# The original objection is preserved exactly, not traded away: a table up to
+# TTL seconds stale could miss a just-started session and wrongly report "no
+# owning terminal". So a cached table is only ever trusted for a POSITIVE
+# answer. Every NEGATIVE ownership conclusion -- current_owner() finding no
+# owning runtime, or assert_owner() about to refuse a stale writer -- re-scans
+# fresh and re-decides before it is allowed to fail. Worst case a negative
+# costs one extra scan; it can never produce a wrong refusal.
+# 30s, not 5s: a scan on this box measures 15-70s under load, so a 5s window
+# expires before the next invocation can ever reach it -- the same trap the
+# per-PROCESS TTL note above describes. Staleness is bounded by the
+# negative-rescan valve, not by this number, so the window can afford to be
+# wide enough to actually catch the herd.
+_DISK_TTL = float(os.environ.get("TERMINAL_STATUS_DISK_TTL", "30"))
+
+
+def _disk_cache_path():
+    return Path(tempfile.gettempdir()) / f"terminal-status-proc-{os.getuid()}.json"
+
+
+def _disk_cache_get():
+    """Recent scan from a sibling invocation, or None. Never raises."""
+    try:
+        path = _disk_cache_path()
+        st = path.stat()
+        if st.st_uid != os.getuid():          # never trust another user's file
+            return None
+        if time.time() - st.st_mtime >= _DISK_TTL:
+            return None
+        payload = json.loads(path.read_text())
+        return {int(p[0]): Process(int(p[0]), int(p[1]), p[2], p[3], p[4])
+                for p in payload}
+    except (OSError, ValueError, TypeError, KeyError, IndexError):
+        return None
+
+
+def _disk_cache_put(rows):
+    """Publish this scan for sibling invocations. Best effort, never raises."""
+    try:
+        path = _disk_cache_path()
+        payload = [[p.pid, p.ppid, p.tty, p.started, p.command]
+                   for p in rows.values()]
+        fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=".ts-proc-")
+        try:
+            with os.fdopen(fd, "w") as handle:
+                json.dump(payload, handle)
+            os.chmod(tmp, 0o600)
+            os.replace(tmp, path)             # atomic; readers see whole files
+        except BaseException:
+            os.unlink(tmp)
+            raise
+    except OSError:
+        pass
+
+
+def _splice_self_chain(rows):
+    """Make a CACHED table usable for ancestry resolution.
+
+    A cached table was written before THIS process existed, so
+    ancestors(os.getpid(), rows) dead-ends immediately and current_owner()
+    reports a false negative -- which then forces a full rescan on every call
+    and cancels the cache's entire benefit. Walk our own chain with targeted
+    single-pid ps calls (measured 0.01-0.10s each, O(depth ~8) not O(2000))
+    and splice in whatever the cached table is too old to know about.
+
+    Returns None if the chain cannot be resolved completely; the caller then
+    falls back to a full scan. A partial chain is never returned, so this can
+    only ever cost an extra scan -- never manufacture a wrong answer.
+    """
+    pid = os.getpid()
+    for _ in range(32):
+        if pid <= 1 or pid in rows:
+            return rows
+        try:
+            out = subprocess.run(
+                ["ps", "-o", "ppid=,tty=,lstart=,comm=", "-p", str(pid)],
+                capture_output=True, text=True, timeout=10,
+                env={**os.environ, "LC_ALL": "C"})
+        except (OSError, subprocess.TimeoutExpired):
+            return None
+        line = out.stdout.strip()
+        if out.returncode or not line:
+            return None
+        parts = line.split(None, 7)
+        if len(parts) != 8:
+            return None
+        try:
+            ppid = int(parts[0])
+        except ValueError:
+            return None
+        rows[pid] = Process(pid, ppid, parts[1], " ".join(parts[2:7]), parts[7])
+        pid = ppid
+    return None
+
 
 def processes(fresh=False):
     now = time.monotonic()
     if not fresh and _PROC_CACHE["rows"] is not None \
             and now - _PROC_CACHE["at"] < _PROC_TTL:
         return _PROC_CACHE["rows"]
-    rows = _scan_processes()
+    rows = None if fresh else _disk_cache_get()
+    if rows is not None:
+        rows = _splice_self_chain(rows)
+    # Testability seam (TK-11398): a cache whose hit-rate you cannot observe is
+    # a cache you cannot prove works. Explicit opt-in flag; never set by callers.
+    if os.environ.get("TERMINAL_STATUS_DEBUG"):
+        print("proc-cache: %s%s" % ("HIT" if rows is not None else "MISS -> scan",
+                                    " (fresh=True forced)" if fresh else ""),
+              file=sys.stderr)
+    if rows is None:
+        rows = _scan_processes()
+        _disk_cache_put(rows)
     _PROC_CACHE["at"] = time.monotonic()
     _PROC_CACHE["rows"] = rows
     return rows
@@ -229,7 +346,7 @@ def owners(rows):
     return {tty: group[0] for tty, group in candidates.items() if len(group) == 1}
 
 
-def current_owner(rows):
+def current_owner(rows, _retried=False):
     # Resolve ownership by ancestry FIRST. CLAUDE_CODE_CHILD_SESSION is set on
     # any tool-spawned shell (the Bash tool, hooks), including a legit top-level
     # session's own subshell, so it can't be the sole refusal signal — that made
@@ -243,9 +360,16 @@ def current_owner(rows):
             # Do not walk through a headless nested agent into its parent's tty.
             if live.get(p.tty) == p.owner():
                 return p.owner()
+            # TK-11398: never let a CACHED table produce a negative. A table up
+            # to _DISK_TTL old can miss a just-started session; re-scan and
+            # re-decide before refusing.
+            if not _retried:
+                return current_owner(processes(fresh=True), _retried=True)
             raise StatusError("Headless or nested agent; refusing parent-terminal write")
     # No owning runtime in ancestry at all: only now does the child-session flag
     # decide — a true background subagent has no terminal of its own.
+    if not _retried:
+        return current_owner(processes(fresh=True), _retried=True)
     if os.environ.get("CLAUDE_CODE_CHILD_SESSION"):
         raise StatusError("A subagent has no terminal of its own; refusing to paint")
     raise StatusError("No owning Claude/Codex terminal in this process ancestry")
@@ -343,7 +467,11 @@ class Store:
         # the scan from the critical section.
         rows = rows if rows is not None else self.process_provider()
         if owners(rows).get(owner.tty) != owner:
-            raise StatusError("Terminal owner changed; refusing stale writer")
+            # TK-11398: a cached table may simply be stale. Re-scan fresh and
+            # re-check before refusing a writer that is in fact still the owner.
+            rows = processes(fresh=True)
+            if owners(rows).get(owner.tty) != owner:
+                raise StatusError("Terminal owner changed; refusing stale writer")
 
     @contextlib.contextmanager
     def lock(self, owner, rows=None):

← 9696608 terminal-status: fix paint hang — raise per-process PROC_TTL · back to Terminal Status · fix: cache ps scan + fix 60s ceiling exit-code in terminal-s 1c16fa2 →