← back to Terminal Status
backfill: always-on dot self-heal (repaint dropped dots + green-floor recordless live tabs)
3790923d76da03ee46dabb02fdd98a0593dec360 · 2026-09-16 17:47:02 -0700 · Steve Abrams
TK-11870. Under host overload (~49 sessions each scanning ps -ax) the owner
scan times out >90s, repaint no-ops, and a tab silently loses its dot -- 68%
of tabs were dark. Adds a 'backfill' subcommand + Store.backfill() that a
scheduled external supervisor runs to (a) warm the cross-process disk cache
with ONE fresh scan so interactive sessions read it instead of each racing
their own ps -ax (collapses the scan herd), and (b) re-assert every live tab's
dot: repaint valid records (never mutated), green-floor owner_changed/missing
live owners (same rule as the start command), skip invalid/busy. Iterates
owners(rows) so it paints each owner's OWN tty from its OWN record -- never
guesses a tty from a label map. Ships negative tests (selftest 7/7): valid
gated dot is never flattened to green; recordless owner is floored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012hTBDZLzS7vZzHrjZwVfVd
Files touched
Diff
commit 3790923d76da03ee46dabb02fdd98a0593dec360
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Wed Sep 16 17:47:02 2026 -0700
backfill: always-on dot self-heal (repaint dropped dots + green-floor recordless live tabs)
TK-11870. Under host overload (~49 sessions each scanning ps -ax) the owner
scan times out >90s, repaint no-ops, and a tab silently loses its dot -- 68%
of tabs were dark. Adds a 'backfill' subcommand + Store.backfill() that a
scheduled external supervisor runs to (a) warm the cross-process disk cache
with ONE fresh scan so interactive sessions read it instead of each racing
their own ps -ax (collapses the scan herd), and (b) re-assert every live tab's
dot: repaint valid records (never mutated), green-floor owner_changed/missing
live owners (same rule as the start command), skip invalid/busy. Iterates
owners(rows) so it paints each owner's OWN tty from its OWN record -- never
guesses a tty from a label map. Ships negative tests (selftest 7/7): valid
gated dot is never flattened to green; recordless owner is floored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012hTBDZLzS7vZzHrjZwVfVd
---
terminal_status.py | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 86 insertions(+), 1 deletion(-)
diff --git a/terminal_status.py b/terminal_status.py
index bbe8e98..f7f0346 100644
--- a/terminal_status.py
+++ b/terminal_status.py
@@ -540,6 +540,29 @@ def selftest():
except StatusError as exc:
case("--force opt-in PAINTS (got refusal: %s)" % exc, False)
+ # 5. TK-11870 backfill: (a) repaints an owner WITH a valid record, (b) NEVER
+ # flattens that valid gated dot to green (the key safety invariant — the
+ # injected fault is a purple/gated tab that must survive a sweep), and (c)
+ # green-floors a recordless/owner_changed live owner so no live tab stays dark.
+ # In-memory painter so it runs headless (no real tty); its own tempdir Store.
+ painted_ttys = []
+ have = Process(5252, 1, "ttys091", "Wed Sep 9 08:35:08 2026", "/usr/bin/claude")
+ lack = Process(5353, 1, "ttys092", "Wed Sep 9 08:35:08 2026", "/usr/bin/claude")
+ bf_rows = {5252: have, 5353: lack}
+ with tempfile.TemporaryDirectory() as bf_dir:
+ st = Store(user_root=bf_dir, process_provider=lambda **_: bf_rows,
+ painter=lambda owner, record: painted_ttys.append(owner.tty))
+ st.set(have.owner(), "purple", "gated reason", rows=bf_rows) # ttys091 = valid purple record
+ painted_ttys.clear() # ignore set()'s own paint
+ p, f, s = st.backfill(bf_rows)
+ purple_rec, _ = st.load(have.owner())
+ green_rec, _ = st.load(lack.owner())
+ case("backfill repaints the owner WITH a record", "ttys091" in painted_ttys and p == 1)
+ case("backfill NEVER flattens a valid gated dot to green",
+ purple_rec is not None and purple_rec["state"] == "purple")
+ case("backfill green-floors the recordless live owner",
+ f == 1 and green_rec is not None and green_rec["state"] == "green")
+
ok = all(v for _, v in checks)
print(("selftest: PASS (%d/%d)" if ok else "selftest: FAIL (%d/%d)")
% (sum(v for _, v in checks), len(checks)))
@@ -888,6 +911,50 @@ class Store:
# Never refresh timestamps, rewrite mirrors, or resurrect cleared state.
return record
+ def backfill(self, rows, *, lock_wait=2.5):
+ """Make Steve's HARD rule true: EVERY live terminal shows a dot at all times.
+
+ Two failures this repairs (TK-11870):
+ 1. DROPPED DOT — under host overload a paint raises a ps-timeout StatusError
+ and NO-OPS, so the tab loses its dot while the persisted record still holds
+ the truth. -> pure `repaint` re-emits the stored OSC (no mutation).
+ 2. RECORDLESS/OWNER-CHANGED LIVE TAB — a live claude/codex session whose only
+ on-disk record belongs to a PRIOR session on the reused tty (the signature
+ of a boot paint refused by the child-session rail), so it shows ⚪ "status
+ not set". owner_changed means a genuinely NEW process (pid/started differ),
+ so there is no prior state to preserve. -> `set` a GREEN floor (working),
+ exactly what the `start` command already does for owner_changed. The
+ session's own hooks refine it on its next turn; a green floor beats a dark
+ ⚪ tab. (`missing` is treated the same — a live owner with no record of its
+ own.)
+
+ Safe by construction: it iterates owners(rows) (the authoritative
+ process-table -> tty resolution), so it repaints/floors each owner's OWN tty
+ from its OWN record — it NEVER guesses a tty from a label map (memory
+ every-terminal-always-shows-a-dot: that mis-map nearly clobbered a GATED tab).
+ A VALID record is only ever repainted, never overwritten, so a real
+ gated/parked/needs-Steve dot can never be flattened to green. `invalid`/
+ conflicting records and mid-turn busy tabs are left for manual refresh / the
+ next cycle. Runs as an EXTERNAL supervisor (clean launchd env), so it passes
+ the resolved `rows` (no per-owner rescan) and a SHORT lock_wait so a busy tab
+ is skipped this cycle, not waited on the 65s self-writer backstop.
+ Returns (painted, floored, skipped)."""
+ painted = floored = skipped = 0
+ for owner in owners(rows).values():
+ _, reason = self.load(owner)
+ try:
+ if reason == "canonical":
+ self.repaint(owner, rows=rows, lock_wait=lock_wait)
+ painted += 1
+ elif reason in ("missing", "owner_changed"):
+ self.set(owner, "green", rows=rows, lock_wait=lock_wait)
+ floored += 1
+ else:
+ skipped += 1 # invalid / legacy_conflict -> manual refresh
+ except (StatusError, OSError):
+ skipped += 1 # busy tab this cycle -> caught next cycle
+ return painted, floored, skipped
+
def set_variant(self, owner, variant, *, rows=None, lock_wait=None):
"""Toggle ONLY the variant (e.g. the additive 🔵 stopped marker) on an existing
status — colour, label, ticket and timestamps are preserved. This is what the
@@ -1090,11 +1157,29 @@ def main(argv=None):
cmd = sub.add_parser(name)
cmd.add_argument("--json", action="store_true")
cmd.add_argument("--tsv", action="store_true")
- sub.add_parser("selftest") # TK-11791 negative test (never run by a plist)
+ bf = sub.add_parser("backfill") # TK-11870 scheduled always-on self-heal
+ bf.add_argument("--quiet", action="store_true")
+ sub.add_parser("selftest") # TK-11791/TK-11870 negative tests (never run by a plist)
args = parser.parse_args(argv)
if args.command == "selftest":
return selftest()
store = Store()
+ if args.command == "backfill":
+ # TK-11870: the SOLE scheduled process-table PRODUCER + the always-on
+ # self-heal. One fresh scan here (a) warms the cross-process disk cache
+ # (_disk_cache_put) so the ~49 interactive sessions read IT instead of each
+ # racing their own `ps -ax` — collapsing the scan herd that drove the >90s
+ # timeout that silently dropped dots — and (b) re-asserts every live tab's
+ # dot from its stored record. No tickets.discover() (that adds a ~4.4s
+ # `ps eww` env dump we don't need): backfill only REPAINTS existing records,
+ # it never resolves or creates a ticket label. Runs from a clean launchd env
+ # (no CLAUDE_CODE_CHILD_SESSION), so the child-session paint rail never trips.
+ rows = processes(fresh=True)
+ painted, floored, skipped = store.backfill(rows)
+ if not getattr(args, "quiet", False):
+ print("backfill: repainted %d, floored %d, skipped %d"
+ % (painted, floored, skipped))
+ return 0
rows = processes()
# TK-11835: tickets.discover() runs TWO `ps` subprocess calls (a `ps -p` argv
# read AND a `ps eww` ENV dump) over every live session PURELY to enrich the
← c6b317d TK-11831: harden the ps-enrichment cache against the codex-c
·
back to Terminal Status
·
backfill: write a bounded liveness heartbeat after the paint 9a4cc58 →