terminal-status: retry transient EAGAIN in write_terminal so a busy tty can't latch a 24h health FAIL

1286106d060867af4459e5c603a6cd4463d1909c · 2026-09-15 12:44:34 -0700 · Steve Abrams

write_terminal opened the tty O_NONBLOCK then did a single os.write with no EAGAIN
handling, so a momentarily-full tty buffer raised BlockingIOError (Errno 35) that
escaped to the top handler; dot-floor.sh then latched the fleet-health panel FAIL
for 24h off a ~1.4% transient. Now: bounded 0.5s retry loop handling EAGAIN + partial
writes, raising the already-benign 'Terminal busy; retry' only on a genuine >0.5s wedge.
Verified: py_compile OK + 3 behavioral tests (real-pty positive, 2xEAGAIN+partial, wedged=0.51s bounded).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A5jmQ4a4VhZNPTRkVUSeML

Files touched

M terminal_status.py

Diff

commit 1286106d060867af4459e5c603a6cd4463d1909c
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Tue Sep 15 12:44:34 2026 -0700

    terminal-status: retry transient EAGAIN in write_terminal so a busy tty can't latch a 24h health FAIL
    
    write_terminal opened the tty O_NONBLOCK then did a single os.write with no EAGAIN
    handling, so a momentarily-full tty buffer raised BlockingIOError (Errno 35) that
    escaped to the top handler; dot-floor.sh then latched the fleet-health panel FAIL
    for 24h off a ~1.4% transient. Now: bounded 0.5s retry loop handling EAGAIN + partial
    writes, raising the already-benign 'Terminal busy; retry' only on a genuine >0.5s wedge.
    Verified: py_compile OK + 3 behavioral tests (real-pty positive, 2xEAGAIN+partial, wedged=0.51s bounded).
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01A5jmQ4a4VhZNPTRkVUSeML
---
 terminal_status.py | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/terminal_status.py b/terminal_status.py
index e92291d..b11e54a 100644
--- a/terminal_status.py
+++ b/terminal_status.py
@@ -469,8 +469,24 @@ def write_terminal(owner, payload):
         if not stat.S_ISCHR(os.fstat(fd).st_mode) or not os.isatty(fd):
             raise StatusError("Paint target is not a terminal")
         # One write, no background loop, no terminal input injection.
-        if os.write(fd, payload) != len(payload):
-            raise StatusError("Incomplete terminal paint")
+        # Non-blocking write: a momentarily busy tty (input buffer full, reader not
+        # draining) makes os.write raise BlockingIOError (EAGAIN / Errno 35) or return
+        # a PARTIAL count. Retry the remaining bytes with a short bounded backoff rather
+        # than turning a transient full buffer into a hard paint failure — an unretried
+        # EAGAIN latched a 24h FAIL on the fleet-health panel (~1.4% of paints hit it).
+        # Keep O_NONBLOCK so a dead/gone reader can never hang us; bound the wait instead.
+        # On genuine wedge (>0.5s) raise "Terminal busy; retry" — the same transient class
+        # dot-floor.sh already treats as benign, not a false engine fault.
+        view = memoryview(payload)
+        sent = 0
+        deadline = time.monotonic() + 0.5
+        while sent < len(view):
+            try:
+                sent += os.write(fd, view[sent:])
+            except BlockingIOError:
+                if time.monotonic() >= deadline:
+                    raise StatusError("Terminal busy; retry")
+                time.sleep(0.01)
     finally:
         os.close(fd)

← 9731e7f TK-11666: measure the TK-REQUIRED false-positive rate (the m · back to Terminal Status · T2 (TK-11779): add dead-tty .dot/.json reaper (ps-liveness a df6f28c →