← back to Model Arena
TK-11723: preflight the :9758 dependency and unsilence the alert arm
614ac9a4baae79613b0e2c0013018d3f9f310846 · 2026-09-16 14:14:20 -0700 · Steve Abrams
The daily cron was never broken - it fired on schedule every day. What died
was an undeclared dependency: daily-challenge.js is a pure HTTP client of
:9758 and cannot start the server, which had been running unmanaged. It died
after 2026-09-13 and nothing restarted it, so 48 consecutive battles became
3 days of ECONNREFUSED that took a human sweep to attribute.
model-arena is now pm2-managed (separate, non-code change) so the server is
resident and self-healing. This commit covers the two code-side halves:
- Dependency preflight: asserts :9758 answers 200 before burning the run,
and on failure NAMES the dependency plus the inspect/repair commands
instead of emitting an opaque errno. Exits 3 (distinct from the battle's
own codes). It deliberately does NOT self-start the server - that would
race the pm2-managed process for the port and mask pm2 having stopped
managing it, which is the regression we most need to stay visible.
- Alert arm: the old arm was 'curl -s ... || true'. curl -s exits 0 on HTTP
404/500 and '|| true' discards the code regardless, so it was structurally
incapable of reporting its own failure. Now posts through the shared
senders that assert a real 2xx and write a delivery receipt.
- Model availability is surfaced as a WARN rather than a gate, since partial
battles are historically normal - but a silently reduced field must be
named, or the ELO ledger takes a clean-looking row from a short contest.
Negative-tested per CLAUDE.md: with a dead port injected the wrapper exits 3,
names the dependency, blocks the battle, and logs its own alert-delivery
failure. Testability seam is env-only (WLOG/BASE); the plist passes neither.
DTD verdict C, 4/6. Dissent (Qwen, minimalism lens) held pm2 alone suffices;
their objection is what made the preflight name-and-fail rather than repair.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YCf8ygWbqXvbqsKv1zLsLW
Files touched
M scripts/daily-challenge-wrapped.sh
Diff
commit 614ac9a4baae79613b0e2c0013018d3f9f310846
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Wed Sep 16 14:14:20 2026 -0700
TK-11723: preflight the :9758 dependency and unsilence the alert arm
The daily cron was never broken - it fired on schedule every day. What died
was an undeclared dependency: daily-challenge.js is a pure HTTP client of
:9758 and cannot start the server, which had been running unmanaged. It died
after 2026-09-13 and nothing restarted it, so 48 consecutive battles became
3 days of ECONNREFUSED that took a human sweep to attribute.
model-arena is now pm2-managed (separate, non-code change) so the server is
resident and self-healing. This commit covers the two code-side halves:
- Dependency preflight: asserts :9758 answers 200 before burning the run,
and on failure NAMES the dependency plus the inspect/repair commands
instead of emitting an opaque errno. Exits 3 (distinct from the battle's
own codes). It deliberately does NOT self-start the server - that would
race the pm2-managed process for the port and mask pm2 having stopped
managing it, which is the regression we most need to stay visible.
- Alert arm: the old arm was 'curl -s ... || true'. curl -s exits 0 on HTTP
404/500 and '|| true' discards the code regardless, so it was structurally
incapable of reporting its own failure. Now posts through the shared
senders that assert a real 2xx and write a delivery receipt.
- Model availability is surfaced as a WARN rather than a gate, since partial
battles are historically normal - but a silently reduced field must be
named, or the ELO ledger takes a clean-looking row from a short contest.
Negative-tested per CLAUDE.md: with a dead port injected the wrapper exits 3,
names the dependency, blocks the battle, and logs its own alert-delivery
failure. Testability seam is env-only (WLOG/BASE); the plist passes neither.
DTD verdict C, 4/6. Dissent (Qwen, minimalism lens) held pm2 alone suffices;
their objection is what made the preflight name-and-fail rather than repair.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YCf8ygWbqXvbqsKv1zLsLW
---
scripts/daily-challenge-wrapped.sh | 101 +++++++++++++++++++++++++++++++------
1 file changed, 86 insertions(+), 15 deletions(-)
diff --git a/scripts/daily-challenge-wrapped.sh b/scripts/daily-challenge-wrapped.sh
index e77ec03..c37a149 100755
--- a/scripts/daily-challenge-wrapped.sh
+++ b/scripts/daily-challenge-wrapped.sh
@@ -9,19 +9,101 @@
#
# Alert channels chosen because they work headless from launchd (no MCP tools):
# 1. persistent log file (NOT /tmp) — audit trail that survives reboot
-# 2. CNCP parking-lot card (curl :3333) — always-visible on Steve's board
+# 2. CNCP parking-lot card — always-visible on Steve's board
# 3. George email (best-effort) — matches canary doctrine
# Alerts fire on a nonzero exit OR when no FRESH daily-log line was appended
# (catches the silent fs.appendFile failure class too). $0 — no metered calls.
+#
+# ---------------------------------------------------------------------------
+# TK-11723 (2026-09-16) — DEPENDENCY PREFLIGHT + UNSILENCED ALERT ARM.
+# DTD verdict C (4/6: Claude, Codex, ops-lens, failure-mode-lens; dissent Qwen +
+# minimalism-lens argued pm2 registration alone suffices).
+#
+# WHAT BROKE: daily-challenge.js is a PURE HTTP CLIENT of $BASE (:9758) and cannot
+# start the server. The server was running UNMANAGED, died after 2026-09-13, and
+# nothing restarted it — 48 consecutive battles then 3 days of ECONNREFUSED. The
+# fix registered model-arena in pm2 (`pm2 start server.js --name model-arena`,
+# saved to the dump that ~/Projects/_shared/pm2-boot-serialize.sh resurrects at
+# boot via the com.PM2 LaunchAgent), which is why this wrapper does NOT start it.
+#
+# WHY THE PREFLIGHT NAMES AND FAILS RATHER THAN SELF-STARTS — this was the panel's
+# one point of agreement across BOTH dissenters and the majority: a self-healing
+# preflight would (a) race the pm2-managed process and can bind a duplicate server
+# on the same port, and (b) MASK pm2 having stopped managing it, hiding the real
+# regression behind a workaround. So it asserts and reports; it never repairs.
+# Its whole job is to turn an opaque `ECONNREFUSED 127.0.0.1:9758` stack trace —
+# which cost 3 days of attribution and 3 unrecoverable days of ledger accrual —
+# into a line that names the dependency and the command that inspects it.
+#
+# WHY THE ALERT ARM CHANGED: the old arm was `curl -s ... || true`. Per CLAUDE.md
+# (TK-11431 amendment 2) `curl -s` EXITS 0 on HTTP 404/500, and `|| true` discards
+# the code even when it doesn't — so that arm was structurally incapable of
+# reporting its own failure. It now posts through the shared senders, which assert
+# a real 2xx and write a delivery receipt to data/alert-delivery.jsonl.
set -uo pipefail
DIR="$HOME/Projects/model-arena"
NODE=/opt/homebrew/bin/node
DAYLOG="$DIR/yolo/daily-log.jsonl"
-WLOG="$DIR/yolo/daily-wrapper.log" # persistent, survives reboot
-mkdir -p "$DIR/yolo"
+# TESTABILITY SEAM (TK-11723, CLAUDE.md amendment 3): WLOG is env-overridable so the
+# negative test can prove this wrapper goes RED on an injected dependency fault without
+# writing test ALERTs into the operational audit trail. The launchd plist passes NEITHER
+# WLOG nor BASE, so a scheduled run always measures the real server and the real log.
+WLOG="${WLOG:-$DIR/yolo/daily-wrapper.log}" # persistent, survives reboot
+BASE="${BASE:-http://127.0.0.1:9758}"
+AUTH="${AUTH:-admin:DW2024!}"
+mkdir -p "$DIR/yolo" "$DIR/data"
TS="$(/bin/date -u +%Y-%m-%dT%H:%M:%SZ)"
+# Delivery receipts land beside this project's data, not in a skill dir.
+export SKILL="$DIR"
+. "$HOME/.claude/skills/_shared/cncp_post.sh" 2>/dev/null || true
+. "$HOME/.claude/skills/_shared/george-send.sh" 2>/dev/null || true
+
+# alert <msg> — log it, then fan out through senders that VERIFY delivery.
+# Never `|| true` a sender: a discarded return code is a silenced alarm.
+alert() {
+ local msg="$1"
+ echo "$TS ALERT $msg" >>"$WLOG"
+ if command -v cncp_post >/dev/null 2>&1; then
+ cncp_post "health://model-arena" "[ALERT] $msg" \
+ || echo "$TS ALERT-DELIVERY-FAILED cncp rc=$? (card did NOT land)" >>"$WLOG"
+ else
+ echo "$TS ALERT-DELIVERY-FAILED cncp helper not sourced" >>"$WLOG"
+ fi
+ if command -v george_send >/dev/null 2>&1; then
+ george_send "steve-office" "info@designerwallcoverings.com" \
+ "[ALERT] Model Arena daily job failed" "$msg" >/dev/null \
+ || echo "$TS ALERT-DELIVERY-FAILED george rc=$? (email did NOT send)" >>"$WLOG"
+ fi
+}
+
+# --- PREFLIGHT: assert the dependency BEFORE burning the run ------------------
+# `curl -fsS` so a 4xx/5xx is a real non-zero, not a silent exit 0.
+# NOTE: curl prints %{http_code} ("000" on transport failure) AND exits non-zero, so a
+# `|| echo 000` fallback here would CONCATENATE into "000000". Take curl's own output and
+# normalise an empty/garbled value to 000 instead.
+PRE_CODE="$(/usr/bin/curl -s -o /dev/null -w '%{http_code}' -m 15 -u "$AUTH" "$BASE/api/models" 2>/dev/null)"
+case "$PRE_CODE" in ''|*[!0-9]*) PRE_CODE=000 ;; esac
+if [ "$PRE_CODE" != "200" ]; then
+ MSG="Model Arena PREFLIGHT FAILED $TS — dependency DOWN: the model-arena server is not answering on $BASE (HTTP $PRE_CODE). daily-challenge.js is an HTTP client and cannot start it. Inspect: pm2 describe model-arena ; lsof -nP -iTCP:9758 -sTCP:LISTEN ; pm2 logs model-arena --lines 50. Repair: pm2 restart model-arena (it is pm2-managed as of TK-11723). NOT auto-started here on purpose — self-starting would mask pm2 having stopped managing it."
+ alert "$MSG"
+ exit 3 # distinct from the battle's own exit codes: 3 == dependency down
+fi
+
+# Model availability is a WARN, not a gate — partial battles are normal and the
+# historical log shows done:1..3 runs that were still worth recording. But an
+# unavailable model must be NAMED here, because otherwise a battle that quietly
+# drops a contender still writes a clean-looking row into the ELO ledger.
+UNAVAIL="$(/usr/bin/curl -s -m 15 -u "$AUTH" "$BASE/api/models" 2>/dev/null \
+ | /usr/bin/python3 -c 'import json,sys
+try:
+ m=json.load(sys.stdin)
+except Exception:
+ print(""); raise SystemExit
+print(",".join(x["id"] for x in m if x.get("kind")=="local" and not x.get("available")))' 2>/dev/null || echo "")"
+[ -n "$UNAVAIL" ] && echo "$TS WARN local model(s) UNAVAILABLE, battle will run short-handed: $UNAVAIL" >>"$WLOG"
+
# snapshot the last judged line BEFORE the run, to detect a fresh append after
BEFORE="$(/usr/bin/tail -1 "$DAYLOG" 2>/dev/null || true)"
@@ -37,16 +119,5 @@ if [ "$CODE" -eq 0 ] && [ "$FRESH" = "yes" ]; then
fi
# --- FAILURE: nonzero exit OR no fresh judged entry (silent-append guard) ---
-MSG="Model Arena daily battle FAILED $TS (exit=$CODE, freshLog=$FRESH). Tail: $WLOG"
-echo "$TS ALERT $MSG" >>"$WLOG"
-
-# 1) CNCP parking-lot card (note is a JSON body value, not an HTTP header — safe for unicode)
-/usr/bin/curl -s -m 10 -X POST http://127.0.0.1:3333/api/parking-lot \
- -H 'Content-Type: application/json' \
- --data-binary "$(/usr/bin/printf '{"note":"[ALERT] %s"}' "$MSG")" >/dev/null 2>&1 || true
-
-# 2) George email (best-effort; account auth baked into the CLI)
-"$HOME/bin/george" send to=info@designerwallcoverings.com \
- subject="[ALERT] Model Arena daily job failed" body="$MSG" >/dev/null 2>&1 || true
-
+alert "Model Arena daily battle FAILED $TS (exit=$CODE, freshLog=$FRESH). Tail: $WLOG"
exit "$CODE"
← 64f8cdf auto-data-snapshot: 2026-09-13T10:00:49 (1 data files) — dat
·
back to Model Arena
·
auto-data-snapshot: 2026-09-16T14:34:47 (2 data files) — dat 80f1b9c →