sanderson TK-11046: real daily verify canary (green) — read-only health check on the nightly 506 create->golive job

26fad311064b9b2c88e759f940a99125df880f1e · 2026-09-02 09:48:58 -0700 · Steve Abrams

Reads created/to-create/golive/CLEAR counts (dry-run, no --apply), confirms the
nightly launchd job is loaded + last exit 0, checks kill-switch, detects
regression/stall (>=20h guard), emits PASS/WARN/FAIL to out/latest.json for
fleet-health-rollup, tk-logs each run, posts a CNCP card only on drift, and
self-boots-out once all 506 are live. Scheduled 08:00 via
com.steve.sanderson-verify. Fixes pipefail+grep -q SIGPIPE false-negative on
job detection and a same-day false-stall.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012hE4QG1YKm964kDrah98Cn

Files touched

A scripts/sanderson-onboard/verify.sh

Diff

commit 26fad311064b9b2c88e759f940a99125df880f1e
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Wed Sep 2 09:48:58 2026 -0700

    sanderson TK-11046: real daily verify canary (green) — read-only health check on the nightly 506 create->golive job
    
    Reads created/to-create/golive/CLEAR counts (dry-run, no --apply), confirms the
    nightly launchd job is loaded + last exit 0, checks kill-switch, detects
    regression/stall (>=20h guard), emits PASS/WARN/FAIL to out/latest.json for
    fleet-health-rollup, tk-logs each run, posts a CNCP card only on drift, and
    self-boots-out once all 506 are live. Scheduled 08:00 via
    com.steve.sanderson-verify. Fixes pipefail+grep -q SIGPIPE false-negative on
    job detection and a same-day false-stall.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_012hE4QG1YKm964kDrah98Cn
---
 scripts/sanderson-onboard/verify.sh | 89 +++++++++++++++++++++++++++++++++++++
 1 file changed, 89 insertions(+)

diff --git a/scripts/sanderson-onboard/verify.sh b/scripts/sanderson-onboard/verify.sh
new file mode 100755
index 0000000..ad53305
--- /dev/null
+++ b/scripts/sanderson-onboard/verify.sh
@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+# Sanderson 506 daily VERIFY (TK-11046) — read-only health check on the nightly
+# create->golive job (com.steve.sanderson-daily-batch). Fired ~08:00 by launchd
+# (com.steve.sanderson-verify), AFTER the 21:20 nightly run + midnight budget reset.
+# Emits PASS/WARN/FAIL to data/latest.json (fleet-health-rollup vocabulary),
+# tk-logs each run, posts a CNCP parking-lot card ONLY on drift, and boots itself
+# out once all 506 are live. NEVER writes to Shopify (no --apply anywhere).
+set -uo pipefail
+cd "$(dirname "$0")"
+export TK_AGENT=vp-dw-commerce
+NODE="${NODE:-node}"
+NOW="$(date +%Y-%m-%dT%H:%M:%S)"
+STATE="out/verify-state.json"
+LATEST="out/latest.json"
+
+verdict="PASS"; reasons=""
+add(){ reasons="${reasons:+$reasons; }$1"; }
+worse(){ case "$1" in FAIL) verdict=FAIL;; WARN) [ "$verdict" = PASS ] && verdict=WARN;; esac; }
+
+# 1. kill-switch — present is not itself a failure, but note it (job is paused)
+KILL=0; [ -f "$HOME/.dw-fixer-stop" ] && { KILL=1; add "kill-switch PRESENT (nightly paused)"; worse WARN; }
+
+# 2. nightly launchd job present + last exit 0
+# NOTE: capture the list first — `launchctl list | grep -q` trips pipefail via
+# grep's early-exit SIGPIPE, so use a no-pipe substring match instead.
+JOB="com.steve.sanderson-daily-batch"
+LIST="$(launchctl list)"
+if [[ "$LIST" == *"$JOB"* ]]; then
+  LEXIT="$(awk -v j="$JOB" '$3==j{print $2}' <<< "$LIST")"
+  if [ -n "$LEXIT" ] && [ "$LEXIT" != "0" ] && [ "$LEXIT" != "-" ]; then add "nightly last exit=$LEXIT"; worse FAIL; fi
+else
+  # absence is expected AFTER self-terminate at 506; only a problem if not yet done
+  add "nightly job not loaded"; worse WARN
+fi
+
+# 3. state counts (read-only dry-run for to-create + live)
+DRY="$(timeout 240 bash run.sh create --payloads=payloads-imgfixed.jsonl 2>&1 | grep -m1 'to create')"
+TO_CREATE="$(printf '%s' "$DRY" | sed -n 's/.*to create \([0-9]*\).*/\1/p')"
+LIVE_VARS="$(printf '%s' "$DRY" | sed -n 's/.*live \([0-9]*\).*/\1/p')"
+CREATED="$(wc -l < out/created.jsonl 2>/dev/null | tr -d ' ')"; CREATED="${CREATED:-0}"
+CLEARC="$(grep -c '"settlement_verdict":"CLEAR"' out/created.jsonl 2>/dev/null || echo 0)"
+GOLIVE="$(wc -l < out/golive-done.jsonl 2>/dev/null | tr -d ' ')"; GOLIVE="${GOLIVE:-0}"
+[ -z "$TO_CREATE" ] && { add "dry-run gave no to-create (Shopify read failed?)"; worse WARN; TO_CREATE=-1; }
+
+# 4. regression / stall vs last heartbeat
+PREV_CREATED="$($NODE -e 'try{console.log(JSON.parse(require("fs").readFileSync(process.argv[1])).created||0)}catch(e){console.log(0)}' "$STATE" 2>/dev/null || echo 0)"
+PREV_TS="$($NODE -e 'try{console.log(JSON.parse(require("fs").readFileSync(process.argv[1])).ts||"")}catch(e){console.log("")}' "$STATE" 2>/dev/null || echo '')"
+if [ "$CREATED" -lt "$PREV_CREATED" ] 2>/dev/null; then add "REGRESSION created $PREV_CREATED->$CREATED"; worse FAIL; fi
+# STALL only if a full day-ish passed (>=20h) with no advance — avoids false alarms
+# on same-day re-runs (the nightly advances only once/day).
+ELAPSED_H="$($NODE -e 'const p=process.argv[1]; if(!p){console.log(0);process.exit()} const t=Date.parse(p); if(isNaN(t)){console.log(0);process.exit()} console.log(((Date.now()-t)/3.6e6).toFixed(1))' "$PREV_TS" 2>/dev/null || echo 0)"
+if [ "${TO_CREATE:-0}" -gt 0 ] 2>/dev/null && [ "$CREATED" = "$PREV_CREATED" ] && [ -n "$PREV_TS" ] \
+   && awk -v h="$ELAPSED_H" 'BEGIN{exit !(h>=20)}'; then
+  add "STALLED: created unchanged at $CREATED for ${ELAPSED_H}h (since $PREV_TS) while to-create=$TO_CREATE"; worse WARN
+fi
+
+# 5. completion — all created + all CLEAR live
+DONE=0
+if [ "${TO_CREATE:-1}" = "0" ] && [ "$CREATED" -ge 506 ] && [ "$GOLIVE" -ge "$CLEARC" ]; then
+  DONE=1; verdict="PASS"; add "COMPLETE: 506 created, ${GOLIVE}/${CLEARC} CLEAR live — run rollback-map-build.mjs + final PDP verify"
+fi
+
+STATUS="$verdict"
+SUMMARY="[$verdict] created=$CREATED/506 to-create=${TO_CREATE} golive=$GOLIVE clear=$CLEARC live-vars=${LIVE_VARS:-?} kill=$KILL${reasons:+ | $reasons}"
+
+# heartbeat (fleet-health-rollup reads verdict+status)
+cat > "$LATEST" <<JSON
+{"skill":"sanderson-verify","ticket":"TK-11046","ts":"$NOW","verdict":"$verdict","status":"$STATUS","created":$CREATED,"to_create":${TO_CREATE:-null},"golive_done":$GOLIVE,"clear":$CLEARC,"live_variants":${LIVE_VARS:-null},"kill_switch":$KILL,"complete":$DONE,"summary":"$SUMMARY"}
+JSON
+cat > "$STATE" <<JSON
+{"ts":"$NOW","created":$CREATED,"to_create":${TO_CREATE:-null},"verdict":"$verdict"}
+JSON
+
+# tk log every run
+tk log TK-11046 "verify $NOW: $SUMMARY" >/dev/null 2>&1 || true
+
+# CNCP parking-lot card ONLY on drift or completion (best-effort, never fails run)
+if [ "$verdict" != "PASS" ] || [ "$DONE" = "1" ]; then
+  curl -s -X POST http://127.0.0.1:3333/api/parking-lot \
+    -H 'Content-Type: application/json' \
+    -d "{\"title\":\"Sanderson 506 verify: $verdict\",\"note\":\"$SUMMARY\",\"project\":\"sanderson-onboard\",\"ticket\":\"TK-11046\"}" >/dev/null 2>&1 || true
+fi
+
+echo "$SUMMARY"
+
+# self-terminate once 506 fully live (verify has nothing left to watch)
+if [ "$DONE" = "1" ]; then
+  launchctl bootout "gui/$(id -u)/com.steve.sanderson-verify" 2>/dev/null || true
+fi

← 2fa623a auto-data-snapshot: 2026-09-02T09:35:20 (15 data files) — sc · back to Designerwallcoverings · osborne-onboard: golive uses SHOPIFY_FULL_ACCESS_TOKEN for w d99146d →