← back to Tk 11331 Exec
TK-11331: hands-off D2 daily-sweep automation (wrapper + plist + state + README) — files only, not loaded
53413a1604e0a19b076c3e4230218039a46630f9 · 2026-09-09 13:57:54 -0700 · Steve Abrams
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X3W2EKx4Yu1hZfiEdQPkQf
Files touched
A data/d2-sweep-state.jsonA deploy/com.steve.tk11331-d2-sweep.plistA scripts/README-d2-sweep.mdA scripts/d2_daily_sweep.sh
Diff
commit 53413a1604e0a19b076c3e4230218039a46630f9
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Wed Sep 9 13:57:54 2026 -0700
TK-11331: hands-off D2 daily-sweep automation (wrapper + plist + state + README) — files only, not loaded
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X3W2EKx4Yu1hZfiEdQPkQf
---
data/d2-sweep-state.json | 6 ++
deploy/com.steve.tk11331-d2-sweep.plist | 56 +++++++++++
scripts/README-d2-sweep.md | 81 ++++++++++++++++
scripts/d2_daily_sweep.sh | 166 ++++++++++++++++++++++++++++++++
4 files changed, 309 insertions(+)
diff --git a/data/d2-sweep-state.json b/data/d2-sweep-state.json
new file mode 100644
index 0000000..207c61b
--- /dev/null
+++ b/data/d2-sweep-state.json
@@ -0,0 +1,6 @@
+{
+ "offset": 431,
+ "last_batch": null,
+ "total": 3600,
+ "note": "431 products (d2_set indices 0..430) already applied+verified on Day-1. Sweep resumes at 431. Advances +1000/run until offset>=3600."
+}
diff --git a/deploy/com.steve.tk11331-d2-sweep.plist b/deploy/com.steve.tk11331-d2-sweep.plist
new file mode 100644
index 0000000..c3a5137
--- /dev/null
+++ b/deploy/com.steve.tk11331-d2-sweep.plist
@@ -0,0 +1,56 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<!--
+ TK-11331 Decision 2 hands-off daily sweep.
+ Runs scripts/d2_daily_sweep.sh once per day at 04:45 local (off-peak; after the
+ 03:00 pg_dump and the 04:20 scratch janitor, before business hours).
+ ONE batch (<=1000 product-writes) per fire; the wrapper self-stops at 3600 and
+ boots itself out. RunAtLoad is FALSE so a reboot/bootstrap never fires an
+ unattended batch immediately — it waits for the next 04:45.
+
+ Install (Steve):
+ cp deploy/com.steve.tk11331-d2-sweep.plist ~/Library/LaunchAgents/
+ launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.steve.tk11331-d2-sweep.plist
+ Kill switch (Steve):
+ launchctl bootout gui/$(id -u)/com.steve.tk11331-d2-sweep
+-->
+<plist version="1.0">
+<dict>
+ <key>Label</key>
+ <string>com.steve.tk11331-d2-sweep</string>
+
+ <key>ProgramArguments</key>
+ <array>
+ <string>/bin/bash</string>
+ <string>/Users/macstudio3/Projects/tk-11331-exec/scripts/d2_daily_sweep.sh</string>
+ </array>
+
+ <key>WorkingDirectory</key>
+ <string>/Users/macstudio3/Projects/tk-11331-exec</string>
+
+ <key>StartCalendarInterval</key>
+ <dict>
+ <key>Hour</key>
+ <integer>4</integer>
+ <key>Minute</key>
+ <integer>45</integer>
+ </dict>
+
+ <key>RunAtLoad</key>
+ <false/>
+
+ <key>StandardOutPath</key>
+ <string>/Users/macstudio3/Projects/tk-11331-exec/data/sweep-stdout.log</string>
+
+ <key>StandardErrorPath</key>
+ <string>/Users/macstudio3/Projects/tk-11331-exec/data/sweep-stderr.log</string>
+
+ <key>EnvironmentVariables</key>
+ <dict>
+ <key>PATH</key>
+ <string>/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
+ <key>HOME</key>
+ <string>/Users/macstudio3</string>
+ </dict>
+</dict>
+</plist>
diff --git a/scripts/README-d2-sweep.md b/scripts/README-d2-sweep.md
new file mode 100644
index 0000000..36b9a66
--- /dev/null
+++ b/scripts/README-d2-sweep.md
@@ -0,0 +1,81 @@
+# TK-11331 Decision 2 — hands-off daily sweep
+
+Finishes D2 (set `global.v_prods_quantity_order_min="5"` + `global.unit_of_measure="Sold Per Yard"`
+on the non-compliant X-prefix @52/54 products) automatically, one throttled batch per day, with a
+fail-loud verify gate and an auto-stop.
+
+## Files
+- `scripts/d2_daily_sweep.sh` — the wrapper (one batch per invocation).
+- `deploy/com.steve.tk11331-d2-sweep.plist` — launchd job (04:45 daily). **NOT installed** — Steve installs.
+- `data/d2-sweep-state.json` — persisted offset (source of truth for "where are we").
+- `data/d2-sweep-runs.log` — human-readable per-run trail.
+- `data/sweep-stdout.log` / `data/sweep-stderr.log` — launchd captured output.
+- `data/d2-sweep-FAILED` — halt marker (present ⇒ wrapper refuses to run; **fail-loud**).
+- `data/d2-sweep-DONE` — completion marker (present ⇒ nothing to do).
+
+## Offset math
+- `data/d2_set.json` holds **3600** non-compliant products (indices `0..3599`).
+- **431** already applied+verified on Day-1 (indices `0..430`). Sweep resumes at offset **431**.
+- Each run: apply `--offset=$OFFSET --limit=1000`, verify, then advance the offset by **1000**.
+
+| Run | offset | rows applied | indices |
+|-----|--------|--------------|---------|
+| (Day-1, done) | 0 | 431 | 0..430 |
+| Day-2 | 431 | 1000 | 431..1430 |
+| Day-3 | 1431 | 1000 | 1431..2430 |
+| Day-4 | 2431 | 1000 | 2431..3430 |
+| Day-5 | 3431 | 169 | 3431..3599 |
+
+3169 remaining ÷ ≤1000/day = **~4 daily runs**. The last day is a partial (169) because
+`limit=1000` caps at the end of the set (`rows = min(1000, total - offset)`).
+
+## Per-run behavior
+1. Single-flight **lock** (`data/.d2-sweep.lock`) — a second fire while one is running just exits.
+2. If `data/d2-sweep-FAILED` exists → **HALT** (exit 1), do nothing. Manual clear required.
+3. If `data/d2-sweep-DONE` exists → nothing to do.
+4. If `offset >= 3600` → write DONE, self-disable, ledger completion, exit.
+5. `node d2_metafields.mjs --apply --offset=$OFFSET --limit=1000`
+ — **PG mirror first, then Shopify** (authoritative), per rule 5. Records old values to `data/d2-restore-map.jsonl`.
+6. `node verify_d2.mjs $OFFSET 1000` — re-reads via the **Shopify Admin API (authoritative, NOT the CDN)**.
+7. **Fail-loud verify gate:** if `NON-compliant != 0` (or verify output is unreadable / `checked=0`)
+ → write `data/d2-sweep-FAILED`, **do NOT advance the offset**, exit 1. The offset stays put so a
+ fixed re-run reprocesses the same slice (writes are idempotent — same values).
+8. Append the batch to `~/.claude/yolo-queue/executed-reversible/ledger.jsonl` via `log-exec.mjs`
+ (`ticket=TK-11331`, `action=d2-daily-batch`, `blast_radius=<rows>`, `undo`, `verify`).
+9. Advance the offset in `data/d2-sweep-state.json`.
+10. When the advance reaches `>= 3600` → write DONE, ledger completion, **self-disable** the launchd
+ job (`launchctl bootout gui/$UID/com.steve.tk11331-d2-sweep`).
+
+## Idempotency & safety
+- Re-running the same offset re-writes the identical values ("5" / "Sold Per Yard") — safe, no drift.
+- On a bad verify the offset does **not** advance, so nothing is skipped.
+- `RunAtLoad=false` — a reboot never fires an unattended batch on boot; it waits for the next 04:45.
+
+## Install (Steve — GATED, one approve)
+```sh
+cp ~/Projects/tk-11331-exec/deploy/com.steve.tk11331-d2-sweep.plist ~/Library/LaunchAgents/
+launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.steve.tk11331-d2-sweep.plist
+```
+
+## Manual kill switch
+```sh
+launchctl bootout gui/$(id -u)/com.steve.tk11331-d2-sweep
+# optional, after DONE: rm ~/Library/LaunchAgents/com.steve.tk11331-d2-sweep.plist
+```
+
+## Clear a FAILED halt (after investigating)
+```sh
+cat ~/Projects/tk-11331-exec/data/d2-sweep-FAILED # read the reason first
+rm ~/Projects/tk-11331-exec/data/d2-sweep-FAILED # then the next 04:45 run resumes at the same offset
+```
+
+## Rollback
+```sh
+cd ~/Projects/tk-11331-exec && node d2_rollback.mjs --apply # reverts the full data/d2-restore-map.jsonl
+```
+
+## Run one batch manually (test / catch-up, outside launchd)
+```sh
+bash ~/Projects/tk-11331-exec/scripts/d2_daily_sweep.sh
+```
+(Still throttled to one ≤1000-row batch per invocation.)
diff --git a/scripts/d2_daily_sweep.sh b/scripts/d2_daily_sweep.sh
new file mode 100755
index 0000000..eec609b
--- /dev/null
+++ b/scripts/d2_daily_sweep.sh
@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+# TK-11331 Decision 2 — hands-off DAILY SWEEP wrapper.
+# One batch per invocation (launchd fires this once/day). Idempotent, fail-loud,
+# self-stopping at 3600, self-disabling launchd job when done.
+#
+# Per run:
+# 1. read persisted offset (init 431)
+# 2. node d2_metafields.mjs --apply --offset=$OFFSET --limit=1000 (PG-mirror-first, then Shopify)
+# 3. node verify_d2.mjs $OFFSET 1000 (Admin API = authoritative, NOT CDN)
+# -> ABORT + write FAILED marker + DO NOT advance offset if any NON-compliant
+# 4. append the batch to ~/.claude/yolo-queue/executed-reversible/ledger.jsonl (via log-exec.mjs)
+# 5. advance the offset state
+# 6. when offset >= 3600: write DONE marker + self-disable (launchctl bootout own job) + ledger completion
+#
+# Kill switch (manual): launchctl bootout gui/$(id -u)/com.steve.tk11331-d2-sweep
+# Clear a FAILED halt (manual, after investigating): rm data/d2-sweep-FAILED
+set -euo pipefail
+
+REPO="/Users/macstudio3/Projects/tk-11331-exec"
+DATA="$REPO/data"
+LEDGER_LOGGER="$HOME/.claude/yolo-queue/executed-reversible/log-exec.mjs"
+JOB_LABEL="com.steve.tk11331-d2-sweep"
+LIMIT=1000
+AGENT="vp-dw-commerce"
+TICKET="TK-11331"
+
+STATE="$DATA/d2-sweep-state.json"
+FAILED="$DATA/d2-sweep-FAILED"
+DONE="$DATA/d2-sweep-DONE"
+RUNLOG="$DATA/d2-sweep-runs.log"
+LOCK="$DATA/.d2-sweep.lock"
+
+cd "$REPO"
+mkdir -p "$DATA"
+
+ts(){ date '+%Y-%m-%dT%H:%M:%S%z'; }
+say(){ echo "[$(ts)] $*" | tee -a "$RUNLOG"; }
+
+# ---- single-flight lock (mkdir is atomic) ----
+if ! mkdir "$LOCK" 2>/dev/null; then
+ say "LOCK held ($LOCK) — a run is already in progress. Exiting."
+ exit 0
+fi
+trap 'rmdir "$LOCK" 2>/dev/null || true' EXIT
+
+# ---- fail-loud halt: refuse to run while a prior FAILED marker stands ----
+if [[ -f "$FAILED" ]]; then
+ say "HALTED — prior FAILED marker present: $FAILED"
+ say " $(cat "$FAILED" 2>/dev/null | tr '\n' ' ')"
+ say " Investigate, then 'rm $FAILED' to resume. NOT advancing."
+ exit 1
+fi
+
+# ---- already complete? ----
+if [[ -f "$DONE" ]]; then
+ say "DONE marker present — nothing to do. (Job should already be booted-out.)"
+ exit 0
+fi
+
+# ---- total rows in the D2 set (dynamic; expected 3600) ----
+TOTAL="$(node -e 'process.stdout.write(String(JSON.parse(require("fs").readFileSync("data/d2_set.json","utf8")).length))')"
+if ! [[ "$TOTAL" =~ ^[0-9]+$ ]]; then
+ say "ERROR — could not read TOTAL from data/d2_set.json"; exit 1
+fi
+
+# ---- read / init offset ----
+if [[ -f "$STATE" ]]; then
+ OFFSET="$(node -e 'process.stdout.write(String(JSON.parse(require("fs").readFileSync(process.argv[1],"utf8")).offset))' "$STATE")"
+else
+ OFFSET=431 # 431 already done+verified on Day-1 (indices 0..430)
+ say "No state file — initializing offset=$OFFSET"
+fi
+if ! [[ "$OFFSET" =~ ^[0-9]+$ ]]; then
+ say "ERROR — bad offset in state: '$OFFSET'"; exit 1
+fi
+
+# ---- stop condition: everything swept ----
+if (( OFFSET >= TOTAL )); then
+ say "COMPLETE — offset=$OFFSET >= total=$TOTAL. Writing DONE marker + self-disabling."
+ printf 'D2 sweep complete: offset=%s reached total=%s at %s\n' "$OFFSET" "$TOTAL" "$(ts)" > "$DONE"
+ node "$LEDGER_LOGGER" --agent "$AGENT" --ticket "$TICKET" \
+ --action "D2 daily-sweep COMPLETE — all $TOTAL non-compliant products swept (min->5 + unit_of_measure 'Sold Per Yard'). Job self-disabling." \
+ --blast "$TOTAL" \
+ --undo "cd $REPO && node d2_rollback.mjs --apply" \
+ --verify "cd $REPO && node verify_d2.mjs 0 $TOTAL # expect NON-compliant=0" \
+ 2>&1 | tee -a "$RUNLOG" || say "WARN — completion ledger append failed (non-fatal)"
+ launchctl bootout "gui/$(id -u)/$JOB_LABEL" 2>&1 | tee -a "$RUNLOG" || \
+ say "NOTE — bootout returned non-zero (job may already be unloaded)."
+ say "Self-disabled. Steve may now 'rm ~/Library/LaunchAgents/$JOB_LABEL.plist' to fully remove."
+ exit 0
+fi
+
+# ---- how many rows THIS batch actually touches (last day is a partial) ----
+REMAIN=$(( TOTAL - OFFSET ))
+ROWS=$(( REMAIN < LIMIT ? REMAIN : LIMIT ))
+say "START batch offset=$OFFSET limit=$LIMIT (rows=$ROWS, total=$TOTAL) — throttle <=1000/day"
+
+# ---- 1) APPLY (PG mirror first, then Shopify — inside d2_metafields.mjs) ----
+if ! node d2_metafields.mjs --apply --offset="$OFFSET" --limit="$LIMIT" 2>&1 | tee -a "$RUNLOG"; then
+ printf 'APPLY FAILED at offset=%s (rows=%s) at %s — d2_metafields.mjs exited non-zero\n' \
+ "$OFFSET" "$ROWS" "$(ts)" > "$FAILED"
+ say "APPLY FAILED — wrote $FAILED, NOT advancing offset."
+ exit 1
+fi
+
+# ---- 2) VERIFY via Admin API (authoritative). Fail loud on any non-compliant. ----
+VERIFY_OUT="$(node verify_d2.mjs "$OFFSET" "$LIMIT" 2>&1)"
+echo "$VERIFY_OUT" | tee -a "$RUNLOG"
+NONCOMPLIANT="$(printf '%s' "$VERIFY_OUT" | grep -oE 'NON-compliant=[0-9]+' | grep -oE '[0-9]+' | tail -1)"
+CHECKED="$(printf '%s' "$VERIFY_OUT" | grep -oE 'checked=[0-9]+' | grep -oE '[0-9]+' | tail -1)"
+
+if [[ -z "$NONCOMPLIANT" || -z "$CHECKED" ]]; then
+ printf 'VERIFY UNREADABLE at offset=%s (rows=%s) at %s — no NON-compliant/checked line from verify_d2.mjs. Output: %s\n' \
+ "$OFFSET" "$ROWS" "$(ts)" "$(printf '%s' "$VERIFY_OUT" | tr '\n' ' ')" > "$FAILED"
+ say "VERIFY UNREADABLE — wrote $FAILED, NOT advancing offset."
+ exit 1
+fi
+if (( CHECKED == 0 )); then
+ printf 'VERIFY checked=0 at offset=%s at %s — nothing was verified (suspicious).\n' \
+ "$OFFSET" "$(ts)" > "$FAILED"
+ say "VERIFY checked=0 — wrote $FAILED, NOT advancing offset."
+ exit 1
+fi
+if (( NONCOMPLIANT != 0 )); then
+ printf 'VERIFY FAILED at offset=%s (rows=%s) at %s — NON-compliant=%s (expected 0).\n' \
+ "$OFFSET" "$ROWS" "$(ts)" "$NONCOMPLIANT" > "$FAILED"
+ say "VERIFY FAILED — NON-compliant=$NONCOMPLIANT. Wrote $FAILED, NOT advancing offset. HALTED."
+ exit 1
+fi
+say "VERIFY OK — checked=$CHECKED NON-compliant=0"
+
+# ---- 3) LEDGER this batch (reversible tier) ----
+NEXT=$(( OFFSET + LIMIT ))
+node "$LEDGER_LOGGER" --agent "$AGENT" --ticket "$TICKET" \
+ --action "D2 daily-batch: global.v_prods_quantity_order_min->5 + global.unit_of_measure->'Sold Per Yard' on $ROWS products (d2_set offset=$OFFSET..$(( OFFSET+ROWS-1 )); PG mirror first, then Shopify). Admin-API verified NON-compliant=0." \
+ --blast "$ROWS" \
+ --undo "cd $REPO && node d2_rollback.mjs --apply # reverts full data/d2-restore-map.jsonl" \
+ --verify "cd $REPO && node verify_d2.mjs $OFFSET $LIMIT # expect NON-compliant=0" \
+ 2>&1 | tee -a "$RUNLOG" || say "WARN — ledger append failed (non-fatal); batch already applied+verified."
+
+# ---- 4) ADVANCE offset state ----
+node -e '
+ const fs=require("fs");
+ const [state,next,rows,total,ts]=[process.argv[1],+process.argv[2],+process.argv[3],+process.argv[4],process.argv[5]];
+ fs.writeFileSync(state, JSON.stringify({offset:next,last_batch:{applied_rows:rows,advanced_at:ts},total},null,2)+"\n");
+' "$STATE" "$NEXT" "$ROWS" "$TOTAL" "$(ts)"
+say "ADVANCED offset $OFFSET -> $NEXT (applied $ROWS rows)."
+
+# ---- 5) if that was the last batch, finish now (DONE + self-disable) ----
+if (( NEXT >= TOTAL )); then
+ say "That was the final batch — finalizing."
+ printf 'D2 sweep complete: last batch offset=%s advanced to %s (total=%s) at %s\n' \
+ "$OFFSET" "$NEXT" "$TOTAL" "$(ts)" > "$DONE"
+ node "$LEDGER_LOGGER" --agent "$AGENT" --ticket "$TICKET" \
+ --action "D2 daily-sweep COMPLETE — final batch landed; all $TOTAL non-compliant products swept. Job self-disabling." \
+ --blast "$ROWS" \
+ --undo "cd $REPO && node d2_rollback.mjs --apply" \
+ --verify "cd $REPO && node verify_d2.mjs 0 $TOTAL # expect NON-compliant=0" \
+ 2>&1 | tee -a "$RUNLOG" || say "WARN — completion ledger append failed (non-fatal)"
+ launchctl bootout "gui/$(id -u)/$JOB_LABEL" 2>&1 | tee -a "$RUNLOG" || \
+ say "NOTE — bootout returned non-zero (job may already be unloaded)."
+ say "Self-disabled. Steve may 'rm ~/Library/LaunchAgents/$JOB_LABEL.plist' to fully remove."
+fi
+
+say "END batch — success."
+exit 0
← 362d57e TK-11331 D3a: reprice 3 diff sellable variants to hw_price (
·
back to Tk 11331 Exec
·
auto-data-snapshot: 2026-09-09T14:29:35 (3 data files) — dat 384c85a →