← back to Ticket System
Verify pinned installed watchdog bytes and reject unsafe source drift
0917d3917d32d3914c9def915f202bdd1381122c · 2026-09-08 18:47:34 -0700 · Steve Abrams
Files touched
A verification/tk11129-alert-empty-notes-20260909/.gitignoreM verification/tk11129-alert-empty-notes-20260909/README.mdM verification/tk11129-alert-empty-notes-20260909/approval-draft.mdM verification/tk11129-alert-empty-notes-20260909/verify.py
Diff
commit 0917d3917d32d3914c9def915f202bdd1381122c
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Tue Sep 8 18:47:34 2026 -0700
Verify pinned installed watchdog bytes and reject unsafe source drift
---
.../tk11129-alert-empty-notes-20260909/.gitignore | 1 +
.../tk11129-alert-empty-notes-20260909/README.md | 25 +++++++++++--
.../approval-draft.md | 13 +++++++
.../tk11129-alert-empty-notes-20260909/verify.py | 42 +++++++++++++++++-----
4 files changed, 70 insertions(+), 11 deletions(-)
diff --git a/verification/tk11129-alert-empty-notes-20260909/.gitignore b/verification/tk11129-alert-empty-notes-20260909/.gitignore
new file mode 100644
index 00000000..c18dd8d8
--- /dev/null
+++ b/verification/tk11129-alert-empty-notes-20260909/.gitignore
@@ -0,0 +1 @@
+__pycache__/
diff --git a/verification/tk11129-alert-empty-notes-20260909/README.md b/verification/tk11129-alert-empty-notes-20260909/README.md
index 297c9338..3ab614ca 100644
--- a/verification/tk11129-alert-empty-notes-20260909/README.md
+++ b/verification/tk11129-alert-empty-notes-20260909/README.md
@@ -2,13 +2,32 @@
The watchdog alert branch crashes under macOS Bash 3.2 when `notes` is empty. The retained candidate changes one expansion to accept an empty array. It has not been installed.
-Run the exact alert-block regression from this directory, supplying a fresh evidence filename:
+Run the exact alert-block regression from this directory against the current original installed source:
```sh
-python3 verify.py --evidence /private/tmp/tk11129-proof-UNIQUE.json
+TK11129_PROOF_DIR="$(mktemp -d /private/tmp/tk11129-proof.XXXXXX)"
+python3 verify.py --evidence "$TK11129_PROOF_DIR/original-proof.json"
```
-The runner retains a fresh `mktemp -d` directory, reproduces the baseline failure, verifies candidate append and state behavior in eight cases, and checks both complete scripts for syntax without executing them. Notifications are stubbed. It refuses changed source structure and existing evidence filenames. Current-source and retained-candidate hashes are recorded in `e2e-proof-mktemp.json`. The first proof is retained as historical evidence; the mktemp proof supersedes its scratch method.
+`--source` defaults to `/Users/macstudio3/Projects/exo-cluster-watchdog/watchdog.sh`; `--expect-source original|candidate` defaults to `original`. The original hash is pinned to `96aba9b28c3bab39eb86f256f571d2cdd3b78f4a92f829f0e8d1c5c21d6d8a60`, and candidate hash to `f661e30b22281685358afc886360db949e147402882e81026853945cb7e5318b`; neither expectation is derived from the inspected input. The retained original baseline is independently checked against its pinned hash on every run.
+
+To verify the retained candidate before installation:
+
+```sh
+TK11129_PROOF_DIR="$(mktemp -d /private/tmp/tk11129-candidate-proof.XXXXXX)"
+python3 verify.py --source ./watchdog.candidate.sh --expect-source candidate --evidence "$TK11129_PROOF_DIR/candidate-proof.json"
+```
+
+After Steve approves the exact change and the candidate is installed, run this exact command against the default **live** path:
+
+```sh
+TK11129_PROOF_DIR="$(mktemp -d /private/tmp/tk11129-installed-proof.XXXXXX)"
+python3 /Users/macstudio3/Projects/ticket-system/verification/tk11129-alert-empty-notes-20260909/verify.py --expect-source candidate --evidence "$TK11129_PROOF_DIR/installed-proof.json"
+```
+
+Candidate mode exercises the actual checked candidate input bytes, alongside the pinned original baseline; it does not manufacture a substitute candidate for testing. Before installation, the command above intentionally refuses the unsafe original. Candidate drift and retained-baseline drift also refuse before running shell tests or writing success evidence.
+
+The runner retains a fresh `mktemp -d` directory, reproduces the baseline failure, verifies candidate append and state behavior in eight cases, and checks both complete scripts for syntax without executing them. Notifications are stubbed. It refuses existing evidence filenames. Current-source and retained-candidate hashes are recorded in `e2e-proof-mktemp.json`. The first proof is retained as historical evidence; the mktemp proof supersedes its scratch method. Increment 2 mode/negative-path evidence is `../../data/codex-yoloforever/cycle-20260909T0123Z.sqpb5o/ops-increment2-verifier-proof.json`.
Full watchdog execution can heal or restart services and is outside this preparation. Live installation and operational recovery remain blocked by the exact approval in `approval-draft.md`. Current healthy scheduler runs do not prove recovery of the alert branch.
diff --git a/verification/tk11129-alert-empty-notes-20260909/approval-draft.md b/verification/tk11129-alert-empty-notes-20260909/approval-draft.md
index a0eb463e..451b603e 100644
--- a/verification/tk11129-alert-empty-notes-20260909/approval-draft.md
+++ b/verification/tk11129-alert-empty-notes-20260909/approval-draft.md
@@ -42,6 +42,19 @@ A peer dropping between the first and second probe can plausibly yield WARN with
5. Observe at least two naturally scheduled launchd runs: run counters increase, last exit is zero, heartbeat generation advances within cadence, and direct GET `/state` results agree with the current state. Compare newly appended stderr with the pre-install offset. Preserve evidence and commit only the approved source fix.
6. Do not label the incident recovered merely because PASS heartbeats succeed. Record the isolated installed alert-branch proof separately from natural scheduler proof, and retain any unresolved incident-correlation gap.
+## Exact installed-file verification command
+
+After approval and installation, this checks the default live watchdog path against the independently pinned candidate hash, then runs its actual alert-block bytes in isolated scratch alongside the pinned original failure baseline:
+
+```sh
+TK11129_PROOF_DIR="$(mktemp -d /private/tmp/tk11129-installed-proof.XXXXXX)"
+python3 /Users/macstudio3/Projects/ticket-system/verification/tk11129-alert-empty-notes-20260909/verify.py --expect-source candidate --evidence "$TK11129_PROOF_DIR/installed-proof.json"
+```
+
+The verifier defaults to `--source /Users/macstudio3/Projects/exo-cluster-watchdog/watchdog.sh`; `--expect-source original` is its preparation-mode default. Candidate mode intentionally refuses the presently installed unsafe source, any candidate byte drift, and a changed retained original baseline. Expected hashes are pinned independently in the verifier, not calculated from whichever input is presented. Retained-candidate mode passes all eight alert tests; default original mode preserves the failing baseline plus candidate proof. This command never invokes the full watchdog or sends a notification; natural scheduler evidence is still required separately.
+
+Increment 2 proof: `/Users/macstudio3/Projects/ticket-system/data/codex-yoloforever/cycle-20260909T0123Z.sqpb5o/ops-increment2-verifier-proof.json`.
+
## Rollback after approved installation
If the approved change causes a regression, require the installed hash still equals the candidate hash (stop if another writer changed it), copy the retained pre-install original over the installed watchdog with permissions preserved, and verify the original SHA-256. Observe the next natural launchd run and heartbeat; no restart, deletion, or forced watchdog invocation is needed. Log the regression and rollback on TK-11129 and retain the ticket as blocked.
diff --git a/verification/tk11129-alert-empty-notes-20260909/verify.py b/verification/tk11129-alert-empty-notes-20260909/verify.py
index fd7102b7..5d381db9 100644
--- a/verification/tk11129-alert-empty-notes-20260909/verify.py
+++ b/verification/tk11129-alert-empty-notes-20260909/verify.py
@@ -11,6 +11,9 @@ import shlex
import subprocess
SOURCE = Path('/Users/macstudio3/Projects/exo-cluster-watchdog/watchdog.sh')
+BASELINE = Path(__file__).resolve().with_name('watchdog.original.sh')
+PINNED_ORIGINAL_SHA256 = '96aba9b28c3bab39eb86f256f571d2cdd3b78f4a92f829f0e8d1c5c21d6d8a60'
+PINNED_CANDIDATE_SHA256 = 'f661e30b22281685358afc886360db949e147402882e81026853945cb7e5318b'
OLD = '${notes[*]}'
NEW = '${notes[*]:-}'
@@ -20,20 +23,38 @@ def sha(data):
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--evidence', required=True, type=Path)
+ parser.add_argument('--source', type=Path, default=SOURCE,
+ help='File to verify; defaults to the installed watchdog')
+ parser.add_argument('--expect-source', choices=('original', 'candidate'), default='original',
+ help='Required independently pinned source version (default: original)')
args = parser.parse_args()
evidence = args.evidence
if evidence.exists():
raise SystemExit('Refusing to overwrite existing evidence: ' + str(evidence))
- original_bytes = SOURCE.read_bytes()
+ source = args.source
+ source_bytes = source.read_bytes()
+ original_bytes = BASELINE.read_bytes()
+ if sha(original_bytes) != PINNED_ORIGINAL_SHA256:
+ raise SystemExit('Refusing baseline drift: retained original SHA-256 must equal ' + PINNED_ORIGINAL_SHA256)
+ expected_sha = PINNED_ORIGINAL_SHA256 if args.expect_source == 'original' else PINNED_CANDIDATE_SHA256
+ if sha(source_bytes) != expected_sha:
+ raise SystemExit('Refusing source mismatch: expected pinned ' + args.expect_source +
+ ' SHA-256 ' + expected_sha + ', got ' + sha(source_bytes))
original = original_bytes.decode()
assert original.count(OLD) == 1, 'Source drift: expected exactly one unsafe notes expansion'
- candidate = original.replace(OLD, NEW)
+ expected_candidate_bytes = original.replace(OLD, NEW).encode()
+ if sha(expected_candidate_bytes) != PINNED_CANDIDATE_SHA256:
+ raise SystemExit('Refusing candidate derivation drift: expected pinned candidate SHA-256')
+ candidate_bytes = source_bytes if args.expect_source == 'candidate' else expected_candidate_bytes
+ if candidate_bytes != expected_candidate_bytes:
+ raise SystemExit('Refusing candidate byte mismatch against independently pinned one-line replacement')
+ candidate = candidate_bytes.decode()
scratch = Path(subprocess.check_output(
['mktemp', '-d', '/private/tmp/tk11129-alert-regression.XXXXXX'], text=True).strip())
original_path = scratch / 'watchdog.original.sh'
candidate_path = scratch / 'watchdog.candidate.sh'
original_path.write_bytes(original_bytes)
- candidate_path.write_text(candidate)
+ candidate_path.write_bytes(candidate_bytes)
patch = ''.join(difflib.unified_diff(original.splitlines(True), candidate.splitlines(True),
fromfile='a/watchdog.sh', tofile='b/watchdog.sh'))
(scratch / 'watchdog.patch').write_text(patch)
@@ -97,17 +118,22 @@ def main():
'intent': 'Prove one-line notes expansion fix at exact isolated Bash alert append boundary',
'risk_tier': 'R1 isolated candidate; live installation remains R4 gated',
'timestamp': datetime.datetime.now(datetime.timezone.utc).isoformat(),
- 'source': str(SOURCE), 'source_sha256_before': sha(original_bytes),
- 'source_sha256_after': sha(SOURCE.read_bytes()), 'candidate_sha256': sha(candidate.encode()),
+ 'source': str(source), 'expect_source': args.expect_source,
+ 'source_sha256_before': sha(source_bytes),
+ 'source_sha256_after': sha(source.read_bytes()), 'candidate_sha256': sha(candidate_bytes),
+ 'baseline': str(BASELINE), 'baseline_sha256': sha(original_bytes),
+ 'pinned_original_sha256': PINNED_ORIGINAL_SHA256,
+ 'pinned_candidate_sha256': PINNED_CANDIDATE_SHA256,
+ 'candidate_bytes_origin': 'actual checked source bytes' if args.expect_source == 'candidate' else 'pinned one-line replacement of pinned baseline',
'original': str(original_path), 'candidate': str(candidate_path), 'patch': str(scratch / 'watchdog.patch'),
'retained_scratch': str(scratch),
'bash_version': subprocess.run(['/bin/bash', '--version'], capture_output=True, text=True).stdout.splitlines()[0],
'syntax': syntax, 'cases': rows,
'boundary': 'Exact original/candidate alert block, real filesystem append/prev-state; notification invocation stubbed',
'cleanup': 'All scratch retained; no deletion or cleanup commands performed',
- 'critical_live_journey': {'verdict': 'SKIP', 'reason': 'No live installation authorized; no full watchdog run or healing attempted'},
- 'source_unchanged': original_bytes == SOURCE.read_bytes(),
- 'isolated_verdict': 'PASS' if all(r['verdict'] == 'PASS' for r in rows) and original_bytes == SOURCE.read_bytes() else 'FAIL',
+ 'critical_live_journey': {'verdict': 'SKIP', 'reason': 'This verifier neither authorizes installation nor runs the full watchdog; natural scheduler recovery requires separate evidence'},
+ 'source_unchanged': source_bytes == source.read_bytes(),
+ 'isolated_verdict': 'PASS' if all(r['verdict'] == 'PASS' for r in rows) and source_bytes == source.read_bytes() else 'FAIL',
'overall_status': 'partial',
}
evidence.write_text(json.dumps(result, indent=2) + '\n')
← fef7ba46 Prepare Exo watchdog empty-notes alert fix with isolated reg
·
back to Ticket System
·
Record ordered overnight cycle and gated Exo verification ev 7c9e6afc →