hourly local-model leaderboard watcher (no-Bash agent, recommend-only, gated)

7070cc0bc08c7bf0dfc9ccbceaad7cf8ce40ffe2 · 2026-08-17 16:01:45 -0700 · steve
Files touched

A .gitignore
A review-prompt.md
A review.sh
Diff

commit 7070cc0bc08c7bf0dfc9ccbceaad7cf8ce40ffe2
Author: steve <steve@designerwallcoverings.com>
Date:   Mon Aug 17 16:01:45 2026 -0700

    hourly local-model leaderboard watcher (no-Bash agent, recommend-only, gated)
---
 .gitignore       |  4 ++++
 review-prompt.md | 16 ++++++++++++++++
 review.sh        | 31 +++++++++++++++++++++++++++++++
 3 files changed, 51 insertions(+)

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..ee3dae2
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,4 @@
+data/*.log
+data/reviews.jsonl
+data/local-state.txt
+.DS_Store
diff --git a/review-prompt.md b/review-prompt.md
new file mode 100644
index 0000000..500cd60
--- /dev/null
+++ b/review-prompt.md
@@ -0,0 +1,16 @@
+You are the hourly LOCAL-MODEL LEADERBOARD WATCHER for Steve's exo Mac cluster. Your job: make sure the cluster is running the BEST open-weight model that actually FITS the hardware. READ-ONLY + RECOMMEND-ONLY — never download a model, never change the loaded model, never touch the cluster. Draft recommendations only.
+
+HARDWARE REALITY (the binding constraint — do not forget it):
+- exo cluster = 3 Macs: one 96 GB M3 Ultra (primary) + two 32 GB nodes.
+- Models ≤ ~90 GB run BEST single-node on the M3 Ultra (faster — no inter-node hops). exo only shards when a model exceeds 96 GB.
+- So "best local model for us" = the highest-ranked OPEN-WEIGHT model that fits ~90 GB at a good quant (4–8 bit) AND exists as an MLX build. Frontier MoEs (DeepSeek V4.x ~671B, Llama 4 Maverick ~400B) DO NOT FIT even the whole 160 GB cluster — exclude them, note them as "too big for this hardware."
+
+STEPS:
+1. Get the current arena.ai / LMArena leaderboard. It is JS-rendered, so WebFetch returns only the shell — rely on WebSearch for current rankings (query e.g. "LMArena arena.ai leaderboard top open-weight models <this month> Qwen DeepSeek Llama Mistral Gemma"), cross-checking 2 sources (swfte.com/lmarena, localaimaster, HF space lmarena-ai/arena-leaderboard).
+2. From the leaderboard, take the top OPEN-WEIGHT models, then FILTER to those that (a) fit ~90 GB single-node at 4–8 bit and (b) have an MLX-community build. That's the candidate set.
+3. Read what we currently have from the file `data/local-state.txt` (the wrapper already wrote it: the loaded exo model + downloaded models + hardware). You have NO shell — read that file with the Read tool. Current default = Qwen3.6-35B-A3B-8bit.
+4. COMPARE. If a candidate open model clearly beats our current default AND fits AND has an MLX build we don't yet have → that's a RECOMMENDATION.
+5. Write a dated one-line entry to data/reviews.jsonl (date, top-fitting-open-model, our-default, verdict: KEEP or UPGRADE-AVAILABLE). ALWAYS write this heartbeat even if nothing changed.
+6. ONLY if UPGRADE-AVAILABLE and it's a NEW finding (not already recommended in a prior review): write a recommendation memo to ~/.claude/yolo-queue/pending-approval/ named `local-model-upgrade-<date>.md` — what model, why it's better, its MLX id, download size, and the exact (gated) command to pull it. Do NOT pull it.
+
+RULES: $0 where possible (WebSearch is free; openclaw is free/local). If you must note any paid call, show its cost. Never download (a pull is 35–120 GB — always Steve-gated). Never change the loaded model. Keep the whole run tight — this fires every hour, so if the leaderboard's top-fitting open model is unchanged from the last review, just write the heartbeat line and stop.
\ No newline at end of file
diff --git a/review.sh b/review.sh
new file mode 100755
index 0000000..27a59f3
--- /dev/null
+++ b/review.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+# Hourly local-model leaderboard watcher.
+# The WRAPPER (this deterministic script) gathers LOCAL cluster state.
+# The claude review then runs with NO shell/Bash — only WebSearch/WebFetch/Read/Write
+# (research + report + gated recommendation). READ-ONLY: never downloads, never changes the cluster.
+set -uo pipefail
+DIR="/Users/macstudio3/Projects/local-model-leaderboard-watch"
+TS="$(date '+%FT%T%z')"
+echo "[$TS] === review start ===" >> "$DIR/data/run.log"
+
+# 1) wrapper collects local state into a file for claude to READ (no Bash needed in the agent)
+{
+  echo "# Local cluster state @ $TS"
+  echo "## Loaded exo model (curl :52415/state):"
+  curl -s -m8 http://127.0.0.1:52415/state 2>/dev/null \
+    | python3 -c "import json,sys
+try:
+  d=json.load(sys.stdin); inst=d.get('instances',{})
+  print(list(inst.values())[0]['MlxRingInstance']['shardAssignments']['modelId'] if inst else 'none loaded')
+except Exception as e: print('state-unreachable')" 2>/dev/null
+  echo "## Downloaded models (~/.exo/models):"
+  ls -la ~/.exo/models 2>/dev/null | awk '{print $5, $NF}' | grep -viE '^\s*$|caches|\.$'
+  echo "## Hardware: 96GB M3 Ultra (primary) + 2x 32GB nodes; target = fits ~90GB single-node."
+} > "$DIR/data/local-state.txt" 2>&1
+
+# 2) claude reviews — NO Bash, NO Skill: research + read the state file + write report/recommendation
+timeout 600 claude -p "$(cat "$DIR/review-prompt.md")" \
+  --allowedTools "WebSearch,WebFetch,Read,Write" \
+  >> "$DIR/data/run.log" 2>&1
+rc=$?
+echo "[$TS] === review done (exit $rc) ===" >> "$DIR/data/run.log"
(oldest) · back to Local Model Leaderboard Watch · daily 4:45am + choose-best-downloaded-model-for-the-day (wra c96a2b6 →