TB-bridge runbook: fix required 4th router arg (""), reshuffle-proof .lan hosts, verified-state header + mac2 pubkey prereq + STP/firewall caveats

0e43d841b91bf1c7d534d2f10eaca062c0ab208a · 2026-08-25 12:03:32 -0700 · claude-run-10749
Files touched

M TB-BRIDGE-RUNBOOK.md
Diff

commit 0e43d841b91bf1c7d534d2f10eaca062c0ab208a
Author: claude-run-10749 <steve@designerwallcoverings.com>
Date:   Tue Aug 25 12:03:32 2026 -0700

    TB-bridge runbook: fix required 4th router arg (""), reshuffle-proof .lan hosts, verified-state header + mac2 pubkey prereq + STP/firewall caveats
---
 TB-BRIDGE-RUNBOOK.md | 88 ++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 71 insertions(+), 17 deletions(-)

diff --git a/TB-BRIDGE-RUNBOOK.md b/TB-BRIDGE-RUNBOOK.md
index 7d49e55..5eb3bb5 100644
--- a/TB-BRIDGE-RUNBOOK.md
+++ b/TB-BRIDGE-RUNBOOK.md
@@ -1,5 +1,22 @@
 # Put the exo cluster on the 40 Gb/s Thunderbolt fabric
 
+> **Status:** OPTIONAL perf upgrade (TK-10749). Cluster serves fine on LAN today —
+> this is pure throughput (gigabit LAN → 40 Gb/s TB), nothing is broken.
+> **Gated on Steve:** needs `sudo` on all 3 Studios (can't run non-interactively) +
+> one GUI step on this box + pubkey to mac2. An agent verifies + hands you the paste; you run it.
+
+## Verified current state (claude-run-10749, 2026-08-25)
+- **Cluster health:** 3/3 nodes HTTP 200 on `:52415`; local node actively serving an
+  MLX ring (`gpt-oss-20b`). No serving urgency.
+- **This box** (`Macs-Mac-Studio-2`, en0 `192.168.1.151`) → target `10.0.0.3`:
+  still on **6 individual `EXO Thunderbolt 1-6` services, NO bridge** — Step 1 (create bridge) still required.
+- **mac1** (`stevesmcstudio2.lan`, currently `192.168.1.133`) → target `10.0.0.1`:
+  already has `bridge0`; **passwordless SSH works** — its step is ready.
+- **mac2** (`stevestacstudio.lan`, currently `192.168.1.54`) → target `10.0.0.2`:
+  **passwordless SSH = Permission denied (publickey).** This is the one live blocker
+  (the ticket's "mac2 pubkey authorization"). Do its step physically at mac2, OR run
+  `ssh-copy-id steve@stevestacstudio.lan` first (Prereq below).
+
 **Why it's not on TB now:** the TB links are set to DHCP; a direct Mac-to-Mac
 cable has no DHCP server, so they fall to `169.254` link-local (unroutable) and
 exo's zero-config discovery rides the regular LAN (`en0` / 192.168.1.x) instead.
@@ -8,43 +25,80 @@ exo's zero-config discovery rides the regular LAN (`en0` / 192.168.1.x) instead.
 `10.0.0.0/24` subnet. Bridging each Mac's TB ports turns the ring into one L2
 broadcast domain, which is exactly what exo's zero-config discovery expects.
 
-**Per-node config discovered:**
-- mac1 (.133): already has a "Thunderbolt Bridge" (bridge0). Just needs a static IP.
-- mac2 (.54): LOCKED — unknown, but almost certainly also default "Thunderbolt Bridge". Do it once logged in.
-- macstudio3 (this box): on 6 individual "EXO Thunderbolt N" services, NO bridge — must CREATE a bridge first (one GUI step), then set the IP.
+**Addressing:**  mac1 = 10.0.0.1 · mac2 = 10.0.0.2 · this box = 10.0.0.3  (mask 255.255.255.0, no router)
 
-**Addressing:**  mac1 = 10.0.0.1 · mac2 = 10.0.0.2 · macstudio3 = 10.0.0.3  (mask 255.255.255.0, no router)
+> **Reshuffle-proof note:** the peers are on DHCP and their `192.168.1.x` IPs *reshuffle
+> on reboot* (this is why the health canary uses `.lan` hostnames, not hardcoded IPs).
+> All `ssh` commands below therefore target the stable **`.lan` hostnames**, never a
+> hardcoded IP — so a reshuffle can't point a `sudo` at the wrong Studio.
 
 ---
 
-## Step 1 — macstudio3 (this box): create the bridge (GUI, once)
+## Prereq (only if you'll drive mac2 over SSH instead of at the machine)
+```bash
+# One-time: authorize this box's key on mac2 (it will prompt for mac2's login password once).
+ssh-copy-id steve@stevestacstudio.lan
+# Verify: this must print OK with no password prompt.
+ssh -o BatchMode=yes steve@stevestacstudio.lan 'echo OK'
+```
+
+## Step 1 — this box: create the bridge (GUI, once)
 System Settings → Network → ⋯ (bottom) → **Manage Virtual Interfaces…** → **＋** →
 **New Thunderbolt Bridge** → add all the `EXO Thunderbolt` ports → Create.
 (This replaces the 6 individual services with one bridge, matching mac1.)
 
+Then confirm the exact service name the GUI created (usually `Thunderbolt Bridge`):
+```bash
+networksetup -listallnetworkservices | grep -i bridge   # note the exact name for Step 2
+```
+
 ## Step 2 — set the static IP on each node
-Run AT each machine's terminal (or `ssh -t steve@<ip> '<cmd>'` so `sudo` gets a TTY).
-`sudo` will prompt for that machine's password — that's why you run these, not me.
+Run AT each machine's terminal (or `ssh -t steve@<host> '<cmd>'` so `sudo` gets a TTY).
+`sudo` prompts for that machine's password — that's why you run these, not an agent.
+If Step 1 named the service something other than `Thunderbolt Bridge`, substitute it below.
+
+> **The 4th argument (router) is REQUIRED and MUST be `""` (empty).** Verified signature:
+> `networksetup -setmanual <service> <ip> <subnet> <router>`. On a direct-cable subnet
+> there is no gateway — an empty router leaves the interface with NO default route, so it
+> can't compete with `en0`'s default. Passing a real IP for router (even the node's own)
+> would install a `0.0.0.0/0` route via the TB bridge and could hijack LAN/internet routing.
 
 ```bash
-# macstudio3 (this box) — after Step 1 created the bridge:
-sudo networksetup -setmanual "Thunderbolt Bridge" 10.0.0.3 255.255.255.0
+# this box — after Step 1 created the bridge:
+sudo networksetup -setmanual "Thunderbolt Bridge" 10.0.0.3 255.255.255.0 ""
 
-# mac1 (.133) — already has the bridge:
-ssh -t steve@192.168.1.133 'sudo networksetup -setmanual "Thunderbolt Bridge" 10.0.0.1 255.255.255.0'
+# mac1 (already has the bridge, pubkey OK):
+ssh -t steve@stevesmcstudio2.lan 'sudo networksetup -setmanual "Thunderbolt Bridge" 10.0.0.1 255.255.255.0 ""'
 
-# mac2 (.54) — ONLY after it's logged in and reachable:
-ssh -t steve@192.168.1.54  'sudo networksetup -setmanual "Thunderbolt Bridge" 10.0.0.2 255.255.255.0'
+# mac2 — do the Prereq first, or run this line physically at mac2:
+ssh -t steve@stevestacstudio.lan 'sudo networksetup -setmanual "Thunderbolt Bridge" 10.0.0.2 255.255.255.0 ""'
 ```
 
 Safe: this touches only the Thunderbolt Bridge, never `en0` — your LAN/SSH stays up.
+The `ssh` targets are `.lan` hostnames that resolve to the **LAN** `192.168.1.x` addresses
+(not the TB subnet), so the SSH session won't drop when the bridge reconfigures.
 
-## Step 3 — verify (I do this part, no sudo needed)
-- `route get 10.0.0.1` on macstudio3 must resolve to a TB `en*` / `bridge0`, NOT `en0`.
+## Step 3 — verify (an agent can do this part, no sudo needed)
+- `route get 10.0.0.1` on this box must resolve to a TB `en*` / `bridge*`, NOT `en0`.
 - `ping 10.0.0.1` / `10.0.0.2` succeed over TB.
 - exo `:52415/state` shows the peers, and node-to-node traffic rides the bridge.
 - Then a sharded model run should show the interconnect on TB, not gigabit LAN.
 
+## Known caveats (observe during Step 3, not blockers)
+- **L2 loop / STP:** if the 3 Studios are cabled in a triangle (each node 2 TB links), the
+  per-host bridges form one loop. macOS STP will block one link (~30s to converge), leaving
+  all 3 mutually reachable in one L2 domain but with the blocked pair hairpinning through the
+  third node (a bandwidth caveat, not a failure). A ring/hub layout avoids it.
+- **macOS firewall:** if "Block all incoming connections" is on, it applies per-interface and
+  will block exo's discovery/listen ports on the new bridge. If exo can't see peers over TB
+  but `ping 10.0.0.x` works, that's the cause.
+
+## Rollback (if anything misbehaves)
+```bash
+# Drop the static config back to DHCP on any node — reverts cleanly, en0 was never touched:
+sudo networksetup -setdhcp "Thunderbolt Bridge"
+```
+
 ## After this: RDMA (separate, later)
-Per the topology notes — match macOS 26.6.1 across nodes + `rdma_ctl enable` in
+Per the topology notes — match macOS 26.6.1 across all nodes + `rdma_ctl enable` in
 Recovery. Only meaningful once the TB subnet above is live and exo shards across nodes.
← e81d84f auto-data-snapshot: 2026-08-25T11:28:19 (1 data files) — dat · back to Exo Cluster Watchdog · auto-data-snapshot: 2026-08-25T12:06:42 (1 data files) — dat d275fae →