`TB-BRIDGE-RUNBOOK.md`

107 lines
# Put the exo cluster on the 40 Gb/s Thunderbolt fabric

> **Status:** OPTIONAL perf upgrade (TK-10749). Cluster serves fine on LAN today —
> this is pure throughput (gigabit LAN → 40 Gb/s TB), nothing is broken.
> **Gated on Steve:** needs `sudo` on all 3 Studios (can't run non-interactively) +
> one GUI step on this box + pubkey to mac2. An agent verifies + hands you the paste; you run it.

## Verified current state (claude-run-10749, 2026-08-25)
- **Cluster health:** 3/3 nodes HTTP 200 on `:52415`; local node actively serving an
  MLX ring (`gpt-oss-20b`). No serving urgency.
- **This box** (`Macs-Mac-Studio-2`, en0 `192.168.1.151`) → target `10.0.0.3`:
  still on **6 individual `EXO Thunderbolt 1-6` services, NO bridge** — Step 1 (create bridge) still required.
- **mac1** (`stevesmcstudio2.lan`, currently `192.168.1.133`) → target `10.0.0.1`:
  **ALREADY DONE** — `Thunderbolt Bridge` is already manual `10.0.0.1/24`, `Router: (null)`
  (verified read-only over SSH). Its Step-2 line is a no-op; verify only. (This live node also
  confirms the empty-`""` router below is correct: a working TB bridge shows `Router: (null)`.)
- **mac2** (`stevestacstudio.lan`, currently `192.168.1.54`) → target `10.0.0.2`:
  **passwordless SSH = Permission denied (publickey).** This is the one live blocker
  (the ticket's "mac2 pubkey authorization"). Do its step physically at mac2, OR run
  `ssh-copy-id steve@stevestacstudio.lan` first (Prereq below).

**Why it's not on TB now:** the TB links are set to DHCP; a direct Mac-to-Mac
cable has no DHCP server, so they fall to `169.254` link-local (unroutable) and
exo's zero-config discovery rides the regular LAN (`en0` / 192.168.1.x) instead.

**Target:** a Thunderbolt Bridge on every node, each with a static IP on one
`10.0.0.0/24` subnet. Bridging each Mac's TB ports turns the ring into one L2
broadcast domain, which is exactly what exo's zero-config discovery expects.

**Addressing:**  mac1 = 10.0.0.1 · mac2 = 10.0.0.2 · this box = 10.0.0.3  (mask 255.255.255.0, no router)

> **Reshuffle-proof note:** the peers are on DHCP and their `192.168.1.x` IPs *reshuffle
> on reboot* (this is why the health canary uses `.lan` hostnames, not hardcoded IPs).
> All `ssh` commands below therefore target the stable **`.lan` hostnames**, never a
> hardcoded IP — so a reshuffle can't point a `sudo` at the wrong Studio.

---

## Prereq (only if you'll drive mac2 over SSH instead of at the machine)
```bash
# One-time: authorize this box's key on mac2 (it will prompt for mac2's login password once).
ssh-copy-id steve@stevestacstudio.lan
# Verify: this must print OK with no password prompt.
ssh -o BatchMode=yes steve@stevestacstudio.lan 'echo OK'
```

## Step 1 — this box: create the bridge (GUI, once)
System Settings → Network → ⋯ (bottom) → **Manage Virtual Interfaces…** → **＋** →
**New Thunderbolt Bridge** → add all the `EXO Thunderbolt` ports → Create.
(This replaces the 6 individual services with one bridge, matching mac1.)

Then confirm the exact service name the GUI created (usually `Thunderbolt Bridge`):
```bash
networksetup -listallnetworkservices | grep -i bridge   # note the exact name for Step 2
```

## Step 2 — set the static IP on each node
Run AT each machine's terminal (or `ssh -t steve@<host> '<cmd>'` so `sudo` gets a TTY).
`sudo` prompts for that machine's password — that's why you run these, not an agent.
If Step 1 named the service something other than `Thunderbolt Bridge`, substitute it below.

> **The 4th argument (router) is REQUIRED and MUST be `""` (empty).** Verified signature:
> `networksetup -setmanual <service> <ip> <subnet> <router>`. On a direct-cable subnet
> there is no gateway — an empty router leaves the interface with NO default route, so it
> can't compete with `en0`'s default. Passing a real IP for router (even the node's own)
> would install a `0.0.0.0/0` route via the TB bridge and could hijack LAN/internet routing.

```bash
# this box — after Step 1 created the bridge:
sudo networksetup -setmanual "Thunderbolt Bridge" 10.0.0.3 255.255.255.0 ""

# mac1 — ALREADY manual 10.0.0.1/24 (router null). SKIP; verify only:
ssh steve@stevesmcstudio2.lan 'networksetup -getinfo "Thunderbolt Bridge"'   # expect 10.0.0.1 / (null)

# mac2 — do the Prereq first, or run this line physically at mac2:
ssh -t steve@stevestacstudio.lan 'sudo networksetup -setmanual "Thunderbolt Bridge" 10.0.0.2 255.255.255.0 ""'
```

Safe: this touches only the Thunderbolt Bridge, never `en0` — your LAN/SSH stays up.
The `ssh` targets are `.lan` hostnames that resolve to the **LAN** `192.168.1.x` addresses
(not the TB subnet), so the SSH session won't drop when the bridge reconfigures.

## Step 3 — verify (an agent can do this part, no sudo needed)
- `route get 10.0.0.1` on this box must resolve to a TB `en*` / `bridge*`, NOT `en0`.
- `ping 10.0.0.1` / `10.0.0.2` succeed over TB.
- exo `:52415/state` shows the peers, and node-to-node traffic rides the bridge.
- Then a sharded model run should show the interconnect on TB, not gigabit LAN.

## Known caveats (observe during Step 3, not blockers)
- **L2 loop / STP:** if the 3 Studios are cabled in a triangle (each node 2 TB links), the
  per-host bridges form one loop. macOS STP will block one link (~30s to converge), leaving
  all 3 mutually reachable in one L2 domain but with the blocked pair hairpinning through the
  third node (a bandwidth caveat, not a failure). A ring/hub layout avoids it.
- **macOS firewall:** if "Block all incoming connections" is on, it applies per-interface and
  will block exo's discovery/listen ports on the new bridge. If exo can't see peers over TB
  but `ping 10.0.0.x` works, that's the cause.

## Rollback (if anything misbehaves)
```bash
# Drop the static config back to DHCP on any node — reverts cleanly, en0 was never touched:
sudo networksetup -setdhcp "Thunderbolt Bridge"
```

## After this: RDMA (separate, later)
Per the topology notes — match macOS 26.6.1 across all nodes + `rdma_ctl enable` in
Recovery. Only meaningful once the TB subnet above is live and exo shards across nodes.