Fix out of order event idx causing fatal crashes (#1894)

2cd66ae4cf82d81e5f166c3092582fe177aa2f45 · 2026-04-15 09:46:02 +0100 · rltakashige

## Motivation

<img width="828" height="373" alt="Screenshot 2026-04-14 at 22 56 52"
src="https://github.com/user-attachments/assets/f8f48c1d-68c5-4acc-a6de-9d180672da9d"
/>

if is_new_master=True, _elect_loop creates a new EventRouter before the
worker has receivers. Then, event router runs _run_ext_in and
buf.drain_indexed() will pick off events, even though
self.internal_outbound is not populated fully.

Finally, when the worker does try requesting events, the next event it
receives is not the first event, meaning the worker crashes.

## Changes

Start the event router after all the receivers are registered

## Why It Works

self.internal_outbound is populated before the loop begins.

## Test Plan

### Manual Testing
No more crashes observed in testing (it's actually quite easy to
reproduce the issue if you have one node with this fix but the other
node on main).

I'm convinced this is a fix, at least.

Files touched

M src/exo/main.py

Diff

commit 2cd66ae4cf82d81e5f166c3092582fe177aa2f45
Author: rltakashige <rl.takashige@gmail.com>
Date:   Wed Apr 15 09:46:02 2026 +0100

    Fix out of order event idx causing fatal crashes (#1894)
    
    ## Motivation
    
    <img width="828" height="373" alt="Screenshot 2026-04-14 at 22 56 52"
    src="https://github.com/user-attachments/assets/f8f48c1d-68c5-4acc-a6de-9d180672da9d"
    />
    
    if is_new_master=True, _elect_loop creates a new EventRouter before the
    worker has receivers. Then, event router runs _run_ext_in and
    buf.drain_indexed() will pick off events, even though
    self.internal_outbound is not populated fully.
    
    Finally, when the worker does try requesting events, the next event it
    receives is not the first event, meaning the worker crashes.
    
    ## Changes
    
    Start the event router after all the receivers are registered
    
    ## Why It Works
    
    self.internal_outbound is populated before the loop begins.
    
    ## Test Plan
    
    ### Manual Testing
    No more crashes observed in testing (it's actually quite easy to
    reproduce the issue if you have one node with this fix but the other
    node on main).
    
    I'm convinced this is a fix, at least.
---
 src/exo/main.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/exo/main.py b/src/exo/main.py
index 49e21895..903cca6f 100644
--- a/src/exo/main.py
+++ b/src/exo/main.py
@@ -192,7 +192,6 @@ class Node:
                         self.router.receiver(topics.GLOBAL_EVENTS),
                         self.router.sender(topics.LOCAL_EVENTS),
                     )
-                    self._tg.start_soon(self.event_router.run)
 
                 if (
                     result.session_id.master_node_id == self.node_id
@@ -258,6 +257,7 @@ class Node:
                         self._tg.start_soon(self.worker.run)
                     if self.api:
                         self.api.reset(result.won_clock, self.event_router.receiver())
+                    self._tg.start_soon(self.event_router.run)
                 else:
                     if self.api:
                         self.api.unpause(result.won_clock)

← 2ecefa0c Fix Qwen3-VL and autodetect vision config (#1893) · back to Exo · Drain tokens silently skipped in thinking parsing (#1898) 8cdc8338 →