fix: route by in-flight tasks only — completed tasks were skewing load balance (#1989)

5d10188d3abe0c4cc5bb4365eddf7dd819f0c269 · 2026-04-27 11:03:12 -0500 · Adam Durham

The load balancer counted ALL tasks (Complete, Cancelled, TimedOut,
Failed) instead of only Pending/Running ones. With 138 accumulated tasks
and only 7 active, routing decisions were based on historical
distribution, causing one node to appear permanently 'busier' and
starving the other of work.

Co-authored-by: Adam Durham <adam@example.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Files touched

M src/exo/master/main.py

Diff

commit 5d10188d3abe0c4cc5bb4365eddf7dd819f0c269
Author: Adam Durham <amdnative@gmail.com>
Date:   Mon Apr 27 11:03:12 2026 -0500

    fix: route by in-flight tasks only — completed tasks were skewing load balance (#1989)
    
    The load balancer counted ALL tasks (Complete, Cancelled, TimedOut,
    Failed) instead of only Pending/Running ones. With 138 accumulated tasks
    and only 7 active, routing decisions were based on historical
    distribution, causing one node to appear permanently 'busier' and
    starving the other of work.
    
    Co-authored-by: Adam Durham <adam@example.com>
    Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 src/exo/master/main.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/exo/master/main.py b/src/exo/master/main.py
index a3e465fe..35c49390 100644
--- a/src/exo/master/main.py
+++ b/src/exo/master/main.py
@@ -133,10 +133,12 @@ class Master:
                                     instance.shard_assignments.model_id
                                     == command.task_params.model
                                 ):
+                                    in_flight = {TaskStatus.Pending, TaskStatus.Running}
                                     task_count = sum(
                                         1
                                         for task in self.state.tasks.values()
                                         if task.instance_id == instance.instance_id
+                                        and task.task_status in in_flight
                                     )
                                     instance_task_counts[instance.instance_id] = (
                                         task_count
@@ -175,10 +177,12 @@ class Master:
                                     instance.shard_assignments.model_id
                                     == command.task_params.model
                                 ):
+                                    in_flight = {TaskStatus.Pending, TaskStatus.Running}
                                     task_count = sum(
                                         1
                                         for task in self.state.tasks.values()
                                         if task.instance_id == instance.instance_id
+                                        and task.task_status in in_flight
                                     )
                                     instance_task_counts[instance.instance_id] = (
                                         task_count
@@ -229,10 +233,12 @@ class Master:
                                     instance.shard_assignments.model_id
                                     == command.task_params.model
                                 ):
+                                    in_flight = {TaskStatus.Pending, TaskStatus.Running}
                                     task_count = sum(
                                         1
                                         for task in self.state.tasks.values()
                                         if task.instance_id == instance.instance_id
+                                        and task.task_status in in_flight
                                     )
                                     instance_task_counts[instance.instance_id] = (
                                         task_count

← f2a0db4e Extend bench/eval tooling (#1905) · back to Exo · MLX P/D (#1993) f0d1371d →