Placement filters for tensor parallel supports_tensor, tensor dimension and pipeline parallel deepseek v3.1 (#1058)

283c0e39e4f9fe5db91b7cc90c904fbdf32cc43b · 2025-12-31 00:33:40 +0000 · Alex Cheema

## Motivation

Certain placements are not valid. Added filters to exclude these placements. There were invalid placement previews being shown in the dashboard which would then fail when the user actually tries to launch an instance with that placement.


## Changes

Three filters added:

1. Certain models do not support tensor parallel at all. Checks `supports_tensor` on the model_meta.
2. For models that do support tensor parallelism, certain tensor parallel sizes are not valid. This check is actually not correct right now but it works fine for now. The actual correct check is more involved.
3. For unknown reasons, deepseek v3.1 (8-bit) does not work with tensor parallelism.

## Why It Works

`place_instance` now raises an `Exception` for invalid placements.

## Test Plan

### Manual Testing
Since `/instance/previews` enumerates all possible placements and runs `place_instance`, I checked the dashboard to see if invalid placements are still shown.

Files touched

M src/exo/master/placement.py

Diff

commit 283c0e39e4f9fe5db91b7cc90c904fbdf32cc43b
Author: Alex Cheema <41707476+AlexCheema@users.noreply.github.com>
Date:   Wed Dec 31 00:33:40 2025 +0000

     Placement filters for tensor parallel supports_tensor, tensor dimension and pipeline parallel deepseek v3.1 (#1058)
    
    ## Motivation
    
    Certain placements are not valid. Added filters to exclude these placements. There were invalid placement previews being shown in the dashboard which would then fail when the user actually tries to launch an instance with that placement.
    
    
    ## Changes
    
    Three filters added:
    
    1. Certain models do not support tensor parallel at all. Checks `supports_tensor` on the model_meta.
    2. For models that do support tensor parallelism, certain tensor parallel sizes are not valid. This check is actually not correct right now but it works fine for now. The actual correct check is more involved.
    3. For unknown reasons, deepseek v3.1 (8-bit) does not work with tensor parallelism.
    
    ## Why It Works
    
    `place_instance` now raises an `Exception` for invalid placements.
    
    ## Test Plan
    
    ### Manual Testing
    Since `/instance/previews` enumerates all possible placements and runs `place_instance`, I checked the dashboard to see if invalid placements are still shown.
---
 src/exo/master/placement.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/src/exo/master/placement.py b/src/exo/master/placement.py
index e7efa51e..04a78bd0 100644
--- a/src/exo/master/placement.py
+++ b/src/exo/master/placement.py
@@ -21,6 +21,7 @@ from exo.shared.types.commands import (
 )
 from exo.shared.types.events import Event, InstanceCreated, InstanceDeleted
 from exo.shared.types.memory import Memory
+from exo.shared.types.models import ModelId
 from exo.shared.types.topology import NodeInfo
 from exo.shared.types.worker.instances import (
     Instance,
@@ -29,6 +30,7 @@ from exo.shared.types.worker.instances import (
     MlxJacclInstance,
     MlxRingInstance,
 )
+from exo.shared.types.worker.shards import Sharding
 
 
 def random_ephemeral_port() -> int:
@@ -59,6 +61,23 @@ def place_instance(
     candidate_cycles = list(
         filter(lambda it: len(it) >= command.min_nodes, cycles + singleton_cycles)
     )
+    if command.sharding == Sharding.Tensor:
+        if not command.model_meta.supports_tensor:
+            raise ValueError(
+                f"Requested Tensor sharding but this model does not support tensor parallelism: {command.model_meta.model_id}"
+            )
+        # TODO: the condition here for tensor parallel is not correct, but it works good enough for now.
+        candidate_cycles = [
+            cycle
+            for cycle in candidate_cycles
+            if command.model_meta.hidden_size % len(cycle) == 0
+        ]
+    if command.sharding == Sharding.Pipeline and command.model_meta.model_id == ModelId(
+        "mlx-community/DeepSeek-V3.1-8bit"
+    ):
+        raise ValueError(
+            "Pipeline parallelism is not supported for DeepSeek V3.1 (8-bit)"
+        )
     cycles_with_sufficient_memory = filter_cycles_by_memory(
         candidate_cycles, command.model_meta.storage_size
     )

← 35be4c55 prioritise mlx jaccl coordinator ip (en0 -> en1 -> non-TB5 - · back to Exo · Emit download progress on start, and change downloads to be 1ec550df →