[object Object]

← back to Exo

Map GLM 4.7 stop tokens to GLM 4 IDs (#2061)

08ffa5f6378c6ae887c5964e4c3ce1701f0bd79d · 2026-05-10 10:02:22 -0700 · team-wcv

## Motivation

GLM 4.7 reuses the GLM 4 chat-template tokenizer, but the model card and
EOS-detection path didn't have an explicit mapping for it, so
OpenAI-compatible clients didn't see a clean stop and the runner emitted
follow-on role turns (e.g. \`<|user|>\` continuations after
\`<|assistant|>\`'s output).

## Changes

\`src/exo/worker/engines/mlx/utils_mlx.py\` — add the GLM 4 stop-token
IDs as the EOS set when the loaded model's tokenizer matches GLM 4 / 4.7
chat templates.

## Why It Works

The GLM 4 tokenizer's \`<|user|>\`, \`<|observation|>\`, and
\`<|endoftext|>\` IDs are stable across the GLM 4 / 4.7 line; treating
any of them as EOS lets the runner stop at the assistant turn boundary
the same way it stops at \`</s>\` for Llama-style models. No
prompt-template changes — only the stop set widens.

## Test Plan

### Automated Testing

New unit test
\`src/exo/worker/tests/unittests/test_mlx/test_eos_token_ids.py\`
covering: GLM 4 / 4.7 path returns the expected stop ID set; non-GLM
path returns the standard EOS only.

\`\`\`
src/exo/worker/tests/unittests/test_mlx/test_eos_token_ids.py ..
=== 2 passed in 0.01s ===
\`\`\`

\`uv run basedpyright\` and \`uv run ruff check\` both clean.

### Manual Testing

Hardware: 4-node Apple Silicon cluster, M5 Max master.

- Loaded \`mlx-community/GLM-4.7-Air-mlx-4bit\`, ran chat completion via
\`/v1/chat/completions\`. Before this fix the assistant turn ran on into
a synthetic \`<|user|>\` continuation; after the fix the response stops
cleanly at the assistant boundary.

---------

Co-authored-by: jw-wcv <101585096+jw-wcv@users.noreply.github.com>
Co-authored-by: Evan Quiney <evanev7@gmail.com>

Files touched

Diff

commit 08ffa5f6378c6ae887c5964e4c3ce1701f0bd79d
Author: team-wcv <team@whaleconnected.io>
Date:   Sun May 10 10:02:22 2026 -0700

    Map GLM 4.7 stop tokens to GLM 4 IDs (#2061)
    
    ## Motivation
    
    GLM 4.7 reuses the GLM 4 chat-template tokenizer, but the model card and
    EOS-detection path didn't have an explicit mapping for it, so
    OpenAI-compatible clients didn't see a clean stop and the runner emitted
    follow-on role turns (e.g. \`<|user|>\` continuations after
    \`<|assistant|>\`'s output).
    
    ## Changes
    
    \`src/exo/worker/engines/mlx/utils_mlx.py\` — add the GLM 4 stop-token
    IDs as the EOS set when the loaded model's tokenizer matches GLM 4 / 4.7
    chat templates.
    
    ## Why It Works
    
    The GLM 4 tokenizer's \`<|user|>\`, \`<|observation|>\`, and
    \`<|endoftext|>\` IDs are stable across the GLM 4 / 4.7 line; treating
    any of them as EOS lets the runner stop at the assistant turn boundary
    the same way it stops at \`</s>\` for Llama-style models. No
    prompt-template changes — only the stop set widens.
    
    ## Test Plan
    
    ### Automated Testing
    
    New unit test
    \`src/exo/worker/tests/unittests/test_mlx/test_eos_token_ids.py\`
    covering: GLM 4 / 4.7 path returns the expected stop ID set; non-GLM
    path returns the standard EOS only.
    
    \`\`\`
    src/exo/worker/tests/unittests/test_mlx/test_eos_token_ids.py ..
    === 2 passed in 0.01s ===
    \`\`\`
    
    \`uv run basedpyright\` and \`uv run ruff check\` both clean.
    
    ### Manual Testing
    
    Hardware: 4-node Apple Silicon cluster, M5 Max master.
    
    - Loaded \`mlx-community/GLM-4.7-Air-mlx-4bit\`, ran chat completion via
    \`/v1/chat/completions\`. Before this fix the assistant turn ran on into
    a synthetic \`<|user|>\` continuation; after the fix the response stops
    cleanly at the assistant boundary.
    
    ---------
    
    Co-authored-by: jw-wcv <101585096+jw-wcv@users.noreply.github.com>
    Co-authored-by: Evan Quiney <evanev7@gmail.com>
---
 src/exo/worker/engines/mlx/utils_mlx.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/src/exo/worker/engines/mlx/utils_mlx.py b/src/exo/worker/engines/mlx/utils_mlx.py
index 1dddad2a..730abf64 100644
--- a/src/exo/worker/engines/mlx/utils_mlx.py
+++ b/src/exo/worker/engines/mlx/utils_mlx.py
@@ -311,12 +311,11 @@ def get_eos_token_ids_for_model(model_id: ModelId) -> list[int] | None:
     model_id_lower = model_id.lower()
     if "kimi-k2" in model_id_lower:
         return [163586]
-    elif "glm-5" in model_id_lower or "glm-4.7" in model_id_lower:
-        # For GLM-5 and GLM-4.7
+    elif "glm-5" in model_id_lower:
         # 154820: <|endoftext|>, 154827: <|user|>, 154829: <|observation|>
         return [154820, 154827, 154829]
     elif "glm" in model_id_lower:
-        # For GLM-4.5 and older
+        # For GLM-4.7 and older
         return [151336, 151329, 151338]
     elif "gpt-oss" in model_id_lower:
         return [200002, 200012]

← 45df74ba Andrei/mp capture stdio (#2056)  ·  back to Exo  ·  bump rust versions (#2081) b76bc301 →