bench: fix KeyError on DownloadCompleted total field

8d94eab6c681fb3e39e12b86ad8a929fdc2c711b · 2026-02-21 16:58:57 +0000 · Jake Hillion

The bench harness accessed the serialized DownloadCompleted field as
"totalBytes", but the Python field is `total: Memory` which serializes
to "total" (not snake_case, so the camelCase alias generator leaves it
unchanged). This caused a KeyError when --danger-delete-downloads
needed to free disk space by deleting existing models.

Changed the key from "totalBytes" to "total" to match the actual
serialized JSON structure.

Test plan:
- CI
- Reproduced KeyError on unfixed code by filling disk on a test node
  to 14GB free and running exo-bench with a 16GB model
  (Meta-Llama-3.1-8B-Instruct-bf16) with --danger-delete-downloads
- Verified fixed code successfully deletes smaller models to free
  space and completes the benchmark run

Files touched

M bench/harness.py

Diff

commit 8d94eab6c681fb3e39e12b86ad8a929fdc2c711b
Author: Jake Hillion <jake@hillion.co.uk>
Date:   Sat Feb 21 16:58:57 2026 +0000

    bench: fix KeyError on DownloadCompleted total field
    
    The bench harness accessed the serialized DownloadCompleted field as
    "totalBytes", but the Python field is `total: Memory` which serializes
    to "total" (not snake_case, so the camelCase alias generator leaves it
    unchanged). This caused a KeyError when --danger-delete-downloads
    needed to free disk space by deleting existing models.
    
    Changed the key from "totalBytes" to "total" to match the actual
    serialized JSON structure.
    
    Test plan:
    - CI
    - Reproduced KeyError on unfixed code by filling disk on a test node
      to 14GB free and running exo-bench with a 16GB model
      (Meta-Llama-3.1-8B-Instruct-bf16) with --danger-delete-downloads
    - Verified fixed code successfully deletes smaller models to free
      space and completes the benchmark run
---
 bench/harness.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bench/harness.py b/bench/harness.py
index 95f133f7..0fb01b68 100644
--- a/bench/harness.py
+++ b/bench/harness.py
@@ -371,7 +371,7 @@ def run_planning_phase(
                 unwrap_instance(p["DownloadCompleted"]["shardMetadata"])["modelCard"][
                     "modelId"
                 ],
-                p["DownloadCompleted"]["totalBytes"]["inBytes"],
+                p["DownloadCompleted"]["total"]["inBytes"],
             )
             for p in node_downloads
             if "DownloadCompleted" in p

← f370452d Better onboarding UX (#1533) · back to Exo · Address ring slowdown by turning on FAST SYNCH (#1594) fbb80e1c →