fix: `KVPrefixCache` Regression (#1668)

afab3095b04f2adecd1c2605eb9e9951aa6afcc2 · 2026-03-06 05:11:08 -0500 · Michael Harrigan

Files touched

M src/exo/worker/runner/llm_inference/runner.py

Diff

commit afab3095b04f2adecd1c2605eb9e9951aa6afcc2
Author: Michael Harrigan <9045677+michaelharrigan@users.noreply.github.com>
Date:   Fri Mar 6 05:11:08 2026 -0500

    fix: `KVPrefixCache` Regression (#1668)
---
 src/exo/worker/runner/llm_inference/runner.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/exo/worker/runner/llm_inference/runner.py b/src/exo/worker/runner/llm_inference/runner.py
index cb56322f..9ff744ac 100644
--- a/src/exo/worker/runner/llm_inference/runner.py
+++ b/src/exo/worker/runner/llm_inference/runner.py
@@ -216,6 +216,8 @@ class Runner:
                         tok.tool_parser,  # pyright: ignore[reportAny]
                     )
 
+                self.generator.kv_prefix_cache = KVPrefixCache(self.generator.group)
+
                 self.generator = self.generator.build()
 
                 self.send_task_status(task.task_id, TaskStatus.Complete)

← b9d40e8e Ciaran/re download bug (#1658) · back to Exo · feat(dashboard): improve model picker feedback and HuggingFa e8c3a873 →