← back to Exo Keepalive
exo: --no-sync runner + notes; inference needs full Xcode to build pinned mlx fork (official wheels drift on model archs)
12c2b347ce11138de17ed570b0da9175d57ff85c · 2026-07-06 21:21:30 -0700 · steve
Files touched
Diff
commit 12c2b347ce11138de17ed570b0da9175d57ff85c
Author: steve <steve@designerwallcoverings.com>
Date: Mon Jul 6 21:21:30 2026 -0700
exo: --no-sync runner + notes; inference needs full Xcode to build pinned mlx fork (official wheels drift on model archs)
---
exo.log | 429 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
run-exo.sh | 2 +-
2 files changed, 430 insertions(+), 1 deletion(-)
diff --git a/exo.log b/exo.log
index 5b566c5..902bb90 100644
--- a/exo.log
+++ b/exo.log
@@ -131527,3 +131527,432 @@ hint: Build failures usually indicate a problem with the package or the build en
[ 09:10:40.7452PM | [1mINFO [0m] [1mWaiting for other campaign to finish[0m
[ 09:10:43.7466PM | [1mINFO [0m] [1mNode elected Master - maintaining self[0m
[ 09:10:43.7469PM | [1mINFO [0m] [1mUnpausing API[0m
+[ 09:13:11.8766PM | [1mINFO [0m] [1mRotated event log: /Users/macstudio3/.exo/event_log/master/events.bin -> /Users/macstudio3/.exo/event_log/master/events.2026-07-07_04-13-11_872623.bin.zst[0m
+[ 09:13:11.8851PM | [1mINFO [0m] [1mRotated event log: /Users/macstudio3/.exo/event_log/api/events.bin -> /Users/macstudio3/.exo/event_log/api/events.2026-07-07_04-13-11_880752.bin.zst[0m
+[ 09:13:11.8875PM | [1mINFO [0m] [1mStopping Worker[0m
+[ 09:13:12.8741PM | [1mINFO [0m] [1mEXO Shutdown complete[0m
+[2026-07-06T21:13:13-0700] === exo start (pid 22704) ===
+[ 09:13:13.6636PM | [1mINFO [0m] [1mpid = 22709[0m
+[ 09:13:13.6639PM | [1mINFO [0m] [1mEXO_ZENOH_NAMESPACE: None[0m
+[ 09:13:13.6676PM | [1mINFO [0m] [1mUsing ZID: f7079ec9dd5378e262ae17cad0a74ed7[0m
+[ 09:13:13.6683PM | [1mINFO [0m] [1mStarting required plugin "storage_manager"[0m
+[ 09:13:14.1760PM | [1mINFO [0m] [1mStarting node f7079ec9dd5378e262ae17cad0a74ed7[0m
+[ 09:13:14.2360PM | [1mINFO [0m] [1mStarting Election[0m
+[ 09:13:14.2363PM | [1mINFO [0m] [1mStarting DownloadCoordinator[0m
+
+╔═══════════════════════════════════════════════════════════════════════╗
+║ ║
+║ ███████╗██╗ ██╗ ██████╗ ║
+║ ██╔════╝╚██╗██╔╝██╔═══██╗ ║
+║ █████╗ ╚███╔╝ ██║ ██║ ║
+║ ██╔══╝ ██╔██╗ ██║ ██║ ║
+║ ███████╗██╔╝ ██╗╚██████╔╝ ║
+║ ╚══════╝╚═╝ ╚═╝ ╚═════╝ ║
+║ ║
+║ Distributed AI Inference Cluster ║
+║ ║
+╚═══════════════════════════════════════════════════════════════════════╝
+
+╔═══════════════════════════════════════════════════════════════════════╗
+║ ║
+║ Dashboard & API Ready ║
+║ ║
+║ http://localhost:52415 ║
+║ ║
+║ Click the URL above to open the dashboard in your browser ║
+║ ║
+╚═══════════════════════════════════════════════════════════════════════╝
+
+
+[ 09:13:14.2364PM | [1mINFO [0m] [1mStarting Worker[0m
+[ 09:13:14.2366PM | [1mINFO [0m] [1mStarting Master[0m
+[ 09:13:14.2366PM | [1mINFO [0m] [1mStarting API[0m
+[ 09:13:14.2377PM | [1mINFO [0m] [1mdiscovered: Some("f7079ec9dd5378e262ae17cad0a74ed7")[0m
+[ 09:13:14.2680PM | [1mINFO [0m] [1mNode elected Master - maintaining self[0m
+[ 09:13:14.2682PM | [1mINFO [0m] [1mUnpausing API[0m
+[ 09:13:14.2687PM | [1mINFO [0m] [1mRunning on http://0.0.0.0:52415 (CTRL + C to quit)[0m
+[ 09:13:14.2687PM | [1mINFO [0m] [1mRunning on http://0.0.0.0:52415 (CTRL + C to quit)[0m
+[ 09:13:14.2688PM | [1mINFO [0m] [1mSubscribed to global_events[0m
+[ 09:13:14.2690PM | [1mINFO [0m] [1mSubscribed to local_events[0m
+[ 09:13:14.2696PM | [1mINFO [0m] [1mSubscribed to commands[0m
+[ 09:13:14.2698PM | [1mINFO [0m] [1mSubscribed to election_messages[0m
+[ 09:13:14.2713PM | [1mINFO [0m] [1mSubscribed to connection_messages[0m
+[ 09:13:14.2714PM | [1mINFO [0m] [1mSubscribed to download_commands[0m
+[ 09:13:14.4677PM | [1mINFO [0m] [1mWaiting for other campaign to finish[0m
+[ 09:13:17.4693PM | [1mINFO [0m] [1mNode elected Master - maintaining self[0m
+[ 09:13:17.4697PM | [1mINFO [0m] [1mUnpausing API[0m
+[ 09:18:38.2421PM | [1mINFO [0m] [1mExecuting command: CreateInstance(command_id='17009bb6-3f20-457b-b126-238e392a3876' instance=MlxRingInstance(instance_id='82a4bac4-ca51-4840-b349-a912030d5370', shard_assignments=ShardAssignments(model_id='mlx-community/Qwen3-VL-4B-Instruct-4bit', runner_to_shard={'903bb6ec-7b40-4da5-b34c-45a9eaf62b06': PipelineShardMetadata(model_card=ModelCard(model_id='mlx-community/Qwen3-VL-4B-Instruct-4bit', storage_size=Memory.from_bytes(3340000000), n_layers=36, hidden_size=2560, supports_tensor=True, num_key_value_heads=None, tasks=[<ModelTask.TextGeneration: 'TextGeneration'>], components=None, family='qwen', quantization='4bit', base_model='Qwen3-VL 4B', capabilities=['text', 'thinking', 'vision'], backends=[<Backend.MlxMetal: 'MlxMetal'>, <Backend.MlxCuda: 'MlxCuda'>, <Backend.MlxCpu: 'MlxCpu'>], reasoning_dialect='none', context_length=262144, uses_cfg=False, trust_remote_code=True, is_custom=False, vision=VisionCardConfig(image_token_id=151655, model_type='qwen3_vl', weights_repo='mlx-community/Qwen3-VL-4B-Instruct-4bit', image_token=None, processor_repo=None), sampling_defaults=SamplingDefaults(temperature=0.7, top_p=0.8, top_k=20, min_p=None, repetition_penalty=1.0, presence_penalty=1.5, frequency_penalty=None, thinking=None, non_thinking=None)), device_rank=0, world_size=1, immediate_exception=False, should_timeout=None, start_layer=0, end_layer=36, n_layers=36)}, node_to_runner={'f7079ec9dd5378e262ae17cad0a74ed7': '903bb6ec-7b40-4da5-b34c-45a9eaf62b06'}), hosts_by_node={'f7079ec9dd5378e262ae17cad0a74ed7': [Host(ip='0.0.0.0', port=53957)]}, ephemeral_port=53957))[0m
+[ 09:18:38.3130PM | [1mINFO [0m] [1mWorker plan: CreateRunner[0m
+[ 09:18:38.4227PM | [1mINFO [0m] [1mWorker plan: DownloadModel[0m
+[ 09:18:38.8339PM | [1mINFO [0m] [1mFast synch flag: 1[0m
+[ 09:18:39.7484PM | [33m[1mWARNING [0m] [33m[1mRunner 903bb6ec-7b40-4da5-b34c-45a9eaf62b06 crashed with critical exception No module named 'mlx_lm.models.deepseek_v4'[0m
+[33m[1mTraceback (most recent call last):[0m
+
+ File "<string>", line 1, in <module>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/spawn.py", line 122, in spawn_main
+ exitcode = _main(fd, parent_sentinel)
+ │ │ └ 3
+ │ └ 101
+ └ <function _main at 0x108a4f600>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/spawn.py", line 135, in _main
+ return self._bootstrap(parent_sentinel)
+ │ │ └ 3
+ │ └ <function BaseProcess._bootstrap at 0x103fbe7a0>
+ └ <Process name='Process-1' parent=22709 started daemon>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
+ self.run()
+ │ └ <function BaseProcess.run at 0x103fbdd00>
+ └ <Process name='Process-1' parent=22709 started daemon>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/process.py", line 108, in run
+ self._target(*self._args, **self._kwargs)
+ │ │ │ │ │ └ {}
+ │ │ │ │ └ <Process name='Process-1' parent=22709 started daemon>
+ │ │ │ └ (<multiprocessing.resource_sharer.DupFd object at 0x108ac82f0>, <multiprocessing.resource_sharer.DupFd object at 0x10ce5c050>...
+ │ │ └ <Process name='Process-1' parent=22709 started daemon>
+ │ └ <function _run_with_captured_stdio at 0x10ce51da0>
+ └ <Process name='Process-1' parent=22709 started daemon>
+
+ File "[32m/Users/macstudio3/exo/src/exo/utils/[0m[32m[1masync_process.py[0m", line [33m267[0m, in [35m_run_with_captured_stdio[0m
+ [1mtarget[0m[1m([0m[35m[1m*[0m[1mtarget_args[0m[1m,[0m [35m[1m**[0m[1mtarget_kwargs[0m[1m)[0m
+ [36m│ │ └ [0m[36m[1m{}[0m
+ [36m│ └ [0m[36m[1m(BoundInstance(instance=MlxRingInstance(instance_id='82a4bac4-ca51-4840-b349-a912030d5370', shard_assignments=ShardAssignment...[0m
+ [36m└ [0m[36m[1m<function entrypoint at 0x10cd371a0>[0m
+
+> File "[32m/Users/macstudio3/exo/src/exo/worker/runner/[0m[32m[1mbootstrap.py[0m", line [33m79[0m, in [35mentrypoint[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mengines[0m[35m[1m.[0m[1mmlx[0m[35m[1m.[0m[1mbuilder[0m [35m[1mimport[0m [1mMlxBuilder[0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/engines/mlx/[0m[32m[1mbuilder.py[0m", line [33m17[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mrunner[0m[35m[1m.[0m[1mllm_inference[0m[35m[1m.[0m[1mbatch_generator[0m [35m[1mimport[0m [1m([0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/runner/llm_inference/[0m[32m[1mbatch_generator.py[0m", line [33m30[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mengines[0m[35m[1m.[0m[1mmlx[0m[35m[1m.[0m[1mcache[0m [35m[1mimport[0m [1mKVPrefixCache[0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/engines/mlx/[0m[32m[1mcache.py[0m", line [33m16[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mmlx_lm[0m[35m[1m.[0m[1mmodels[0m[35m[1m.[0m[1mdeepseek_v4[0m [35m[1mimport[0m [1m([0m
+
+[31m[1mModuleNotFoundError[0m:[1m No module named 'mlx_lm.models.deepseek_v4'[0m
+[ 09:18:39.7513PM | [1mINFO [0m] [1mbye from the runner[0m
+[ 09:18:39.7514PM | [1mINFO [0m] [1mChecking runner's status[0m
+[ 09:18:39.7515PM | [1mINFO [0m] [1mRunner was found to be alive, stopping process[0m
+[ 09:18:39.8904PM | [1mINFO [0m] [1mRunner exited with exit code 1[0m
+[ 09:18:39.8906PM | [31m[1mERROR [0m] [31m[1mRunner terminated with exitcode=1
+Runner error: ModuleNotFoundError: No module named 'mlx_lm.models.deepseek_v4'
+Traceback (most recent call last):
+ File "/Users/macstudio3/exo/src/exo/worker/runner/bootstrap.py", line 79, in entrypoint
+ from exo.worker.engines.mlx.builder import MlxBuilder
+ File "/Users/macstudio3/exo/src/exo/worker/engines/mlx/builder.py", line 17, in <module>
+ from exo.worker.runner.llm_inference.batch_generator import (
+ ...<2 lines>...
+ )
+ File "/Users/macstudio3/exo/src/exo/worker/runner/llm_inference/batch_generator.py", line 30, in <module>
+ from exo.worker.engines.mlx.cache import KVPrefixCache
+ File "/Users/macstudio3/exo/src/exo/worker/engines/mlx/cache.py", line 16, in <module>
+ from mlx_lm.models.deepseek_v4 import (
+ DeepseekV4Cache,
+ )
+ModuleNotFoundError: No module named 'mlx_lm.models.deepseek_v4'
+[0m
+[ 09:18:39.8912PM | [1mINFO [0m] [1mRunner supervisor shutting down[0m
+[ 09:18:39.8918PM | [1mINFO [0m] [1mRunner process successfully terminated: 1[0m
+[ 09:18:39.9384PM | [1mINFO [0m] [1mWorker plan: Shutdown[0m
+[ 09:18:39.9388PM | [1mINFO [0m] [1mStarting task Shutdown(task_id='00e48781-dcfc-4d67-87fa-13960c55f312' task_status=<TaskStatus.Pending: 'Pending'> instance_id='82a4bac4-ca51-4840-b349-a912030d5370' runner_id='903bb6ec-7b40-4da5-b34c-45a9eaf62b06')[0m
+[ 09:18:39.9392PM | [33m[1mWARNING [0m] [33m[1mTask Shutdown(task_id='00e48781-dcfc-4d67-87fa-13960c55f312' task_status=<TaskStatus.Pending: 'Pending'> instance_id='82a4bac4-ca51-4840-b349-a912030d5370' runner_id='903bb6ec-7b40-4da5-b34c-45a9eaf62b06') dropped, runner closed communication.[0m
+[ 09:18:40.0406PM | [1mINFO [0m] [1mWorker plan: CreateRunner[0m
+[ 09:18:40.4671PM | [1mINFO [0m] [1mFast synch flag: 1[0m
+[ 09:18:41.1514PM | [33m[1mWARNING [0m] [33m[1mRunner 903bb6ec-7b40-4da5-b34c-45a9eaf62b06 crashed with critical exception No module named 'mlx_lm.models.deepseek_v4'[0m
+[33m[1mTraceback (most recent call last):[0m
+
+ File "<string>", line 1, in <module>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/spawn.py", line 122, in spawn_main
+ exitcode = _main(fd, parent_sentinel)
+ │ │ └ 3
+ │ └ 131
+ └ <function _main at 0x1087af600>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/spawn.py", line 135, in _main
+ return self._bootstrap(parent_sentinel)
+ │ │ └ 3
+ │ └ <function BaseProcess._bootstrap at 0x103d2e7a0>
+ └ <Process name='Process-2' parent=22709 started daemon>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
+ self.run()
+ │ └ <function BaseProcess.run at 0x103d2dd00>
+ └ <Process name='Process-2' parent=22709 started daemon>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/process.py", line 108, in run
+ self._target(*self._args, **self._kwargs)
+ │ │ │ │ │ └ {}
+ │ │ │ │ └ <Process name='Process-2' parent=22709 started daemon>
+ │ │ │ └ (<multiprocessing.resource_sharer.DupFd object at 0x1088282f0>, <multiprocessing.resource_sharer.DupFd object at 0x10cc5c050>...
+ │ │ └ <Process name='Process-2' parent=22709 started daemon>
+ │ └ <function _run_with_captured_stdio at 0x10cc51da0>
+ └ <Process name='Process-2' parent=22709 started daemon>
+
+ File "[32m/Users/macstudio3/exo/src/exo/utils/[0m[32m[1masync_process.py[0m", line [33m267[0m, in [35m_run_with_captured_stdio[0m
+ [1mtarget[0m[1m([0m[35m[1m*[0m[1mtarget_args[0m[1m,[0m [35m[1m**[0m[1mtarget_kwargs[0m[1m)[0m
+ [36m│ │ └ [0m[36m[1m{}[0m
+ [36m│ └ [0m[36m[1m(BoundInstance(instance=MlxRingInstance(instance_id='82a4bac4-ca51-4840-b349-a912030d5370', shard_assignments=ShardAssignment...[0m
+ [36m└ [0m[36m[1m<function entrypoint at 0x10cb371a0>[0m
+
+> File "[32m/Users/macstudio3/exo/src/exo/worker/runner/[0m[32m[1mbootstrap.py[0m", line [33m79[0m, in [35mentrypoint[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mengines[0m[35m[1m.[0m[1mmlx[0m[35m[1m.[0m[1mbuilder[0m [35m[1mimport[0m [1mMlxBuilder[0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/engines/mlx/[0m[32m[1mbuilder.py[0m", line [33m17[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mrunner[0m[35m[1m.[0m[1mllm_inference[0m[35m[1m.[0m[1mbatch_generator[0m [35m[1mimport[0m [1m([0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/runner/llm_inference/[0m[32m[1mbatch_generator.py[0m", line [33m30[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mengines[0m[35m[1m.[0m[1mmlx[0m[35m[1m.[0m[1mcache[0m [35m[1mimport[0m [1mKVPrefixCache[0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/engines/mlx/[0m[32m[1mcache.py[0m", line [33m16[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mmlx_lm[0m[35m[1m.[0m[1mmodels[0m[35m[1m.[0m[1mdeepseek_v4[0m [35m[1mimport[0m [1m([0m
+
+[31m[1mModuleNotFoundError[0m:[1m No module named 'mlx_lm.models.deepseek_v4'[0m
+[ 09:18:41.1537PM | [1mINFO [0m] [1mbye from the runner[0m
+[ 09:18:41.1537PM | [1mINFO [0m] [1mChecking runner's status[0m
+[ 09:18:41.1539PM | [1mINFO [0m] [1mRunner was found to be alive, stopping process[0m
+[ 09:18:41.3015PM | [1mINFO [0m] [1mRunner exited with exit code 1[0m
+[ 09:18:41.3017PM | [31m[1mERROR [0m] [31m[1mRunner terminated with exitcode=1
+Runner error: ModuleNotFoundError: No module named 'mlx_lm.models.deepseek_v4'
+Traceback (most recent call last):
+ File "/Users/macstudio3/exo/src/exo/worker/runner/bootstrap.py", line 79, in entrypoint
+ from exo.worker.engines.mlx.builder import MlxBuilder
+ File "/Users/macstudio3/exo/src/exo/worker/engines/mlx/builder.py", line 17, in <module>
+ from exo.worker.runner.llm_inference.batch_generator import (
+ ...<2 lines>...
+ )
+ File "/Users/macstudio3/exo/src/exo/worker/runner/llm_inference/batch_generator.py", line 30, in <module>
+ from exo.worker.engines.mlx.cache import KVPrefixCache
+ File "/Users/macstudio3/exo/src/exo/worker/engines/mlx/cache.py", line 16, in <module>
+ from mlx_lm.models.deepseek_v4 import (
+ DeepseekV4Cache,
+ )
+ModuleNotFoundError: No module named 'mlx_lm.models.deepseek_v4'
+[0m
+[ 09:18:41.3023PM | [1mINFO [0m] [1mRunner supervisor shutting down[0m
+[ 09:18:41.3027PM | [1mINFO [0m] [1mRunner process successfully terminated: 1[0m
+[ 09:18:41.3554PM | [1mINFO [0m] [1mWorker plan: Shutdown[0m
+[ 09:18:41.3557PM | [1mINFO [0m] [1mStarting task Shutdown(task_id='e758504b-fd28-479d-950d-d9680f506b90' task_status=<TaskStatus.Pending: 'Pending'> instance_id='82a4bac4-ca51-4840-b349-a912030d5370' runner_id='903bb6ec-7b40-4da5-b34c-45a9eaf62b06')[0m
+[ 09:18:41.3561PM | [33m[1mWARNING [0m] [33m[1mTask Shutdown(task_id='e758504b-fd28-479d-950d-d9680f506b90' task_status=<TaskStatus.Pending: 'Pending'> instance_id='82a4bac4-ca51-4840-b349-a912030d5370' runner_id='903bb6ec-7b40-4da5-b34c-45a9eaf62b06') dropped, runner closed communication.[0m
+[ 09:18:42.0621PM | [1mINFO [0m] [1mWorker plan: CreateRunner[0m
+[ 09:18:42.5163PM | [1mINFO [0m] [1mFast synch flag: 1[0m
+[ 09:18:43.1776PM | [33m[1mWARNING [0m] [33m[1mRunner 903bb6ec-7b40-4da5-b34c-45a9eaf62b06 crashed with critical exception No module named 'mlx_lm.models.deepseek_v4'[0m
+[33m[1mTraceback (most recent call last):[0m
+
+ File "<string>", line 1, in <module>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/spawn.py", line 122, in spawn_main
+ exitcode = _main(fd, parent_sentinel)
+ │ │ └ 3
+ │ └ 155
+ └ <function _main at 0x106063600>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/spawn.py", line 135, in _main
+ return self._bootstrap(parent_sentinel)
+ │ │ └ 3
+ │ └ <function BaseProcess._bootstrap at 0x105f5a7a0>
+ └ <Process name='Process-3' parent=22709 started daemon>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
+ self.run()
+ │ └ <function BaseProcess.run at 0x105f59d00>
+ └ <Process name='Process-3' parent=22709 started daemon>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/process.py", line 108, in run
+ self._target(*self._args, **self._kwargs)
+ │ │ │ │ │ └ {}
+ │ │ │ │ └ <Process name='Process-3' parent=22709 started daemon>
+ │ │ │ └ (<multiprocessing.resource_sharer.DupFd object at 0x1060dc2f0>, <multiprocessing.resource_sharer.DupFd object at 0x10a4d0050>...
+ │ │ └ <Process name='Process-3' parent=22709 started daemon>
+ │ └ <function _run_with_captured_stdio at 0x10a4c5da0>
+ └ <Process name='Process-3' parent=22709 started daemon>
+
+ File "[32m/Users/macstudio3/exo/src/exo/utils/[0m[32m[1masync_process.py[0m", line [33m267[0m, in [35m_run_with_captured_stdio[0m
+ [1mtarget[0m[1m([0m[35m[1m*[0m[1mtarget_args[0m[1m,[0m [35m[1m**[0m[1mtarget_kwargs[0m[1m)[0m
+ [36m│ │ └ [0m[36m[1m{}[0m
+ [36m│ └ [0m[36m[1m(BoundInstance(instance=MlxRingInstance(instance_id='82a4bac4-ca51-4840-b349-a912030d5370', shard_assignments=ShardAssignment...[0m
+ [36m└ [0m[36m[1m<function entrypoint at 0x10a3ab1a0>[0m
+
+> File "[32m/Users/macstudio3/exo/src/exo/worker/runner/[0m[32m[1mbootstrap.py[0m", line [33m79[0m, in [35mentrypoint[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mengines[0m[35m[1m.[0m[1mmlx[0m[35m[1m.[0m[1mbuilder[0m [35m[1mimport[0m [1mMlxBuilder[0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/engines/mlx/[0m[32m[1mbuilder.py[0m", line [33m17[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mrunner[0m[35m[1m.[0m[1mllm_inference[0m[35m[1m.[0m[1mbatch_generator[0m [35m[1mimport[0m [1m([0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/runner/llm_inference/[0m[32m[1mbatch_generator.py[0m", line [33m30[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mengines[0m[35m[1m.[0m[1mmlx[0m[35m[1m.[0m[1mcache[0m [35m[1mimport[0m [1mKVPrefixCache[0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/engines/mlx/[0m[32m[1mcache.py[0m", line [33m16[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mmlx_lm[0m[35m[1m.[0m[1mmodels[0m[35m[1m.[0m[1mdeepseek_v4[0m [35m[1mimport[0m [1m([0m
+
+[31m[1mModuleNotFoundError[0m:[1m No module named 'mlx_lm.models.deepseek_v4'[0m
+[ 09:18:43.1797PM | [1mINFO [0m] [1mbye from the runner[0m
+[ 09:18:43.1798PM | [1mINFO [0m] [1mChecking runner's status[0m
+[ 09:18:43.1799PM | [1mINFO [0m] [1mRunner was found to be alive, stopping process[0m
+[ 09:18:43.3301PM | [1mINFO [0m] [1mRunner exited with exit code 1[0m
+[ 09:18:43.3303PM | [31m[1mERROR [0m] [31m[1mRunner terminated with exitcode=1
+Runner error: ModuleNotFoundError: No module named 'mlx_lm.models.deepseek_v4'
+Traceback (most recent call last):
+ File "/Users/macstudio3/exo/src/exo/worker/runner/bootstrap.py", line 79, in entrypoint
+ from exo.worker.engines.mlx.builder import MlxBuilder
+ File "/Users/macstudio3/exo/src/exo/worker/engines/mlx/builder.py", line 17, in <module>
+ from exo.worker.runner.llm_inference.batch_generator import (
+ ...<2 lines>...
+ )
+ File "/Users/macstudio3/exo/src/exo/worker/runner/llm_inference/batch_generator.py", line 30, in <module>
+ from exo.worker.engines.mlx.cache import KVPrefixCache
+ File "/Users/macstudio3/exo/src/exo/worker/engines/mlx/cache.py", line 16, in <module>
+ from mlx_lm.models.deepseek_v4 import (
+ DeepseekV4Cache,
+ )
+ModuleNotFoundError: No module named 'mlx_lm.models.deepseek_v4'
+[0m
+[ 09:18:43.3311PM | [1mINFO [0m] [1mRunner supervisor shutting down[0m
+[ 09:18:43.3316PM | [1mINFO [0m] [1mRunner process successfully terminated: 1[0m
+[ 09:18:43.3753PM | [1mINFO [0m] [1mWorker plan: Shutdown[0m
+[ 09:18:43.3756PM | [1mINFO [0m] [1mStarting task Shutdown(task_id='12ce86d1-3117-4ee2-8f67-81f168b96048' task_status=<TaskStatus.Pending: 'Pending'> instance_id='82a4bac4-ca51-4840-b349-a912030d5370' runner_id='903bb6ec-7b40-4da5-b34c-45a9eaf62b06')[0m
+[ 09:18:43.3760PM | [33m[1mWARNING [0m] [33m[1mTask Shutdown(task_id='12ce86d1-3117-4ee2-8f67-81f168b96048' task_status=<TaskStatus.Pending: 'Pending'> instance_id='82a4bac4-ca51-4840-b349-a912030d5370' runner_id='903bb6ec-7b40-4da5-b34c-45a9eaf62b06') dropped, runner closed communication.[0m
+[ 09:18:46.0991PM | [1mINFO [0m] [1mWorker plan: CreateRunner[0m
+[ 09:18:46.5319PM | [1mINFO [0m] [1mFast synch flag: 1[0m
+[ 09:18:47.2031PM | [33m[1mWARNING [0m] [33m[1mRunner 903bb6ec-7b40-4da5-b34c-45a9eaf62b06 crashed with critical exception No module named 'mlx_lm.models.deepseek_v4'[0m
+[33m[1mTraceback (most recent call last):[0m
+
+ File "<string>", line 1, in <module>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/spawn.py", line 122, in spawn_main
+ exitcode = _main(fd, parent_sentinel)
+ │ │ └ 3
+ │ └ 177
+ └ <function _main at 0x105f37600>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/spawn.py", line 135, in _main
+ return self._bootstrap(parent_sentinel)
+ │ │ └ 3
+ │ └ <function BaseProcess._bootstrap at 0x1015227a0>
+ └ <Process name='Process-4' parent=22709 started daemon>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
+ self.run()
+ │ └ <function BaseProcess.run at 0x101521d00>
+ └ <Process name='Process-4' parent=22709 started daemon>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/process.py", line 108, in run
+ self._target(*self._args, **self._kwargs)
+ │ │ │ │ │ └ {}
+ │ │ │ │ └ <Process name='Process-4' parent=22709 started daemon>
+ │ │ │ └ (<multiprocessing.resource_sharer.DupFd object at 0x105fb02f0>, <multiprocessing.resource_sharer.DupFd object at 0x10a384050>...
+ │ │ └ <Process name='Process-4' parent=22709 started daemon>
+ │ └ <function _run_with_captured_stdio at 0x10a379da0>
+ └ <Process name='Process-4' parent=22709 started daemon>
+
+ File "[32m/Users/macstudio3/exo/src/exo/utils/[0m[32m[1masync_process.py[0m", line [33m267[0m, in [35m_run_with_captured_stdio[0m
+ [1mtarget[0m[1m([0m[35m[1m*[0m[1mtarget_args[0m[1m,[0m [35m[1m**[0m[1mtarget_kwargs[0m[1m)[0m
+ [36m│ │ └ [0m[36m[1m{}[0m
+ [36m│ └ [0m[36m[1m(BoundInstance(instance=MlxRingInstance(instance_id='82a4bac4-ca51-4840-b349-a912030d5370', shard_assignments=ShardAssignment...[0m
+ [36m└ [0m[36m[1m<function entrypoint at 0x10a25f1a0>[0m
+
+> File "[32m/Users/macstudio3/exo/src/exo/worker/runner/[0m[32m[1mbootstrap.py[0m", line [33m79[0m, in [35mentrypoint[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mengines[0m[35m[1m.[0m[1mmlx[0m[35m[1m.[0m[1mbuilder[0m [35m[1mimport[0m [1mMlxBuilder[0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/engines/mlx/[0m[32m[1mbuilder.py[0m", line [33m17[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mrunner[0m[35m[1m.[0m[1mllm_inference[0m[35m[1m.[0m[1mbatch_generator[0m [35m[1mimport[0m [1m([0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/runner/llm_inference/[0m[32m[1mbatch_generator.py[0m", line [33m30[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mengines[0m[35m[1m.[0m[1mmlx[0m[35m[1m.[0m[1mcache[0m [35m[1mimport[0m [1mKVPrefixCache[0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/engines/mlx/[0m[32m[1mcache.py[0m", line [33m16[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mmlx_lm[0m[35m[1m.[0m[1mmodels[0m[35m[1m.[0m[1mdeepseek_v4[0m [35m[1mimport[0m [1m([0m
+
+[31m[1mModuleNotFoundError[0m:[1m No module named 'mlx_lm.models.deepseek_v4'[0m
+[ 09:18:47.2052PM | [1mINFO [0m] [1mbye from the runner[0m
+[ 09:18:47.2053PM | [1mINFO [0m] [1mChecking runner's status[0m
+[ 09:18:47.2055PM | [1mINFO [0m] [1mRunner was found to be alive, stopping process[0m
+[ 09:18:47.3486PM | [1mINFO [0m] [1mRunner exited with exit code 1[0m
+[ 09:18:47.3487PM | [31m[1mERROR [0m] [31m[1mRunner terminated with exitcode=1
+Runner error: ModuleNotFoundError: No module named 'mlx_lm.models.deepseek_v4'
+Traceback (most recent call last):
+ File "/Users/macstudio3/exo/src/exo/worker/runner/bootstrap.py", line 79, in entrypoint
+ from exo.worker.engines.mlx.builder import MlxBuilder
+ File "/Users/macstudio3/exo/src/exo/worker/engines/mlx/builder.py", line 17, in <module>
+ from exo.worker.runner.llm_inference.batch_generator import (
+ ...<2 lines>...
+ )
+ File "/Users/macstudio3/exo/src/exo/worker/runner/llm_inference/batch_generator.py", line 30, in <module>
+ from exo.worker.engines.mlx.cache import KVPrefixCache
+ File "/Users/macstudio3/exo/src/exo/worker/engines/mlx/cache.py", line 16, in <module>
+ from mlx_lm.models.deepseek_v4 import (
+ DeepseekV4Cache,
+ )
+ModuleNotFoundError: No module named 'mlx_lm.models.deepseek_v4'
+[0m
+[ 09:18:47.3494PM | [1mINFO [0m] [1mRunner supervisor shutting down[0m
+[ 09:18:47.3498PM | [1mINFO [0m] [1mRunner process successfully terminated: 1[0m
+[ 09:18:47.4107PM | [1mINFO [0m] [1mWorker plan: Shutdown[0m
+[ 09:18:47.4111PM | [1mINFO [0m] [1mStarting task Shutdown(task_id='30e86e9e-e9f6-4ef8-8a21-599337d1974d' task_status=<TaskStatus.Pending: 'Pending'> instance_id='82a4bac4-ca51-4840-b349-a912030d5370' runner_id='903bb6ec-7b40-4da5-b34c-45a9eaf62b06')[0m
+[ 09:18:47.4115PM | [33m[1mWARNING [0m] [33m[1mTask Shutdown(task_id='30e86e9e-e9f6-4ef8-8a21-599337d1974d' task_status=<TaskStatus.Pending: 'Pending'> instance_id='82a4bac4-ca51-4840-b349-a912030d5370' runner_id='903bb6ec-7b40-4da5-b34c-45a9eaf62b06') dropped, runner closed communication.[0m
+[ 09:18:54.1504PM | [1mINFO [0m] [1mWorker plan: CreateRunner[0m
+[ 09:18:54.6211PM | [1mINFO [0m] [1mFast synch flag: 1[0m
+[ 09:18:55.3238PM | [33m[1mWARNING [0m] [33m[1mRunner 903bb6ec-7b40-4da5-b34c-45a9eaf62b06 crashed with critical exception No module named 'mlx_lm.models.deepseek_v4'[0m
+[33m[1mTraceback (most recent call last):[0m
+
+ File "<string>", line 1, in <module>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/spawn.py", line 122, in spawn_main
+ exitcode = _main(fd, parent_sentinel)
+ │ │ └ 3
+ │ └ 107
+ └ <function _main at 0x10a2df600>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/spawn.py", line 135, in _main
+ return self._bootstrap(parent_sentinel)
+ │ │ └ 3
+ │ └ <function BaseProcess._bootstrap at 0x10a1d67a0>
+ └ <Process name='Process-5' parent=22709 started daemon>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
+ self.run()
+ │ └ <function BaseProcess.run at 0x10a1d5d00>
+ └ <Process name='Process-5' parent=22709 started daemon>
+ File "/Users/macstudio3/.local/share/uv/python/cpython-3.13.14-macos-aarch64-none/lib/python3.13/multiprocessing/process.py", line 108, in run
+ self._target(*self._args, **self._kwargs)
+ │ │ │ │ │ └ {}
+ │ │ │ │ └ <Process name='Process-5' parent=22709 started daemon>
+ │ │ │ └ (<multiprocessing.resource_sharer.DupFd object at 0x10a3582f0>, <multiprocessing.resource_sharer.DupFd object at 0x10e680050>...
+ │ │ └ <Process name='Process-5' parent=22709 started daemon>
+ │ └ <function _run_with_captured_stdio at 0x10e675da0>
+ └ <Process name='Process-5' parent=22709 started daemon>
+
+ File "[32m/Users/macstudio3/exo/src/exo/utils/[0m[32m[1masync_process.py[0m", line [33m267[0m, in [35m_run_with_captured_stdio[0m
+ [1mtarget[0m[1m([0m[35m[1m*[0m[1mtarget_args[0m[1m,[0m [35m[1m**[0m[1mtarget_kwargs[0m[1m)[0m
+ [36m│ │ └ [0m[36m[1m{}[0m
+ [36m│ └ [0m[36m[1m(BoundInstance(instance=MlxRingInstance(instance_id='82a4bac4-ca51-4840-b349-a912030d5370', shard_assignments=ShardAssignment...[0m
+ [36m└ [0m[36m[1m<function entrypoint at 0x10e55b1a0>[0m
+
+> File "[32m/Users/macstudio3/exo/src/exo/worker/runner/[0m[32m[1mbootstrap.py[0m", line [33m79[0m, in [35mentrypoint[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mengines[0m[35m[1m.[0m[1mmlx[0m[35m[1m.[0m[1mbuilder[0m [35m[1mimport[0m [1mMlxBuilder[0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/engines/mlx/[0m[32m[1mbuilder.py[0m", line [33m17[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mrunner[0m[35m[1m.[0m[1mllm_inference[0m[35m[1m.[0m[1mbatch_generator[0m [35m[1mimport[0m [1m([0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/runner/llm_inference/[0m[32m[1mbatch_generator.py[0m", line [33m30[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mexo[0m[35m[1m.[0m[1mworker[0m[35m[1m.[0m[1mengines[0m[35m[1m.[0m[1mmlx[0m[35m[1m.[0m[1mcache[0m [35m[1mimport[0m [1mKVPrefixCache[0m
+
+ File "[32m/Users/macstudio3/exo/src/exo/worker/engines/mlx/[0m[32m[1mcache.py[0m", line [33m16[0m, in [35m<module>[0m
+ [35m[1mfrom[0m [1mmlx_lm[0m[35m[1m.[0m[1mmodels[0m[35m[1m.[0m[1mdeepseek_v4[0m [35m[1mimport[0m [1m([0m
+
+[31m[1mModuleNotFoundError[0m:[1m No module named 'mlx_lm.models.deepseek_v4'[0m
+[ 09:18:55.3259PM | [1mINFO [0m] [1mbye from the runner[0m
+[ 09:18:55.3260PM | [1mINFO [0m] [1mChecking runner's status[0m
+[ 09:18:55.3262PM | [1mINFO [0m] [1mRunner was found to be alive, stopping process[0m
+[ 09:18:55.4787PM | [1mINFO [0m] [1mRunner exited with exit code 1[0m
+[ 09:18:55.4789PM | [31m[1mERROR [0m] [31m[1mRunner terminated with exitcode=1
+Runner error: ModuleNotFoundError: No module named 'mlx_lm.models.deepseek_v4'
+Traceback (most recent call last):
+ File "/Users/macstudio3/exo/src/exo/worker/runner/bootstrap.py", line 79, in entrypoint
+ from exo.worker.engines.mlx.builder import MlxBuilder
+ File "/Users/macstudio3/exo/src/exo/worker/engines/mlx/builder.py", line 17, in <module>
+ from exo.worker.runner.llm_inference.batch_generator import (
+ ...<2 lines>...
+ )
+ File "/Users/macstudio3/exo/src/exo/worker/runner/llm_inference/batch_generator.py", line 30, in <module>
+ from exo.worker.engines.mlx.cache import KVPrefixCache
+ File "/Users/macstudio3/exo/src/exo/worker/engines/mlx/cache.py", line 16, in <module>
+ from mlx_lm.models.deepseek_v4 import (
+ DeepseekV4Cache,
+ )
+ModuleNotFoundError: No module named 'mlx_lm.models.deepseek_v4'
+[0m
+[ 09:18:55.4795PM | [1mINFO [0m] [1mRunner supervisor shutting down[0m
+[ 09:18:55.4800PM | [1mINFO [0m] [1mRunner process successfully terminated: 1[0m
+[ 09:18:55.5680PM | [1mINFO [0m] [1mWorker plan: Shutdown[0m
+[ 09:18:55.5689PM | [1mINFO [0m] [1mStarting task Shutdown(task_id='71f03e7c-a4b3-4e76-b334-63ef0a2170b8' task_status=<TaskStatus.Pending: 'Pending'> instance_id='82a4bac4-ca51-4840-b349-a912030d5370' runner_id='903bb6ec-7b40-4da5-b34c-45a9eaf62b06')[0m
+[ 09:18:55.5694PM | [33m[1mWARNING [0m] [33m[1mTask Shutdown(task_id='71f03e7c-a4b3-4e76-b334-63ef0a2170b8' task_status=<TaskStatus.Pending: 'Pending'> instance_id='82a4bac4-ca51-4840-b349-a912030d5370' runner_id='903bb6ec-7b40-4da5-b34c-45a9eaf62b06') dropped, runner closed communication.[0m
+[ 09:19:04.2387PM | [33m[1mWARNING [0m] [33m[1mInstance 82a4bac4-ca51-4840-b349-a912030d5370 exceeded 5 retries, requesting deletion[0m
+[ 09:19:04.2401PM | [1mINFO [0m] [1mExecuting command: DeleteInstance(command_id='631bb142-ccfe-45ef-9a93-62afc936801c' instance_id='82a4bac4-ca51-4840-b349-a912030d5370')[0m
+[ 09:19:04.2416PM | [1mINFO [0m] [1mCancelling download for mlx-community/Qwen3-VL-4B-Instruct-4bit[0m
diff --git a/run-exo.sh b/run-exo.sh
index 2641e72..1a08273 100755
--- a/run-exo.sh
+++ b/run-exo.sh
@@ -9,4 +9,4 @@ cd "$HOME/exo" || exit 1
LOG="$HOME/Projects/exo-keepalive/exo.log"
echo "[$(date +%FT%T%z)] === exo start (pid $$) ===" >> "$LOG"
# exec so launchd tracks the real exo process (respawns on crash)
-exec /Users/macstudio3/.local/bin/uv run exo >> "$LOG" 2>&1
+exec /Users/macstudio3/.local/bin/uv run --no-sync exo >> "$LOG" 2>&1
← e985522 exo: add /usr/sbin to PATH (networksetup); revert --extra ml
·
back to Exo Keepalive
·
exo: restore clean locked daemon (self-syncing runner); offi e842362 →