exo: handle -c flag for multiprocessing helpers in frozen apps

9afc1043ef93f28322646621a1dffb99893b12e4 · 2025-12-23 15:11:49 +0000 · Jake Hillion

When Python's multiprocessing spawns child processes on macOS (using the
"spawn" method), it also spawns helper processes like the resource tracker
by executing:

    ./frozen_app -c "from multiprocessing.resource_tracker import main; main()"

A frozen PyInstaller app doesn't understand `-c` natively - it just runs
main(). This causes the resource tracker to fail silently.

This adds a minimal `-c` handler that intercepts the flag, extracts the
inline code, and exec()s it before main() runs. This is required for the
Process() spawn in runner_supervisor.py to work correctly in the DMG.

Note that the pyinstaller docs say `freeze_support` is supposed to make
this work, but it doesn't.

Test plan:

Hardware setup: 3x Mac Studio M3 Ultra connected all-to-all with TB5

- Built a DMG[0].
- Installed on the Macs.
- Started an instance. Got an error this time in ~/.exo/exo.log. The
  last DMG from main doesn't show anything when an instance starts, this
  now shows the errors.

[0] https://github.com/exo-explore/exo/actions/runs/20464409279/job/58804485197

Files touched

M src/exo/__main__.py

Diff

commit 9afc1043ef93f28322646621a1dffb99893b12e4
Author: Jake Hillion <jake@hillion.co.uk>
Date:   Tue Dec 23 15:11:49 2025 +0000

    exo: handle -c flag for multiprocessing helpers in frozen apps
    
    When Python's multiprocessing spawns child processes on macOS (using the
    "spawn" method), it also spawns helper processes like the resource tracker
    by executing:
    
        ./frozen_app -c "from multiprocessing.resource_tracker import main; main()"
    
    A frozen PyInstaller app doesn't understand `-c` natively - it just runs
    main(). This causes the resource tracker to fail silently.
    
    This adds a minimal `-c` handler that intercepts the flag, extracts the
    inline code, and exec()s it before main() runs. This is required for the
    Process() spawn in runner_supervisor.py to work correctly in the DMG.
    
    Note that the pyinstaller docs say `freeze_support` is supposed to make
    this work, but it doesn't.
    
    Test plan:
    
    Hardware setup: 3x Mac Studio M3 Ultra connected all-to-all with TB5
    
    - Built a DMG[0].
    - Installed on the Macs.
    - Started an instance. Got an error this time in ~/.exo/exo.log. The
      last DMG from main doesn't show anything when an instance starts, this
      now shows the errors.
    
    [0] https://github.com/exo-explore/exo/actions/runs/20464409279/job/58804485197
---
 src/exo/__main__.py | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/src/exo/__main__.py b/src/exo/__main__.py
index bd18145c..d2febaee 100644
--- a/src/exo/__main__.py
+++ b/src/exo/__main__.py
@@ -1,7 +1,39 @@
+from __future__ import annotations
+
+import sys
+from collections.abc import Sequence
 from multiprocessing import freeze_support
+from typing import Final
 
 from exo.main import main
 
+INLINE_CODE_FLAG: Final[str] = "-c"
+
+
+def _maybe_run_inline_code(argv: Sequence[str]) -> bool:
+    """
+    Reproduce the bare minimum of Python's `-c` flag so multiprocessing
+    helper processes (for example the resource tracker) can execute.
+    """
+
+    try:
+        flag_index = argv.index(INLINE_CODE_FLAG)
+    except ValueError:
+        return False
+
+    code_index = flag_index + 1
+    if code_index >= len(argv):
+        return False
+
+    inline_code = argv[code_index]
+    sys.argv = ["-c", *argv[code_index + 1 :]]
+    namespace: dict[str, object] = {"__name__": "__main__"}
+    exec(inline_code, namespace, namespace)
+    return True
+
+
 if __name__ == "__main__":
+    if _maybe_run_inline_code(sys.argv):
+        sys.exit(0)
     freeze_support()
     main()

← 70c423f5 feat: conform to XDG Base Directory Specification on Linux ( · back to Exo · mlx: update to 0.30.1 and align coordinator naming with MLX 1c1792f5 →