← back to Exo

fix: enable psutil fallback for memory monitoring when macmon is missing (#1478)

e32b649d2f5691f41a64c4ba872dd2b7cfe36f03 · 2026-02-20 05:12:22 -0800 · Alex Cheema

## Summary
- On macOS, memory monitoring relied exclusively on `macmon` — the
psutil fallback was explicitly disabled (`memory_poll_rate = None`)
- When `macmon` is not installed (e.g., mac-mini-2 through mac-mini-4 in
our cluster), **no memory data was reported**, causing nodes to show 0GB
memory in the cluster state
- This blocked the scheduler from placing shards on those nodes since it
had no memory data to work with
- Fix: when `macmon` is not found on Darwin, fall back to psutil-based
memory polling (`memory_poll_rate = 1`)

## Root cause
`InfoGatherer` has two memory monitoring paths:
1. `macmon` (Darwin-only): provides memory + GPU/CPU/power stats
2. `psutil` (non-Darwin fallback): provides memory via
`MemoryUsage.from_psutil()`

Line 378 disabled psutil on Darwin: `memory_poll_rate = None if
IS_DARWIN else 1`
Line 389 only starts macmon if the binary exists: `if
shutil.which("macmon") is not None`

If macmon is missing on Darwin, **neither path runs** — zero memory
reported.

## Test plan
- [ ] Verify `uv run basedpyright` passes (0 errors confirmed)
- [ ] Verify `uv run ruff check` passes (confirmed)
- [ ] Verify `uv run pytest src/exo/utils/info_gatherer/` passes (2/2
confirmed)
- [ ] Deploy to cluster nodes without macmon and verify memory appears
in `/state`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>

Files touched

• M src/exo/utils/info_gatherer/info_gatherer.py

Diff

commit e32b649d2f5691f41a64c4ba872dd2b7cfe36f03
Author: Alex Cheema <41707476+AlexCheema@users.noreply.github.com>
Date:   Fri Feb 20 05:12:22 2026 -0800

    fix: enable psutil fallback for memory monitoring when macmon is missing (#1478)
    
    ## Summary
    - On macOS, memory monitoring relied exclusively on `macmon` — the
    psutil fallback was explicitly disabled (`memory_poll_rate = None`)
    - When `macmon` is not installed (e.g., mac-mini-2 through mac-mini-4 in
    our cluster), **no memory data was reported**, causing nodes to show 0GB
    memory in the cluster state
    - This blocked the scheduler from placing shards on those nodes since it
    had no memory data to work with
    - Fix: when `macmon` is not found on Darwin, fall back to psutil-based
    memory polling (`memory_poll_rate = 1`)
    
    ## Root cause
    `InfoGatherer` has two memory monitoring paths:
    1. `macmon` (Darwin-only): provides memory + GPU/CPU/power stats
    2. `psutil` (non-Darwin fallback): provides memory via
    `MemoryUsage.from_psutil()`
    
    Line 378 disabled psutil on Darwin: `memory_poll_rate = None if
    IS_DARWIN else 1`
    Line 389 only starts macmon if the binary exists: `if
    shutil.which("macmon") is not None`
    
    If macmon is missing on Darwin, **neither path runs** — zero memory
    reported.
    
    ## Test plan
    - [ ] Verify `uv run basedpyright` passes (0 errors confirmed)
    - [ ] Verify `uv run ruff check` passes (confirmed)
    - [ ] Verify `uv run pytest src/exo/utils/info_gatherer/` passes (2/2
    confirmed)
    - [ ] Deploy to cluster nodes without macmon and verify memory appears
    in `/state`
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
    
    ---------
    
    Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
    Co-authored-by: rltakashige <rl.takashige@gmail.com>
---
 src/exo/utils/info_gatherer/info_gatherer.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/src/exo/utils/info_gatherer/info_gatherer.py b/src/exo/utils/info_gatherer/info_gatherer.py
index 8f31369b..a43e54f1 100644
--- a/src/exo/utils/info_gatherer/info_gatherer.py
+++ b/src/exo/utils/info_gatherer/info_gatherer.py
@@ -388,6 +388,12 @@ class InfoGatherer:
             if IS_DARWIN:
                 if (macmon_path := shutil.which("macmon")) is not None:
                     tg.start_soon(self._monitor_macmon, macmon_path)
+                else:
+                    # macmon not installed — fall back to psutil for memory
+                    logger.warning(
+                        "macmon not found, falling back to psutil for memory monitoring"
+                    )
+                    self.memory_poll_rate = 1
                 tg.start_soon(self._monitor_system_profiler_thunderbolt_data)
                 tg.start_soon(self._monitor_thunderbolt_bridge_status)
                 tg.start_soon(self._monitor_rdma_ctl_status)

← bddad7e7 feat: show ETA on prefill progress bar (#1557)  ·  back to Exo  ·  bench: add download timing to bench output (#1566) d484b062 →