improve logging: add dates to verbose stderr, match file log level to verbosity (#1772)

e9fdd8d4afe0c6123d0dcfb0658f5dc240e739c8 · 2026-03-23 19:06:26 +0300 · vskiwi

## Summary

- **Add ISO date to verbose stderr format**: when running with `-v`, the
stderr timestamp changes from `HH:mm:ss.SSS` to `YYYY-MM-DD
HH:mm:ss.SSS`, matching the file log format. This makes it possible to
correlate entries across days when stderr is captured by launchd,
systemd, Docker, or file redirection.
- **File log respects verbosity**: `exo.log` now uses DEBUG level when
`-v` is passed, so the persistent log has the same detail as stderr.
Previously it was always INFO regardless of verbosity.
- **Startup banner with PID**: adds a visual separator and process ID to
the startup message, making it easy to identify session boundaries in
long-running logs (e.g. `grep "Starting EXO"`).

The non-verbose (default) stderr format is unchanged — end-user terminal
experience is not affected.

## Motivation

When exo runs as a service (launchd, systemd, Docker), stderr is
typically captured to a file. Without calendar dates in the timestamp,
it is impossible to tell which day a log entry belongs to. This caused
misidentification of log entries during a multi-day RDMA debugging
session on a 4-node cluster.

The file log (`exo.log`) already had dates and rotation, but only
captured INFO level — missing the DEBUG output needed for postmortem
analysis.

## Test plan

- [x] `basedpyright` — 0 errors, 0 warnings, 0 notes
- [x] `ruff check` — all checks passed
- [x] `pytest` — 249 passed, 1 skipped

Co-authored-by: user <user@m1.note>

Files touched

M src/exo/main.py
M src/exo/shared/logging.py

Diff

commit e9fdd8d4afe0c6123d0dcfb0658f5dc240e739c8
Author: vskiwi <141816715+vskiwi@users.noreply.github.com>
Date:   Mon Mar 23 19:06:26 2026 +0300

    improve logging: add dates to verbose stderr, match file log level to verbosity (#1772)
    
    ## Summary
    
    - **Add ISO date to verbose stderr format**: when running with `-v`, the
    stderr timestamp changes from `HH:mm:ss.SSS` to `YYYY-MM-DD
    HH:mm:ss.SSS`, matching the file log format. This makes it possible to
    correlate entries across days when stderr is captured by launchd,
    systemd, Docker, or file redirection.
    - **File log respects verbosity**: `exo.log` now uses DEBUG level when
    `-v` is passed, so the persistent log has the same detail as stderr.
    Previously it was always INFO regardless of verbosity.
    - **Startup banner with PID**: adds a visual separator and process ID to
    the startup message, making it easy to identify session boundaries in
    long-running logs (e.g. `grep "Starting EXO"`).
    
    The non-verbose (default) stderr format is unchanged — end-user terminal
    experience is not affected.
    
    ## Motivation
    
    When exo runs as a service (launchd, systemd, Docker), stderr is
    typically captured to a file. Without calendar dates in the timestamp,
    it is impossible to tell which day a log entry belongs to. This caused
    misidentification of log entries during a multi-day RDMA debugging
    session on a 4-node cluster.
    
    The file log (`exo.log`) already had dates and rotation, but only
    captured INFO level — missing the DEBUG output needed for postmortem
    analysis.
    
    ## Test plan
    
    - [x] `basedpyright` — 0 errors, 0 warnings, 0 notes
    - [x] `ruff check` — all checks passed
    - [x] `pytest` — 249 passed, 1 skipped
    
    Co-authored-by: user <user@m1.note>
---
 src/exo/main.py           | 4 +++-
 src/exo/shared/logging.py | 4 ++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/exo/main.py b/src/exo/main.py
index 3e08fa99..bf5ae688 100644
--- a/src/exo/main.py
+++ b/src/exo/main.py
@@ -264,7 +264,9 @@ def main():
     mp.set_start_method("spawn", force=True)
     # TODO: Refactor the current verbosity system
     logger_setup(EXO_LOG, args.verbosity)
-    logger.info("Starting EXO")
+    logger.info(f"{'=' * 40}")
+    logger.info(f"Starting EXO | pid={os.getpid()}")
+    logger.info(f"{'=' * 40}")
     logger.info(f"EXO_LIBP2P_NAMESPACE: {os.getenv('EXO_LIBP2P_NAMESPACE')}")
 
     if args.offline:
diff --git a/src/exo/shared/logging.py b/src/exo/shared/logging.py
index e08fe2c5..9e37ea5a 100644
--- a/src/exo/shared/logging.py
+++ b/src/exo/shared/logging.py
@@ -66,7 +66,7 @@ def logger_setup(log_file: Path | None, verbosity: int = 0):
     else:
         logger.add(
             sys.__stderr__,  # type: ignore
-            format="[ {time:HH:mm:ss.SSS} | <level>{level: <8}</level> | {name}:{function}:{line} ] <level>{message}</level>",
+            format="[ {time:YYYY-MM-DD HH:mm:ss.SSS} | <level>{level: <8}</level> | {name}:{function}:{line} ] <level>{message}</level>",
             level="DEBUG",
             colorize=True,
             enqueue=True,
@@ -76,7 +76,7 @@ def logger_setup(log_file: Path | None, verbosity: int = 0):
         logger.add(
             log_file,
             format="[ {time:YYYY-MM-DD HH:mm:ss.SSS} | {level: <8} | {name}:{function}:{line} ] {message}",
-            level="INFO",
+            level="DEBUG" if verbosity > 0 else "INFO",
             colorize=False,
             enqueue=True,
             rotation=lambda _, __: next(rotate_once),

← 07598a3a teeny refactor (#1753) · back to Exo · Prefer higher model download % for placement (#1767) e06e70a8 →