← back to Exo

fix: retry downloads on transient errors instead of breaking (#1398)

9f502793c1194bfcbbb652c61849d8e17467cd91 · 2026-02-06 05:51:54 -0600 · Hunter Bown

## Motivation

`download_file_with_retry()` has a `break` in the generic exception
handler that exits the retry loop after the first transient failure.
This means network timeouts, connection resets, and server errors all
cause an immediate download failure — the two remaining retry attempts
never run.

## Changes

**download_utils.py**: Replaced `break` with logging and exponential
backoff in the generic exception handler, matching the existing
rate-limit handler behavior.

Before:
```python
except Exception as e:
    on_connection_lost()
    if attempt == n_attempts - 1:
        raise e
    break  # exits loop immediately
```

After:
```python
except Exception as e:
    on_connection_lost()
    if attempt == n_attempts - 1:
        raise e
    logger.error(f"Download error on attempt {attempt + 1}/{n_attempts} ...")
    logger.error(traceback.format_exc())
    await asyncio.sleep(2.0**attempt)
```

## Why It Works

The `break` statement was bypassing the retry mechanism entirely.
Replacing it with the same log-and-backoff pattern used by the
`HuggingFaceRateLimitError` handler means all 3 attempts are actually
used before giving up. The exponential backoff (1s, 2s) gives transient
issues time to resolve between attempts.

## Test Plan

### Manual Testing
- Downloads that hit transient network errors now retry instead of
failing immediately

### Automated Testing
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — passes
- `uv run pytest src/exo/download/tests/ -v` — 11 tests pass

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>

Files touched

• M src/exo/download/download_utils.py

Diff

commit 9f502793c1194bfcbbb652c61849d8e17467cd91
Author: Hunter Bown <hmbown@gmail.com>
Date:   Fri Feb 6 05:51:54 2026 -0600

    fix: retry downloads on transient errors instead of breaking (#1398)
    
    ## Motivation
    
    `download_file_with_retry()` has a `break` in the generic exception
    handler that exits the retry loop after the first transient failure.
    This means network timeouts, connection resets, and server errors all
    cause an immediate download failure — the two remaining retry attempts
    never run.
    
    ## Changes
    
    **download_utils.py**: Replaced `break` with logging and exponential
    backoff in the generic exception handler, matching the existing
    rate-limit handler behavior.
    
    Before:
    ```python
    except Exception as e:
        on_connection_lost()
        if attempt == n_attempts - 1:
            raise e
        break  # exits loop immediately
    ```
    
    After:
    ```python
    except Exception as e:
        on_connection_lost()
        if attempt == n_attempts - 1:
            raise e
        logger.error(f"Download error on attempt {attempt + 1}/{n_attempts} ...")
        logger.error(traceback.format_exc())
        await asyncio.sleep(2.0**attempt)
    ```
    
    ## Why It Works
    
    The `break` statement was bypassing the retry mechanism entirely.
    Replacing it with the same log-and-backoff pattern used by the
    `HuggingFaceRateLimitError` handler means all 3 attempts are actually
    used before giving up. The exponential backoff (1s, 2s) gives transient
    issues time to resolve between attempts.
    
    ## Test Plan
    
    ### Manual Testing
    - Downloads that hit transient network errors now retry instead of
    failing immediately
    
    ### Automated Testing
    - `uv run basedpyright` — 0 errors
    - `uv run ruff check` — passes
    - `uv run pytest src/exo/download/tests/ -v` — 11 tests pass
    
    ---------
    
    Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
    Co-authored-by: rltakashige <rl.takashige@gmail.com>
---
 src/exo/download/download_utils.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/src/exo/download/download_utils.py b/src/exo/download/download_utils.py
index 618e4f38..5a8feeb9 100644
--- a/src/exo/download/download_utils.py
+++ b/src/exo/download/download_utils.py
@@ -378,10 +378,14 @@ async def download_file_with_retry(
             logger.error(traceback.format_exc())
             await asyncio.sleep(2.0**attempt)
         except Exception as e:
-            on_connection_lost()
             if attempt == n_attempts - 1:
+                on_connection_lost()
                 raise e
-            break
+            logger.error(
+                f"Download error on attempt {attempt + 1}/{n_attempts} for {model_id=} {revision=} {path=} {target_dir=}"
+            )
+            logger.error(traceback.format_exc())
+            await asyncio.sleep(2.0**attempt)
     raise Exception(
         f"Failed to download file {model_id=} {revision=} {path=} {target_dir=}"
     )

← c8371349 add scripts (#1401)  ·  back to Exo  ·  Fix offline no cache (#1402) f0107e96 →