← back to Exo
fix(bug): no longer repeated _trigger_notify_user_to_download_model (#2114)
a8602ea6d5c5a7636e66008f6da86f437b6e9692 · 2026-05-26 14:42:39 +0100 · Andrei Cravtov
## Motivation
Partially fixes [this](https://github.com/exo-explore/exo/issues/2098)
issue. Removed erroneous logic for telling user to download when they
already downloaded.
Could not figure out about the "spontaneous crashes" in that issue,
author should consolidate more logs and open a new issue dedicated to
that. I believe
[this](https://github.com/exo-explore/exo/commit/74e9fe15e62fe189dc7e019db86e75c83eca2721)
commit solved some EventRouter-related crashes, which was mentioned in
[this](https://github.com/exo-explore/exo/issues/2098) issue, so it may
have already been solved. If not, should be re-submitted as a new issue.
## Changes
- Consolidated _resolve_and_validate_text_model and
_validate_image_model into one function: _validate_model_has_instance;
- + They already had virtually identical logic, it being different seems
to be an artifact of history
- + Added logic to ensure that _trigger_notify_user_to_download_model is
only called when no such model is downloaded, not just if there is no
instance of it
- Added a new `/instance/await` SSE streaming endpoint to wait for when
a model has an instance available. Complements instance-placement API,
so we can wait till that is done without client-side polling.
- Updated docs and a /tmp script to reflect some of the changes
- Updated dashboard `getModelForRequest` to only return model ID if an
instance exists for it, and updated bits to use `handleChatSend` instead
of `sendMessage` because that checks for if a model instance exists
first.
## Why It Works
The problem was that there was erroneous logging for model not
downloaded. I fixed that logic. The rest is extra.
Files touched
M README.mdM dashboard/src/lib/stores/app.svelte.tsM dashboard/src/routes/+page.svelteM docs/api.mdM src/exo/api/main.pyM src/exo/api/types/__init__.pyM src/exo/api/types/api.pyM src/exo/master/main.pyM tmp/test_trust_remote_code_attack.sh
Diff
commit a8602ea6d5c5a7636e66008f6da86f437b6e9692
Author: Andrei Cravtov <the.andrei.cravtov@gmail.com>
Date: Tue May 26 14:42:39 2026 +0100
fix(bug): no longer repeated _trigger_notify_user_to_download_model (#2114)
## Motivation
Partially fixes [this](https://github.com/exo-explore/exo/issues/2098)
issue. Removed erroneous logic for telling user to download when they
already downloaded.
Could not figure out about the "spontaneous crashes" in that issue,
author should consolidate more logs and open a new issue dedicated to
that. I believe
[this](https://github.com/exo-explore/exo/commit/74e9fe15e62fe189dc7e019db86e75c83eca2721)
commit solved some EventRouter-related crashes, which was mentioned in
[this](https://github.com/exo-explore/exo/issues/2098) issue, so it may
have already been solved. If not, should be re-submitted as a new issue.
## Changes
- Consolidated _resolve_and_validate_text_model and
_validate_image_model into one function: _validate_model_has_instance;
- + They already had virtually identical logic, it being different seems
to be an artifact of history
- + Added logic to ensure that _trigger_notify_user_to_download_model is
only called when no such model is downloaded, not just if there is no
instance of it
- Added a new `/instance/await` SSE streaming endpoint to wait for when
a model has an instance available. Complements instance-placement API,
so we can wait till that is done without client-side polling.
- Updated docs and a /tmp script to reflect some of the changes
- Updated dashboard `getModelForRequest` to only return model ID if an
instance exists for it, and updated bits to use `handleChatSend` instead
of `sendMessage` because that checks for if a model instance exists
first.
## Why It Works
The problem was that there was erroneous logging for model not
downloaded. I fixed that logic. The rest is extra.
---
README.md | 12 ++++
dashboard/src/lib/stores/app.svelte.ts | 16 +++--
dashboard/src/routes/+page.svelte | 26 ++++---
docs/api.md | 41 +++++++++--
src/exo/api/main.py | 120 ++++++++++++++++++++++-----------
src/exo/api/types/__init__.py | 2 +
src/exo/api/types/api.py | 10 +++
src/exo/master/main.py | 4 ++
tmp/test_trust_remote_code_attack.sh | 30 ++++++++-
9 files changed, 199 insertions(+), 62 deletions(-)
diff --git a/README.md b/README.md
index 9e009c6a..4449c52f 100644
--- a/README.md
+++ b/README.md
@@ -401,6 +401,18 @@ Sample response:
}
```
+This command is asynchronous. Before sending inference requests, wait until the
+API sees the new instance for this model:
+
+```bash
+curl -N "http://localhost:52415/instance/await?model_id=mlx-community/Llama-3.2-1B-Instruct-4bit"
+```
+
+The endpoint returns an SSE stream. A successful wait emits a message with
+`"type": "ready"` and the matching instance; a timeout emits `"type": "timeout"`.
+By default it waits indefinitely. Set `timeout_seconds` to a positive value to
+bound the wait.
+
---
**3. Send a chat completion**
diff --git a/dashboard/src/lib/stores/app.svelte.ts b/dashboard/src/lib/stores/app.svelte.ts
index 5b28b3f0..a717c3af 100644
--- a/dashboard/src/lib/stores/app.svelte.ts
+++ b/dashboard/src/lib/stores/app.svelte.ts
@@ -2253,10 +2253,9 @@ class AppStore {
* @returns The model ID to use, or null if none available
*/
private getModelForRequest(modelId?: string): string | null {
- if (modelId) return modelId;
- if (this.selectedChatModel) return this.selectedChatModel;
+ const requestedModelId = modelId || this.selectedChatModel;
- // Try to get model from first running instance
+ // Only models with a placed instance can receive requests; disk downloads alone are not enough.
for (const [, instanceWrapper] of Object.entries(this.instances)) {
if (instanceWrapper && typeof instanceWrapper === "object") {
const keys = Object.keys(instanceWrapper as Record<string, unknown>);
@@ -2264,8 +2263,15 @@ class AppStore {
const instance = (instanceWrapper as Record<string, unknown>)[
keys[0]
] as { shardAssignments?: { modelId?: string } };
- if (instance?.shardAssignments?.modelId) {
- return instance.shardAssignments.modelId;
+ const instanceModelId = instance?.shardAssignments?.modelId;
+
+ // ensure to only return requestedModelId that matches an instance
+ // or fall back to first instance
+ if (
+ instanceModelId &&
+ (!requestedModelId || requestedModelId === instanceModelId)
+ ) {
+ return instanceModelId;
}
}
}
diff --git a/dashboard/src/routes/+page.svelte b/dashboard/src/routes/+page.svelte
index f69cdd15..5db62079 100644
--- a/dashboard/src/routes/+page.svelte
+++ b/dashboard/src/routes/+page.svelte
@@ -1461,6 +1461,9 @@
addToast({ type: "info", message: `Launching model...` });
// Always auto-select the newly launched model so the user chats to what they just launched
setSelectedChatModel(modelId);
+ userForcedIdle = false;
+ pendingChatModelId = modelId;
+ chatLaunchState = "launching";
// Record the launch in recent models history
recordRecentLaunch(modelId);
@@ -2547,12 +2550,10 @@
];
// ── Seamless chat: launch models from chat view ──
- type ChatLaunchState =
- | "idle"
- | "launching"
- | "downloading"
- | "loading"
- | "ready";
+ type InFlightChatLaunchState = "launching" | "downloading" | "loading";
+ type ReadyLikeChatLaunchState = "idle" | "ready";
+ type ChatLaunchState = InFlightChatLaunchState | ReadyLikeChatLaunchState;
+
let chatLaunchState = $state<ChatLaunchState>("idle");
let pendingChatModelId = $state<string | null>(null);
let selectedChatCategory = $state<string | null>(null);
@@ -3129,6 +3130,15 @@
if (model) {
pendingAutoMessage = { content, files };
userForcedIdle = false;
+ // The selected model is already being placed or loaded; keep the queued
+ // message and let the existing launch state effects send it once ready.
+ if (
+ pendingChatModelId === model &&
+ chatLaunchState !== "idle" &&
+ chatLaunchState !== "ready"
+ ) {
+ return;
+ }
launchModelForChat(model, "picker", messages().length > 0);
return;
}
@@ -4603,7 +4613,7 @@
type="button"
onclick={() => {
completeOnboarding();
- sendMessage(chip, undefined, thinkingEnabled());
+ handleChatSend(chip);
}}
class="px-4 py-2 rounded-full border border-white/10 bg-white/5 text-sm text-white/60 hover:bg-white/10 hover:text-white/80 hover:border-white/20 transition-all duration-200 cursor-pointer"
>
@@ -6100,7 +6110,7 @@
onclick={() => {
chatLaunchState = "idle";
selectedChatCategory = null;
- sendMessage(prompt, undefined, thinkingEnabled());
+ handleChatSend(prompt);
}}
class="text-left px-3 py-2.5 text-xs text-exo-light-gray hover:text-white font-mono rounded-lg border border-exo-medium-gray/30 hover:border-exo-yellow/30 bg-exo-dark-gray/30 hover:bg-exo-dark-gray/60 transition-all duration-200 cursor-pointer"
>
diff --git a/docs/api.md b/docs/api.md
index 85d0cd3f..d7952487 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -66,7 +66,9 @@ Creates a new model instance in the cluster.
```
**Response:**
-JSON description of the created instance.
+Command acknowledgement. Instance creation is asynchronous; clients should wait
+for the model to appear through `/instance/await` before sending inference
+requests for that model.
### Delete Instance
@@ -94,6 +96,31 @@ Returns details of a specific instance.
**Response:**
JSON description of the instance.
+### Await Instance
+
+**GET** `/instance/await?model_id=...&timeout_seconds=0`
+
+Waits until API state contains an instance for the requested model. The response
+is an SSE stream so clients receive keep-alive comments while waiting.
+
+**Query parameters:**
+
+* `model_id`: string, required
+* `timeout_seconds`: float, optional, default `0`. `0` waits indefinitely;
+ positive values time out after that many seconds. Maximum positive value:
+ `300`.
+
+**Stream messages:**
+
+```text
+data: {"type": "ready", "instance": {...}}
+
+data: {"type": "timeout", "message": "No instance found for model ..."}
+```
+
+The HTTP status is `200` for both messages because the stream starts before the
+final result is known. The `type` field disambiguates the terminal message.
+
### Preview Placements
**GET** `/instance/previews?model_id=...`
@@ -123,17 +150,18 @@ Computes a placement for a potential instance without creating it.
**Response:**
JSON object describing the proposed placement / instance configuration.
-### Place Instance (Dry Operation)
+### Place Instance
**POST** `/place_instance`
-Performs a placement operation for an instance (planning step), without necessarily creating it.
+Places an instance for a model using the server's placement logic.
**Request body:**
JSON describing the instance to be placed.
**Response:**
-Placement result.
+Command acknowledgement. The instance may not be ready immediately; wait for it
+to appear through `/instance/await` before sending inference requests.
## 3. Models
@@ -639,10 +667,11 @@ GET /events
# Instance Management
POST /instance
-GET /instance/{instance_id}
-DELETE /instance/{instance_id}
+GET /instance/await
GET /instance/previews
GET /instance/placement
+GET /instance/{instance_id}
+DELETE /instance/{instance_id}
POST /place_instance
# Models
diff --git a/src/exo/api/main.py b/src/exo/api/main.py
index b836e149..d15882b1 100644
--- a/src/exo/api/main.py
+++ b/src/exo/api/main.py
@@ -50,6 +50,8 @@ from exo.api.keepalive import with_sse_keepalive
from exo.api.types import (
AddCustomModelParams,
AdvancedImageParams,
+ AwaitInstanceReadyMessage,
+ AwaitInstanceTimeoutMessage,
BenchChatCompletionRequest,
BenchChatCompletionResponse,
BenchImageGenerationResponse,
@@ -344,6 +346,7 @@ class API:
self.app.post("/place_instance")(self.place_instance)
self.app.get("/instance/placement")(self.get_placement)
self.app.get("/instance/previews")(self.get_placement_previews)
+ self.app.get("/instance/await", response_model=None)(self.await_instance)
self.app.get("/instance/{instance_id}")(self.get_instance)
self.app.delete("/instance/{instance_id}")(self.delete_instance)
self.app.get("/v1/instance-links")(self.list_instance_links)
@@ -633,6 +636,48 @@ class API:
raise HTTPException(status_code=404, detail="Instance not found")
return self.state.instances[instance_id]
+ async def await_instance(
+ self,
+ model_id: ModelId,
+ timeout_seconds: float = Query(default=0.0, ge=0.0, le=300.0),
+ ) -> StreamingResponse:
+ _sleep = 0.1
+
+ async def _stream() -> AsyncGenerator[str, None]:
+ deadline = (
+ None if timeout_seconds == 0 else anyio.current_time() + timeout_seconds
+ )
+
+ while True:
+ for instance in self.state.instances.values():
+ if instance.shard_assignments.model_id == model_id:
+ payload = AwaitInstanceReadyMessage(instance=instance)
+ yield f"data: {payload.model_dump_json()}\n\n"
+ return
+
+ if deadline is None:
+ await anyio.sleep(_sleep)
+ else:
+ remaining = deadline - anyio.current_time()
+ if remaining <= 0:
+ payload = AwaitInstanceTimeoutMessage(
+ message=f"No instance found for model {model_id}"
+ )
+ yield f"data: {payload.model_dump_json()}\n\n"
+ return
+
+ await anyio.sleep(min(_sleep, remaining))
+
+ return StreamingResponse(
+ with_sse_keepalive(_stream()),
+ media_type="text/event-stream",
+ headers={
+ "Cache-Control": "no-cache",
+ "Connection": "close",
+ "X-Accel-Buffering": "no",
+ },
+ )
+
async def delete_instance(self, instance_id: InstanceId) -> DeleteInstanceResponse:
if instance_id not in self.state.instances:
raise HTTPException(status_code=404, detail="Instance not found")
@@ -871,10 +916,8 @@ class API:
) -> ChatCompletionResponse | StreamingResponse:
"""OpenAI Chat Completions API - adapter."""
task_params = await chat_request_to_text_generation(payload)
- resolved_model = await self._resolve_and_validate_text_model(
- ModelId(task_params.model)
- )
- task_params = task_params.model_copy(update={"model": resolved_model})
+ validated_model = await self._validate_model_has_instance(task_params.model)
+ task_params = task_params.model_copy(update={"model": validated_model})
command = await self._send_text_generation_with_images(task_params)
@@ -906,10 +949,10 @@ class API:
self, payload: BenchChatCompletionRequest
) -> BenchChatCompletionResponse | StreamingResponse:
task_params = await chat_request_to_text_generation(payload)
- resolved_model = await self._resolve_and_validate_text_model(
+ validated_model = await self._validate_model_has_instance(
ModelId(task_params.model)
)
- task_params = task_params.model_copy(update={"model": resolved_model})
+ task_params = task_params.model_copy(update={"model": validated_model})
task_params = task_params.model_copy(
update={
@@ -939,8 +982,10 @@ class API:
return await self._collect_text_generation_with_stats(command.command_id)
- async def _resolve_and_validate_text_model(self, model_id: ModelId) -> ModelId:
- """Validate a text model exists and return the resolved model ID.
+ async def _validate_model_has_instance(self, model_id: ModelId) -> ModelId:
+ """Validate a model has an active instance.
+ If the model isn't even downloaded, triggers notification to user to download model.
+
Raises HTTPException 404 if no instance is found for the model.
"""
@@ -948,29 +993,20 @@ class API:
instance.shard_assignments.model_id == model_id
for instance in self.state.instances.values()
):
- await self._trigger_notify_user_to_download_model(model_id)
- raise HTTPException(
- status_code=404,
- detail=f"No instance found for model {model_id}",
+ # Check if model is actually downloaded
+ model_is_downloaded = any(
+ isinstance(download, DownloadCompleted)
+ and download.shard_metadata.model_card.model_id == model_id
+ for node_downloads in self.state.downloads.values()
+ for download in node_downloads
)
- return model_id
+ if not model_is_downloaded:
+ await self._trigger_notify_user_to_download_model(model_id)
- async def _validate_image_model(self, model: ModelId) -> ModelId:
- """Validate model exists and return resolved model ID.
-
- Raises HTTPException 404 if no instance is found for the model.
- """
- model_card = await ModelCard.load(model)
- resolved_model = model_card.model_id
- if not any(
- instance.shard_assignments.model_id == resolved_model
- for instance in self.state.instances.values()
- ):
- await self._trigger_notify_user_to_download_model(resolved_model)
raise HTTPException(
- status_code=404, detail=f"No instance found for model {resolved_model}"
+ status_code=404, detail=f"No instance found for model {model_id}"
)
- return resolved_model
+ return model_id
def stream_events(self) -> StreamingResponse:
def _generate_json_array(events: Iterable[Event]) -> Iterable[str]:
@@ -1024,7 +1060,9 @@ class API:
"""
payload = payload.model_copy(
update={
- "model": await self._validate_image_model(ModelId(payload.model)),
+ "model": await self._validate_model_has_instance(
+ ModelId(payload.model)
+ ),
"advanced_params": _ensure_seed(payload.advanced_params),
}
)
@@ -1292,7 +1330,9 @@ class API:
) -> BenchImageGenerationResponse:
payload = payload.model_copy(
update={
- "model": await self._validate_image_model(ModelId(payload.model)),
+ "model": await self._validate_model_has_instance(
+ ModelId(payload.model)
+ ),
"stream": False,
"partial_images": 0,
"advanced_params": _ensure_seed(payload.advanced_params),
@@ -1328,7 +1368,7 @@ class API:
advanced_params: AdvancedImageParams | None,
) -> ImageEdits:
"""Prepare and send an image edits command with chunked image upload."""
- resolved_model = await self._validate_image_model(model)
+ validated_model = await self._validate_model_has_instance(model)
advanced_params = _ensure_seed(advanced_params)
image_content = await image.read()
@@ -1347,7 +1387,7 @@ class API:
image_data="",
total_input_chunks=total_chunks,
prompt=prompt,
- model=resolved_model,
+ model=validated_model,
n=n,
size=size,
response_format=response_format,
@@ -1368,7 +1408,7 @@ class API:
await self._send(
SendInputChunk(
chunk=InputImageChunk(
- model=resolved_model,
+ model=validated_model,
command_id=command.command_id,
data=chunk_data,
chunk_index=chunk_index,
@@ -1492,10 +1532,10 @@ class API:
) -> ClaudeMessagesResponse | StreamingResponse:
"""Claude Messages API - adapter."""
task_params = await claude_request_to_text_generation(payload)
- resolved_model = await self._resolve_and_validate_text_model(
+ validated_model = await self._validate_model_has_instance(
ModelId(task_params.model)
)
- task_params = task_params.model_copy(update={"model": resolved_model})
+ task_params = task_params.model_copy(update={"model": validated_model})
command = await self._send_text_generation_with_images(task_params)
@@ -1530,8 +1570,8 @@ class API:
) -> ResponsesResponse | StreamingResponse:
"""OpenAI Responses API."""
task_params = await responses_request_to_text_generation(payload)
- resolved_model = await self._resolve_and_validate_text_model(task_params.model)
- task_params = task_params.model_copy(update={"model": resolved_model})
+ validated_model = await self._validate_model_has_instance(task_params.model)
+ task_params = task_params.model_copy(update={"model": validated_model})
command = await self._send_text_generation_with_images(task_params)
@@ -1573,10 +1613,10 @@ class API:
body = await request.body()
payload = OllamaChatRequest.model_validate_json(body)
task_params = ollama_request_to_text_generation(payload)
- resolved_model = await self._resolve_and_validate_text_model(
+ validated_model = await self._validate_model_has_instance(
ModelId(task_params.model)
)
- task_params = task_params.model_copy(update={"model": resolved_model})
+ task_params = task_params.model_copy(update={"model": validated_model})
command = await self._send_text_generation_with_images(task_params)
@@ -1609,10 +1649,10 @@ class API:
body = await request.body()
payload = OllamaGenerateRequest.model_validate_json(body)
task_params = ollama_generate_request_to_text_generation(payload)
- resolved_model = await self._resolve_and_validate_text_model(
+ validated_model = await self._validate_model_has_instance(
ModelId(task_params.model)
)
- task_params = task_params.model_copy(update={"model": resolved_model})
+ task_params = task_params.model_copy(update={"model": validated_model})
command = await self._send_text_generation_with_images(task_params)
diff --git a/src/exo/api/types/__init__.py b/src/exo/api/types/__init__.py
index 9cb2f834..03f29c41 100644
--- a/src/exo/api/types/__init__.py
+++ b/src/exo/api/types/__init__.py
@@ -1,5 +1,7 @@
from .api import AddCustomModelParams as AddCustomModelParams
from .api import AdvancedImageParams as AdvancedImageParams
+from .api import AwaitInstanceReadyMessage as AwaitInstanceReadyMessage
+from .api import AwaitInstanceTimeoutMessage as AwaitInstanceTimeoutMessage
from .api import BenchChatCompletionRequest as BenchChatCompletionRequest
from .api import BenchChatCompletionResponse as BenchChatCompletionResponse
from .api import BenchImageGenerationResponse as BenchImageGenerationResponse
diff --git a/src/exo/api/types/api.py b/src/exo/api/types/api.py
index 8cfa10dd..45782267 100644
--- a/src/exo/api/types/api.py
+++ b/src/exo/api/types/api.py
@@ -291,6 +291,16 @@ class DeleteInstanceResponse(BaseModel):
instance_id: InstanceId
+class AwaitInstanceReadyMessage(BaseModel):
+ type: Literal["ready"] = "ready"
+ instance: Instance
+
+
+class AwaitInstanceTimeoutMessage(BaseModel):
+ type: Literal["timeout"] = "timeout"
+ message: str
+
+
class CancelCommandResponse(BaseModel):
message: str
command_id: CommandId
diff --git a/src/exo/master/main.py b/src/exo/master/main.py
index 55fed7e6..dd5fbf0f 100644
--- a/src/exo/master/main.py
+++ b/src/exo/master/main.py
@@ -181,6 +181,7 @@ class Master:
case TestCommand():
pass
case TextGeneration():
+ # set-difference => prefill-only nodes
prefill_only: set[InstanceId] = set()
for link in self.state.instance_links.values():
prefill_only.update(link.prefill_instances)
@@ -188,11 +189,13 @@ class Master:
prefill_only.difference_update(link.decode_instances)
for instance in self.state.instances.values():
+ # NON-prefill-only instances matching the model ID
if (
instance.shard_assignments.model_id
== command.task_params.model
and instance.instance_id not in prefill_only
):
+ # count in-flight tasks of that instance
in_flight = {TaskStatus.Pending, TaskStatus.Running}
task_count = sum(
1
@@ -204,6 +207,7 @@ class Master:
task_count
)
+ # there are no NON-prefill-only instances matching this model ID
if not instance_task_counts:
raise ValueError(
f"No instance found for model {command.task_params.model}"
diff --git a/tmp/test_trust_remote_code_attack.sh b/tmp/test_trust_remote_code_attack.sh
index df9641ff..31e7c37d 100755
--- a/tmp/test_trust_remote_code_attack.sh
+++ b/tmp/test_trust_remote_code_attack.sh
@@ -7,6 +7,9 @@ set -uo pipefail
HOST="${1:-localhost:52415}"
MODEL_ID="KevTheHermit/security-testing"
+ENCODED_MODEL_ID=$(
+ python3 -c 'import sys, urllib.parse; print(urllib.parse.quote(sys.argv[1], safe=""))' "$MODEL_ID"
+)
CUSTOM_CARDS_DIR="$HOME/.exo/custom_model_cards"
CARD_FILE="$CUSTOM_CARDS_DIR/KevTheHermit--security-testing.toml"
@@ -71,9 +74,30 @@ PLACE_BODY=$(echo "$PLACE_RESPONSE" | sed '$d')
echo " HTTP $PLACE_CODE"
echo " Response: $PLACE_BODY"
-# Step 3b: Send a chat completion to actually trigger tokenizer loading
+if [ "$PLACE_CODE" -ge 400 ]; then
+ echo " Placement failed; cannot trigger tokenizer loading."
+ exit 1
+fi
+
+# Step 3b: Wait for placement to materialize before inference.
+echo ""
+echo "[3b] Waiting for placed instance ..."
+if ! AWAIT_RESPONSE=$(curl -fsS --max-time 65 \
+ "http://$HOST/instance/await?model_id=$ENCODED_MODEL_ID&timeout_seconds=60" |
+ awk '/^data: / { sub(/^data: /, ""); print; exit }'); then
+ echo " Timed out waiting for an instance for $MODEL_ID"
+ exit 1
+fi
+
+if ! printf '%s' "$AWAIT_RESPONSE" | grep -q '"type":"ready"'; then
+ echo " Timed out waiting for an instance for $MODEL_ID"
+ exit 1
+fi
+echo " Instance ready"
+
+# Step 3c: Send a chat completion to actually trigger tokenizer loading
echo ""
-echo "[3b] Sending chat completion to trigger tokenizer load ..."
+echo "[3c] Sending chat completion to trigger tokenizer load ..."
CHAT_RESPONSE=$(curl -s -w "\n%{http_code}" --max-time 30 -X POST "http://$HOST/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\":\"$MODEL_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}],\"max_tokens\":1}")
@@ -82,7 +106,7 @@ CHAT_BODY=$(echo "$CHAT_RESPONSE" | sed '$d')
echo " HTTP $CHAT_CODE"
echo " Response: $CHAT_BODY"
echo ""
-echo "[3c] Checking for RCE proof ..."
+echo "[3d] Checking for RCE proof ..."
sleep 5
if [ -f /tmp/exo-rce-proof.txt ]; then
echo " VULNERABLE: Remote code executed!"
← a1a22b5f feat: added background/daemon support (#2106)
·
back to Exo
·
Capture energy in prefill and ageneration separately (#2124) 051a64e3 →