[object Object]

← back to Butlr

predeploy-guard: abort deploy if any call is in flight

6fe29bf2bff2033927f2efcbb7f75b9436af38fb · 2026-05-13 07:22:53 -0700 · SteveStudio2

Discovered 2026-05-13 in the 5-call live test — a mid-test deploy
killed 3 active calls' WSS audio capture because pm2 reload drops the
Media Streams lane. The AI agent loop survives (it's stateless from
Twilio's POV — they just keep POSTing /twilio/gather), but the PCM
stream goes dark and the MP3 finalize never fires.

scripts/predeploy-guard.sh queries prod's calls.json for any call in
status dialing/on_hold/connected, prints them, and exits 2 unless
FORCE_DEPLOY=1 is set. Hooked in via .deploy.conf's new PREDEPLOY_HOOK
variable, which the shared deploy.sh (also in this commit) now runs
before the rsync step.

Stale-on-hold cleanup: 3 Wells Fargo calls (6o3PMv5s, npTGTE7T,
GNNCY2co) had been stuck in status=on_hold since 2026-05-12 because
their /twilio/status callbacks were lost to a prior pm2 restart. One-
shot Twilio API query confirmed all 3 actually completed (45/125/106s
durations) — synced to status=done.

Audio path proof (post-this-commit): NIST call lbd8VV_r and Steve's
MacEnthusiasts quick-dial call 7pCAWx5A both produced full MP3 +
whisper.cpp transcript. The /calls inline UI works end-to-end.

Files touched

Diff

commit 6fe29bf2bff2033927f2efcbb7f75b9436af38fb
Author: SteveStudio2 <stevestudio2@SteveStacStudio.lan>
Date:   Wed May 13 07:22:53 2026 -0700

    predeploy-guard: abort deploy if any call is in flight
    
    Discovered 2026-05-13 in the 5-call live test — a mid-test deploy
    killed 3 active calls' WSS audio capture because pm2 reload drops the
    Media Streams lane. The AI agent loop survives (it's stateless from
    Twilio's POV — they just keep POSTing /twilio/gather), but the PCM
    stream goes dark and the MP3 finalize never fires.
    
    scripts/predeploy-guard.sh queries prod's calls.json for any call in
    status dialing/on_hold/connected, prints them, and exits 2 unless
    FORCE_DEPLOY=1 is set. Hooked in via .deploy.conf's new PREDEPLOY_HOOK
    variable, which the shared deploy.sh (also in this commit) now runs
    before the rsync step.
    
    Stale-on-hold cleanup: 3 Wells Fargo calls (6o3PMv5s, npTGTE7T,
    GNNCY2co) had been stuck in status=on_hold since 2026-05-12 because
    their /twilio/status callbacks were lost to a prior pm2 restart. One-
    shot Twilio API query confirmed all 3 actually completed (45/125/106s
    durations) — synced to status=done.
    
    Audio path proof (post-this-commit): NIST call lbd8VV_r and Steve's
    MacEnthusiasts quick-dial call 7pCAWx5A both produced full MP3 +
    whisper.cpp transcript. The /calls inline UI works end-to-end.
---
 .deploy.conf               |  4 ++++
 scripts/predeploy-guard.sh | 50 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)

diff --git a/.deploy.conf b/.deploy.conf
index 3a29fa0..9bd18c4 100644
--- a/.deploy.conf
+++ b/.deploy.conf
@@ -5,5 +5,9 @@ HEALTH_URL=https://butlr.agentabrams.com/healthz
 REQUIRED_ENVS=""
 RSYNC_EXTRA_EXCLUDES="data/users.json data/users.json.* data/calls.json data/calls.json.* data/recordings data/transcripts data/audio-cache data/menu-map"
 
+# Block deploys while calls are in flight — pm2 reload kills the WSS Media
+# Stream lane and silently drops audio capture. Override with FORCE_DEPLOY=1.
+PREDEPLOY_HOOK="bash scripts/predeploy-guard.sh"
+
 # Idempotent every deploy: migrate user_ids, finalize any orphaned PCM → MP3.
 INSTALL_CMD="npm ci --production && node scripts/migrate-calls-to-user.js && node scripts/finalize-recordings.js"
diff --git a/scripts/predeploy-guard.sh b/scripts/predeploy-guard.sh
new file mode 100755
index 0000000..ab7964f
--- /dev/null
+++ b/scripts/predeploy-guard.sh
@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# Pre-deploy guard for Butlr: abort if any call is currently active.
+# A pm2 reload kills the WSS Media Stream lane, which silently drops
+# the PCM audio capture and any in-flight ffmpeg→MP3 finalize. Discovered
+# 2026-05-13 when a mid-call deploy lost 5 recordings (3 calls completed
+# but had no audio archived).
+#
+# Usage (called from .deploy.conf):
+#   PREDEPLOY_HOOK="bash scripts/predeploy-guard.sh"
+#
+# Exits 0 if safe. Exits 2 with explanation if at least one call is
+# in dialing / on_hold / connected. Override with FORCE_DEPLOY=1 if
+# you've decided the deploy is worth dropping any active recordings.
+
+set -euo pipefail
+
+DEPLOY_HOST="${DEPLOY_HOST:-45.61.58.125}"
+DEPLOY_PATH="${DEPLOY_PATH:-/root/public-projects/butlr}"
+
+if [[ "${FORCE_DEPLOY:-}" == "1" ]]; then
+  echo "  ⚠️  FORCE_DEPLOY=1 — skipping active-call guard"
+  exit 0
+fi
+
+# Read prod calls.json over ssh. Statuses considered active:
+#   dialing  — call is being placed
+#   on_hold  — Twilio Stream is open, capturing audio
+#   connected — Stream + AI loop running, audio still flowing
+active=$(ssh -o ConnectTimeout=5 "root@${DEPLOY_HOST}" \
+  "node -e 'const d=require(\"${DEPLOY_PATH}/lib/data\"); const a=d.listCallsUnscoped(); const live=a.filter(c=>[\"dialing\",\"on_hold\",\"connected\"].includes(c.status)); console.log(JSON.stringify(live.map(c=>({id:c.id,status:c.status,biz:c.business_name}))))'" \
+  2>/dev/null || echo "[]")
+
+count=$(node -e "process.stdout.write(String(JSON.parse(process.argv[1]).length))" "$active")
+
+if [[ "$count" == "0" ]]; then
+  echo "  ✓ no active calls — safe to deploy"
+  exit 0
+fi
+
+echo
+echo "  ✗ ABORTING DEPLOY — $count active call(s) on prod:"
+node -e "
+const a = JSON.parse(process.argv[1]);
+for (const c of a) console.log('     ' + c.id.padEnd(10) + ' | ' + c.status.padEnd(10) + ' | ' + (c.biz || ''));
+" "$active"
+echo
+echo "  Wait for calls to complete (pm2 logs butlr | grep status), or override:"
+echo "    FORCE_DEPLOY=1 bash ~/Projects/_shared/scripts/deploy.sh"
+echo
+exit 2

← 3f6963c calls: quick-dial form + inline transcript viewer + listen-l  ·  back to Butlr  ·  ai-agent: fix 3 bugs exposed by AAA call PSmRTAJ5 d68c3bf →