Butlr

Twilio AMD mislabels chatty voicemail systems as AnsweredBy=human, so the
existing machine-hangup in /twilio/amd never fires and the AI agent navigates
the voicemail menu (call Tjz8cfFd burned turns inside Civil Coffee's mailbox).
Adds a deterministic looksLikeVoicemail() guard in both /twilio/gather (silent
Hangup TwiML, pre-LLM) and /twilio/partial (earliest call.update Status=completed
fast-bail) as belt-and-suspenders. Voicemail = wasted money.

Fixes "can't hear anything on live-listen". The bridge only called
broadcastPcm for the inbound track, so the AI (outbound) and on-hold/silent
business produced no audio. Now every decoded 20ms frame is broadcast as it
arrives; the browser's nextPlayTime queue interleaves them in order. Phone
calls are mostly half-duplex so this sounds clean with no per-sample summing,
no clipping, no buffering latency, and zero client change (still mono 8kHz
Int16). Stereo MP3 archive (archiveAppend both tracks) untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds a 'Completed calls' history section below the live grid on
live.agentabrams.com — fetches /api/done?limit=24 every 30s and renders
each done call with business name, status pill, time-ago, goal snippet,
and an inline <audio> player streaming from /api/calls/:id/recording
(same-origin session-cookie auth, token stays out of the DOM). onerror
gracefully swaps a failed player for a 'no recording archived' note.

Also reconciles 31 lines of prod drift: the Kamatera copy had auto-play-
on-connect + the shared-AudioContext unlock banner that were never
committed back to this repo. This commit pulls prod's full current state
(342 lines) plus the new history section (→408) so Mac2 = prod again.

Prod (root@45.61.58.125:/root/public-projects/butlr) was substantially
ahead of local after Steve's direct-on-box edits. Pulled prod → local
(excludes: .env, node_modules, data/, *.log, *.bak-*, *.pre-*; no --delete).

This supersedes my redundant local AMD commit (f4b29f0) — Steve's prod
version of the answering-machine auto-hangup is cleaner and broader:
  - triggers on [machine_start, machine_end_beep, machine_end_silence,
    machine_end_other, fax] (mine missed fax)
  - hangs up via Status=completed (cleaner than my Twiml=<Hangup/>)
  - patches the call row notes with [auto-hangup: AMD <kind>]

Also brought down prod-only work that was never on local:
  - lib/scheduling.js + routes/scheduled.js (appointment scheduling)
  - routes/live.js (live-calls surface)
  - views/public/calendar.ejs
  - public/audio/intro-steve.mp3 (call intro)
  - scripts/dial-sequence.sh
  - modified: server.js, lib/data.js, lib/twilio.js, lib/vapi.js,
    routes/listen.js, routes/public.js, several views

All key files node -c clean. Local is now a faithful mirror of the live
butlr service, so a future /deploy won't clobber prod's work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Steve 2026-05-26: "if answering machine beep, HANG UP."

The /twilio/amd/:call_id handler was previously log-only with a "WE DO NOT
AUTO-BRIDGE" note explaining the human-detection false-positive risk on
chatty IVRs. That stays true for the HUMAN branch.

This patch adds a MACHINE branch: when Twilio's AnsweredBy starts with
"machine_" (with our DetectMessageEnd mode that's machine_end_beep,
machine_end_silence, or machine_end_other), the handler fires the same
proven call.update fast-path as /partial/:call_id's goal-satisfied
hangup — but with BARE <Response><Hangup/></Response>, no <Say>/<Play>.
Bare = no audio gets recorded onto the voicemail. Silent disconnect.

State tracked in MACHINE_HUNGUP Set (callId dedupe) so a delayed
duplicate AMD callback doesn't re-fire. On Twilio API failure we
delete from the set to allow retry on next webhook.

Status updates to 'done' on successful hangup (matches the partial-hangup
pattern at line 428).

Acked the webhook before doing async work (res.type('text/xml').send
moved to top) so Twilio doesn't retry the AMD callback while we're
mid-call.update.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- broaden .gitignore to cover *.bak, *.bak.*, *.pre-*, .pre-*
- add Express middleware that 404s any .bak/.pre- path before
  express.static so a stray snapshot dropped in public/ can never
  leak. Pre-existing target=_blank links already carry rel=noopener.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

lib/listen-bridge.js decoded Twilio's G.711 µ-law media frames with
MULAW_BIAS=33. The G.711 decode bias is 0x84 (132), added inside the
<< exponent shift. At 33 the bias was under-counted 4×: quiet samples
(exponent 0) stayed correct but every loud sample was nonlinearly
crushed (0x00 decoded to -19551 instead of -32124). That amplitude-
dependent distortion made both the recorded .mp3 and the live-listen
feed sound horrible. Verified: the decoder now matches canonical
G.711 on all 256 µ-law byte values.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

POST /api/external/place-call lets NationalPaperHangers.com place a Butlr
call without the owner-authed wizard. Guarded by a shared-secret header
(X-Butlr-External-Secret); fail-closed when BUTLR_EXTERNAL_SECRET is unset.

Three modes — hold / bridge / ai_agent — map onto the existing addCall +
dialCall path. Calls are owned by a dedicated auto-created service user so
per-user scoping holds. allow_repeat:true is passed so different customers
can call the same installer; the 4-call/number hard cap still applies.
Honors TWILIO_DRY_RUN like every other call path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Per Steve's standing instruction (2026-05-14): Bridge-me-now should dial
the human's personal cell, NOT the Twilio outbound FROM line. The
quick-dial handler was reading req.user.email_phone, a field that does
not exist in users.js's schema, falling all the way through to the
hardcoded +16067130489 (the Twilio FROM number).

Fix has two halves:
  1. seedLegacyAdmin() now persists callback_phone from a new env var
     BUTLR_OWNER_CALLBACK_PHONE so every fresh boot picks it up.
  2. /api/quick-dial's callback chain becomes:
       body.callback_phone           (per-call override from /butlr skill)
       → req.user.callback_phone     (seeded from env)
       → process.env.BUTLR_OWNER_CALLBACK_PHONE
       → process.env.TWILIO_PHONE_NUMBER  (last-resort fallback)
       → +16067130489

Steve set BUTLR_OWNER_CALLBACK_PHONE=+13107130489 in /root/public-projects/butlr/.env
and pm2-restarts butlr, the admin user record auto-persists callback_phone.

The /butlr Claude Code skill was confidently saying 'DIALED' when the
server was actually in TWILIO_DRY_RUN=ON mode (simulated state machine,
no real Twilio call). The skill now hits this probe BEFORE POSTing to
/api/quick-dial; if dry_run is true it warns Steve and refuses to
claim 'DIALED' without explicit override.

Owner-only. Surfaces nothing sensitive — only boolean flags.

tokenFromReq() already accepted the cookie and ?owner_token= query.
The /butlr Claude Code skill needs to POST /api/quick-dial without
the token landing in URL/nginx logs. Adding two header transports:
  - X-Butlr-Owner-Token (explicit)
  - Authorization: Bearer <token> (standard)
Fallback chain stays: cookie → query → header → bearer.

Coupling: the /butlr skill prefers the header path now and falls
back to ?owner_token= only when the server returns 401 (pre-deploy
of this commit). Existing browser cookies + the legacy /login?t=
fast-path are unaffected.

Was blocking the wizard-2-business 'Use my location' button with
Chrome err.message 'Geolocation has been disabled in this document
by permissions policy.' Same-origin top-level only — mic/camera stay
denied. Flipped payment to (self) too so Stripe Payment Request
API stops console-warning.

Surfaces what scripts/status-snapshot.sh writes to ~/status, but live
per-request (no dependence on cron persistence). Auto-refreshes every
30s. Shows butlr + wallco-ai with HTTP, uptime, restarts, pm2 status,
memory, CPU. Reads recent ALERT-*.txt files if present.

Admin-only. URL: https://butlr.agentabrams.com/admin/status

POST /twilio/sms-inbound — TCPA §64.1200(d) compliant inbound-SMS
handler. Twilio's Messaging Service routes inbound SMS here.

Keywords (case-insensitive, word-boundary match anywhere in body):
- STOP / STOPALL / UNSUBSCRIBE / CANCEL / END / QUIT / REVOKE →
  dnc.addToInternalSuppression(from, 'sms_stop_keyword', 'twilio_inbound')
  Reply: "You are unsubscribed from Butlr. No further messages will be
  sent. Reply START to re-subscribe."
- START / UNSTOP / YES → removes from suppression (re-opt-in).
  Reply: "You are re-subscribed to Butlr. Reply STOP to opt out."
- Anything else → generic identity + opt-out reply (no engagement).

Atomic write on suppression updates (tmp + rename). No-op gracefully
if From is missing. dnc-check's addToInternalSuppression is already
idempotent so repeat STOPs don't duplicate.

test/sms-stop.test.js: 8/8 — STOP/UNSUBSCRIBE/CANCEL add, repeat STOP
is idempotent, START removes, random body doesn't change list, missing
From → 200 no-op, lowercase "stop" matches.

Configure on Twilio side: Messaging Service → Inbound webhook →
https://butlr.agentabrams.com/twilio/sms-inbound

npm test now: 49/49 across 6 files (orphan 8 + admin-gate 6 +
vapi-webhook 11 + dnc 9 + upload-watcher 7 + sms-stop 8).

Replaces manual "Sync now" click with a background scan that runs on
the server startup loop.

- lib/upload-watcher.js: scanOnce() walks meta.folder, dedups by
  (size+mtime) per source_path, copies new allowed-ext files into
  data/uploads/ with unguessable hex filenames. Logged with source:
  'auto_watcher' so manual + watcher imports are distinguishable.
  start({intervalMs:30000}) sets the recurring tick. HFM_NO_WATCHER=1
  disables (used by tests). First scan fires 1s after server boot.
- server.js: wires uploadWatcher.start() right after twilio.start().
- test/upload-watcher.test.js: 7/7 — imports allowed exts, skips bad
  ext + hidden, idempotent on second scan, picks up new file, meta
  source_* fields populated, missing folder gracefully reported,
  no-folder config is a noop.
- package.json: npm test now runs 41 assertions across 5 files.

Steve drops images in the configured folder; within 30s they appear
in /admin/uploads with copy-URL chips.

Second half of Steve's "connect to a folder" ask.

- POST /admin/uploads/folder/sync — admin-gated. Walks meta.folder,
  imports any file (allowed ext, ≤10MB) not already in the gallery.
  Dedup via (size + mtime) key per source path. Returns
  {ok, imported, skipped, errors}. Atomic per-file via copyFileSync.
- /admin/uploads page: when a folder is configured, shows a "Sync
  now" button + status line next to the path display. Reload page
  on first new import.
- New gallery entries from folder-sync carry source_path /
  source_size / source_mtime so future syncs skip them automatically.

Watcher-on-cron is a follow-up tick (TODO: 30s interval scan on the
worker tick). For now Steve presses "Sync now" when he drops files
into the watched folder.

GET  /admin/uploads        — drop-zone + gallery + folder-connect field
POST /admin/uploads/file   — multipart, 10MB cap, 20 files at once, ext-whitelist
POST /admin/uploads/folder — record a local folder path for future watcher
POST /admin/uploads/delete — admin delete by filename
GET  /uploads/<filename>   — public serve (unguessable 24-hex names),
                              immutable cache, no auth (content-addressed token)

Per Steve's standing rule: "create a button to upload images to url, or
connect to a folder". Button + drag-drop ships now; the folder side
records a path (watcher implementation deferred — TODO surface in next
tick once Steve confirms which folder he wants watched).

Storage: data/uploads/<uuid>.<ext> + data/uploads-meta.json. Both
rsync-excluded + gitignored so prod state persists across deploys.

Allowed exts: png/jpg/jpeg/gif/webp/svg/pdf/mp3/wav/mp4/txt/csv/json.

Frontend: tile gallery, copy-URL chip per file (writes
location.origin + /uploads/<filename> to clipboard), inline preview
for images, link-out for non-image types. Delete confirmed via
window.confirm.

Per Steve's standing rule — every paid API call appends to the ledger.

- ~/.claude/skills/cost-tracker/pricing.json: new vapi_voice entry at
  $0.10/min conservative midpoint (tune from real invoice). Twilio
  rates (sms $0.0083 / voice_min $0.014) already present.
- routes/vapi-webhooks.js: on end-of-call-report, spawn cost-tracker
  log.js detached with durationSeconds/60 voice_min units. Best-
  effort — log failure doesn't affect call data.
- routes/twilio-webhooks.js: on POST /status/<id> with CallStatus=
  completed, log the CallDuration as voice_min via twilio.logCost().
- lib/twilio.js: new logCost(api, qty, unit, note) helper. Spawns
  detached + unref so SMS send / call hangup return immediately.
  sendSms now logs 1 sms unit on successful send.

Smoke-tested: log.js --api vapi_voice --units 1:voice_min logs
$0.100000 and writes the jsonl entry as expected.

views/public/wizard-2-business.ejs:
- Custom-phone input fires /api/dnc-check on blur — shows 🚫 + red
  explainer + disables "Next" button if number is on DNC.
- Selecting a directory preset radio also pre-checks (pulls phone from
  the muted sibling line). Catches the case where Steve's curated
  businesses.json includes a number that's been added to suppression.
- ✓ "not on DNC" or "(dry-run mode)" confirmation on allowed.

Same contract as the /calls quick-dial form (deployed prior). Now
users see DNC status BEFORE submitting either dial path.

- GET /api/dnc-check?phone=… returns {allowed, reason, dry_run_bypass}.
  Auth-gated. Doesn't log a block — checks-only, no dial attempt
  happened. Gate-error (snapshot missing) surfaces as allowed=false
  with reason=gate_error + code, so UI can render the right message.
- /calls quick-dial form: phone-blur hits /api/dnc-check. On block,
  dial button is disabled + red 🚫 explainer ("on federal DNC" /
  "in internal suppression" / "DNC list unavailable"). On allowed,
  blue ✓ confirmation.

User now gets pre-flight feedback before pressing Dial; they don't
waste a submit cycle finding out the number is suppressed.

Two visual halves of the DNC compliance work shipped in the prior commit:

- /admin/dnc — owner+role=admin gated page:
  * federal snapshot status (size on disk, MISSING warning if absent)
  * add-to-suppression form (phone + reason, manual entry)
  * current suppression table with per-row "remove" button
  * POST /admin/dnc/add and /admin/dnc/remove are also admin-gated;
    /remove uses atomic write (tmp+rename) like the main lib does
- /calls rows now render:
  * 🚫 DNC pill next to the status pill (red, hover-tooltip with
    reason+source)
  * Red sub-line explaining what was blocked + link to /admin/dnc
  * Only renders when row has dnc_flagged=true (set by tick() at the
    pre-flight gate)

Together this gives Steve a complete loop: dial → DNC gate blocks →
row is flagged with explanation in his UI → he can remove from
suppression in admin if false-positive, or leave it as a permanent
block.

data/dnc-blocks.jsonl is append-only audit log (per-call); should not
be in git or rsynced. data/dnc-suppression.json is prod-side state
(per-call adds via STOP keyword / manual admin). Both excluded from
rsync via RSYNC_EXTRA_EXCLUDES. .gitignore updated to match.

Steve's standing rule 2026-05-13: "Do not allow butlr to call anyone
on the do not call list for each state and do not sms too. Any do not
on either, remove from list. Always check this before dialing."

Implementation (per comms-compliance subagent's policy review):

- lib/dnc-check.js: checkPhone(num, channel) returns {allowed} after
  consulting (a) federal DNC snapshot at $DNC_FILE (default
  ~/.claude/data/dnc/national-dnc.txt), (b) internal suppression list
  at data/dnc-suppression.json. Missing snapshot THROWS (fail closed).
  TWILIO_DRY_RUN=1 bypasses for dev. Block events written to
  data/dnc-blocks.jsonl (last-4 + sha256 only, no PII).

- lib/twilio.js tick(): pre-flight checks BOTH business_phone AND
  callback_phone before any Twilio/Vapi dispatch. Block → status=
  'cancelled' + dnc_flagged=true + dnc_reason/source/field on the row.
  Gate-error (snapshot missing) also fail-closed, same row patches.

- lib/twilio.js sendSms(): pre-flight checks `to` against DNC.
  Returns {ok:false, error:'dnc_block'} on hit. Gate-error fails closed.

- test/dnc-check.test.js: 9/9 — normalize variants, federal hit,
  internal suppression, DRY_RUN bypass, missing snapshot throws,
  addToInternalSuppression idempotent + atomic, non-US passthrough,
  no-phone validation.

- npm test now: orphan-recordings 8/8 + admin-gate 6/6 +
  vapi-webhook 11/11 + dnc-check 9/9 = 34/34.

State DNC lists (IN/LA/MO/OK/WY) require paid subscriber portals same
as federal; not API-accessible. Sticking with federal + internal. CA
uses federal + stacks penalties via Cal. Bus. & Prof. § 17592.

Federal DNC SAN subscription: https://telemarketing.donotcall.gov/.
Until the SAN file is at $DNC_FILE, the gate fails closed on every
live dial. Prod seeded with a placeholder so dev mode works.

The 4AnkhRRI smoke call exposed that Butlr's worker is now dialing
through Vapi instead of Twilio — the report was false-flagging
3 red on a fully-successful call because it only knew the Twilio
shape (twilio_call_sid, local MP3 in data/recordings/, local
whisper transcript).

Now auto-detects provider by which sid is present and branches:
- TWILIO path: existing Twilio API cross-check + local MP3/PCM +
  data/transcripts/*.json + TwiML validate.
- VAPI path: HEAD probe on c.recording_url (storage.vapi.ai),
  validates c.transcript_path starts with "vapi:", confirms the
  /vapi/webhook route is mounted (covered by test/vapi-webhook.test.js).
- UNKNOWN provider stays red.

Header now shows "(VAPI)" or "(TWILIO)" so the operator immediately
knows which infrastructure the call used.

Verified by re-running against 4AnkhRRI: 7/7 green, recording confirmed
1141.9KB audio/wav on storage.vapi.ai (publicly fetchable).

Locks in the Vapi event-shape contract before Steve starts routing
real call traffic through /vapi/webhook. Tests cover:
- status-update mappings (queued/ringing/in-progress/ended/failed →
  Butlr's internal statuses dialing/connected/done/failed)
- skips DB write when metadata.butlr_call_id is absent
- final transcript appends to data/agent-log/<id>.json
- partial/non-final transcript chunks do NOT log
- end-of-call-report stores recording_url + summary in DB + log
- unknown event types return 200 with no side effects
- empty body returns 200, no crash

Uses require.cache injection to stub lib/data + fs read/write
redirect to a tmp dir so the real calls.json/agent-log isn't touched.

npm test now runs 25 assertions across 3 files (orphan-recordings 8/8
+ admin-gate 6/6 + vapi-webhook 11/11).

Cross-checks 4 systems for any call:
- DB record exists + user-scoped
- Twilio API status/duration matches local DB
- Recording (MP3) exists + size; warns if PCM left unfinalized
- Transcript exists + content sample
- Live TwiML still serves the inline-Gather/both_tracks/no-Redirect
  contract for this call_id

Green/yellow/red per check + exit code 0 only on all-green. Supports
--remote to ssh into prod for you. Wired into npm: `npm run report --
n87ResHr`.

First run against n87ResHr (NIST both_tracks proof) — 7/7 green. The
transcript starts with "Hello!" (Butlr's greeting via ElevenLabs) +
NIST time-reading (Whisper mis-recognized as "singing in foreign
language"), confirming track="both_tracks" captures both sides.

inbound_track only captured the business side — the listener heard
the IVR and rep but Butlr's own TTS responses were silent. AI and rep
speak in alternation, not overlap, so there's no echo to avoid by
splitting tracks. Switch to both_tracks. Bonus: Steve can now verify
the consent announcement actually played at call start.

Exposed by Steve testing the post-prompt-fix AAA + Capital One calls
2026-05-13 — he reported "nothing is talking back" while logs showed
the AI was correctly responding "No" / "I don't know" / etc.

The AAA Roadside call (Steve testing membership verification) exposed
three prompt-engineering bugs:

1. AI greeted rep "Carol" as "Sarah" because Sarah was in the worked
   example and the model copied it literally. Fix: explicit "DO NOT
   memorize names from examples below. Extract from each input. The
   name CHANGES per call" instruction + 4 worked examples with 4
   different names (Sarah/Carol/Marcus/no-name).

2. AI pressed digits not present in the menu — pressed 4 when "for
   roadside press 1, for travel press 2" was offered. Fix: "ONLY
   press a digit that was EXPLICITLY mentioned. The digit MUST
   literally appear in the input." Plus new yes/no question rule
   (the original Kentucky prompt got pressed-0 when it should have
   been "No" spoken aloud).

3. AI hung up claiming "goal achieved" on empty input right after
   greeting Carol — never actually heard the answer. Fix: GOAL
   ACHIEVED now requires "the rep EXPLICITLY answered the goal
   question". Empty input now ALWAYS routes to wait, never hangup.

Also added rule 7: rep follow-up handler so when the rep asks for
phone-on-account or name-on-account, we answer concisely with the
real data.

Discovered 2026-05-13 in the 5-call live test — a mid-test deploy
killed 3 active calls' WSS audio capture because pm2 reload drops the
Media Streams lane. The AI agent loop survives (it's stateless from
Twilio's POV — they just keep POSTing /twilio/gather), but the PCM
stream goes dark and the MP3 finalize never fires.

scripts/predeploy-guard.sh queries prod's calls.json for any call in
status dialing/on_hold/connected, prints them, and exits 2 unless
FORCE_DEPLOY=1 is set. Hooked in via .deploy.conf's new PREDEPLOY_HOOK
variable, which the shared deploy.sh (also in this commit) now runs
before the rsync step.

Stale-on-hold cleanup: 3 Wells Fargo calls (6o3PMv5s, npTGTE7T,
GNNCY2co) had been stuck in status=on_hold since 2026-05-12 because
their /twilio/status callbacks were lost to a prior pm2 restart. One-
shot Twilio API query confirmed all 3 actually completed (45/125/106s
durations) — synced to status=done.

Audio path proof (post-this-commit): NIST call lbd8VV_r and Steve's
MacEnthusiasts quick-dial call 7pCAWx5A both produced full MP3 +
whisper.cpp transcript. The /calls inline UI works end-to-end.

Per Steve's "allow user to dial from here and listen" — the /calls page
now does all three:
- Top-of-page Quick dial form (business name + phone + goal). POSTs
  /api/quick-dial which uses immediate-dial path so call moves the
  instant the form returns.
- Per-row "🔊 Listen live →" chip when status is queued/dialing/
  on_hold/connected, links to /calls/<id>/live (gesture-unlock + WSS).
- Per-row inline <audio> when status is done/failed/completed (hidden
  unless /has-recording confirms MP3 exists).
- Per-row "📝 transcript" toggle button (hidden unless /has-transcript
  says it exists). Click to expand inline below the audio; renders
  segments with start-time prefixes, falls back to raw text or JSON.
- Auto-poll status every 5s so rows transition live→archived without
  page reload.

routes/public.js: new POST /api/quick-dial handler, validates body,
falls back to env phone if user has no email_phone, kicks immediate
dial in-process, returns {ok, id}.

Tick 6 work:
- lib/admin-gate.js: extract requireAdmin middleware. Pure function,
  unit-testable, no Express deps.
- test/admin-gate.test.js: 6/6 — no-user/wrong-role/no-role/case-sensitive
  all 403; role==='admin' calls next(). Asserts 404 template body so the
  status leaks 403 but the body looks like a 404 (no enumeration).
- routes/listen.js: re-add /api/calls/<id>/has-recording (lost in an
  earlier external rollback — was documented in ead4134 but the route
  body never made it to disk), and add new /api/calls/<id>/has-transcript.
  Both owner-scoped via data.getCall, cheap existence probes.
- views/public/calls.ejs: each row now shows a "📝 transcript" chip
  next to the inline player when /has-transcript says it exists.
- routes/admin.js: delegate to lib/admin-gate.requireAdmin.
- package.json: npm test runs orphan-recordings (8/8) + admin-gate (6/6).

YOLO tick 6 — all reversible, local-only.

- lib/orphan-recordings.js: scanOrphans({dir, knownIds}) extracted from
  routes/admin.js. No Express deps so it can be unit-tested in isolation.
  Accepts optional knownIds injection for tests; defaults to live calls.json.
- routes/admin.js: delegates to scanOrphans(), 20 lines lighter.
- test/orphan-recordings.test.js: 8 assertions in a temp dir —
  returns-exactly-N, mtime-desc-sort, known-ID exclusion, .pcm rejection,
  empty/nonexistent dirs. npm test exits 0 on full pass.
- package.json: wire "npm test" script.

Tick 5 validation findings:
- validate-twiml.js: 11/11 green on UozqdFJg + JsAqO2by (two different
  prod call IDs); confirms inline-Gather/no-Redirect/inbound_track holds
  across calls, not just US33rQWN.

YOLO tick 5 — all reversible, local-only.

- routes/admin.js: new requireAdmin-gated route lists MP3s in
  data/recordings/ with no matching call_id. Two orphans currently
  on prod (D5y8T2YV, gdRSfYO5) — leftovers from a pre-RSYNC_EXTRA_
  EXCLUDES rsync wipe. Inline <audio> for each so Steve can ID by
  ear, then delete/relink manually.
- server.js: mount /admin behind requireOwner; admin.js double-checks
  req.user.role === 'admin' per-route.
- lib/listen-bridge.js: external add of TwilioProxy UA guard on
  /stream WS lane had a regex that wouldn't match the actual UA
  ("Twilio.TmeWs/1.0" per nginx logs) — would have broken every live
  call. Fixed regex to match both observed Twilio UA forms while
  still blocking everything else.
- Plus external add of orphan-PCM recovery on boot — finalizes any
  .pcm files left from a mid-call process death. Kept.

YOLO tick 4 — all reversible, local-only.

POSTs /twilio/twiml/<call_id> against prod and asserts:
- track="inbound_track" on <Stream>
- no <Redirect> (would tear down <Stream>)
- exactly one <Gather> with actionOnEmptyResult=true
- gather action URL matches /twilio/gather/<id>
- announce present (Play or Say fallback)

Used in YOLO tick 3 to confirm cd10f95 deploy was correct on prod —
returned 11/11 green. Exit code = 0 on all-pass for CI use.

- views/public/calls.ejs: each done/failed/completed row now renders an
  <audio controls> inline. JS probes /api/calls/<id>/has-recording on
  load and only reveals the audio block if MP3 exists. Skips the rows
  with no recording (dry-run calls, lost-audio calls like DTCI7PuZ).
- routes/listen.js: new GET /api/calls/<id>/has-recording returns
  {ok, exists, size_bytes} — cheap probe without streaming.
- views/public/listen.ejs: minor button-text consistency fix on disconnect.
- server.js: pre-warm Ollama at startup with keep_alive=24h so first
  real call doesn't hit BUTLR_LLM_TIMEOUT_MS on cold cache.

YOLO tick 2 — all reversible, local-only.

Three fixes that together unblock live audio on Steve's calls:

1. routes/twilio-webhooks.js — DROP <Redirect> after Gather. Redirect ends
   the TwiML execution context and tears down <Start><Stream/>. Switch to
   self-loop inline Gather: /gather POST returns a new <Say> + inline
   <Gather> in the same response, so the stream persists across all turns
   of the conversation. Also add track="inbound_track" to <Stream> — the
   default both_tracks mixes Butlr's outbound TTS into the listen feed.
   Plus action-first system prompt with worked examples for press/wait/
   escalate/hangup so qwen3 doesn't deflect with "One moment please".

2. views/public/call-live.ejs — beep on status=connected now guards on
   window.__butlrAudioCtx (only fires if user already clicked "Listen live"
   to satisfy Chrome's autoplay gesture policy). Listen-live block exposes
   audioCtx as window.__butlrAudioCtx so the guard works.

3. server.js — add uncaughtException + unhandledRejection + signal
   handlers. pm2 was restarting butlr ~5×/10min on 2026-05-12 with clean
   SIGINT exits and no stack traces; explicit handlers surface the
   cause on next crash.

Also: views/public/listen.ejs split into live/archive sections so the
post-call MP3 player auto-shows once call.status hits done/failed.

After Steve's 'i heard nothing' report, traced through Twilio Alerts
API → nginx access logs → pm2 logs to confirm Media Streams audio path
is actually working post-StatusCallbackEvent fix.

The latest call US33rQWN captured 44s of audio to data/recordings/
and triggered the AI agent loop. Steve heard nothing only because he
wasn't on the live listen page during the call.

  • scripts/finalize-recordings.js — convert leftover .pcm → .mp3 via
    ffmpeg on every deploy, so a crashed/restarted pm2 doesn't strand
    raw PCM. Idempotent — skips when .mp3 already exists.
  • scripts/diagnose-audio.js + diagnose-wss.js — kept for future
    audit. Print pm2 logs, nginx access/error, Twilio Recordings API.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Twilio's REST API rejects 'initiated ringing answered completed' as
a single string with error 21626 (Invalid Events) and silently falls
back to firing only the 'completed' webhook. This is exactly why our
/twilio/status/:id route never received ringing/answered events all
day. Fix: use URLSearchParams.append() to pass StatusCallbackEvent
as a repeated param.

Found by checking Monitor v1 Alerts API on the 23:45 UTC calls today.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF

After today's data-loss incident (prod's calls.json was overwritten by
local test fixture during an earlier rsync --delete), reinforce the
RSYNC_EXTRA_EXCLUDES list and document the why so a future deploy.conf
diff doesn't quietly drop the excludes again.

Restored 4 Wells Fargo records from ~/Desktop/butlr-calls-LATEST.json
in the prior deploy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

When users.json is wiped between deploys (which happened during the
v2 rollout — fixed by RSYNC_EXTRA_EXCLUDES in .deploy.conf), existing
calls keep pointing at the old admin's user_id. The new admin gets a
fresh id, and the calls become invisible to Steve. This pass detects
user_ids missing from users.json and re-stamps them to the current
legacy admin. Idempotent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Replace single BUTLR_OWNER_TOKEN with per-user accounts backed by
data/users.json (atomic writes, bcrypt password hashes, 64-char hex
owner_tokens used as cookie values).

  • lib/users.js — create/verify/lookup, bcrypt cost 12, sha256-keyed
    token lookup for constant-time auth without bcrypt per-request
  • lib/owner-auth.js — req.user populated from cookie → users.findByOwnerToken;
    new GET/POST /signup + /login forms with email + password
  • lib/data.js — addCall/listCalls/getCall all take userId now;
    rows include user_id; getCall returns null when user_id mismatches
    (404 not 403, no oracle); listCallsUnscoped/getCallUnscoped reserved
    for Twilio webhook + background worker contexts only
  • server.js — hard-gates /calls, /listen, /api/calls, /new behind
    requireOwner; seeds legacy BUTLR_OWNER_TOKEN as admin user on boot
  • scripts/migrate-calls-to-user.js — one-shot stamp existing calls
    with legacy admin user_id (idempotent, dry-run flag, backs up
    calls.json before write)

Backward compat: existing browser cookies keep working because the
seeded admin's owner_token == BUTLR_OWNER_TOKEN env value.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

pm2 runs butlr from /root/public-projects/butlr (not the original
holdforme path). Earlier deploys rsynced to /root/Projects/holdforme
which was a dead mirror, so new routes never reached the live process.
Confirmed via embedded pm2 jlist discovery.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds two Twilio webhook fallback routes used by the +16067130489
incoming-phone-number itself (not per-call):
  POST/GET /twilio/voice-default  — polite inbound rejection TwiML
  POST     /twilio/status-default — number-level status log-only

Wired via Twilio REST API on the number, so callbacks land at
butlr.agentabrams.com regardless of whether a worker has the right
PUBLIC_URL set. Belt-and-suspenders for the 2026-05-12 outage where
the local .env had a stale trycloudflare PUBLIC_URL.

Also adds .deploy.conf so /deploy can ship to Kamatera, and gitignores
runtime artifact dirs (data/recordings, transcripts, audio-cache,
menu-map) to prevent committed test debris.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Persists every Twilio MediaStream to data/recordings/<id>.pcm even
when no browser is connected. On close, shells out to ffmpeg to
produce an MP3 + (optionally) kicks local Whisper. Prevents repeat of
the 2026-05-12 DTCI7PuZ Wells Fargo loss where audio path failed and
the call left zero artifact on disk.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Fixes the GPTBot exposure at 22:51:46 UTC where /calls/DTCI7PuZ?reveal=1
returned the decrypted Wells Fargo account number (0209203652) in plain
HTML to anyone with the call ID — no auth, no robots.txt, no noindex.

Changes:
1. lib/owner-auth.js — new middleware. Gates /calls, /calls/:id,
   /listen/:id, /api/calls/:id/* behind a BUTLR_OWNER_TOKEN cookie
   (or ?owner_token=). Constant-time compare. /login route renders a
   paste-token form. /logout clears.
2. routes/public.js — reveal=1 now requires req.ownerAuthed=true.
   Always sets X-Robots-Tag: noindex.
3. server.js — mounts /login + /logout publicly, then requireOwner
   on every call-data surface BEFORE the route handlers see the
   request. Soft-auth everywhere else for view locals.
4. public/robots.txt — Disallow: /calls /listen /api /twilio /login.
   Explicit Disallow: / for GPTBot, ChatGPT-User, OAI-SearchBot,
   ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User,
   CCBot, Bytespider, Amazonbot, cohere-ai, Google-Extended,
   FacebookBot, Applebot-Extended, Diffbot, ImagesiftBot, TimpiBot.
5. views/partials/head.ejs — noindex,nofollow,noarchive,nosnippet
   meta on every page by default; routes opt in to indexing.

Verified post-deploy:
  - GET /calls/DTCI7PuZ?reveal=1 (no cookie) → 302 /login (was 200 + acct#)
  - GET /api/calls/DTCI7PuZ/status (no cookie) → 302 /login (was 200 JSON)
  - GET /listen/DTCI7PuZ (no cookie) → 302 /login (was 200 page)
  - GET /robots.txt → 200 (was 404)
  - GET /calls/DTCI7PuZ?reveal=1 + owner cookie → 200 + acct visible (owner OK)
  - GET / (no cookie) → 200 (homepage still public)

BUTLR_OWNER_TOKEN registered with /secrets (digest ...da8c:693bb5f9).
Token persisted at /tmp/butlr_owner_token + Kamatera .env.

/api/calls/:id/agent-log greps pm2 logs for [ai-agent] heard/reply lines and
returns parsed turns. Listen page polls every 3s and renders bubbles
(They said / Butlr replied). Works while Twilio Media Streams is disabled
at the account tier — gives Steve real-time visibility into the call
without depending on the audio bridge.

DTCI7PuZ Wells Fargo call confirmed working: turn 1 heard recording
disclaimer, Butlr replied requesting $100K limit increase.

Adds the MacEnthusiasts row in data/businesses.json with ivr_dtmf:
[{delay_sec:6, digit:'2'}]. The twimlHandler() picks this up via
ivrDtmfForPhone() and emits <Pause length=6/><Play digits=2/> after
the consent announcement.

Discovered the IVR layout from the 2026-05-12 test-call transcript
(CAb64d5b1ee, whisper.cpp local) — 1=rentals, 2=sales, 3=service,
4=accounting, 5=address/hours, *=directory, #=repeat.

Reason — 2026-05-12 MacEnthusiasts test call (CAb64d5b1ee, 93s) landed
fine, MP3 saved, but OpenAI Whisper returned 429 insufficient_quota and
left us blind. whisper-cpp on the local M2 Max transcribed the same file
in 1.6s. Local-first removes the OpenAI quota dependency entirely.

transcribeLocal() — runs ffmpeg → whisper-cli with ggml-base.en model.
transcribeRemote() — original OpenAI path, kept as a fallback only.
transcribe() — dispatcher: try local, fall back to remote, error if both fail.

Env knobs (all optional, sensible Homebrew defaults):
  WHISPER_CLI_PATH=/opt/homebrew/bin/whisper-cli
  FFMPEG_PATH=/opt/homebrew/bin/ffmpeg
  WHISPER_MODEL_PATH=~/.cache/whisper-cpp/ggml-base.en.bin

twilioAuth() resolver picks the best available credential:
  1) TWILIO_API_KEY_SID + TWILIO_API_KEY_SECRET (preferred, revocable)
  2) TWILIO_ACCOUNT_SID + TWILIO_AUTH_TOKEN (fallback, master cred)
  3) error if neither

Wires through dialCall(), sendSms(), and transcribe.downloadRecording().
log() now records which auth mode was used per call.

End-to-end pipeline for Phase 2 going live:

  lib/elevenlabs.js   — TTS for the announce preamble, sha256 cached,
                        graceful fallback to Twilio <Say> on 401/network err
  lib/listen-bridge.js — single ws server, two lanes:
                        /stream/<id>   = Twilio Media Streams (µ-law → PCM16)
                        /listen-ws/<id> = browser playback (raw PCM frames)
  lib/transcribe.js   — Twilio Recording download (basic-auth) + Whisper
                        verbose_json, stored at data/transcripts/<id>.json
  lib/twilio.js       — LIVE dialCall() now real, POSTs to Twilio Calls API
                        with Url=/twilio/twiml/<id>, status/AMD callbacks,
                        AsyncAmd for human-vs-voicemail branch
  routes/twilio-webhooks.js — /twiml /status /recording /amd /audio/announce
                              TwiML: <Stream> + announce + <Record/> with
                              recordingStatusCallback → kicks Whisper
  routes/listen.js    — /listen/<id> browser page + /calls/<id>/{transcript,recording}
  views/public/listen.ejs — Web Audio API playback page with consent banner
  server.js           — http.createServer, attachListenBridge(server),
                        CSP allows ws:/wss: + media-src blob:
  lib/data.js         — setTwilioSid/setRecordingMeta/setTranscriptMeta

All gated by feedback_holdforme_always_announce.md — every outbound call
opens with two-party-consent announcement, no skip flag, ever.