[object Object]

← back to Dw Domain Fleet

deploy-fleet: smoke test cried wolf on every healthy deploy (TK-11307)

4f65cef945ae295c5aa980f0766c0d1fa03da609 · 2026-09-11 08:27:47 -0700 · Steve Abrams

The origin probe curls http://127.0.0.1/health and demanded 200, but certbot
rewrites each vhost to redirect HTTP->HTTPS, so a healthy host answers 301.
Today's deploy therefore reported "43 failures" while all 43 domains were
serving 200 over HTTPS with correct per-host certs.

A check that is red when everything is fine is as corrosive as one that is
green when it isn't -- it trains everyone to ignore the only signal that would
catch a real outage. Now accepts 2xx/301/302 at the origin and follows through
to the public HTTPS URL, so a pass means the site actually answers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011FAPArHdMKRiNHqJiorUFm

Files touched

Diff

commit 4f65cef945ae295c5aa980f0766c0d1fa03da609
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Fri Sep 11 08:27:47 2026 -0700

    deploy-fleet: smoke test cried wolf on every healthy deploy (TK-11307)
    
    The origin probe curls http://127.0.0.1/health and demanded 200, but certbot
    rewrites each vhost to redirect HTTP->HTTPS, so a healthy host answers 301.
    Today's deploy therefore reported "43 failures" while all 43 domains were
    serving 200 over HTTPS with correct per-host certs.
    
    A check that is red when everything is fine is as corrosive as one that is
    green when it isn't -- it trains everyone to ignore the only signal that would
    catch a real outage. Now accepts 2xx/301/302 at the origin and follows through
    to the public HTTPS URL, so a pass means the site actually answers.
    
    Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_011FAPArHdMKRiNHqJiorUFm
---
 scripts/deploy-fleet.sh | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/scripts/deploy-fleet.sh b/scripts/deploy-fleet.sh
index bfea484..dbe346d 100755
--- a/scripts/deploy-fleet.sh
+++ b/scripts/deploy-fleet.sh
@@ -30,8 +30,19 @@ ssh "$HOST" "for f in $REMOTE/data/nginx/*.conf; do \
 echo "==> smoke test (origin, via Host header)"
 FAIL=0
 for d in $(ls "$LOCAL/data/nginx/" | sed 's/.conf//'); do
+  # 301 is HEALTHY here: certbot rewrites each vhost to redirect HTTP->HTTPS, so an origin
+  # probe over plain HTTP answers 301, not 200. Demanding 200 made every deploy report
+  # "43 failures" on a completely healthy fleet — a false alarm that trains everyone to
+  # ignore this smoke test, which is how a real outage gets waved through. Accept 2xx/301,
+  # and follow the redirect to confirm the app really answers.
   CODE=$(ssh "$HOST" "curl -s -o /dev/null -w '%{http_code}' -H 'Host: $d' http://127.0.0.1/health" || echo 000)
-  if [ "$CODE" != "200" ]; then echo "  FAIL $d ($CODE)"; FAIL=$((FAIL+1)); else echo "  ok   $d"; fi
+  case "$CODE" in
+    200|301|302)
+      LIVE=$(curl -s --http1.1 --max-time 12 -o /dev/null -w '%{http_code}' "https://$d/" 2>/dev/null || echo 000)
+      if [ "$LIVE" = "200" ]; then echo "  ok   $d (origin $CODE, https $LIVE)"
+      else echo "  FAIL $d (origin $CODE but https $LIVE)"; FAIL=$((FAIL+1)); fi ;;
+    *) echo "  FAIL $d ($CODE)"; FAIL=$((FAIL+1)) ;;
+  esac
 done
 echo "==> deploy complete · $FAIL failures"
 exit $FAIL

← 11085c2 de-mail-merge articles for thehotelwallpaper + 3 sister mone  ·  back to Dw Domain Fleet  ·  auto-data-snapshot: 2026-09-11T08:57:34 (1 data files) — dat b363266 →