← back to Allnewsdaily
check-live: direct-first / residential-proxy-fallback for resilient YT live-check
ba93d26be41d937fca0249f678a999314dd07fc3 · 2026-09-12 07:57:27 -0700 · Steve Abrams
Prod's datacenter IP gets the real YouTube page most of the time (the wall is
intermittent), so try direct first (free, no proxy bandwidth) and fall back to
the Webshare static-residential proxy only when the page is walled (no
ytInitialPlayerResponse). Degrades gracefully — if the proxy is down/lapsed,
direct still covers every unwalled fetch instead of a hard 0-live outage.
fetchHtml now takes an explicit agent arg. TK-11340
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KdYQbarNfqKNkgQLQffDS4
Files touched
Diff
commit ba93d26be41d937fca0249f678a999314dd07fc3
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Sat Sep 12 07:57:27 2026 -0700
check-live: direct-first / residential-proxy-fallback for resilient YT live-check
Prod's datacenter IP gets the real YouTube page most of the time (the wall is
intermittent), so try direct first (free, no proxy bandwidth) and fall back to
the Webshare static-residential proxy only when the page is walled (no
ytInitialPlayerResponse). Degrades gracefully — if the proxy is down/lapsed,
direct still covers every unwalled fetch instead of a hard 0-live outage.
fetchHtml now takes an explicit agent arg. TK-11340
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KdYQbarNfqKNkgQLQffDS4
---
scripts/check-live.js | 26 +++++++++++++++++++++++---
1 file changed, 23 insertions(+), 3 deletions(-)
diff --git a/scripts/check-live.js b/scripts/check-live.js
index 3367816..c57f9e0 100644
--- a/scripts/check-live.js
+++ b/scripts/check-live.js
@@ -37,7 +37,7 @@ const UA =
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ' +
'(KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36';
-function fetchHtml(url, timeoutMs = 10000) {
+function fetchHtml(url, timeoutMs = 10000, agent = proxyAgent) {
return new Promise((resolve, reject) => {
let done = false;
let req;
@@ -54,7 +54,7 @@ function fetchHtml(url, timeoutMs = 10000) {
port: u.port || 443,
path: u.pathname + u.search,
method: 'GET',
- agent: proxyAgent || undefined,
+ agent: agent || undefined,
headers: {
'User-Agent': UA,
'Accept-Language': 'en-US,en;q=0.9',
@@ -132,11 +132,31 @@ async function checkOne(outlet) {
return { id: outlet.id, isLive: false, videoId: null, skipped: true };
}
const url = `https://www.youtube.com/channel/${outlet.youtube}/live?gl=US&hl=en`;
+ // DIRECT-FIRST, PROXY-FALLBACK (TK-11340): prod's datacenter IP gets the real YouTube
+ // watch page MOST of the time — the consent/bot wall is intermittent — so try direct
+ // (free, no proxy bandwidth) and only fall back to the residential proxy when the page
+ // is walled. A real watch page (live OR not) always carries ytInitialPlayerResponse; a
+ // consent/bot wall does not — that's the reliable "walled" signal. This also degrades
+ // gracefully: if the proxy is ever down/lapsed, direct still covers every unwalled fetch
+ // instead of a hard 0-live outage.
try {
- const html = await fetchHtml(url);
+ let html = await fetchHtml(url, 10000, null); // direct (no proxy)
+ if (!/ytInitialPlayerResponse/.test(html) && proxyAgent) {
+ html = await fetchHtml(url, 10000, proxyAgent); // walled → residential fallback
+ }
const { isLive, videoId } = detectLive(html);
return { id: outlet.id, isLive, videoId };
} catch (e) {
+ // Direct threw (timeout/reset — often itself a soft block): try the proxy before giving up.
+ if (proxyAgent) {
+ try {
+ const html = await fetchHtml(url, 10000, proxyAgent);
+ const { isLive, videoId } = detectLive(html);
+ return { id: outlet.id, isLive, videoId };
+ } catch (e2) {
+ return { id: outlet.id, isLive: false, videoId: null, error: e2.message };
+ }
+ }
return { id: outlet.id, isLive: false, videoId: null, error: e.message };
}
}
← acde7bd Fix live detection through residential proxy: anchor on live
·
back to Allnewsdaily
·
chore: commit code deployed in 2026-09-15 fleet push c1ab0a9 →