[object Object]

← back to Abrams Report

abrams-report: add scheduled scrape cron (3x/day, local $0)

a58d942069278ba9944553303bdd6cc4676ce52e · 2026-08-18 12:21:23 -0700 · Steve Abrams

TK-10655 (Steve: 'run on cron to keep current'). abrams-report had no scheduler
(hadn't scraped since May). Adds scripts/run-scrape.sh + LaunchAgent
com.steve.abrams-report-scrape (07:15/13:15/19:15 daily, headless, ProcessType
Background). RSS + all HTML + Schumacher refresh at $0 via local Chrome; Wallquest
(CF-hard) needs a headed manual 'npm run scrape' — documented in the wrapper.
Plist mirrored under deploy/launchd/ for reinstall. Scrape logs gitignored.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

Diff

commit a58d942069278ba9944553303bdd6cc4676ce52e
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Tue Aug 18 12:21:23 2026 -0700

    abrams-report: add scheduled scrape cron (3x/day, local $0)
    
    TK-10655 (Steve: 'run on cron to keep current'). abrams-report had no scheduler
    (hadn't scraped since May). Adds scripts/run-scrape.sh + LaunchAgent
    com.steve.abrams-report-scrape (07:15/13:15/19:15 daily, headless, ProcessType
    Background). RSS + all HTML + Schumacher refresh at $0 via local Chrome; Wallquest
    (CF-hard) needs a headed manual 'npm run scrape' — documented in the wrapper.
    Plist mirrored under deploy/launchd/ for reinstall. Scrape logs gitignored.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 .gitignore                                         |  5 ++++
 .../launchd/com.steve.abrams-report-scrape.plist   | 27 ++++++++++++++++++++++
 scripts/run-scrape.sh                              | 16 +++++++++++++
 3 files changed, 48 insertions(+)

diff --git a/.gitignore b/.gitignore
index 1977b5d..bdd110d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,3 +16,8 @@ data/*.db-wal
 *.orig
 *.rej
 *~
+
+# scrape logs
+data/scrape.log
+data/scrape.out.log
+data/scrape.err.log
diff --git a/deploy/launchd/com.steve.abrams-report-scrape.plist b/deploy/launchd/com.steve.abrams-report-scrape.plist
new file mode 100644
index 0000000..dce4ef3
--- /dev/null
+++ b/deploy/launchd/com.steve.abrams-report-scrape.plist
@@ -0,0 +1,27 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+  <key>Label</key>
+  <string>com.steve.abrams-report-scrape</string>
+  <key>ProgramArguments</key>
+  <array>
+    <string>/bin/zsh</string>
+    <string>/Users/macstudio3/Projects/abrams-report/scripts/run-scrape.sh</string>
+  </array>
+  <key>StartCalendarInterval</key>
+  <array>
+    <dict><key>Hour</key><integer>7</integer><key>Minute</key><integer>15</integer></dict>
+    <dict><key>Hour</key><integer>13</integer><key>Minute</key><integer>15</integer></dict>
+    <dict><key>Hour</key><integer>19</integer><key>Minute</key><integer>15</integer></dict>
+  </array>
+  <key>StandardOutPath</key>
+  <string>/Users/macstudio3/Projects/abrams-report/data/scrape.out.log</string>
+  <key>StandardErrorPath</key>
+  <string>/Users/macstudio3/Projects/abrams-report/data/scrape.err.log</string>
+  <key>RunAtLoad</key>
+  <false/>
+  <key>ProcessType</key>
+  <string>Background</string>
+</dict>
+</plist>
diff --git a/scripts/run-scrape.sh b/scripts/run-scrape.sh
new file mode 100755
index 0000000..d32d736
--- /dev/null
+++ b/scripts/run-scrape.sh
@@ -0,0 +1,16 @@
+#!/bin/zsh
+# abrams-report headline scrape (RSS + HTML). The 2 anti-bot sources (Schumacher, Wallquest) go
+# through LOCAL real Chrome — $0, no Browserbase (TK-10655).
+#
+# Unattended = HEADLESS (ABR_BROWSER_HEADLESS=1): clean, never steals window focus on Steve's active
+# workstation. Tradeoff: Wallquest's Cloudflare "managed challenge" only auto-clears for a HEADED
+# Chrome, so Wallquest degrades to 0 in the cron run (its "headlines" are product-sitemap slugs — the
+# least newsworthy source; Schumacher + all 30 RSS/HTML sources work headless). To capture Wallquest,
+# run headed manually:  cd ~/Projects/abrams-report && npm run scrape
+export PATH="/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin"
+export ABR_BROWSER_HEADLESS=1
+cd "$HOME/Projects/abrams-report" || exit 1
+LOG="$HOME/Projects/abrams-report/data/scrape.log"
+echo "[$(date '+%F %T')] scrape start" >> "$LOG"
+/opt/homebrew/bin/node scrapers/scrape-all.js >> "$LOG" 2>&1
+echo "[$(date '+%F %T')] scrape done (exit $?)" >> "$LOG"

← 4127e9b abrams-report: rescrape — refresh headlines (300 new, all-lo  ·  back to Abrams Report  ·  auto-data-snapshot: 2026-08-18T19:20:00 (1 data files) — dat cd8475f →