Creative ideas + design notes
Commits with substantial prose (≥120 chars) — the rationale behind each move.
a58d942 · 2026-08-18 · abrams-report: add scheduled scrape cron (3x/day, local $0)
TK-10655 (Steve: 'run on cron to keep current'). abrams-report had no scheduler
(hadn't scraped since May). Adds scripts/run-scrape.sh + LaunchAgent
com.steve.abrams-report-scrape (07:15/13:15/19:15 daily, headless, ProcessType
Background). RSS + all HTML + Schumacher refresh at $0 via local Chrome; Wallquest
(CF-hard) needs a headed manual 'npm run scrape' — documented in the wrapper.
Plist mirrored under deploy/launchd/ for reinstall. Scrape logs gitignored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
e85230b · 2026-08-18 · abrams-report: bump better-sqlite3 9.4.3 -> 13.0.3 for node v26 support
The pinned ^9.4.3 had no prebuilt binary for node v26 (ABI 147) and its source
won't compile against node 26's V8 API — so the DB write step was broken (app
hadn't scraped since May). 13.0.3 ships a node-26 prebuilt; loads instantly,
same Database API (no app-code change). Full scrape now runs end-to-end: 300
headlines written, $0 (local Chrome for the 2 anti-bot sources). Completes the
TK-10655 reroute (fetch path was already proven; this unblocks the write path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4c1bd95 · 2026-08-18 · abrams-report: reroute anti-bot scrapes off Browserbase to local real Chrome ($0)
TK-10655 (Steve-approved): the 2 Cloudflare-protected sources (Schumacher blog,
Wallquest sitemap) went through paid Browserbase (~$46.80 all-time, the #1 spend
line). Replace with scrapers/fetch-local-browser.js — Playwright driving the
installed Google Chrome via a persistent profile (residential IP + real-Chrome
fingerprint clears CF's managed challenge where headless/datacenter is bounced).
Raw XML captured from page.goto().text() before Chrome's viewer transforms it.
Verified live: Schumacher -> 13 blog links, Wallquest -> 394 sitemap URLs, $0.
Deleted fetch-browserbase.js + dropped @browserbasehq/sdk dep (now unused).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
a0688b4 · 2026-05-19 · chore(refactor): broaden .gitignore for snapshot files + add .bak/.pre 404 guard middleware
- .gitignore now excludes *.bak, *.bak.*, *.pre-*, *.orig, *.rej, *~, .next/
so accidental snapshot files never enter the tree.
- New Express middleware (before express.static) returns 404 for any path
matching .bak / .pre- / .orig / .rej / trailing-tilde patterns, so even if
a snapshot file slips into public/ it can't serve.
No behavior change for normal traffic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7257efe · 2026-05-11 · fix(schumacher): route through Browserbase
CloudFront 403s data-center IPs (Kamatera prod) but allows residential
(Mac2). Force Browserbase so the scheduled scrape works on Kamatera.
2e118fb · 2026-05-11 · fix(fetch-browserbase): prefer project-local .env over home-dir paths
abrams-report's own .env (written by secrets-manager) is the canonical
location on both Mac2 and Kamatera. Move it to the front of the
candidate list so a per-project key rotation propagates immediately
instead of being shadowed by an older skill .env on Mac2 or /root/.env
on Kamatera.
20113db · 2026-05-11 · deps(wallpaper): add @browserbasehq/sdk + playwright-core for Wallquest scrape
Wallquest is behind Cloudflare bot protection; only path to scrape its
sitemap is via Browserbase's residential-IP Chromium cluster. Adding
the SDK + playwright-core as direct deps so 'npm install --omit=dev'
on Kamatera pulls them in. fetch-browserbase.js falls back to loading
from ~/.claude/skills/browserbase/node_modules on Mac2 when these are
absent, so this dep addition is additive, not a hard requirement on
the dev box.
42ab728 · 2026-05-11 · fix(wallpaper-watch): rewrite all 10 wallpaper-house scrapers
Each source now yields 3-6 real headlines per run. Changes:
- Schumacher: switched fschumacher.com/search to schumacher.com/blog
(parent brand publishes a real editorial blog; selector 'h3 a').
- Thibaut: news/blog paths all 404, switched to /sitemaps/collections.xml
with sitemap parser (lastmod = newest collection drop).
- Phillip Jeffries: /news redirects to /shop; switched to blog subdomain
blog.phillipjeffries.com with 'h2 a' selector.
- Scalamandre: /news page literally says 'No news!'; switched to flat
sitemap.xml filtered to /brands/.+/.+\.html collection pages, sorted
by lastmod (newest collection updates).
- Maya Romanoff: kept /news/ URL but tightened selector to 'h2 a' and
link filter to /news/<slug>/ (was matching every nav link).
- Arte International: switched from /en/collections to /en/stories with
selector a[href*=/stories/] and slug filter (rejects ?category= nav).
- Brewster: /new-arrivals 404s; switched to /blog with 'h3 a' selector
and a host-anchored slug filter.
- York: tightened to a[href*=/blog/] + excludeFilter for tag/month/rss
(was matching navigation tag clouds).
- Wallquest: blocked by Cloudflare bot challenge AND publishes nothing
to /news or /blog. Routed through Browserbase, fetching sitemap.xml
via the same-origin fetch (page.evaluate) to bypass Chrome's XML
pretty-print wrapper; filtered slugs to human-readable collection
names (≥1 dash, no SKU codes).
- Cole & Son: /en/news redirects to homepage; switched to
sitemap_blogs_1.xml, parse <image:title> as headline.
Infrastructure:
- New scrapers/fetch-browserbase.js for cloud-Chromium fetches; loads
creds from ~/.claude/skills/browserbase/.env, /root/.env, or process.
When URL contains 'sitemap' or '.xml', uses page.evaluate(fetch) to
return raw XML (not Chrome's wrapper HTML).
- scrape-html.js: real Chrome UA (was scraper bot UA — many sites
403/404 a bot), redirect: 'follow', new parsers for 'sitemap' and
'sitemap-index', new 'excludeFilter' for rejecting category/tag/etc,
cap of 6 headlines per source (per copyright-safety brief).
