← back to Abrams Report
fix(wallpaper-watch): rewrite all 10 wallpaper-house scrapers
42ab7288283e66563e36fa75f3a2284cdbe5d55c · 2026-05-11 20:38:16 -0700 · Steve Abrams
Each source now yields 3-6 real headlines per run. Changes:
- Schumacher: switched fschumacher.com/search to schumacher.com/blog
(parent brand publishes a real editorial blog; selector 'h3 a').
- Thibaut: news/blog paths all 404, switched to /sitemaps/collections.xml
with sitemap parser (lastmod = newest collection drop).
- Phillip Jeffries: /news redirects to /shop; switched to blog subdomain
blog.phillipjeffries.com with 'h2 a' selector.
- Scalamandre: /news page literally says 'No news!'; switched to flat
sitemap.xml filtered to /brands/.+/.+\.html collection pages, sorted
by lastmod (newest collection updates).
- Maya Romanoff: kept /news/ URL but tightened selector to 'h2 a' and
link filter to /news/<slug>/ (was matching every nav link).
- Arte International: switched from /en/collections to /en/stories with
selector a[href*=/stories/] and slug filter (rejects ?category= nav).
- Brewster: /new-arrivals 404s; switched to /blog with 'h3 a' selector
and a host-anchored slug filter.
- York: tightened to a[href*=/blog/] + excludeFilter for tag/month/rss
(was matching navigation tag clouds).
- Wallquest: blocked by Cloudflare bot challenge AND publishes nothing
to /news or /blog. Routed through Browserbase, fetching sitemap.xml
via the same-origin fetch (page.evaluate) to bypass Chrome's XML
pretty-print wrapper; filtered slugs to human-readable collection
names (≥1 dash, no SKU codes).
- Cole & Son: /en/news redirects to homepage; switched to
sitemap_blogs_1.xml, parse <image:title> as headline.
Infrastructure:
- New scrapers/fetch-browserbase.js for cloud-Chromium fetches; loads
creds from ~/.claude/skills/browserbase/.env, /root/.env, or process.
When URL contains 'sitemap' or '.xml', uses page.evaluate(fetch) to
return raw XML (not Chrome's wrapper HTML).
- scrape-html.js: real Chrome UA (was scraper bot UA — many sites
403/404 a bot), redirect: 'follow', new parsers for 'sitemap' and
'sitemap-index', new 'excludeFilter' for rejecting category/tag/etc,
cap of 6 headlines per source (per copyright-safety brief).
Files touched
A scrapers/fetch-browserbase.jsM scrapers/scrape-html.jsM scrapers/sources.js
Diff
commit 42ab7288283e66563e36fa75f3a2284cdbe5d55c
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Mon May 11 20:38:16 2026 -0700
fix(wallpaper-watch): rewrite all 10 wallpaper-house scrapers
Each source now yields 3-6 real headlines per run. Changes:
- Schumacher: switched fschumacher.com/search to schumacher.com/blog
(parent brand publishes a real editorial blog; selector 'h3 a').
- Thibaut: news/blog paths all 404, switched to /sitemaps/collections.xml
with sitemap parser (lastmod = newest collection drop).
- Phillip Jeffries: /news redirects to /shop; switched to blog subdomain
blog.phillipjeffries.com with 'h2 a' selector.
- Scalamandre: /news page literally says 'No news!'; switched to flat
sitemap.xml filtered to /brands/.+/.+\.html collection pages, sorted
by lastmod (newest collection updates).
- Maya Romanoff: kept /news/ URL but tightened selector to 'h2 a' and
link filter to /news/<slug>/ (was matching every nav link).
- Arte International: switched from /en/collections to /en/stories with
selector a[href*=/stories/] and slug filter (rejects ?category= nav).
- Brewster: /new-arrivals 404s; switched to /blog with 'h3 a' selector
and a host-anchored slug filter.
- York: tightened to a[href*=/blog/] + excludeFilter for tag/month/rss
(was matching navigation tag clouds).
- Wallquest: blocked by Cloudflare bot challenge AND publishes nothing
to /news or /blog. Routed through Browserbase, fetching sitemap.xml
via the same-origin fetch (page.evaluate) to bypass Chrome's XML
pretty-print wrapper; filtered slugs to human-readable collection
names (≥1 dash, no SKU codes).
- Cole & Son: /en/news redirects to homepage; switched to
sitemap_blogs_1.xml, parse <image:title> as headline.
Infrastructure:
- New scrapers/fetch-browserbase.js for cloud-Chromium fetches; loads
creds from ~/.claude/skills/browserbase/.env, /root/.env, or process.
When URL contains 'sitemap' or '.xml', uses page.evaluate(fetch) to
return raw XML (not Chrome's wrapper HTML).
- scrape-html.js: real Chrome UA (was scraper bot UA — many sites
403/404 a bot), redirect: 'follow', new parsers for 'sitemap' and
'sitemap-index', new 'excludeFilter' for rejecting category/tag/etc,
cap of 6 headlines per source (per copyright-safety brief).
---
scrapers/fetch-browserbase.js | 94 ++++++++++++++++++++++++
scrapers/scrape-html.js | 161 ++++++++++++++++++++++++++++++++++--------
scrapers/sources.js | 83 +++++++++++++---------
3 files changed, 275 insertions(+), 63 deletions(-)
diff --git a/scrapers/fetch-browserbase.js b/scrapers/fetch-browserbase.js
new file mode 100644
index 0000000..4fd8636
--- /dev/null
+++ b/scrapers/fetch-browserbase.js
@@ -0,0 +1,94 @@
+'use strict';
+// Cloud Chromium fetch via Browserbase — bypasses Cloudflare / anti-bot for sites like Wallquest.
+// Credentials live in ~/.claude/skills/browserbase/.env on Mac2 and /root/.env on Kamatera.
+
+const path = require('path');
+const fs = require('fs');
+const os = require('os');
+
+// Try local skill env first, then ~/.env, then process.env
+function loadCreds() {
+ if (process.env.BROWSERBASE_API_KEY && process.env.BROWSERBASE_PROJECT_ID) {
+ return {
+ apiKey: process.env.BROWSERBASE_API_KEY,
+ projectId: process.env.BROWSERBASE_PROJECT_ID,
+ };
+ }
+ const candidates = [
+ path.join(os.homedir(), '.claude/skills/browserbase/.env'),
+ path.join(os.homedir(), '.env'),
+ '/root/.env',
+ ];
+ for (const f of candidates) {
+ try {
+ const txt = fs.readFileSync(f, 'utf8');
+ const obj = {};
+ txt.split('\n').forEach(line => {
+ const m = line.match(/^([A-Z_]+)=(.+)$/);
+ if (m) obj[m[1]] = m[2].replace(/^["']|["']$/g, '');
+ });
+ if (obj.BROWSERBASE_API_KEY && obj.BROWSERBASE_PROJECT_ID) {
+ return { apiKey: obj.BROWSERBASE_API_KEY, projectId: obj.BROWSERBASE_PROJECT_ID };
+ }
+ } catch {}
+ }
+ return null;
+}
+
+async function fetchViaBrowserbase(url) {
+ const creds = loadCreds();
+ if (!creds) throw new Error('Browserbase creds not found in env or .env files');
+
+ // Lazy-load to avoid pulling chromium when not needed
+ let Browserbase, chromium;
+ try {
+ Browserbase = require('@browserbasehq/sdk').default || require('@browserbasehq/sdk').Browserbase;
+ chromium = require('playwright-core').chromium;
+ } catch (e) {
+ // Fallback: load from the browserbase skill's node_modules
+ const skillNm = path.join(os.homedir(), '.claude/skills/browserbase/node_modules');
+ try {
+ Browserbase = require(path.join(skillNm, '@browserbasehq/sdk')).default;
+ chromium = require(path.join(skillNm, 'playwright-core')).chromium;
+ } catch (e2) {
+ throw new Error('Browserbase SDK not installed: ' + e.message);
+ }
+ }
+
+ const bb = new Browserbase({ apiKey: creds.apiKey });
+ const session = await bb.sessions.create({ projectId: creds.projectId });
+ const browser = await chromium.connectOverCDP(session.connectUrl);
+
+ try {
+ const ctx = browser.contexts()[0] || (await browser.newContext());
+ const page = ctx.pages()[0] || (await ctx.newPage());
+
+ await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 35000 });
+ // Give Cloudflare a moment to clear the challenge if any
+ await page.waitForTimeout(2500);
+
+ // If this is an XML/sitemap URL, Chrome wraps it in pretty-print HTML.
+ // Try fetching the raw resource through the page context (bypasses the wrapper
+ // and preserves the same-site cookies set by the initial navigation).
+ if (/\.xml(\?|$)|sitemap/i.test(url)) {
+ try {
+ const xml = await page.evaluate(async (u) => {
+ const r = await fetch(u, {
+ headers: { Accept: 'application/xml, text/xml, */*' },
+ });
+ return await r.text();
+ }, url);
+ if (xml && xml.length > 200 && xml.trim().startsWith('<')) {
+ return xml;
+ }
+ } catch {}
+ }
+
+ const html = await page.content();
+ return html;
+ } finally {
+ try { await browser.close(); } catch {}
+ }
+}
+
+module.exports = { fetchViaBrowserbase };
diff --git a/scrapers/scrape-html.js b/scrapers/scrape-html.js
index 4ad6490..737abe9 100644
--- a/scrapers/scrape-html.js
+++ b/scrapers/scrape-html.js
@@ -4,12 +4,20 @@ const cheerio = require('cheerio');
const crypto = require('crypto');
const { upsertHeadline, logScrape } = require('./db');
const { SCRAPE_SOURCES } = require('./sources');
+const { fetchViaBrowserbase } = require('./fetch-browserbase');
+
+// Use a real browser UA — many sites 403/404 a bot UA
+const BROWSER_UA =
+ 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ' +
+ '(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36';
const FETCH_OPTS = {
- timeout: 20000,
+ timeout: 25000,
+ redirect: 'follow',
headers: {
- 'User-Agent': 'Mozilla/5.0 (compatible; AbramsDrReport/1.0; +https://abramsreport.agentabrams.com)',
- 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
+ 'User-Agent': BROWSER_UA,
+ 'Accept':
+ 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
},
};
@@ -34,70 +42,165 @@ function resolveUrl(href, baseUrl) {
}
function cleanTitle(text) {
- return text
- .replace(/\s+/g, ' ')
+ return String(text || '')
.replace(/[\r\n\t]/g, ' ')
+ .replace(/\s+/g, ' ')
.trim()
.slice(0, 200);
}
-async function fetchHtml(url) {
+function slugToTitle(url) {
+ try {
+ const u = new URL(url);
+ const last = u.pathname.split('/').filter(Boolean).pop() || '';
+ return last
+ .replace(/[-_]+/g, ' ')
+ .replace(/\.\w+$/, '')
+ .replace(/\b\w/g, c => c.toUpperCase())
+ .slice(0, 100);
+ } catch {
+ return '';
+ }
+}
+
+async function fetchHtml(url, useBrowserbase = false) {
+ if (useBrowserbase) {
+ return fetchViaBrowserbase(url);
+ }
const res = await fetch(url, FETCH_OPTS);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
return res.text();
}
-async function scrapeSite(src) {
- let html;
- try {
- html = await fetchHtml(src.url);
- } catch (err) {
- console.error(`[html] ${src.key} failed: ${err.message}`);
- logScrape(src.key, 0, err.message);
- return 0;
+// ─── Sitemap parsing (Cole & Son uses this) ─────────────────────────────────
+function parseSitemapXml(xml, src) {
+ // Returns array of { title, url }
+ const $ = cheerio.load(xml, { xmlMode: true });
+ const items = [];
+
+ $('url').each((_, el) => {
+ const loc = $(el).find('loc').first().text().trim();
+ if (!loc) return;
+ if (src.linkFilter && !src.linkFilter.test(loc)) return;
+ if (src.excludeFilter && src.excludeFilter.test(loc)) return;
+
+ let title = '';
+ if (src.sitemapTitleFromImage) {
+ // <image:title> child gives a human-readable title
+ const imgTitle = $(el).find('image\\:title, title').first().text().trim();
+ if (imgTitle) title = imgTitle;
+ }
+ if (!title) title = slugToTitle(loc);
+ if (!title || title.length < 4) return;
+
+ const lastmod = $(el).find('lastmod').first().text().trim();
+ items.push({ title, url: loc, lastmod });
+ });
+
+ // Sort newest first if we have lastmod
+ items.sort((a, b) => (b.lastmod || '').localeCompare(a.lastmod || ''));
+ return items;
+}
+
+async function parseSitemapIndex(xml, src) {
+ // For Scalamandre: pick newest child sitemaps that match filter, then read entries
+ const $ = cheerio.load(xml, { xmlMode: true });
+ const childSitemaps = [];
+ $('sitemap').each((_, el) => {
+ const loc = $(el).find('loc').first().text().trim();
+ const lastmod = $(el).find('lastmod').first().text().trim();
+ if (!loc) return;
+ if (src.sitemapFilter && !src.sitemapFilter.test(loc)) return;
+ childSitemaps.push({ loc, lastmod });
+ });
+ childSitemaps.sort((a, b) => (b.lastmod || '').localeCompare(a.lastmod || ''));
+
+ // Limit to first 2 child sitemaps for cost
+ const items = [];
+ for (const cs of childSitemaps.slice(0, 2)) {
+ try {
+ const childXml = await fetchHtml(cs.loc);
+ items.push(...parseSitemapXml(childXml, src));
+ } catch (e) {
+ console.error(`[sitemap-index] ${src.key} child ${cs.loc} failed: ${e.message}`);
+ }
}
+ return items;
+}
+// ─── HTML scrape with selector + filters ────────────────────────────────────
+function parseHtml(html, src) {
const $ = cheerio.load(html);
const seen = new Set();
- let added = 0;
+ const items = [];
$(src.selector).each((_, el) => {
const href = $(el).attr('href');
if (!href) return;
+ if (href.startsWith('#')) return;
const absUrl = resolveUrl(href, src.baseUrl);
if (!absUrl) return;
if (seen.has(absUrl)) return;
- if (!src.linkFilter.test(absUrl)) return;
+ if (src.linkFilter && !src.linkFilter.test(absUrl)) return;
+ if (src.excludeFilter && src.excludeFilter.test(absUrl)) return;
seen.add(absUrl);
- // Title: try <img alt>, then text content, then URL slug
+ // Title: prefer anchor text, then <img alt>, then URL slug
let title = cleanTitle($(el).text());
if (!title || title.length < 5) {
title = cleanTitle($(el).find('img').attr('alt') || '');
}
if (!title || title.length < 5) {
- // derive from URL slug
- const slug = absUrl.split('/').filter(Boolean).pop() || '';
- title = slug.replace(/-/g, ' ').replace(/\.\w+$/, '').slice(0, 100);
+ title = slugToTitle(absUrl);
}
if (!title || title.length < 5) return;
- const uid = makeUid(src.key, absUrl);
+ items.push({ title, url: absUrl });
+ });
+
+ return items;
+}
+
+async function scrapeSite(src) {
+ let items = [];
+ let errorMsg = null;
+
+ try {
+ const html = await fetchHtml(src.url, !!src.browserbase);
+
+ if (src.parser === 'sitemap') {
+ items = parseSitemapXml(html, src);
+ } else if (src.parser === 'sitemap-index') {
+ items = await parseSitemapIndex(html, src);
+ } else {
+ items = parseHtml(html, src);
+ }
+ } catch (err) {
+ errorMsg = err.message;
+ console.error(`[html] ${src.key} failed: ${err.message}`);
+ }
+
+ // Cap at 6 most recent per source (per spec)
+ items = items.slice(0, 6);
+
+ let added = 0;
+ for (const it of items) {
+ const uid = makeUid(src.key, it.url);
added += upsertHeadline({
uid,
- title,
- url: absUrl,
+ title: it.title,
+ url: it.url,
source: src.name,
category: src.category,
pub_date: null,
- is_breaking: isBreaking(title) ? 1 : 0,
- is_developing: isDeveloping(title) ? 1 : 0,
+ is_breaking: isBreaking(it.title) ? 1 : 0,
+ is_developing: isDeveloping(it.title) ? 1 : 0,
is_top: 0,
});
- });
+ }
- logScrape(src.key, added);
+ logScrape(src.key, added, errorMsg);
return added;
}
@@ -111,8 +214,8 @@ async function scrapeAllHtml(category = null) {
const count = await scrapeSite(src);
console.log(`[html] ${src.name}: +${count}`);
results.push({ source: src.key, added: count });
- // Small polite delay
- await new Promise(r => setTimeout(r, 500));
+ // Polite delay; Browserbase sources need a longer beat
+ await new Promise(r => setTimeout(r, src.browserbase ? 1500 : 500));
}
return results;
}
diff --git a/scrapers/sources.js b/scrapers/sources.js
index 9c7c045..ab2e2f3 100644
--- a/scrapers/sources.js
+++ b/scrapers/sources.js
@@ -181,69 +181,77 @@ const SCRAPE_SOURCES = [
baseUrl: 'https://miamidesigndistrict.net',
},
// --- Wallpaper Houses ---
+ // Each entry has: url, baseUrl, selector, linkFilter (optional reject regex via excludeFilter)
+ // Some use `parser: 'sitemap'` to read a sitemap.xml instead of HTML
+ // `browserbase: true` routes through Browserbase cloud chromium (anti-bot bypass)
{
key: 'schumacher',
name: 'Schumacher',
category: CAT.WALLPAPER,
- url: 'https://www.fschumacher.com/search#q=&t=All%20Products&sort=newest',
- selector: 'a[href]',
- linkFilter: /\/(search|product|collection|new)/i,
- baseUrl: 'https://www.fschumacher.com',
- newsUrl: 'https://www.fschumacher.com/news',
- newsFilter: /\/(news|press|stories)/i,
+ url: 'https://schumacher.com/blog',
+ selector: 'h3 a, h2 a',
+ linkFilter: /\/blog\/[a-z0-9-]+\/?$/i,
+ excludeFilter: /\/blog\/(category|author|search|tag|page)\//i,
+ baseUrl: 'https://schumacher.com',
},
{
key: 'thibaut',
name: 'Thibaut',
category: CAT.WALLPAPER,
- url: 'https://www.thibautdesign.com/inspiration/',
- selector: 'a[href]',
- linkFilter: /\/(inspiration|news|press|new-arrivals|collection|blog)/i,
+ url: 'https://www.thibautdesign.com/sitemaps/collections.xml',
+ parser: 'sitemap',
baseUrl: 'https://www.thibautdesign.com',
},
{
key: 'phillip-jeffries',
name: 'Phillip Jeffries',
category: CAT.WALLPAPER,
- url: 'https://www.phillipjeffries.com/news',
- selector: 'a[href]',
- linkFilter: /\/(news|press|new-arrivals|collection)/i,
- baseUrl: 'https://www.phillipjeffries.com',
+ url: 'https://blog.phillipjeffries.com/',
+ selector: 'h2 a',
+ linkFilter: /blog\.phillipjeffries\.com\/[a-z0-9-]+$/i,
+ excludeFilter: /\/(tag|category|author|page|search)/i,
+ baseUrl: 'https://blog.phillipjeffries.com',
},
{
key: 'scalamandre',
name: 'Scalamandre',
category: CAT.WALLPAPER,
- url: 'https://scalamandre.com/news',
- selector: 'a[href]',
- linkFilter: /\/(news|press|blog|collection)/i,
- baseUrl: 'https://scalamandre.com',
+ // Scalamandre publishes no public news/blog (their /news page literally says "No news!").
+ // Their sitemap.xml is a flat urlset listing brand/collection pages with lastmod —
+ // newest = newest collection drops. Use those as "what's new" headlines.
+ url: 'https://www.scalamandre.com/sitemap.xml',
+ parser: 'sitemap',
+ linkFilter: /\/brands\/[a-z-]+\/[a-z0-9-]+\.html$/i,
+ baseUrl: 'https://www.scalamandre.com',
},
{
key: 'maya-romanoff',
name: 'Maya Romanoff',
category: CAT.WALLPAPER,
url: 'https://www.mayaromanoff.com/news/',
- selector: 'a[href]',
- linkFilter: /\/(news|press|blog|collection)/i,
+ selector: 'h2 a',
+ linkFilter: /\/news\/[a-z0-9-]+\/?$/i,
+ excludeFilter: /\/news\/?$|\/news\/page/i,
baseUrl: 'https://www.mayaromanoff.com',
},
{
key: 'arte-international',
name: 'Arte International',
category: CAT.WALLPAPER,
- url: 'https://www.arte-international.com/en/collections',
- selector: 'a[href]',
- linkFilter: /\/(en\/|collections|products|new)/i,
+ url: 'https://www.arte-international.com/en/stories',
+ selector: 'a[href*="/stories/"]',
+ linkFilter: /\/en\/stories\/[a-z0-9-]{4,}$/i,
+ excludeFilter: /\?category=|\/stories\/?$/i,
baseUrl: 'https://www.arte-international.com',
},
{
key: 'brewster',
name: 'Brewster Home Fashions',
category: CAT.WALLPAPER,
- url: 'https://www.brewsterwallcovering.com/new-arrivals',
- selector: 'a[href]',
- linkFilter: /\/(new-arrivals|collections|wallcovering|catalog)/i,
+ url: 'https://www.brewsterwallcovering.com/blog',
+ selector: 'h3 a, h2 a',
+ linkFilter: /brewsterwallcovering\.com\/[a-z0-9-]+$/i,
+ excludeFilter: /\/(blog|news|about|contact|search|cart|account|page-not-found)$/i,
baseUrl: 'https://www.brewsterwallcovering.com',
},
{
@@ -251,27 +259,34 @@ const SCRAPE_SOURCES = [
name: 'York Wallcoverings',
category: CAT.WALLPAPER,
url: 'https://www.yorkwallcoverings.com/blog',
- selector: 'a[href]',
- linkFilter: /\/(blog|collections|new|inspiration)/i,
+ selector: 'a[href*="/blog/"]',
+ linkFilter: /\/blog\/[a-z0-9-]+$/i,
+ excludeFilter: /\/blog\/(tag|month|rss|category|author|page)\//i,
baseUrl: 'https://www.yorkwallcoverings.com',
},
{
key: 'wallquest',
name: 'Wallquest',
category: CAT.WALLPAPER,
- url: 'https://wallquest.com/latest-news/',
- selector: 'a[href]',
- linkFilter: /\/(news|press|collection|latest)/i,
+ // Wallquest is behind Cloudflare bot protection AND publishes no blog/news posts.
+ // Use the sitemap (fetched via Browserbase) and surface the most recently updated
+ // collection pages with human-readable slugs (≥1 dash, not a SKU code).
+ url: 'https://wallquest.com/sitemap.xml',
+ parser: 'sitemap',
+ linkFilter: /wallquest\.com\/[a-z][a-z-]+-[a-z][a-z0-9-]+$/i,
+ excludeFilter: /\/(news|blog|cart|login|register|cms|about|contact|privacy|terms|sitemap|customer|account|search|forgot|password|category-page|residential|commercial|coloratlas|email|wp-)/i,
baseUrl: 'https://wallquest.com',
+ browserbase: true,
},
{
key: 'cole-and-son',
name: 'Cole & Son',
category: CAT.WALLPAPER,
- url: 'https://www.cole-and-son.com/en/news',
- selector: 'a[href]',
- linkFilter: /\/(news|press|new|collection)/i,
- baseUrl: 'https://www.cole-and-son.com',
+ url: 'https://cole-and-son.com/sitemap_blogs_1.xml',
+ parser: 'sitemap',
+ sitemapTitleFromImage: true,
+ linkFilter: /\/blogs\/[a-z-]+\/[a-z0-9-]+$/i,
+ baseUrl: 'https://cole-and-son.com',
},
];
← 035396a initial scaffold: Drudge-style design industry news aggregat
·
back to Abrams Report
·
deps(wallpaper): add @browserbasehq/sdk + playwright-core fo 20113db →