← back to Designer Wallcoverings
Sangetsu: parsePage proven (plain fetch, no Browserbase) — 15 patterns/238 SKUs; documented swatch+numeric-SKU refinement via variations JSON
be5f55490a88500906c7c59c294ed1d8d9d23b02 · 2026-06-29 16:03:09 -0700 · steve@designerwallcoverings.com
Files touched
M onboarding/sangetsu-lilycolor/DOSSIER.mdM onboarding/sangetsu-lilycolor/sangetsu-sitemap-scraper.cjs
Diff
commit be5f55490a88500906c7c59c294ed1d8d9d23b02
Author: steve@designerwallcoverings.com <steve@designerwallcoverings.com>
Date: Mon Jun 29 16:03:09 2026 -0700
Sangetsu: parsePage proven (plain fetch, no Browserbase) — 15 patterns/238 SKUs; documented swatch+numeric-SKU refinement via variations JSON
---
onboarding/sangetsu-lilycolor/DOSSIER.md | 20 +++
.../sangetsu-sitemap-scraper.cjs | 134 +++++++++++++++++----
2 files changed, 131 insertions(+), 23 deletions(-)
diff --git a/onboarding/sangetsu-lilycolor/DOSSIER.md b/onboarding/sangetsu-lilycolor/DOSSIER.md
index a54c1388..9c4de186 100644
--- a/onboarding/sangetsu-lilycolor/DOSSIER.md
+++ b/onboarding/sangetsu-lilycolor/DOSSIER.md
@@ -108,3 +108,23 @@ wallpaper-wallcovering). US site blocks sitemap/wp-json; the **regional .co.th s
**Customer:** Andrew Glick reply DRAFTED via George (`info@` account, draftId r-7044…) with top-3
Sanshin/Virtuoso/Savannah — awaiting Steve review/send.
+
+## 7. Sangetsu per-page parse — PROVEN (plain fetch, NO Browserbase)
+
+`sangetsu-sitemap-scraper.cjs parse <url>` works on a plain `curl --compressed`-style fetch
+(server-rendered WooCommerce; Node `https` needs manual gzip/br decompress — handled). Proven:
+- **Sanshin** → pattern, decoded overview, spec (Width 52" / Type II / 20 oz / grasscloth+metallic),
+ **24 colorway SKUs** AZ52790–AZ52813, each with full-res image `uploads/YYYY/MM/<SKU>.jpg`.
+- Batch `stage 15` → 15 patterns, **238 colorway SKUs** (Acappella 24, Uncorked 13, Sanshin 24…).
+- **No paid Browserbase needed** — the anti-bot "block" was just a missing decompress.
+
+**Two known refinements before staging the full 623 (gated):**
+1. **Swatch-style pages return 0 SKUs.** ~1/3 of pages use a swatch widget (not `<select>`), and some
+ carry **numeric SKUs** (e.g. 384500, G0167, TF3385) the `<option>` regex skips. Fix = parse the
+ `data-product_variations` JSON blob (authoritative for BOTH dropdown + swatch products) and/or the
+ `uploads/…/<SKU>.jpg` image filenames (catches numeric SKUs). Present on every page.
+2. **`backing` field** captures `OsnaburgWidth` (no separator in source spec block) — split on the
+ next capitalized label.
+
+Both are local parser tweaks (offline). Then: dedup (prefix AZ/Y/GSD/LX/numeric vs dw_sku_registry +
+shopify_products), settlement-gate naturals, source net COST → only then stage to Shopify DRAFT.
diff --git a/onboarding/sangetsu-lilycolor/sangetsu-sitemap-scraper.cjs b/onboarding/sangetsu-lilycolor/sangetsu-sitemap-scraper.cjs
index db8f71a9..da45faa1 100644
--- a/onboarding/sangetsu-lilycolor/sangetsu-sitemap-scraper.cjs
+++ b/onboarding/sangetsu-lilycolor/sangetsu-sitemap-scraper.cjs
@@ -1,58 +1,146 @@
#!/usr/bin/env node
/**
- * Sangetsu (Goodrich) scraper — DISCOVERY + per-page parse scaffold (OFFLINE/STAGING).
+ * Sangetsu (Goodrich) scraper — DISCOVERY + per-page PARSE (OFFLINE/STAGING).
*
- * Discovery (WORKS): the US site blocks wp-json/sitemap, but the regional .co.th site
- * exposes wp-sitemap-posts-product-1.xml (~879 pattern pages). The Sangetsu/Goodrich
+ * Discovery: US site blocks wp-json/sitemap, but the regional .co.th site exposes
+ * wp-sitemap-posts-product-1.xml (~879 pattern pages, 623 wallpaper). The Sangetsu/Goodrich
* catalog is shared across regions, so this is a valid seed for the global product set.
*
- * Per-page parse (STUB): each "product" page is a PATTERN holding N colorway SKUs in a
- * <select> ("Choose an option AZ52790 ..."). The overview/spec text is in the static HTML.
- * If the colorway SKUs turn out to be JS-injected, fall back to dev-browser/Browserbase.
+ * Per-page parse: each PATTERN page is server-rendered WooCommerce (NO JS render needed —
+ * just send Accept-Encoding and decompress). Colorway SKUs live in <option> values, full-res
+ * images at /wp-content/uploads/YYYY/MM/<SKU>.jpg, specs + overview in static HTML, and the
+ * full WooCommerce variation set in the data-product_variations JSON blob.
*
* HARD: writes ONLY to local staging. No Shopify, no dw_unified, no publish.
- * Usage: node sangetsu-sitemap-scraper.cjs discover # enumerate + filter wallpaper URLs
+ * Usage:
+ * node sangetsu-sitemap-scraper.cjs discover # enumerate + filter wallpaper URLs
+ * node sangetsu-sitemap-scraper.cjs parse <url> # parse one pattern page -> JSON (stdout)
+ * node sangetsu-sitemap-scraper.cjs stage [limit] # parse N pattern pages -> staging JSONL
*/
const https = require('https');
+const zlib = require('zlib');
const fs = require('fs');
const path = require('path');
const SITEMAP = 'https://www.sangetsu-goodrich.co.th/wp-sitemap-posts-product-1.xml';
const OUT_DIR = path.join(__dirname, 'staging');
const URLS_OUT = path.join(OUT_DIR, 'sangetsu-wallpaper-urls.txt');
+const STAGE_OUT = path.join(OUT_DIR, 'sangetsu-staging.jsonl');
+// fetch with gzip/br/deflate decompression (Node https does NOT auto-decode)
function get(url) {
return new Promise((resolve, reject) => {
- https.get(url, { headers: { 'User-Agent': 'Mozilla/5.0 (DW-onboarding-recon)' } }, (res) => {
- let body = '';
- res.on('data', (c) => (body += c));
- res.on('end', () => resolve(body));
+ https.get(url, {
+ headers: {
+ 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) AppleWebKit/537.36 Chrome/120 Safari/537.36',
+ 'Accept-Encoding': 'gzip, deflate, br',
+ },
+ }, (res) => {
+ const chunks = [];
+ res.on('data', (c) => chunks.push(c));
+ res.on('end', () => {
+ let buf = Buffer.concat(chunks);
+ const enc = (res.headers['content-encoding'] || '').toLowerCase();
+ try {
+ if (enc === 'gzip') buf = zlib.gunzipSync(buf);
+ else if (enc === 'br') buf = zlib.brotliDecompressSync(buf);
+ else if (enc === 'deflate') buf = zlib.inflateSync(buf);
+ } catch (e) { /* fall through with raw */ }
+ resolve(buf.toString('utf8'));
+ });
}).on('error', reject);
});
}
+const decodeEntities = (s) => (s || '')
+ .replace(/"/g, '"').replace(/–/g, '–').replace(/’/g, "'")
+ .replace(/&/g, '&').replace(/’/g, "'").replace(/</g, '<').replace(/>/g, '>');
+const stripTags = (s) => (s || '').replace(/<[^>]+>/g, '').replace(/\s+/g, ' ').trim();
+
+function parsePage(html, url) {
+ const name = stripTags((html.match(/<h1[^>]*class="product_title[^"]*"[^>]*>([\s\S]*?)<\/h1>/i) || [])[1])
+ .replace(/\s+Wallpapers?$/i, '').trim() || null;
+
+ // colorway SKUs from <option value="...">SKU
+ const skus = [...new Set([...html.matchAll(/<option[^>]*>\s*([A-Z]{1,4}[0-9]{4,6}(?:-[0-9]+)?)\s*</gi)].map((m) => m[1].toUpperCase()))];
+
+ // SKU -> full-res image (non-thumbnail) from uploads path
+ const imgMap = {};
+ for (const m of html.matchAll(/(https:\/\/[^"' ]*\/uploads\/[0-9]{4}\/[0-9]{2}\/([A-Z]{1,4}[0-9]{4,6}))\.jpe?g/gi)) {
+ imgMap[m[2].toUpperCase()] = m[1] + '.jpg';
+ }
+
+ // overview: short-description (percent- or entity-encoded in places) -> best-effort plain text
+ let overview = stripTags((html.match(/woocommerce-product-details__short-description[^>]*>([\s\S]*?)<\/div>/i) || [])[1]);
+ if (!overview) {
+ const enc = (html.match(/Mimicking[\s\S]{0,400}?(?:%2E|\.)/) || [])[0];
+ if (enc) { try { overview = decodeURIComponent(enc.replace(/\+/g, ' ')); } catch (_) {} }
+ }
+ overview = stripTags(decodeEntities(overview || '')) || null;
+
+ // specs: pull common wallcovering spec tokens from static text
+ const text = stripTags(html);
+ const spec = {
+ width: (text.match(/Width[:\s]*(\d{2}(?:\.\d)?)\s*"/i) || text.match(/\b(\d{2})"\s*Type\s+I/i) || [])[1]
+ ? `${(text.match(/Width[:\s]*(\d{2}(?:\.\d)?)\s*"/i) || text.match(/\b(\d{2})"\s*Type\s+I/i))[1]}"` : null,
+ type: (text.match(/Type\s+I{1,3}/i) || [])[0] || null,
+ weight: (text.match(/\b\d{1,2}\s*oz\b/i) || [])[0] || null,
+ backing: (text.match(/Backing[:\s]*([A-Za-z][A-Za-z-]+)/i) || [])[1] || null,
+ is_grasscloth: /grasscloth/i.test(text),
+ is_metallic: /metallic|shimmer/i.test(text),
+ };
+
+ return {
+ source: 'sangetsu',
+ source_url: url,
+ pattern: name,
+ overview,
+ spec,
+ colorway_skus: skus,
+ colorway_count: skus.length,
+ sku_images: imgMap,
+ mfr_prefix: (skus[0] || '').match(/^([A-Z]+)/)?.[1] || null, // dedup key
+ settlement_checked: false, deduped: false, cost_confirmed: false,
+ activation_ready: false, status: 'staged-for-new',
+ };
+}
+
async function discover() {
const xml = await get(SITEMAP);
const all = [...xml.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
- // keep only wallpaper/wallcovering pattern pages (drop carpet / flooring / etc.)
const wp = all.filter((u) => /\/products\/wallpaper-wallcovering\//.test(u));
if (!fs.existsSync(OUT_DIR)) fs.mkdirSync(OUT_DIR, { recursive: true });
fs.writeFileSync(URLS_OUT, wp.join('\n') + '\n');
- const grass = wp.filter((u) => /grass|sanshin|virtuoso|savannah|sass-a-grass|woven/i.test(u));
- console.log(JSON.stringify({
- totalProductPages: all.length,
- wallpaperPages: wp.length,
- out: URLS_OUT,
- grasscloth_line: grass,
- }, null, 2));
+ console.log(JSON.stringify({ totalProductPages: all.length, wallpaperPages: wp.length, out: URLS_OUT }, null, 2));
+ return wp;
}
-// TODO (gated build step): parsePage(url) -> { pattern, overview, specs, colorways:[{sku,name,img}] }
-// selectors observed: H1 = pattern name; ".product_desc" = overview; <select> option values = colorway SKUs.
-// Each colorway SKU -> one DW staging record (dedup vs dw_sku_registry by SKU; settlement-gate naturals).
-
(async () => {
const cmd = process.argv[2] || 'discover';
if (cmd === 'discover') return discover();
+ if (cmd === 'parse') {
+ const url = process.argv[3];
+ if (!url) { console.error('parse needs <url>'); process.exit(1); }
+ const html = await get(url);
+ console.log(JSON.stringify(parsePage(html, url), null, 2));
+ return;
+ }
+ if (cmd === 'stage') {
+ const limit = parseInt(process.argv[3] || '50', 10);
+ const wp = fs.existsSync(URLS_OUT) ? fs.readFileSync(URLS_OUT, 'utf8').trim().split('\n') : await discover();
+ if (!fs.existsSync(OUT_DIR)) fs.mkdirSync(OUT_DIR, { recursive: true });
+ const out = fs.createWriteStream(STAGE_OUT);
+ let n = 0, skuTotal = 0;
+ for (const url of wp.slice(0, limit)) {
+ try {
+ const rec = parsePage(await get(url), url);
+ out.write(JSON.stringify(rec) + '\n'); n++; skuTotal += rec.colorway_count;
+ process.stderr.write(`[${n}] ${rec.pattern} (${rec.colorway_count} skus)\n`);
+ } catch (e) { process.stderr.write(`SKIP ${url}: ${e.message}\n`); }
+ }
+ out.end();
+ console.log(JSON.stringify({ patternsStaged: n, colorwaySkus: skuTotal, out: STAGE_OUT }, null, 2));
+ return;
+ }
console.error('unknown command:', cmd); process.exit(1);
})().catch((e) => { console.error('FAILED:', e.message); process.exit(1); });
← d29bf809 auto-save: 2026-06-29T15:54:24 (5 files) — pending-approval/
·
back to Designer Wallcoverings
·
Sangetsu parser: image-filename SKU harvest (covers swatch+n cda0849d →