← back to Designer Wallcoverings
Coraggio line onboard: vendor-scoped scraper + skill (98 SKUs staged, quote-only)
03b9fb62df6b76d5cea2d867349b7e5e8564d54d · 2026-07-18 07:29:34 -0700 · Steve Abrams
Files touched
A shopify/scripts/coraggio-scraper-manager.SKILL.mdA shopify/scripts/coraggio-scraper.js
Diff
commit 03b9fb62df6b76d5cea2d867349b7e5e8564d54d
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Sat Jul 18 07:29:34 2026 -0700
Coraggio line onboard: vendor-scoped scraper + skill (98 SKUs staged, quote-only)
---
shopify/scripts/coraggio-scraper-manager.SKILL.md | 97 ++++++++++++
shopify/scripts/coraggio-scraper.js | 185 ++++++++++++++++++++++
2 files changed, 282 insertions(+)
diff --git a/shopify/scripts/coraggio-scraper-manager.SKILL.md b/shopify/scripts/coraggio-scraper-manager.SKILL.md
new file mode 100644
index 00000000..7d95811c
--- /dev/null
+++ b/shopify/scripts/coraggio-scraper-manager.SKILL.md
@@ -0,0 +1,97 @@
+---
+name: coraggio-scraper-manager
+description: Coraggio fabric import manager. Use when scraping Coraggio products, refreshing the coraggio_catalog staging table, or re-onboarding the Coraggio line. Coraggio (est. 1979, Seattle) is an independent luxury textile house DISTRIBUTED on thibautdesign.com — mfr codes span C#####, U#####, D##### namespaces, ALL FABRIC (drapery / sheer / upholstery). Customer-facing brand, NOT a Thibaut sub-label and NOT a private label. Scope by the /brand/coraggio + /collections/coraggio-* anchor, NEVER by brand.name (reads "Thibaut" on every page) or code prefix. Vendor-scoped to Coraggio ONLY. Short command alias: /coraggio.
+---
+
+# Coraggio Scraper Manager
+
+> INSTALL: copy this file to `~/.claude/skills/coraggio-scraper-manager/SKILL.md`
+> (skill-dir writes were gated in the onboarding session, so it lives in-repo).
+
+Dedicated vendor-scoped ingest path for the **Coraggio** line. Coraggio is an
+**independent** luxury textile house (founded 1979, Seattle — coraggio.com,
+@coraggio1979) whose catalog is **distributed on thibautdesign.com**, the same
+host as Anna French. It is **customer-facing** — the Coraggio name IS shown to
+shoppers (unlike a private label), but it is NOT Thibaut proper.
+
+## Hard rules
+
+- **All FABRIC.** Coraggio is a textile house — drapery, sheer, upholstery. There is
+ no wallcovering. `product_type = Fabric` for every SKU. The mfr-code letter
+ (`C`/`U`/`D`) is Thibaut's internal category marker, NOT a type or scope signal.
+- **Scope by the Coraggio anchor, NEVER the brand field or code prefix.** Every
+ product page's JSON-LD `brand.name` reads **"Thibaut"** (parent bleed) — the exact
+ trap that mis-ingested 175 Anna French rows AS "Thibaut". Keep a row ONLY if the page
+ carries `/brand/coraggio`, a `coraggio-spring-YYYY` collection, or a `CORAGGIO …`
+ collection field. Anything else is skipped.
+- **Vendor name is `Coraggio`.** Customer-facing. Never "Thibaut", never a private label.
+- **Trade-only / quote-only.** thibautdesign.com exposes **no public price** for Coraggio
+ (`Trade-only = true`, offers.price = null on 98/98 SKUs). Onboard **Needs-Price /
+ sample-only** (ACTIVE + $4.25 memo sample + NULL product price) per Steve's blank-price
+ rule — never force a cost/0.65/0.85 price. Set real prices only when a Coraggio cost
+ list arrives (then price cost/0.65/0.85, standard fabric markup).
+- **mfr# (C/U/D#####) is NEVER customer-facing** — strip it from image filenames; bind it
+ ONLY to `custom.manufacturer_sku` + `dwc.manufacturer_sku`.
+- **NEVER "Unknown" in a title.** Title format: `Pattern Name Real Color | Coraggio`.
+ Clean handle: `pattern-color-coraggio`.
+- **PostgreSQL BEFORE Shopify.** The scraper writes to `coraggio_catalog` (staging) ONLY.
+- **Sold per yard** (textile line). No roll / double-roll rules. No settlement gate needed
+ (contemporary weaves — no botanical/scenic imagery), but run the settlement check anyway
+ if a future collection adds figurative motifs.
+- **DWxx SKU prefix is Steve-gated** (customer-facing identity). Proposed: **`DWCG`**
+ (Coraggio). Left NULL in staging until Steve confirms — mirrors how `DWAF` was
+ "Steve-confirmed" for Anna French.
+
+## Source structure (verified 2026-07-18)
+
+| | |
+|---|---|
+| Host | `https://www.thibautdesign.com` (coraggio.com is the brand site, not the catalog) |
+| Product URL | `/products/fabrics/<CODE>` where CODE ∈ `C#####` / `U#####` / `D#####` |
+| Collections | `/collections/coraggio-spring-2024`, `-2025`, `-2026` (the 3 live books) |
+| Brand anchor | `/brand/coraggio` |
+| Line size | **98 SKUs** (24 + 31 + 43), 31 distinct patterns |
+| Auth | none — plain https GET + browser UA |
+| Discontinued | product URL `301 → /collection/...` = discontinued (ARCHIVE) |
+
+### Per-page extraction (same shape as anna-french / thibaut-repeat-scraper)
+- **JSON-LD `@graph` → `@type:"Product"`:** `name` (`PATTERN - Color`), `sku`, `mpn`,
+ `color`, `material`, `image[]`, `description`. (`brand.name` = "Thibaut" — ignore.)
+- **`additionalProperty` table:** `Pattern Name`, `Collection`, `Use` (style —
+ Modern/Geometric/Stripe/…), `Width`, `Trade-only`.
+- **Images:** every `cdn.shopify.com/...<CODE>...jpg|png` (strip the code at attach time).
+
+## The scraper
+
+`shopify/scripts/coraggio-scraper.js` (self-contained; local PG socket + DATABASE_URL fallback).
+
+```
+node shopify/scripts/coraggio-scraper.js --dry-run # prove extraction, no writes
+node shopify/scripts/coraggio-scraper.js --dry-run --limit 5
+node shopify/scripts/coraggio-scraper.js # stage the full 98-SKU line ($0 local)
+node shopify/scripts/coraggio-scraper.js --urls C98065,U98080 # explicit codes
+```
+
+Enumerates the 3 collection streams for the SKU list, fetches each product page, applies
+the Coraggio scope guard, and upserts `coraggio_catalog` keyed on `mfr_sku`. Idempotent.
+
+## Onboarding / cutover (GATED — Steve sign-off required)
+
+Staging is done (98 rows, 0 blanks except price). Promotion to Shopify runs ONLY through
+the metered hourly cadence (`cadence-import.js --commit --activate`) with a Coraggio entry
+in `cadence/vendors.js`, through the activation gate (`validate-before-activate.js`). Each
+step is Steve-gated:
+1. Steve confirms the DWxx prefix (proposed `DWCG`) → backfill `dw_sku` in staging.
+2. Local AI enrichment (colors/styles) via the LOCAL Ollama path (qwen2.5vl/PIL) — $0.
+3. Cadence create → products land ACTIVE, sample-only ($4.25), NULL price (Needs-Price).
+4. Per Steve's "new line → agent + microsite" rule: a `coraggio.designerwallcoverings.com`
+ editorial microsite (dw-vendor-landing pattern) — GATED deploy.
+5. FileMaker fabric masters follow the standard invoice/WALLPAPER-record rules.
+
+## Reference
+
+- Staging table: `coraggio_catalog` (dw_unified, cloned from `anna_french_catalog`)
+- Scraper: `~/Projects/Designer-Wallcoverings/shopify/scripts/coraggio-scraper.js`
+- Sibling pattern: `anna-french-scraper-manager` (same host, same extraction shape)
+- Activation gate: `shopify/scripts/lib/validate-before-activate.js`
+- Cadence importer: `shopify/scripts/cadence/cadence-import.js`
diff --git a/shopify/scripts/coraggio-scraper.js b/shopify/scripts/coraggio-scraper.js
new file mode 100644
index 00000000..eedca9a7
--- /dev/null
+++ b/shopify/scripts/coraggio-scraper.js
@@ -0,0 +1,185 @@
+#!/usr/bin/env node
+/**
+ * coraggio-scraper.js — vendor-scoped ingest for the CORAGGIO line.
+ *
+ * Coraggio (est. 1979, Seattle) is an independent luxury textile house whose
+ * catalog is DISTRIBUTED on thibautdesign.com (same host as Anna French). It is a
+ * CUSTOMER-FACING brand — NOT a Thibaut sub-label and NOT a private label.
+ *
+ * Source shape (verified 2026-07-18): plain https GET + browser UA, no auth.
+ * - Enumeration: the 3 Coraggio collection pages server-embed the full product
+ * URL list (/products/fabrics/<CODE>). Codes span C/U/D namespaces — ALL FABRIC.
+ * - Per product: server-rendered JSON-LD Product node + an additionalProperty
+ * PropertyValue table (Pattern Name, Collection, Use, Width, Trade-only).
+ *
+ * HARD SCOPE RULE: JSON-LD brand.name reads "Thibaut" (parent bleed) on every page —
+ * NEVER scope on it. Keep a row ONLY if the page carries the /brand/coraggio anchor
+ * or a "CORAGGIO ..." collection. (Same trap that corrupted the Anna French ingest.)
+ *
+ * Writes to coraggio_catalog (staging) ONLY. No Shopify, no FileMaker. dw_sku is
+ * left NULL — the DWxx prefix is a Steve-gated customer-facing decision.
+ *
+ * Usage:
+ * node coraggio-scraper.js --dry-run # enumerate + extract, no DB writes
+ * node coraggio-scraper.js --dry-run --limit 5
+ * node coraggio-scraper.js # stage the full line (98 SKUs, $0 local)
+ * node coraggio-scraper.js --urls C98065,U98080 # explicit codes
+ */
+const { Pool } = require('pg');
+const https = require('https');
+
+const pool = process.env.DATABASE_URL
+ ? new Pool({ connectionString: process.env.DATABASE_URL })
+ : new Pool({ host: '/tmp', database: 'dw_unified', user: process.env.PGUSER || process.env.USER });
+
+const UA = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36';
+const BASE = 'https://www.thibautdesign.com';
+const COLLECTIONS = ['coraggio-spring-2024', 'coraggio-spring-2025', 'coraggio-spring-2026'];
+const DELAY_MS = 1200;
+
+const argv = process.argv.slice(2);
+const DRY_RUN = argv.includes('--dry-run');
+const li = argv.indexOf('--limit');
+const LIMIT = li >= 0 ? Math.max(1, parseInt(argv[li + 1]) || 0) : 0;
+const ui = argv.indexOf('--urls');
+const URLS = ui >= 0 ? String(argv[ui + 1] || '').split(',').map(s => s.trim()).filter(Boolean) : [];
+
+function fetchPage(url) {
+ return new Promise((resolve, reject) => {
+ const req = https.get(url, { headers: { 'User-Agent': UA } }, (res) => {
+ if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
+ return resolve({ status: res.statusCode, body: '', redirect: res.headers.location });
+ }
+ let data = '';
+ res.on('data', d => data += d);
+ res.on('end', () => resolve({ status: res.statusCode, body: data }));
+ });
+ req.on('error', reject);
+ req.setTimeout(25000, () => { req.destroy(); reject(new Error('timeout')); });
+ });
+}
+const sleep = ms => new Promise(r => setTimeout(r, ms));
+const titleCase = s => (s || '').toLowerCase().replace(/\b\w/g, c => c.toUpperCase()).trim();
+
+function jsonLdProduct(html) {
+ const blocks = html.match(/<script[^>]*application\/ld\+json[^>]*>([\s\S]*?)<\/script>/gi) || [];
+ for (const b of blocks) {
+ const raw = b.replace(/<script[^>]*>/i, '').replace(/<\/script>/i, '');
+ let d; try { d = JSON.parse(raw); } catch { continue; }
+ const nodes = Array.isArray(d) ? d : (d['@graph'] || [d]);
+ for (const n of nodes) if (n && n['@type'] === 'Product') return n;
+ }
+ return null;
+}
+function specTable(prod) {
+ const out = {};
+ for (const p of (prod.additionalProperty || [])) {
+ if (p && p.name) out[p.name.trim()] = String(p.value == null ? '' : p.value).trim();
+ }
+ return out;
+}
+// scope guard: page must anchor to the Coraggio brand/collection, never brand.name
+function isCoraggio(html, spec) {
+ if (/\/brand\/coraggio/i.test(html)) return true;
+ if (/coraggio-spring-\d{4}/i.test(html)) return true;
+ if (/coraggio/i.test(spec.Collection || '')) return true;
+ return false;
+}
+
+async function enumerate() {
+ if (URLS.length) {
+ return URLS.map(u => {
+ const code = (u.match(/([A-Z]\d{4,5})/i) || [])[1] || u;
+ return code.toUpperCase();
+ });
+ }
+ const set = new Set();
+ for (const c of COLLECTIONS) {
+ const { body, status } = await fetchPage(`${BASE}/collections/${c}`);
+ const codes = [...(body || '').matchAll(/\/products\/fabrics\/([A-Z]\d{4,5})/g)].map(m => m[1]);
+ console.log(` collection ${c}: HTTP ${status}, ${new Set(codes).size} SKUs`);
+ codes.forEach(x => set.add(x));
+ await sleep(DELAY_MS);
+ }
+ return [...set].sort();
+}
+
+function extract(html, mfr) {
+ const prod = jsonLdProduct(html);
+ if (!prod) return null;
+ const spec = specTable(html && prod ? prod : {});
+ if (!isCoraggio(html, spec)) return { skip: true, mfr };
+ // name = "PATTERN - Color"
+ const name = prod.name || '';
+ let pattern = spec['Pattern Name'] || name.split(' - ')[0] || '';
+ let color = prod.color || name.split(' - ').slice(1).join(' - ') || '';
+ const offers = Array.isArray(prod.offers) ? prod.offers[0] : (prod.offers || {});
+ const priceRaw = offers && offers.price != null ? Number(offers.price) : null;
+ // images: JSON-LD image[] + every shopify cdn asset carrying this code
+ const imgs = new Set();
+ const jl = Array.isArray(prod.image) ? prod.image : (prod.image ? [prod.image] : []);
+ jl.forEach(u => imgs.add(u));
+ [...html.matchAll(new RegExp(`https://cdn\\.shopify\\.com/[^\\s"'\\\\]*${mfr}[^\\s"'\\\\]*\\.(?:jpg|jpeg|png)`, 'g'))].forEach(m => imgs.add(m[0]));
+ const collection = titleCase(spec.Collection || '');
+ return {
+ mfr_sku: mfr,
+ pattern_name: titleCase(pattern),
+ color_name: color.trim(),
+ collection,
+ product_type: 'Fabric',
+ width: spec.Width || '',
+ material: prod.material || '',
+ application: spec.Use || '', // Coraggio "Use" = style/application (e.g. Modern)
+ price_retail: null, // trade-only, no public price → Needs-Price
+ price_trade: null,
+ image_url: [...imgs][0] || '',
+ all_images: [...imgs].join('|'),
+ product_url: `${BASE}/products/fabrics/${mfr}`,
+ trade_only: /true/i.test(spec['Trade-only'] || ''),
+ _priceSeen: priceRaw,
+ };
+}
+
+async function upsert(p) {
+ await pool.query(
+ `INSERT INTO coraggio_catalog
+ (mfr_sku, pattern_name, color_name, collection, product_type, width, material,
+ application, price_retail, price_trade, image_url, all_images, product_url,
+ in_stock, discontinued, last_scraped)
+ VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,true,false,now())
+ ON CONFLICT (mfr_sku) DO UPDATE SET
+ pattern_name=EXCLUDED.pattern_name, color_name=EXCLUDED.color_name,
+ collection=EXCLUDED.collection, product_type=EXCLUDED.product_type,
+ width=EXCLUDED.width, material=EXCLUDED.material, application=EXCLUDED.application,
+ image_url=EXCLUDED.image_url, all_images=EXCLUDED.all_images,
+ product_url=EXCLUDED.product_url, last_scraped=now(), updated_at=now()`,
+ [p.mfr_sku, p.pattern_name, p.color_name, p.collection, p.product_type, p.width,
+ p.material, p.application, p.price_retail, p.price_trade, p.image_url, p.all_images,
+ p.product_url]
+ );
+}
+
+(async () => {
+ console.log(`${DRY_RUN ? '[DRY RUN] ' : ''}Coraggio scrape — enumerating…`);
+ let codes = await enumerate();
+ if (LIMIT) codes = codes.slice(0, LIMIT);
+ console.log(` → ${codes.length} candidate SKUs\n`);
+ let staged = 0, skipped = 0, noprice = 0, failed = 0;
+ for (const mfr of codes) {
+ try {
+ const { body, status, redirect } = await fetchPage(`${BASE}/products/fabrics/${mfr}`);
+ if (redirect || status >= 400) { console.log(` ${mfr} DISCONTINUED (HTTP ${status})`); skipped++; await sleep(DELAY_MS); continue; }
+ const p = extract(body, mfr);
+ if (!p) { console.log(` ${mfr} no JSON-LD — FAIL`); failed++; await sleep(DELAY_MS); continue; }
+ if (p.skip) { console.log(` ${mfr} NOT Coraggio-scoped — skip`); skipped++; await sleep(DELAY_MS); continue; }
+ if (p._priceSeen == null) noprice++;
+ console.log(` ${mfr} ${p.pattern_name} — ${p.color_name} | ${p.collection} | ${p.material.slice(0,28)} | ${[...p.all_images.split('|')].filter(Boolean).length} img${p._priceSeen==null?' | NO-PRICE':''}`);
+ if (!DRY_RUN) { await upsert(p); staged++; }
+ await sleep(DELAY_MS);
+ } catch (e) {
+ console.log(` ${mfr} ERROR ${e.message}`); failed++; await sleep(DELAY_MS);
+ }
+ }
+ console.log(`\n${DRY_RUN ? '[DRY RUN] ' : ''}done — staged:${staged} skipped:${skipped} failed:${failed} no-price:${noprice}/${codes.length}`);
+ await pool.end();
+})();
← 4f9d815a auto-save: 2026-07-18T06:50:05 (4 files) — pending-approval/
·
back to Designer Wallcoverings
·
auto-save: 2026-07-18T07:50:19 (4 files) — pending-approval/ fabcbb7c →