rescrape-todo stubs for romo/arte/rlf/glitter (dw-stale-urls pass)

8ef797758432b120db1a19e8ae2605d59ac5c28e · 2026-05-19 22:00:47 -0700 · Steve Abrams
Files touched

A rescrape-todo/arte-international.md
A rescrape-todo/glitter-walls.md
A rescrape-todo/ralph-lauren-fabric.md
A rescrape-todo/romo.md
Diff

commit 8ef797758432b120db1a19e8ae2605d59ac5c28e
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Tue May 19 22:00:47 2026 -0700

    rescrape-todo stubs for romo/arte/rlf/glitter (dw-stale-urls pass)
---
 rescrape-todo/arte-international.md  | 32 ++++++++++++++++++++++++++++++
 rescrape-todo/glitter-walls.md       | 36 ++++++++++++++++++++++++++++++++++
 rescrape-todo/ralph-lauren-fabric.md | 38 ++++++++++++++++++++++++++++++++++++
 rescrape-todo/romo.md                | 29 +++++++++++++++++++++++++++
 4 files changed, 135 insertions(+)

diff --git a/rescrape-todo/arte-international.md b/rescrape-todo/arte-international.md
new file mode 100644
index 0000000..d83ce88
--- /dev/null
+++ b/rescrape-todo/arte-international.md
@@ -0,0 +1,32 @@
+# Arte International — Re-scrape TODO
+
+**Status**: 133 affected SKUs. 9 already refreshed via PDP og:image extraction this pass. 124 remaining.
+
+## What this pass accomplished
+- Identified 9 affected SKUs that have `pattern_url` entries in `arte_rescrape` (`SKU` 37620-37623 Natura, 37640-37644 Gloria).
+- Fetched each PDP, extracted `og:image`, HEAD-verified all 9 return 200, staged into `image_url_swap_proposals_2026_05_19_v2` with `source_table='arte_pdp_og_image'`.
+- Updated `image_hashes` for the 2 unique URLs to `status='success'`.
+
+## Remaining problem (124 SKUs)
+- 124 of the 133 affected Arte SKUs do NOT appear in `arte_rescrape` (the only source of PDP URLs).
+- `arte_catalog` rows for these SKUs store only the OLD dead Shopify CDN URL — never a vendor-CDN URL.
+- The 1,416 rows with `edge.arte-international.com` URLs in `arte_catalog` are all OTHER SKUs (not the affected ones).
+- Arte's sitemap.xml redirects to a 404 — no easy slug enumeration.
+
+## Recommendation
+Use the existing `arte-international-colorway-scraper` skill (located at `~/.claude/skills/arte-international-colorway-scraper/SKILL.md`) to do a full Arte vendor crawl. Critically, the skill should:
+1. Walk every collection page (e.g., `/en/collections/metal-x-signum/`).
+2. For each PDP URL discovered, parse the SKU from the URL and extract `og:image`.
+3. UPSERT into `arte_rescrape` (which is keyed on `arte_sku`).
+4. After the crawl, re-run a swap-proposal pass joining `image_url_swap_proposals_2026_05_19` → `arte_rescrape` to pick up newly available URLs.
+
+## Affected SKU list (124 remaining)
+`/tmp/arte_affected_skus.csv` minus the 9 already handled this pass.
+
+## What I did this pass (Mac2)
+Bash one-liner extracted og:image from each of the 9 PDPs via:
+```bash
+curl -s -A "Mozilla/5.0" --max-time 10 "$pdp_url" | \
+  grep -oE 'og:image"[^>]*content="[^"]+' | head -1 | sed 's/.*content="//'
+```
+This same approach works for any Arte PDP — the site is statically scrapable.
diff --git a/rescrape-todo/glitter-walls.md b/rescrape-todo/glitter-walls.md
new file mode 100644
index 0000000..d9f2399
--- /dev/null
+++ b/rescrape-todo/glitter-walls.md
@@ -0,0 +1,36 @@
+# Glitter Walls — Re-scrape TODO
+
+**Status**: 237 affected SKUs, NO scraper-manager skill, catalog tables effectively empty for these SKUs.
+
+## The problem
+- `glitter_walls_catalog` has 16 rows total — well below the 554 GW products in Shopify and the 237 SKUs flagged dead.
+- `glitter_catalog` has 121 rows but its SKU column is numeric IDs (e.g., `26048552`) — does NOT match the affected SKU format which is verbose-kebab `BUTTERFLY-GLITTER-WALLCOVERING-RAINBOW`.
+- Only 3 of 237 partially match `glitter_walls_catalog` via ILIKE.
+
+## Recommendation
+Build a `glitter-walls-scraper-manager` skill.
+
+**Vendor URL**: `glitterwalls.com` (no trade portal required — public catalog)
+**Existing scaffolding**: `~/Projects/glitterwalls/` exists (DW microsite for this label). The vendor's public PDPs use slug URLs like `/products/butterfly-glitter-wallcovering` — the SKU format suggests the slug IS the SKU (downcased, no `-` → `-` substitution).
+**Target DB table**: `glitter_walls_catalog` (already exists with proper schema).
+
+## Affected SKU list
+237 SKUs at `/tmp/glitter_walls_affected_skus.csv`.
+
+Sample:
+```
+BUTTERFLY-GLITTER-WALLCOVERING-RAINBOW
+GRUNGE-GLITTER-WALLCOVERING-MULTI
+CRACKLE-GLITTER-WALLCOVERING-GOLD
+CRACKLE-GLITTER-WALLCOVERING-CHAMPAGNE
+FISHNET-SEQUIN-GLITTER-WALLCOVERING-LAVENDER
+```
+
+## SKU → URL mapping hypothesis
+The DW SKU like `BUTTERFLY-GLITTER-WALLCOVERING-RAINBOW` likely maps to a vendor PDP at
+`https://glitterwalls.com/products/butterfly-glitter-wallcovering-rainbow` (lowercase).
+Worth one-shot HEAD-testing 5 SKUs to confirm the pattern before building the full scraper.
+
+## Suggested next step
+1. HEAD-test the slug pattern hypothesis on 5 SKUs.
+2. If it works, write a Node script that walks all 237 SKUs, fetches each PDP, extracts `og:image`, writes to `glitter_walls_catalog.image_url`, then re-runs the swap-proposal logic.
diff --git a/rescrape-todo/ralph-lauren-fabric.md b/rescrape-todo/ralph-lauren-fabric.md
new file mode 100644
index 0000000..53d284e
--- /dev/null
+++ b/rescrape-todo/ralph-lauren-fabric.md
@@ -0,0 +1,38 @@
+# Ralph Lauren Fabric — Re-scrape TODO
+
+**Status**: 1,055 affected SKUs, NO existing scraper-manager skill, NO vendor-side CDN URLs in catalog.
+
+## The problem
+- `rl_catalog` has 3,148 rows but 1,979 (63%) point to dead `cdn.shopify.com/s/files/1/0015/4117/7456/...` URLs.
+- The remaining rows mirror images from sister sites (`bhf-cdn.azureedge.net`, `designersguild.com`) — NOT from `ralphlaurenhome.com`.
+- All 1,055 SKUs flagged by the dead-image audit have ONLY the dead Shopify CDN URL in `rl_catalog`.
+- The `rlf-crawler` Express viewer at `~/Projects/Designer-Wallcoverings/DW-Programming/vendor-crawlers/rlf-crawler/server.js` exists, but the table name it references (`rlf_catalog`) does not exist — and there is NO `fetchFn` passed to its `triggerCrawl()` call. The crawler is a stub.
+
+## Recommendation
+Build a `ralph-lauren-fabric-scraper-manager` skill modeled on `romo-scraper-manager`.
+
+**Trade portal**: `ralphlaurenhome.com` (Pratt & Larson / Ralph Lauren Home / Ralph Lauren Fabric all live here)
+**Login required**: Yes — credentials must be added to `/root/DW-Agents/dw-price-stock/.env` as `RLF_USERNAME` / `RLF_PASSWORD`.
+**SKU prefix in DW**: `DWRR-` (already in use, see `rl_catalog.dw_sku`)
+**Manufacturer SKU format**: Mixed — `PRL5083/11`, `ZA8059M`, `KT2263M`, `Prl026/23`, plus the 1,055 affected SKUs use `RLDF-NNNNN` (e.g., `RLDF-18250`, `RLDF-18614`).
+**Target DB table**: New `rlf_catalog` (currently missing — must `CREATE TABLE` modeled on `romo_catalog` schema).
+
+## Affected SKU list
+1,055 SKUs at `/tmp/rlf_affected_skus.csv` (also persisted to `image_url_swap_proposals_2026_05_19` rows where `vendor='Ralph Lauren Fabric'`).
+
+Sample:
+```
+RLDF-18250
+RLDF-18614
+RLDF-18655
+RLDF-18779
+RLDF-18860
+```
+
+## Why this can't be done from Mac2
+- No login credentials for `ralphlaurenhome.com` available locally.
+- No PDP URL pattern documented.
+- The vendor scraper infrastructure (`vendor-crawlers/lib/crawl-engine.js`) expects a `fetchFn` to be authored per vendor.
+
+## Suggested next step
+Spawn a `ralph-lauren-fabric-scraper-manager` skill build via the `skill-creator` skill, modeled after `romo-scraper-manager`, then have it walk the RL Home trade portal and write to a new `rlf_catalog` PG table.
diff --git a/rescrape-todo/romo.md b/rescrape-todo/romo.md
new file mode 100644
index 0000000..aecb7b1
--- /dev/null
+++ b/rescrape-todo/romo.md
@@ -0,0 +1,29 @@
+# Romo — Re-scrape TODO
+
+**Status**: 930 affected SKUs. 797 refreshed THIS pass via SKU-format normalization (no actual scraping needed). 133 still unmatched.
+
+## What this pass discovered
+The prior agent's swap-proposal join used `UPPER(rc.mfr_sku) = UPPER(p.sku)`, which compares `W920-02` (dash form, as stored in `shopify_products.mfr_sku`) against `W920/02` (slash form, as stored in `romo_catalog.mfr_sku`). **Every Romo affected SKU was missed by the prior join** because of this format mismatch.
+
+When I re-join with `REPLACE(p.sku, '-', '/')`, all 930 SKUs match `romo_catalog`, and 797 of them already have alive `static.theromogroup.com` vendor-CDN URLs sitting unused.
+
+## What was staged for push-A2
+- 797 swap proposals inserted into `image_url_swap_proposals_2026_05_19_v2` with `source_table='romo_catalog'`, `match_strategy='sku_dash_to_slash'`, `new_url_status_in_cache='success'`.
+- These URLs were already in `image_hashes` as `status='success'` from prior cataloging — no further verification needed.
+
+## Remaining 133 unmatched
+- 74 are in `romo_catalog` with vendor-CDN URLs but were flagged `decode_error` in `image_hashes` (`VipsJpeg: Corrupt JPEG data` — transient pipeline error during prior fetch). Sample HEAD-test confirmed all 10 sampled URLs return 200. These should be retried by image-fetcher with a relaxed Sharp/Vips error handler.
+- 59 are NOT in `romo_catalog` at all (the slash-normalized SKU is unknown to the catalog). Likely discontinued or never crawled.
+
+## Recommendation
+1. **Immediate (free)**: Re-run image-fetcher over the 74 `decode_error` URLs with relaxed Sharp `failOn: 'truncated'` config — they'll likely flip to `success`. Then re-stage them into v2 same way as the 797.
+2. **For the remaining 59**: Invoke `romo-scraper-manager` skill via the Kamatera trade-portal login at `45.61.58.125:9830` (not reachable from Mac2). Pass these 59 SKUs in slash form.
+
+## Affected SKU list
+`/tmp/romo_affected_skus.csv` — all 930. The 797 refreshed are recorded in `image_url_swap_proposals_2026_05_19_v2`.
+
+## Standing rule for the swap-proposal pipeline
+The dash-vs-slash SKU normalization issue likely affects OTHER vendors too. The next iteration of the proposal-builder should normalize on both sides:
+```sql
+ON UPPER(REPLACE(REPLACE(rc.mfr_sku, '-', ''), '/', '')) = UPPER(REPLACE(REPLACE(p.sku, '-', ''), '/', ''))
+```
← 5be39bd init: dw-monitoring + daily broken-image regression guard (b · back to Dw Monitoring · fix: correct package.json main entry and add backup-file pat 772aea1 →