`STALE-URL-RESCRAPE-RUNBOOK.md`

95 lines
# DW Stale-URL Re-Scrape Runbook — Tier 2-4 Cleanup

**Generated:** 2026-05-18 · **Mode:** READ-ONLY investigation · **Writes made:** ZERO
**Status:** Awaiting Steve approval before any re-scrape runs.

---

## 1. Confirmed affected-product counts (current, post Tier-1 delete)

Query (read-only) used:

```sql
SELECT vc.vendor_code, count(*) AS affected
FROM vendor_catalog vc
JOIN image_hashes ih ON ih.url = vc.image_url
WHERE ih.status <> 'success'
GROUP BY vc.vendor_code
ORDER BY affected DESC;
```

**Total: 21,563 affected products across 56 vendors.**

`vendor_catalog` has **no `local_image_path` column** — there is no on-disk fallback; the broken `image_url` is the only image these rows have. Fix = refresh `image_url` via re-scrape (or, for Shopify-CDN cases, re-resolve the CDN URL).

The numbers are very close to the earlier estimate — within a few rows per vendor, no material drift since Tier-1.

## 2. Error-type breakdown (drives the recommended action)

| Error class | Meaning | Implication |
|---|---|---|
| `http_404` | URL gone (vendor moved/renamed asset, or our Shopify CDN re-versioned) | Needs fresh URL → re-scrape |
| `fetch_error` (`fetch failed`) | DNS/connection failed at fetch time — often a transient host outage | May just need a retry |
| `decode_error` | Fetched bytes are not a decodable image | Scraper captured a placeholder/wrong asset → fix scraper |
| `http_401` / `http_403` | Auth wall / blocked | Needs authenticated scraper or new access |
| `http_5xx` / `http_52x` / `timeout` | Server-side / CDN flakiness | Retry-only |
| `invalid_url` / `too_small` | Malformed URL or thumbnail-stub captured | Scraper bug → fix scraper |

### Special-case findings (verified this pass)

- **marburg — 3,546 rows, ALL `fetch_error: "fetch failed"`.** URLs are `https://marburg.com/wp-content/uploads/...jpg` — a single host, single failure mode. This is the classic **host-outage pattern**. Strong candidate for **retry-only** (re-run the image fetcher against existing URLs) before committing to a full re-scrape. Biggest quick win in the whole list.
- **rwltd — 905 rows, 881 are `decode_error`.** The stored `image_url` for those rows is a **base64 data-URI SVG placeholder** (`data:image/svg+xml;base64,...` decoding to a 99999×99999 transparent `<rect>` — a lazy-load stub). The scraper grabbed the lazy-load placeholder instead of the real `<img src>`/`data-src`. This is a **scraper bug — fix the scraper first**, then re-scrape. Remaining 24 are `http_502`/`timeout` (retry).
- **romo — 1,045 rows: 916 `http_404` + 127 `decode_error`.** The 127 decode_errors are `content_type: image/jpeg` that won't decode (truncated/corrupt fetches). Romo group spans multiple brand sites (kirkbydesign.com, theromogroup.com, villanova.co.uk) — re-scrape needs the per-brand routing the romo scraper already has.
- **wolf_gordon — 200 rows: 191 `http_401` + 7 `http_403` + 2 timeout.** Confirmed **trade-portal auth wall** (`cdn2-optimize.wolfgordon.com` returns 401 Unauthorized). Cannot re-scrape without authenticated session — needs scraper-side auth wired before any run.
- **pointe-intl** — flagged in the brief as "HTML-as-image scraper bug" — **NOT present** in `vendor_catalog` (0 rows). No action; bug already purged or vendor not loaded.
- **Dead-vendor check** — of et_cie / folia / hermes / laura_ashley, only **laura_ashley** is in `vendor_catalog` (104 rows total) and it has just **1 affected row** (`fetch_error`). et_cie / folia / hermes have zero rows. No affected vendor sits meaningfully on the dead list — laura_ashley's single row is negligible.
- **cowtan_tout — 8,634 rows, ALL `http_404`.** URLs point at `d2mq91o692rj7w.cloudfront.net/images/Products/<SKU>.jpg` (Cowtan's own CloudFront). The whole image path appears retired/renamed. Largest single block — needs a full re-scrape of designs.cowtan.com via the cowtan colorway scraper.
- **ralph_lauren (1,543) + hollywood (1,712) — ALL `http_404`, and the URLs point at OUR OWN Shopify CDN** (`cdn.shopify.com/s/files/1/0015/4117/7456/...`). These rows reference Shopify-hosted files that have since been deleted or re-versioned (`?v=` query changed). This is **not a vendor re-scrape problem** — it's a Shopify-CDN URL refresh. Re-resolve the current CDN URL for each product from `shopify_product_id` (READ via Shopify API), or re-scrape the source vendor. Note: there is **no Ralph Lauren / Hollywood scraper skill** — both are likely Shopify-origin / private-label catalogs. Treat as a Shopify-side image refresh job, not a scraper run.

## 3. Scraper-skill availability

Scraper skills present under `~/.claude/skills/`: 1838, arc-com (colorway), arte-international (colorway), arteriors, atomic50, bespoke, bnwalls, brewster, colefax-fowler (colorway), colouranddesign, command54, contrado, cowtan (colorway), dedar, denovowall, designtex, fabricut, folia, franquemont, hygge, jim-dultz, justindavid, knoll, maya-romanoff, mindthegap, muralsource, naturale54, osborne, reid-witlin, romo, scalamandre, stout, timorous, wallquest, york / york-wallcoverings.

**No pm2 scraper processes are running** — `pm2 list` shows only `national-paper-hangers`. All scraper skills are run-on-demand, so "scraper health" below = "skill exists / skill missing", not a live process.

## 4. Prioritized runbook (ordered by impact; quick wins called out)

| # | Vendor | Affected | Scraper skill | Scraper health | Recommended action | Effort |
|---|---|---:|---|---|---|---|
| 1 | **cowtan_tout** | 8,634 | `cowtan-colorway-scraper` | Skill exists, no live proc | **Re-scrape** designs.cowtan.com — all `http_404`, CloudFront path retired | High (largest block) |
| 2 | **marburg** | 3,546 | `naturale54`? — **no dedicated marburg scraper** | None | **QUICK WIN — retry-only first.** All `fetch_error` on one host = outage. Re-run image fetcher against existing URLs; only re-scrape if retry still fails | Low (retry) / Med (if re-scrape needed) |
| 3 | **hollywood** | 1,712 | NONE | n/a | **Shopify-CDN refresh** — URLs are our own `cdn.shopify.com`, 404 = re-versioned files. Re-resolve via `shopify_product_id` | Med |
| 4 | **ralph_lauren** | 1,545 | NONE | n/a | **Shopify-CDN refresh** — same as hollywood; re-resolve current CDN URLs | Med |
| 5 | **romo** | 1,045 | `romo-scraper-manager` | Skill exists | **Re-scrape** (916 `http_404`) — multi-brand routing already in skill; 127 decode_errors covered by same pass | Med |
| 6 | **rwltd** | 905 | `reid-witlin-scraper-manager` | Skill exists | **FIX SCRAPER FIRST** — 881 rows stored a base64 SVG lazy-load placeholder; patch image-selector (use real `data-src`/`src`), then re-scrape | Med (bug fix + run) |
| 7 | **graham_brown** | 607 | NONE | n/a | **No-scraper — manual / build scraper.** All `http_404`; grahambrown.com | Med |
| 8 | **spoonflower** | 541 | NONE (DW-internal Spoonflower pipeline exists, not a scraper skill) | n/a | **Re-resolve** — `garden.spoonflower.com` asset URLs 404'd; refresh via Spoonflower pipeline / product re-pull | Med |
| 9 | **nina_campbell** | 445 | NONE | n/a | **No-scraper — manual / build scraper.** ninacampbell.com, all `http_404` | Med |
| 10 | **thibaut** | 358 | NONE | n/a | **Re-scrape / manual** — 348 `http_404` + 8 `http_524` (retry the 524s) | Low-Med |
| 11 | **missoni** | 329 | NONE | n/a | No-scraper — manual; all `http_404` | Low-Med |
| 12 | **glitter_walls** | 240 | NONE | n/a | No-scraper — manual; 238 `http_404` | Low |
| 13 | **schumacher** | 224 | `scalamandre-catalog-scraper` (sister brand) | Skill exists | Re-scrape; 221 `http_404` | Low |
| 14 | **osborne** | 222 | `osborne-scraper-manager` | Skill exists | Re-scrape; all `http_404` | Low |
| 15 | **wolf_gordon** | 200 | NONE | n/a | **FIX SCRAPER FIRST — needs trade-portal auth.** 191 `http_401` on `cdn2-optimize.wolfgordon.com`; wire authenticated session before any run | Med-High |
| 16 | **colefax_fowler** | 162 | `colefax-fowler-colorway-scraper` | Skill exists | Re-scrape; all `http_404` | Low |
| 17 | **arte** | 142 | `arte-international-colorway-scraper` | Skill exists | Re-scrape (133 `http_404`) + retry 9 `http_403` | Low |
| 18 | **as_creation** | 138 | NONE | n/a | No-scraper — manual; 134 `http_404` | Low |
| 19 | **bnwalls** | 113 | `bnwalls-scraper-manager` | Skill exists | Re-scrape; all 113 `http_403` (may need UA/header tweak) | Low |
| 20 | **timorous** | 50 | `timorous-scraper-manager` | Skill exists | Re-scrape; all `http_404` | Low |
| 21 | **command54** | 49 | `command54-scraper-manager` | Skill exists | Re-scrape; 42 `http_404` | Low |
| 22 | **wallquest** | 4 | `wallquest-scraper-manager` | Skill exists | Retry-only; 4 `timeout` | Trivial |
| 23 | **mindthegap** | 1 | `mindthegap-scraper-manager` | Skill exists | Retry-only; 1 `fetch_error` | Trivial |
| — | **long tail** (innovations 56, newmor 45, manuel_canovas 65, relativity_textiles 30, villa_nova 22, brewster_york 21, rebel_walls 21, versa_ds 17, maya 13, coordonne 10, plus ~15 vendors with 1-7 rows each) | ~430 total | mixed (maya-romanoff, brewster, york skills exist; most don't) | mixed | Batch retry the `5xx`/`timeout`/`fetch_error` rows fleet-wide first (cheap), then re-scrape the `http_404` residue per vendor where a skill exists | Low (batched) |

### Suggested execution order

1. **Phase 0 — fleet-wide retry sweep (cheapest, highest ROI).** Re-fetch every row whose error is `fetch_error` / `http_5xx` / `http_52x` / `timeout` against its *existing* URL. This alone clears marburg (3,546) if the host is back, plus the long-tail flakiness — potentially ~4,000+ rows with zero scraping.
2. **Phase 1 — scraper-bug fixes.** Patch `reid-witlin` (placeholder-SVG selector) and wire `wolf_gordon` trade-portal auth *before* their re-scrapes.
3. **Phase 2 — big re-scrapes with existing skills.** cowtan_tout, romo, osborne, colefax_fowler, arte, schumacher, bnwalls, timorous, command54.
4. **Phase 3 — Shopify-CDN refresh.** hollywood + ralph_lauren — re-resolve image URLs from `shopify_product_id` (Shopify API read), not a vendor scrape.
5. **Phase 4 — no-scraper vendors.** graham_brown, nina_campbell, missoni, glitter_walls, as_creation, spoonflower — decide build-scraper vs. manual per vendor.

---

*All counts above from live SELECT queries against `dw_unified` on 2026-05-18. No writes were made to `dw_unified`, Shopify, or any production system.*