snapshot — gitify backup 2026-05-19

a52d295a1abdea9fb08fd84bbcaed0dbc8145f13 · 2026-05-19 11:35:45 -0700 · Steve
Files touched

M DW-Websites/LosAngelesFabrics
A STALE-URL-RESCRAPE-RUNBOOK.md
Diff

commit a52d295a1abdea9fb08fd84bbcaed0dbc8145f13
Author: Steve <steve@designerwallcoverings.com>
Date:   Tue May 19 11:35:45 2026 -0700

    snapshot — gitify backup 2026-05-19
---
 DW-Websites/LosAngelesFabrics |  2 +-
 STALE-URL-RESCRAPE-RUNBOOK.md | 94 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+), 1 deletion(-)

diff --git a/DW-Websites/LosAngelesFabrics b/DW-Websites/LosAngelesFabrics
index 13ccc4a4..fecd589b 160000
--- a/DW-Websites/LosAngelesFabrics
+++ b/DW-Websites/LosAngelesFabrics
@@ -1 +1 @@
-Subproject commit 13ccc4a459d9cdf78506297520d77c4957be7f1c
+Subproject commit fecd589bcb9612d6d50b8b775aafd9d287be2d27
diff --git a/STALE-URL-RESCRAPE-RUNBOOK.md b/STALE-URL-RESCRAPE-RUNBOOK.md
new file mode 100644
index 00000000..c972f823
--- /dev/null
+++ b/STALE-URL-RESCRAPE-RUNBOOK.md
@@ -0,0 +1,94 @@
+# DW Stale-URL Re-Scrape Runbook — Tier 2-4 Cleanup
+
+**Generated:** 2026-05-18 · **Mode:** READ-ONLY investigation · **Writes made:** ZERO
+**Status:** Awaiting Steve approval before any re-scrape runs.
+
+---
+
+## 1. Confirmed affected-product counts (current, post Tier-1 delete)
+
+Query (read-only) used:
+
+```sql
+SELECT vc.vendor_code, count(*) AS affected
+FROM vendor_catalog vc
+JOIN image_hashes ih ON ih.url = vc.image_url
+WHERE ih.status <> 'success'
+GROUP BY vc.vendor_code
+ORDER BY affected DESC;
+```
+
+**Total: 21,563 affected products across 56 vendors.**
+
+`vendor_catalog` has **no `local_image_path` column** — there is no on-disk fallback; the broken `image_url` is the only image these rows have. Fix = refresh `image_url` via re-scrape (or, for Shopify-CDN cases, re-resolve the CDN URL).
+
+The numbers are very close to the earlier estimate — within a few rows per vendor, no material drift since Tier-1.
+
+## 2. Error-type breakdown (drives the recommended action)
+
+| Error class | Meaning | Implication |
+|---|---|---|
+| `http_404` | URL gone (vendor moved/renamed asset, or our Shopify CDN re-versioned) | Needs fresh URL → re-scrape |
+| `fetch_error` (`fetch failed`) | DNS/connection failed at fetch time — often a transient host outage | May just need a retry |
+| `decode_error` | Fetched bytes are not a decodable image | Scraper captured a placeholder/wrong asset → fix scraper |
+| `http_401` / `http_403` | Auth wall / blocked | Needs authenticated scraper or new access |
+| `http_5xx` / `http_52x` / `timeout` | Server-side / CDN flakiness | Retry-only |
+| `invalid_url` / `too_small` | Malformed URL or thumbnail-stub captured | Scraper bug → fix scraper |
+
+### Special-case findings (verified this pass)
+
+- **marburg — 3,546 rows, ALL `fetch_error: "fetch failed"`.** URLs are `https://marburg.com/wp-content/uploads/...jpg` — a single host, single failure mode. This is the classic **host-outage pattern**. Strong candidate for **retry-only** (re-run the image fetcher against existing URLs) before committing to a full re-scrape. Biggest quick win in the whole list.
+- **rwltd — 905 rows, 881 are `decode_error`.** The stored `image_url` for those rows is a **base64 data-URI SVG placeholder** (`data:image/svg+xml;base64,...` decoding to a 99999×99999 transparent `<rect>` — a lazy-load stub). The scraper grabbed the lazy-load placeholder instead of the real `<img src>`/`data-src`. This is a **scraper bug — fix the scraper first**, then re-scrape. Remaining 24 are `http_502`/`timeout` (retry).
+- **romo — 1,045 rows: 916 `http_404` + 127 `decode_error`.** The 127 decode_errors are `content_type: image/jpeg` that won't decode (truncated/corrupt fetches). Romo group spans multiple brand sites (kirkbydesign.com, theromogroup.com, villanova.co.uk) — re-scrape needs the per-brand routing the romo scraper already has.
+- **wolf_gordon — 200 rows: 191 `http_401` + 7 `http_403` + 2 timeout.** Confirmed **trade-portal auth wall** (`cdn2-optimize.wolfgordon.com` returns 401 Unauthorized). Cannot re-scrape without authenticated session — needs scraper-side auth wired before any run.
+- **pointe-intl** — flagged in the brief as "HTML-as-image scraper bug" — **NOT present** in `vendor_catalog` (0 rows). No action; bug already purged or vendor not loaded.
+- **Dead-vendor check** — of et_cie / folia / hermes / laura_ashley, only **laura_ashley** is in `vendor_catalog` (104 rows total) and it has just **1 affected row** (`fetch_error`). et_cie / folia / hermes have zero rows. No affected vendor sits meaningfully on the dead list — laura_ashley's single row is negligible.
+- **cowtan_tout — 8,634 rows, ALL `http_404`.** URLs point at `d2mq91o692rj7w.cloudfront.net/images/Products/<SKU>.jpg` (Cowtan's own CloudFront). The whole image path appears retired/renamed. Largest single block — needs a full re-scrape of designs.cowtan.com via the cowtan colorway scraper.
+- **ralph_lauren (1,543) + hollywood (1,712) — ALL `http_404`, and the URLs point at OUR OWN Shopify CDN** (`cdn.shopify.com/s/files/1/0015/4117/7456/...`). These rows reference Shopify-hosted files that have since been deleted or re-versioned (`?v=` query changed). This is **not a vendor re-scrape problem** — it's a Shopify-CDN URL refresh. Re-resolve the current CDN URL for each product from `shopify_product_id` (READ via Shopify API), or re-scrape the source vendor. Note: there is **no Ralph Lauren / Hollywood scraper skill** — both are likely Shopify-origin / private-label catalogs. Treat as a Shopify-side image refresh job, not a scraper run.
+
+## 3. Scraper-skill availability
+
+Scraper skills present under `~/.claude/skills/`: 1838, arc-com (colorway), arte-international (colorway), arteriors, atomic50, bespoke, bnwalls, brewster, colefax-fowler (colorway), colouranddesign, command54, contrado, cowtan (colorway), dedar, denovowall, designtex, fabricut, folia, franquemont, hygge, jim-dultz, justindavid, knoll, maya-romanoff, mindthegap, muralsource, naturale54, osborne, reid-witlin, romo, scalamandre, stout, timorous, wallquest, york / york-wallcoverings.
+
+**No pm2 scraper processes are running** — `pm2 list` shows only `national-paper-hangers`. All scraper skills are run-on-demand, so "scraper health" below = "skill exists / skill missing", not a live process.
+
+## 4. Prioritized runbook (ordered by impact; quick wins called out)
+
+| # | Vendor | Affected | Scraper skill | Scraper health | Recommended action | Effort |
+|---|---|---:|---|---|---|---|
+| 1 | **cowtan_tout** | 8,634 | `cowtan-colorway-scraper` | Skill exists, no live proc | **Re-scrape** designs.cowtan.com — all `http_404`, CloudFront path retired | High (largest block) |
+| 2 | **marburg** | 3,546 | `naturale54`? — **no dedicated marburg scraper** | None | **QUICK WIN — retry-only first.** All `fetch_error` on one host = outage. Re-run image fetcher against existing URLs; only re-scrape if retry still fails | Low (retry) / Med (if re-scrape needed) |
+| 3 | **hollywood** | 1,712 | NONE | n/a | **Shopify-CDN refresh** — URLs are our own `cdn.shopify.com`, 404 = re-versioned files. Re-resolve via `shopify_product_id` | Med |
+| 4 | **ralph_lauren** | 1,545 | NONE | n/a | **Shopify-CDN refresh** — same as hollywood; re-resolve current CDN URLs | Med |
+| 5 | **romo** | 1,045 | `romo-scraper-manager` | Skill exists | **Re-scrape** (916 `http_404`) — multi-brand routing already in skill; 127 decode_errors covered by same pass | Med |
+| 6 | **rwltd** | 905 | `reid-witlin-scraper-manager` | Skill exists | **FIX SCRAPER FIRST** — 881 rows stored a base64 SVG lazy-load placeholder; patch image-selector (use real `data-src`/`src`), then re-scrape | Med (bug fix + run) |
+| 7 | **graham_brown** | 607 | NONE | n/a | **No-scraper — manual / build scraper.** All `http_404`; grahambrown.com | Med |
+| 8 | **spoonflower** | 541 | NONE (DW-internal Spoonflower pipeline exists, not a scraper skill) | n/a | **Re-resolve** — `garden.spoonflower.com` asset URLs 404'd; refresh via Spoonflower pipeline / product re-pull | Med |
+| 9 | **nina_campbell** | 445 | NONE | n/a | **No-scraper — manual / build scraper.** ninacampbell.com, all `http_404` | Med |
+| 10 | **thibaut** | 358 | NONE | n/a | **Re-scrape / manual** — 348 `http_404` + 8 `http_524` (retry the 524s) | Low-Med |
+| 11 | **missoni** | 329 | NONE | n/a | No-scraper — manual; all `http_404` | Low-Med |
+| 12 | **glitter_walls** | 240 | NONE | n/a | No-scraper — manual; 238 `http_404` | Low |
+| 13 | **schumacher** | 224 | `scalamandre-catalog-scraper` (sister brand) | Skill exists | Re-scrape; 221 `http_404` | Low |
+| 14 | **osborne** | 222 | `osborne-scraper-manager` | Skill exists | Re-scrape; all `http_404` | Low |
+| 15 | **wolf_gordon** | 200 | NONE | n/a | **FIX SCRAPER FIRST — needs trade-portal auth.** 191 `http_401` on `cdn2-optimize.wolfgordon.com`; wire authenticated session before any run | Med-High |
+| 16 | **colefax_fowler** | 162 | `colefax-fowler-colorway-scraper` | Skill exists | Re-scrape; all `http_404` | Low |
+| 17 | **arte** | 142 | `arte-international-colorway-scraper` | Skill exists | Re-scrape (133 `http_404`) + retry 9 `http_403` | Low |
+| 18 | **as_creation** | 138 | NONE | n/a | No-scraper — manual; 134 `http_404` | Low |
+| 19 | **bnwalls** | 113 | `bnwalls-scraper-manager` | Skill exists | Re-scrape; all 113 `http_403` (may need UA/header tweak) | Low |
+| 20 | **timorous** | 50 | `timorous-scraper-manager` | Skill exists | Re-scrape; all `http_404` | Low |
+| 21 | **command54** | 49 | `command54-scraper-manager` | Skill exists | Re-scrape; 42 `http_404` | Low |
+| 22 | **wallquest** | 4 | `wallquest-scraper-manager` | Skill exists | Retry-only; 4 `timeout` | Trivial |
+| 23 | **mindthegap** | 1 | `mindthegap-scraper-manager` | Skill exists | Retry-only; 1 `fetch_error` | Trivial |
+| — | **long tail** (innovations 56, newmor 45, manuel_canovas 65, relativity_textiles 30, villa_nova 22, brewster_york 21, rebel_walls 21, versa_ds 17, maya 13, coordonne 10, plus ~15 vendors with 1-7 rows each) | ~430 total | mixed (maya-romanoff, brewster, york skills exist; most don't) | mixed | Batch retry the `5xx`/`timeout`/`fetch_error` rows fleet-wide first (cheap), then re-scrape the `http_404` residue per vendor where a skill exists | Low (batched) |
+
+### Suggested execution order
+
+1. **Phase 0 — fleet-wide retry sweep (cheapest, highest ROI).** Re-fetch every row whose error is `fetch_error` / `http_5xx` / `http_52x` / `timeout` against its *existing* URL. This alone clears marburg (3,546) if the host is back, plus the long-tail flakiness — potentially ~4,000+ rows with zero scraping.
+2. **Phase 1 — scraper-bug fixes.** Patch `reid-witlin` (placeholder-SVG selector) and wire `wolf_gordon` trade-portal auth *before* their re-scrapes.
+3. **Phase 2 — big re-scrapes with existing skills.** cowtan_tout, romo, osborne, colefax_fowler, arte, schumacher, bnwalls, timorous, command54.
+4. **Phase 3 — Shopify-CDN refresh.** hollywood + ralph_lauren — re-resolve image URLs from `shopify_product_id` (Shopify API read), not a vendor scrape.
+5. **Phase 4 — no-scraper vendors.** graham_brown, nina_campbell, missoni, glitter_walls, as_creation, spoonflower — decide build-scraper vs. manual per vendor.
+
+---
+
+*All counts above from live SELECT queries against `dw_unified` on 2026-05-18. No writes were made to `dw_unified`, Shopify, or any production system.*
← 63d21512 Phase 3: Shopify-CDN image-URL refresh script for hollywood/ · back to Designer Wallcoverings · chore(git-surgery): draft surgery plan for slimming .git (pr ff2dcc58 →