← back to Dw Sku Integrity

evidence/TK-10900-rescrape-workplan.md

291 lines

# TK-10900 — canonical `dw_sku` recovery: the "re-scrape" backlog work-plan

**Parent:** TK-10896 (recoverable-now recovery — DONE, 4,547 codes). This plan owns the
**10,671 RESCRAPE + 145 IMPORT_DEFECT** rows that TK-10896's classifier deferred.
**Owner:** vp-dw-commerce. **Status:** DRAFT / read-only planning artifact. Generated 2026-08-31.
All numbers are from **canonical Kamatera** (`ssh root@45.61.58.125 psql dw_unified`) + the
Mac2 staging catalogs (`psql -h /tmp -d dw_unified`).

---

## Headline: this is mostly a LOCAL CONTENT-MATCH job, NOT a web re-scrape

The classifier binned these rows `RESCRAPE` because the blank Shopify rows carry **no `sku`
and no `mfr_sku`**, so the exact `mfr_sku → <vendor>_catalog` join (the staging-link method
already shipped in `staging-link-gen.mjs`) cannot fire. **But the vendor catalogs already hold
the real codes.** They just need a fuzzier join key — `title → pattern_name + color_number` —
i.e. the "EXACT pattern+color content-match" method from the `dw-unbuyable-recovery-methodology`
memo. That is a `$0` local recovery. A true web scrape is only needed for the small residual of
vendors whose catalog is **missing or thinner than the blank set** (~450 rows).

So TK-10900 splits into four programs, not "openclaw-scrape 10,800 products":

| Program | What | Rows | Cost | Path |
|---|---|---|---|---|
| **A · Content-match recovery** | Join Shopify title → catalog pattern+color → real code | ~9,950 | $0 local | extend `staging-link-gen.mjs` |
| **B · Phillipe Romano (Greenland private label)** | Cork-line, own catalog + agent | 3,022 | $0 local (+ scrape gap) | `greenland-scraper-manager` |
| **C · Import-defect (`null-Sample`)** | China Seas staging-linkable; PR folds into B | 145 | $0 local | staging-link / B |
| **D · True web re-scrape** | Catalog missing/thin → fetch fresh identity | ~450 | metered scrape | `*-scraper-manager` skills |

> Program A + B overlap the same catalog-match engine; they're split only because PR is a
> private label with its own agent and is the single biggest owner after Carnegie.

---

## Two hard provenance rails the recovery MUST encode

Both are the never-mint doctrine (`dw-dwsku-keep-old-never-mint`) applied to this data:

1. **Mint-catalog vendors — use `mfr_sku`, NEVER `dw_sku`.** For **Carnegie, Maharam, CMO Paris,
   Stout** every catalog `dw_sku` is a reverted Phase-4 **greenfield mint** (`DWAG-*` etc.).
   Content-matching to `catalog.dw_sku` would silently **re-instate the exact mints we reverted**.
   Recovery for these four pulls `catalog.mfr_sku` (the real code) instead. The other catalog
   vendors carry a clean `dw_sku` and use it directly.
2. **Strip the DW type-suffix to the base code + break the width tie.** Carnegie `mfr_sku` reads
   `655790-panels`, `6557W94-wallcoverings` — the real code is the token before the first `-`
   (`655790`, `6557W94`). After stripping, ~46% of Carnegie titles match a **unique** code; the
   rest are same-title Upholstery-vs-Wallcovering pairs (`7360A11` vs `7360WA11`) that the blank
   row's `product_type` (`Upholstery`) disambiguates. Verified hit rates below.

Every write stays **gated** (canonical customer-facing), `shopify_id`-keyed, blank-guarded,
per-item verified, reversible + ledgered — identical guardrails to the TK-10896 fire scripts.

---

## Per-vendor targets — ordered by row count

`code source` = which catalog column yields the canonical code. `catalog` = staging rows available.
`skill` = the `*-scraper-manager` that owns the web-scrape fallback / provenance.

| # | Vendor | Blank rows | Program | Catalog (rows) | Code source | Scraper skill |
|---|---|--:|---|---|---|---|
| 1 | Carnegie | 3,921 | A | `carnegie_catalog` (5,928) | `mfr_sku` (mint dw_sku) | *(none — `dw-carnegie-price-canary`; build if D needed)* |
| 2 | Phillipe Romano | 3,022¹ | B | `greenland_catalog` (Cork-) | private-label Cork code | `greenland-scraper-manager` |
| 3 | Maharam | 1,429 | A | `maharam_catalog` (1,601) | `mfr_sku` (mint dw_sku) | *(none — catalog-match)* |
| 4 | Knoll | 505 | A | `knoll_catalog` (632) | `dw_sku` (clean) | `knoll-scraper-manager` (login) |
| 5 | Vahallan | 386 | A | `vahallan_catalog` (390) | `dw_sku` (clean) | *(none — catalog-match)* |
| 6 | CMO Paris | 344 | A | `cmo_paris_catalog` (345) | `mfr_sku` (mint dw_sku) | *(none — catalog-match)* |
| 7 | Hollywood Wallcoverings | 286 | A | `hollywood_catalog` (194 dw / 3,466 mfr) | HWC `mfr_sku`² | `dw-hollywood-sku-canary` |
| 8 | Fentucci Naturals | 213 | A+D | `fentucci_naturals_catalog` (148) | `dw_sku` + ~65 scrape gap | `fentucci-scraper-manager` |
| 9 | Stout Textiles | 180 | A | `stout_catalog` (180, exact) | `mfr_sku` (mint dw_sku) | `stout-scraper-manager` (login) |
| 10 | **Majilite** | 160 | **D** | — (no catalog, no skill) | web scrape | ⚠ **new scraper build** |
| 11 | Fabricut | 110 | A | `fabricut_catalog` (1,333) | `dw_sku` (clean) | `fabricut-scraper-manager` (login) |
| 12 | Novasuede | 67 | A | `novasuede_catalog` (142) | `dw_sku` (clean) | `novasuede-updater` |
| 13 | Thibaut | 65 | A | `thibaut_catalog` (6,057) | `dw_sku` (clean) | *(none — catalog-match; login site)* |
| 14 | Rebel Walls | 52 | A | `rebelwalls_catalog` (4,350) | `dw_sku` (clean) | `rebel-walls-scraper-manager` |
| 15 | **Pixels** | 52 | **D** | `vendor_catalog` (7, generic) | POD/print — special | ⚠ POD, not a wallcovering scrape |
| 16 | China Seas | 29¹ | C | `china_seas_catalog` (2,650) | `mfr_sku` present → **staging-link** | *(existing staging-link tool)* |
| 17 | Steve Abrams Studios | 13 | D | — (FAA art prints) | FAA order code | `fineartamerica-scraper-manager` |
| 18 | **Jeffrey Stevens** | 12 | **D** | — (no catalog) | web scrape | ⚠ **new scraper build** |
| 19 | Romo | 11 | A | `romo_catalog` (2,648) | `dw_sku` (clean) | `romo-scraper-manager` (login) |
| 20 | Pierre Frey | 4 | A | `pierre_frey_catalog` (1,161) | `dw_sku` (clean) | `pierre-frey-scraper-manager` |
| 21 | Graduate Collection UK | 4 | A/D | `vendor_catalog` (182, generic) | `dw_sku` partial | *(generic table)* |
| 22 | Arte International | 1 | A | `arte_catalog` (1,991) | `dw_sku` (clean) | `arte-international-colorway-scraper` |
| 23 | Harlequin | 1 | A | `harlequin_catalog` (763) | `dw_sku` (clean) | *(catalog-match)* |
| 24 | Designer Wallcoverings | 1 | manual | — (house SKU) | manual | — |

¹ Includes import-defect (`null-Sample`) rows: **Phillipe Romano 116**, **China Seas 29** = the 145.
² Hollywood is the HWC private-label reconciliation (`hollywood-hwc-line-lbi-boyd`, TK-10633) — its
  real mfr code is unrecovered for the 2019-era line; treat as **best-effort HWC match, gaps → TK-10633**,
  do NOT name-match on DW-invented product names.

**Totals:** Program A ≈ 6,050 · Program B (PR) 3,022 · Program C 29 (CN import-defect) · Program D ≈ 450.

### Program A splits by catalog schema (fleet sweep, 2026-08-31) — corrects the optimistic estimate

The `content-match-gen.mjs` sweep proved the catalogs are heterogeneous, so Program A is two builds:

| Sub | Vendors | Blank | Recoverable now | State |
|---|---|--:|--:|---|
| **A1 · numeric color** (`color_number`, title ends in a number) | Carnegie | 3,921 | **2,745 safe** (+1,176 review) | ✅ **shipped** |
| **A2 · color-name / title** (`color_name`, title ends in a name) | see provenance outcome below | ~3,300 | **~13 + tails** | 🔴 **mostly re-routed to D** (Steve ruling + full ledger) |
| **A0 · deferred schema** (title-only / thin) | Vahallan 386 · Novasuede 67 | 453 | 0 | ⚠ title-match or D |

**A2 final outcome (after Steve's provenance rulings + the full 12,216-code ledger, 2026-08-31):** the
unified concat matcher works mechanically, but per-vendor provenance re-routed almost all of A2 to
Program D: **Maharam 1,429 + CMO 344 → D** (Steve ruled the `MH-*`/`CMO_*` codes are DW-fabricated →
re-scrape for real codes); **Knoll → D** (full ledger showed 492/632 catalog codes are reverted mints;
only 13 native recoverable); **Thibaut 65 + Rebel Walls 52 → title-format mismatch → D/investigate**.
Net Program-A content-match yield ≈ **Carnegie 2,745 + ~13 Knoll + small tails**, NOT ~6,050. The bulk
of the "re-scrape backlog" really is a re-scrape (D). The matcher + full ledger's value was **preventing
wrong writes** (1,235 fabricated Maharam + 492 reverted-mint Knoll codes).

### Program A — FINAL fire-ready roster (all preflight GO, gated; Steve fires)

Completed the sweep across every clean/confirm vendor with the unified matcher + full 12,216 ledger:

| Vendor | Rows | Code sample | Provenance basis | Preflight |
|---|--:|---|---|---|
| **Carnegie** | 2,745 | `655790` | real mfr code — Steve-ruled bare base (2026-08-31) | GO |
| **Stout Textiles** | 180 | `W7853-3` | real mfr code (same class as Carnegie) | GO |
| **Fabricut** | 105 | `DWFC-230457` | DW-namespace code, ledger-cleared native (cf. China Seas `DWCH`) | GO |
| **Knoll** | 13 | `DWKN-250638` | DWKN native (492 reverted-mint siblings excluded by full ledger) | GO |
| **Romo** | 10 | `DWRM-241256` | DW-namespace native, not in reverted ledger | GO |
| **Total A** | **3,053** | | | all GO |

**A3 title-scrub landed (2026-08-31):** a general `scrubTitle` (strip `"Wallcovering(s)"`, `","`→space,
applied to both keys) recovered the storefront-title-noise vendors from their EXISTING catalogs with
**zero regression** (all 5 above unchanged) and verified-correct + provenance-clean (0 in reverted ledger):

| Vendor | Rows | Code | Preflight |
|---|--:|---|---|
| **Thibaut** | 61 | `DWTT-74442` (native) | GO |
| **Rebel Walls** | 52 | `DWRW-361409` (native) | GO |
| **Pierre Frey** | 4 | native | GO |

The same scrub also recovered two more from their existing catalogs (verified, provenance-clean, GO):
**Vahallan 384** (title-mode; `DWYG-*` native — the biggest surprise, was "deferred") and **Arte 1**.

**Program A total fire-ready = 3,555** (3,053 + 117 + 385). **Recoverable-now grand total (A + C China
Seas 29) = 3,584 across 11 vendors** — all gated, reversible, ledgered, verified, provenance-clean. Fire A:
`fire-content-match.sh Carnegie "Stout Textiles" Fabricut Romo Knoll Thibaut "Pierre Frey" "Rebel Walls" Vahallan "Arte International"`.

The catalog-recoverable set is now **exhausted**. Everything remaining genuinely needs re-scrape →
**Program D** (Maharam 1,429 · Knoll-resid 490 · CMO 344 · Hollywood 286 · Fentucci 213 · Majilite 160 ·
Novasuede 67 · Pixels 52 · FAA 13 · Jeffrey Stevens 12 · Harlequin 1 ≈ 3,150) or **Program B** Greenland/PR 3,022.

A1 is done. A2's approach is now PROVEN: build a CONCATENATION key — catalog `norm(pattern_name+' '+color_name)`
matched against the vendor-stripped normalized Shopify title — which sidesteps the ambiguous split entirely
(Maharam: 88.9% unique / 1,271 of 1,429 / 0 no-match). The same key also reproduces Carnegie, so A2 is really a
UNIFIED concat matcher that can replace A1's trailing-number parser — keeping bucket-compat + base-strip + mint
rails, re-verifying Carnegie, and taking a fresh contrarian pass (per the Defect-B lesson). Per-vendor title
normalization still needed (Maharam = `"<colorway> | Maharam"` suffix; Carnegie = `"Carnegie …"` prefix).

---

## Verified content-match evidence (Carnegie, the 3,921-row canary)

Measured this session by the actual tool (`content-match-gen.mjs`), not assumed. Two things the
schemas forced (see the build insights): catalog `product_type` vocab does NOT match Shopify's
(Shopify blanks are 99.8% `Upholstery`; the catalog splits one code across
`Upholstery`/`Windows`/`Museum Display Cases`/`Wallcoverings`), and catalog schemas differ
(`carnegie_catalog` has `color_number`; `knoll_catalog`/`rebelwalls_catalog` only `color_name`) —
so the tool is schema-adaptive and uses `product_type` as a **bucket** (wall/fabric/other), not a key.

```
title→(pattern,color), base-code collapse, bucket TIEBREAK only (v1) .. 3,439 matched (87.7%) — UNSAFE
  └─ contrarian (Cody) Defect B: single-candidate keys skipped the bucket check →
     wall-only catalog codes (Abbey 61 -> 7394A61 'Upholstered Walls/Panels') stamped onto
     Upholstery rows. Blast radius: 1,523 wall-only single-base keys.
title→(pattern,color), bucket-compat HARD PRECONDITION (v2, shipped) ... 2,745 matched (70.0%) — SAFE
     694 cross_class_mismatch → review · 468 variant-fork ambiguous → review · 14 parse-miss
```

**70.0% is the honest, safe rate** — every emitted code is bucket-compatible with the row. The 694
cross-class + 468 ambiguous (1,162 total) defer to human review / Program D re-scrape rather than
risk a wrong write ("a false-positive SKU is worse than a blank"). Expect similar or higher safe
rates on clean-catalog vendors (no mint-`mfr_sku` indirection).

---

## ⚠ Provenance gate — the catalog code is NOT uniformly a "real" code (2026-08-31)

Verification (spot-checking the matcher's output) found the recovered code's provenance varies by
vendor. Program A **cannot be blanket-fired**; each vendor needs a provenance ruling before its draft
is trusted. The matcher is mechanically correct (Carnegie reproduced exactly); this is about *what the
catalog code IS*, not whether the join works.

| Provenance class | Vendors | Sample catalog code | Ruling needed |
|---|---|---|---|
| **Real mfr code** — safe to adopt | Carnegie (`655032`), Stout (`W7899-2`) | numeric / `W####-#` | just confirm base-code form (open-decision #1) |
| **DW-fabricated** (brand-prefix invention, not a mfr code) | Maharam (`MH-GEMMA…-031`), CMO Paris (`CMO_WBO_09_80`) | `MH-<pattern>-###` | is this an acceptable canonical `dw_sku`, or re-scrape (D)? |
| **Mixed-use mint prefix** (native + Phase-4 mint intermixed) | Knoll (`DWKN-*`), Thibaut (`DWTT-*`), Rebel Walls (`DWRW-*`) | `DWKN-250130` | needs the FULL reverted-mint ledger to split native vs mint (= `PROVENANCE_REVIEW`) |
| **DW prefix, provenance TBD** | China Seas (`DWCH/DWCW-*`), Fabricut (`DWFC-*`) | `DWCH-510010` | confirm these are established DW codes, not fabrications |

Consequence: the only vendor cleanly fireable today is **Carnegie** (real codes, 2,745, base-form pending
open-decision #1). Maharam/CMO are provenance-blocked; Knoll/Thibaut/Rebel need the mint ledger; China
Seas/Fabricut need a quick confirm. This is why every write stays gated — the matcher can find a code, but
only Steve can rule whether that code is the canonical identity or a fabrication to be replaced.

## Execution shape (per program, all gated writes fired by Steve)

### Program A — `content-match-gen.mjs` (SHIPPED; sibling of `staging-link-gen.mjs`)
1. Read canonical blank-`dw_sku` active rows for the vendor (title, product_type, shopify_id).
2. Schema-adaptively read `<vendor>_catalog` (detect `color_number` vs `color_name`) → build
   `(pattern_name, color) → {baseCode → set(product_type bucket)}` map, where `code = mfr_sku`
   (mint-catalog vendors) else `dw_sku`, **base-code cleaned** (strip only trailing `-<alpha>`
   suffixes so numeric-hyphenated real codes survive), **excluding greenfield-mint + reverted-ledger
   codes** (`/tmp/_ledcodes.txt`).
3. Parse each Shopify `title` → `pattern + trailing color token`; look up `(pattern, color)`. Emit a
   write **only when the key resolves to exactly ONE base code AND that code's catalog `product_type`
   bucket is compatible with the row's bucket** (wall↔wall, fabric↔fabric; `other` passes). `product_type`
   is a compatibility **precondition + tiebreak**, NOT a key component (its cross-source vocab mismatch
   would otherwise cause misses). Anything else — multi-code fork, cross-class, parse-miss, no-map —
   → **no-write + `review-queue.json`**, never a best-guess. Rationale (codex + contrarian, 2026-08-31):
   a false-positive SKU is worse than a blank; the bucket precondition is what stops the Abbey-61 class.
   Unique+compatible matches → `apply-plans-content-match/<vendor>/{apply,undo}.sql` + `restore-map.json`,
   `shopify_id`-keyed + blank-guarded.
4. `preflight-check.mjs` (GO) → **Steve fires** → `verify` (GO) → ledger. Canary smallest vendor first
   (Harlequin 1 → Arte 1 → Pierre Frey 4 → … → Carnegie 3,921 last).
5. Per-item verify sample (unbuyable-recovery methodology) before scaling each vendor.

### Program B — Phillipe Romano / Greenland
Route to `greenland-scraper-manager`. Content-match blanks → `greenland_catalog` Cork- codes
(same engine as A). The 116 `null-Sample` import-defect rows fold in here. Any true gap → the
existing Greenland feed-first scrape (open JSON API, $0).

### Program C — Import-defect (145)
- **China Seas 29** — these `null-Sample` rows DO carry `mfr_sku`, so they are **staging-linkable
  today** via the existing `staging-link-gen.mjs` (`china_seas_catalog` has 2,650 rows). Cheapest win.
- **Phillipe Romano 116** — folded into Program B.
- **Never** write `null`/`null-Sample` back as a code (the JOB-4 "don't paper over" rule).

### Program A3 — title-normalizer recovery (~70–120 rows, $0, catalog HAS the code)
Vendors whose blank Shopify rows failed the concat match on **title-format noise only** — the product
IS in the catalog (verified). A per-vendor title normalizer feeding the existing concat matcher recovers
them without a scrape:
- **Thibaut 65** — strip `" Wide Width"`, `", "`, `" Wallcoverings | Thibaut"` → `pattern + color`
  (Woolston/Saddle Weave/Taluk Sisal confirmed present). Provenance: `DWTT` mixed-use → ledger-clear per row.
- **Pierre Frey 4** — strip leading `"Wallcovering "`, `", <color> Wallcoverings | Pierre Frey"`.
- **Arte International 1**, **Harlequin 1** — single rows, normalize or hand-place.
- **Rebel Walls 52** — TBC: confirm blank patterns (`Flowing Pride`…) exist in `rebelwalls_catalog` first;
  if yes → A3, else → D.
Build = a small `{vendor → normalize(title)}` map applied before `stripVendor`; same bucket/mint/gated rails.

### Program D — genuine web re-scrape (~3,050 rows, metered) — catalog CANNOT supply a real code
Per-vendor assignment (priority by count). openclaw is serial (one gateway/profile) so batch small; show
`$` per scrape; feed results into the staging catalog → then re-run Program A's matcher.

| Vendor | Rows | Why D (not catalog) | Scraper skill |
|---|--:|---|---|
| **Maharam** | 1,429 | catalog code is DW-fabricated `MH-*` (Steve-ruled re-scrape) | ⚠ **new build** (`maharam-scraper-manager`) |
| **Knoll residual** | ~490 | catalog codes are reverted mints (ledger-excluded) | `knoll-scraper-manager` (login) |
| **CMO Paris** | 344 | catalog code fabricated `CMO_*` (Steve-ruled) | ⚠ **new build** |
| **Hollywood** | 286 | HWC private-label; real codes unrecovered | → **TK-10633** (`dw-hollywood-sku-canary`) |
| **Fentucci Naturals** | 213 | catalog (148) does NOT cover the blank patterns | `fentucci-scraper-manager` |
| **Majilite** | 160 | no catalog, no skill | ⚠ **new build** |
| **Pixels** | 52 | POD/print — not a wallcovering scrape | print pipeline |
| **Steve Abrams / FAA** | 13 | FAA art prints | `fineartamerica-scraper-manager` |
| **Jeffrey Stevens** | 12 | no catalog | ⚠ **new build** |

**Three new scrapers to build:** `maharam-scraper-manager`, `cmo-paris-scraper-manager` (or Majilite),
`jeffrey-stevens` — gate each build + first live run through Steve (live scraping is externality-gated).

---

## Sequencing & effort

1. **Ship `content-match-gen.mjs`** (extends proven staging-link tooling; ~1 build session).
2. **Program C China Seas** — free staging-link, immediate (29 rows).
3. **Program A** — canary tail-first (single-digit vendors) → mid (Knoll/Vahallan/CMO/Hollywood/
   Stout/Fabricut) → **Carnegie + Maharam last** (the 5,350 heavy rows, after the tiebreak is proven).
4. **Program B** Phillipe Romano via greenland agent (3,022 — the second heaviest).
5. **Program D** true scrapes last; build the two missing scrapers (Majilite, Jeffrey Stevens).

**Expected outcome:** ~10,400 of 10,816 recovered locally at `$0` (Programs A+B+C); ~400 via metered
scrape (D). Canonical blank `dw_sku` backlog **10,868 → ~450**, still zero mints, all gated + reversible.

---

## Open decisions for Steve (CLARIFY gate)

1. **Mint-catalog base code as canonical** — is Carnegie's cleaned `mfr_sku` base (`655790`) the
   canonical `dw_sku`, or should it carry the DW type-suffix / a `DW`-prefixed form? (Recovery uses
   the raw base code by default — this is the one "shape" decision that isn't purely mechanical.)
2. **Two new scrapers** — approve building `majilite-scraper-manager` + a Jeffrey Stevens scraper
   (172 rows combined), or defer/discontinue those lines?
3. **Pixels (52)** — confirm these are POD prints to be handled by the print pipeline, not the
   wallcovering catalog.

Related tickets: TK-10896 (parent), TK-10633 (Hollywood HWC reconciliation), TK-10649 (dedup).