← back to Dw Sku Integrity

TK-10900: draft re-scrape work-plan (per-vendor, content-match-first)

ddf3994f7af9ad24effffc9a202d1c2092635acb · 2026-08-31 01:56:35 -0700 · codex-10896

Reframes the 10,671+145 'rescrape' backlog: verified that most is recoverable
by LOCAL content-match (title -> catalog pattern+color -> real code), not a web
scrape. Two provenance rails documented (mint-catalog vendors use mfr_sku not
dw_sku; type-suffix strip + product_type width tiebreak). Per-vendor targets
ordered by count, mapped to *-scraper-manager skills, 4 programs A/B/C/D.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files touched

• A evidence/TK-10900-rescrape-workplan.md

Diff

commit ddf3994f7af9ad24effffc9a202d1c2092635acb
Author: codex-10896 <steve@designerwallcoverings.com>
Date:   Mon Aug 31 01:56:35 2026 -0700

    TK-10900: draft re-scrape work-plan (per-vendor, content-match-first)
    
    Reframes the 10,671+145 'rescrape' backlog: verified that most is recoverable
    by LOCAL content-match (title -> catalog pattern+color -> real code), not a web
    scrape. Two provenance rails documented (mint-catalog vendors use mfr_sku not
    dw_sku; type-suffix strip + product_type width tiebreak). Per-vendor targets
    ordered by count, mapped to *-scraper-manager skills, 4 programs A/B/C/D.
    
    Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 evidence/TK-10900-rescrape-workplan.md | 172 +++++++++++++++++++++++++++++++++
 1 file changed, 172 insertions(+)

diff --git a/evidence/TK-10900-rescrape-workplan.md b/evidence/TK-10900-rescrape-workplan.md
new file mode 100644
index 0000000..6b45cc0
--- /dev/null
+++ b/evidence/TK-10900-rescrape-workplan.md
@@ -0,0 +1,172 @@
+# TK-10900 — canonical `dw_sku` recovery: the "re-scrape" backlog work-plan
+
+**Parent:** TK-10896 (recoverable-now recovery — DONE, 4,547 codes). This plan owns the
+**10,671 RESCRAPE + 145 IMPORT_DEFECT** rows that TK-10896's classifier deferred.
+**Owner:** vp-dw-commerce. **Status:** DRAFT / read-only planning artifact. Generated 2026-08-31.
+All numbers are from **canonical Kamatera** (`ssh root@45.61.58.125 psql dw_unified`) + the
+Mac2 staging catalogs (`psql -h /tmp -d dw_unified`).
+
+---
+
+## Headline: this is mostly a LOCAL CONTENT-MATCH job, NOT a web re-scrape
+
+The classifier binned these rows `RESCRAPE` because the blank Shopify rows carry **no `sku`
+and no `mfr_sku`**, so the exact `mfr_sku → <vendor>_catalog` join (the staging-link method
+already shipped in `staging-link-gen.mjs`) cannot fire. **But the vendor catalogs already hold
+the real codes.** They just need a fuzzier join key — `title → pattern_name + color_number` —
+i.e. the "EXACT pattern+color content-match" method from the `dw-unbuyable-recovery-methodology`
+memo. That is a `$0` local recovery. A true web scrape is only needed for the small residual of
+vendors whose catalog is **missing or thinner than the blank set** (~450 rows).
+
+So TK-10900 splits into four programs, not "openclaw-scrape 10,800 products":
+
+| Program | What | Rows | Cost | Path |
+|---|---|---|---|---|
+| **A · Content-match recovery** | Join Shopify title → catalog pattern+color → real code | ~9,950 | $0 local | extend `staging-link-gen.mjs` |
+| **B · Phillipe Romano (Greenland private label)** | Cork-line, own catalog + agent | 3,022 | $0 local (+ scrape gap) | `greenland-scraper-manager` |
+| **C · Import-defect (`null-Sample`)** | China Seas staging-linkable; PR folds into B | 145 | $0 local | staging-link / B |
+| **D · True web re-scrape** | Catalog missing/thin → fetch fresh identity | ~450 | metered scrape | `*-scraper-manager` skills |
+
+> Program A + B overlap the same catalog-match engine; they're split only because PR is a
+> private label with its own agent and is the single biggest owner after Carnegie.
+
+---
+
+## Two hard provenance rails the recovery MUST encode
+
+Both are the never-mint doctrine (`dw-dwsku-keep-old-never-mint`) applied to this data:
+
+1. **Mint-catalog vendors — use `mfr_sku`, NEVER `dw_sku`.** For **Carnegie, Maharam, CMO Paris,
+   Stout** every catalog `dw_sku` is a reverted Phase-4 **greenfield mint** (`DWAG-*` etc.).
+   Content-matching to `catalog.dw_sku` would silently **re-instate the exact mints we reverted**.
+   Recovery for these four pulls `catalog.mfr_sku` (the real code) instead. The other catalog
+   vendors carry a clean `dw_sku` and use it directly.
+2. **Strip the DW type-suffix to the base code + break the width tie.** Carnegie `mfr_sku` reads
+   `655790-panels`, `6557W94-wallcoverings` — the real code is the token before the first `-`
+   (`655790`, `6557W94`). After stripping, ~46% of Carnegie titles match a **unique** code; the
+   rest are same-title Upholstery-vs-Wallcovering pairs (`7360A11` vs `7360WA11`) that the blank
+   row's `product_type` (`Upholstery`) disambiguates. Verified hit rates below.
+
+Every write stays **gated** (canonical customer-facing), `shopify_id`-keyed, blank-guarded,
+per-item verified, reversible + ledgered — identical guardrails to the TK-10896 fire scripts.
+
+---
+
+## Per-vendor targets — ordered by row count
+
+`code source` = which catalog column yields the canonical code. `catalog` = staging rows available.
+`skill` = the `*-scraper-manager` that owns the web-scrape fallback / provenance.
+
+| # | Vendor | Blank rows | Program | Catalog (rows) | Code source | Scraper skill |
+|---|---|--:|---|---|---|---|
+| 1 | Carnegie | 3,921 | A | `carnegie_catalog` (5,928) | `mfr_sku` (mint dw_sku) | *(none — `dw-carnegie-price-canary`; build if D needed)* |
+| 2 | Phillipe Romano | 3,022¹ | B | `greenland_catalog` (Cork-) | private-label Cork code | `greenland-scraper-manager` |
+| 3 | Maharam | 1,429 | A | `maharam_catalog` (1,601) | `mfr_sku` (mint dw_sku) | *(none — catalog-match)* |
+| 4 | Knoll | 505 | A | `knoll_catalog` (632) | `dw_sku` (clean) | `knoll-scraper-manager` (login) |
+| 5 | Vahallan | 386 | A | `vahallan_catalog` (390) | `dw_sku` (clean) | *(none — catalog-match)* |
+| 6 | CMO Paris | 344 | A | `cmo_paris_catalog` (345) | `mfr_sku` (mint dw_sku) | *(none — catalog-match)* |
+| 7 | Hollywood Wallcoverings | 286 | A | `hollywood_catalog` (194 dw / 3,466 mfr) | HWC `mfr_sku`² | `dw-hollywood-sku-canary` |
+| 8 | Fentucci Naturals | 213 | A+D | `fentucci_naturals_catalog` (148) | `dw_sku` + ~65 scrape gap | `fentucci-scraper-manager` |
+| 9 | Stout Textiles | 180 | A | `stout_catalog` (180, exact) | `mfr_sku` (mint dw_sku) | `stout-scraper-manager` (login) |
+| 10 | **Majilite** | 160 | **D** | — (no catalog, no skill) | web scrape | ⚠ **new scraper build** |
+| 11 | Fabricut | 110 | A | `fabricut_catalog` (1,333) | `dw_sku` (clean) | `fabricut-scraper-manager` (login) |
+| 12 | Novasuede | 67 | A | `novasuede_catalog` (142) | `dw_sku` (clean) | `novasuede-updater` |
+| 13 | Thibaut | 65 | A | `thibaut_catalog` (6,057) | `dw_sku` (clean) | *(none — catalog-match; login site)* |
+| 14 | Rebel Walls | 52 | A | `rebelwalls_catalog` (4,350) | `dw_sku` (clean) | `rebel-walls-scraper-manager` |
+| 15 | **Pixels** | 52 | **D** | `vendor_catalog` (7, generic) | POD/print — special | ⚠ POD, not a wallcovering scrape |
+| 16 | China Seas | 29¹ | C | `china_seas_catalog` (2,650) | `mfr_sku` present → **staging-link** | *(existing staging-link tool)* |
+| 17 | Steve Abrams Studios | 13 | D | — (FAA art prints) | FAA order code | `fineartamerica-scraper-manager` |
+| 18 | **Jeffrey Stevens** | 12 | **D** | — (no catalog) | web scrape | ⚠ **new scraper build** |
+| 19 | Romo | 11 | A | `romo_catalog` (2,648) | `dw_sku` (clean) | `romo-scraper-manager` (login) |
+| 20 | Pierre Frey | 4 | A | `pierre_frey_catalog` (1,161) | `dw_sku` (clean) | `pierre-frey-scraper-manager` |
+| 21 | Graduate Collection UK | 4 | A/D | `vendor_catalog` (182, generic) | `dw_sku` partial | *(generic table)* |
+| 22 | Arte International | 1 | A | `arte_catalog` (1,991) | `dw_sku` (clean) | `arte-international-colorway-scraper` |
+| 23 | Harlequin | 1 | A | `harlequin_catalog` (763) | `dw_sku` (clean) | *(catalog-match)* |
+| 24 | Designer Wallcoverings | 1 | manual | — (house SKU) | manual | — |
+
+¹ Includes import-defect (`null-Sample`) rows: **Phillipe Romano 116**, **China Seas 29** = the 145.
+² Hollywood is the HWC private-label reconciliation (`hollywood-hwc-line-lbi-boyd`, TK-10633) — its
+  real mfr code is unrecovered for the 2019-era line; treat as **best-effort HWC match, gaps → TK-10633**,
+  do NOT name-match on DW-invented product names.
+
+**Totals:** Program A ≈ 9,950 · Program B (PR) 3,022 · Program C 29 (CN import-defect) · Program D ≈ 450.
+
+---
+
+## Verified content-match evidence (Carnegie, the 3,921-row canary)
+
+Measured this session, not assumed (VERIFY-BEFORE-ACTING):
+
+```
+raw-title exact match ....................... 0 / 3,921   (catalog stores no "Carnegie " prefix)
+title→pattern+color, mfr_sku suffix kept .... 35 unique / 3,872 "ambiguous"  (suffix noise)
+title→pattern+color, base code (pre-"-") .... 1,825 unique (46.5%) / 2,082 width-ambiguous / 14 miss
+```
+
+The 2,082 residual are same-title Upholstery/Wallcovering pairs (`Arctic 11 => 7360A11 | 7360WA11`);
+the blank rows are `product_type='Upholstery'`, which selects the non-`W` base variant → expected
+**>90% unique** once the `product_type`/width tiebreak is added. The 14 no-match rows are the true
+tail for Program D (fresh scrape).
+
+---
+
+## Execution shape (per program, all gated writes fired by Steve)
+
+### Program A — `content-match-gen.mjs` (new; sibling of `staging-link-gen.mjs`)
+1. Read canonical blank-`dw_sku` active rows for the vendor (title, product_type, shopify_id).
+2. Read `<vendor>_catalog` → build `pattern_name + color_number → code` map, where
+   `code = mfr_sku` (mint-catalog vendors) else `dw_sku`, **base-code cleaned** (strip `-<suffix>`),
+   **excluding any greenfield-mint code** (reuse the `/tmp/_ledcodes.txt` reverted-mint exclude).
+3. Parse each Shopify `title` → `pattern + trailing color number`; join; break width ties with
+   `product_type`. Emit only **unique** matches to `apply-plans-content-match/<vendor>/{apply,undo}.sql`
+   + `restore-map.json`, `shopify_id`-keyed + blank-guarded. Ambiguous/miss → report, never guess.
+4. `preflight-check.mjs` (GO) → **Steve fires** → `verify` (GO) → ledger. Canary smallest vendor first
+   (Harlequin 1 → Arte 1 → Pierre Frey 4 → … → Carnegie 3,921 last).
+5. Per-item verify sample (unbuyable-recovery methodology) before scaling each vendor.
+
+### Program B — Phillipe Romano / Greenland
+Route to `greenland-scraper-manager`. Content-match blanks → `greenland_catalog` Cork- codes
+(same engine as A). The 116 `null-Sample` import-defect rows fold in here. Any true gap → the
+existing Greenland feed-first scrape (open JSON API, $0).
+
+### Program C — Import-defect (145)
+- **China Seas 29** — these `null-Sample` rows DO carry `mfr_sku`, so they are **staging-linkable
+  today** via the existing `staging-link-gen.mjs` (`china_seas_catalog` has 2,650 rows). Cheapest win.
+- **Phillipe Romano 116** — folded into Program B.
+- **Never** write `null`/`null-Sample` back as a code (the JOB-4 "don't paper over" rule).
+
+### Program D — True web re-scrape (~450 rows, metered)
+Only where the catalog can't supply a code. Priority by count:
+`Majilite 160` (new scraper build) · `Pixels 52` (POD — treat as print-on-demand, not a wallcovering
+scrape) · `Fentucci Naturals ~65 gap` (`fentucci-scraper-manager`) · `Steve Abrams/FAA 13`
+(`fineartamerica-scraper-manager`) · `Jeffrey Stevens 12` (new build) · `Graduate 4`. Show `$` cost
+per scrape; openclaw is serial (one gateway/profile) so batch small. Feed results back through the
+staging catalog → then Program A's content-match.
+
+---
+
+## Sequencing & effort
+
+1. **Ship `content-match-gen.mjs`** (extends proven staging-link tooling; ~1 build session).
+2. **Program C China Seas** — free staging-link, immediate (29 rows).
+3. **Program A** — canary tail-first (single-digit vendors) → mid (Knoll/Vahallan/CMO/Hollywood/
+   Stout/Fabricut) → **Carnegie + Maharam last** (the 5,350 heavy rows, after the tiebreak is proven).
+4. **Program B** Phillipe Romano via greenland agent (3,022 — the second heaviest).
+5. **Program D** true scrapes last; build the two missing scrapers (Majilite, Jeffrey Stevens).
+
+**Expected outcome:** ~10,400 of 10,816 recovered locally at `$0` (Programs A+B+C); ~400 via metered
+scrape (D). Canonical blank `dw_sku` backlog **10,868 → ~450**, still zero mints, all gated + reversible.
+
+---
+
+## Open decisions for Steve (CLARIFY gate)
+
+1. **Mint-catalog base code as canonical** — is Carnegie's cleaned `mfr_sku` base (`655790`) the
+   canonical `dw_sku`, or should it carry the DW type-suffix / a `DW`-prefixed form? (Recovery uses
+   the raw base code by default — this is the one "shape" decision that isn't purely mechanical.)
+2. **Two new scrapers** — approve building `majilite-scraper-manager` + a Jeffrey Stevens scraper
+   (172 rows combined), or defer/discontinue those lines?
+3. **Pixels (52)** — confirm these are POD prints to be handled by the print pipeline, not the
+   wallcovering catalog.
+
+Related tickets: TK-10896 (parent), TK-10633 (Hollywood HWC reconciliation), TK-10649 (dedup).

← b5f98d8 chore: lint (node --check clean) + v0.2.0 (session close — c  ·  back to Dw Sku Integrity  ·  TK-10900: harden content-match to strict 1:1 (pattern,color, 584b4e1 →