`tk11041-innovations-reconcile/rescrape/README.md`

66 lines
# TK-11041 — Innovations RE-SCRAPE (the ticket's original named fix)

## Result: 51/51 identity RECOVERED at $0, read-only. Zero mismatches.

### What this overturns
The 2026-09-02 reconcile concluded `BLOCKED_ON_VENDOR_LINESHEET — 0 trustworthy local
recovery sources`, on the premise that CSO/ENS/FLN/PRO/SKL/TTM/VCO/WHL were
**"DW-internal collection codes"**.

That premise is **false**. They are **real Innovations item-number prefixes**, publicly
resolvable on innovationsusa.com. The earlier lane only searched the stale local
`innovations_catalog` (341 rows, 236 with no provenance URL) — it never scraped the vendor,
which is exactly what the ticket title asked for.

### Method (read-only, $0, robots-compliant)
1. `sitemap.xml` (live, 371 `/item/<slug>` pattern pages). robots.txt **allows** `/item/*/` ;
   `/item/*/*` (per-colorway pages) is disallowed and was **not** fetched.
2. `sweep.sh` — fetch all 371 pattern pages @0.35s delay, extract every `AAA-nnn` item code.
3. Build prefix -> pattern map; join to the 51-item cohort on `mfr_sku`.
4. `extract.py` — pull pattern name, colorway names, full spec block, image URL per item.

### Prefix -> Innovations pattern (all 8 resolved)
| DW mfr prefix | Innovations pattern | n  | site state |
|---|---|---|---|
| CSO | Costine     | 3  | live |
| ENS | Enchase     | 6  | live |
| FLN | Flanders    | 5  | live |
| PRO | Province    | 1  | live |
| SKL | Skylark     | 4  | live |
| TTM | Tatami      | 7  | live |
| VCO | Vinyl Cork  | 13 | live |
| WHL | **Whistler**| 12 | **DISCONTINUED** (404 live; recovered from Wayback 20230127073736) |

### Independent proof the join is correct
Innovations' own colorway names were compared against the DW private-label titles:

* **43/51 EXACT** (e.g. `FLN-01 HEATHER` == "La Bologna **Heather**")
* **7/51 PREFIX** (DW truncated: `VCO-010 Blue Opal` -> "…**Blue** Type 2")
* **1/51** DW title carries no colorway (`PRO-012` = Innovations "Uva")
* **0 MISMATCHES**

DW relabelled only the **pattern** (Flanders->Bologna, Tatami->Udino, Skylark->Sieno,
Enchase->Renegado, Costine->Turino, Whistler->Orvito, Province->Youngstown); the
colorways were kept. `VCO` titles even retain the literal string "Vinyl Cork".

### What is recovered per item
`data/innovations-identity-map.{json,csv}` — mfr_sku, Innovations pattern + colorway,
DW colorway + match class, site state, product_url, image_url, width, full roll,
composition, repeat, fire rating.

### What is STILL gated (unchanged)
**PRICE.** Verified: zero price tokens across all 371 public pages — Innovations trade
pricing is login-only. The drafted vendor line-sheet request (Gmail draft
`draftId=r-1766618972989872280`) remains the correct and only path to net price, and its
send stays Steve's.

### New finding that changes the plan
`WHL` / "La Orvito" (12 of the 51) is a **discontinued** Innovations pattern. Repricing and
activating those 12 would put a dead line back on sale. They should be routed to
discontinue/archive, **not** to the add-variant recovery. Only **39** are live-line candidates.

### Reproduce
```sh
./sweep.sh && python3 extract.py
```