`README.md`

127 lines
# dw-sku-integrity

Read-only, deterministic tooling for the **canonical `dw_sku` backlog** (TK-10896).

## Doctrine (hard rule — Steve, 2026-08-26)

`dw_sku` must **recover an EXISTING code** (self-copy from `sku`, or re-scrape the
real mfr/DW code). It must **NEVER mint a new sequential**. See memory
`dw-dwsku-keep-old-never-mint`. This tooling is built so it *cannot* mint: the
classifier only ever proposes a candidate that is already a prefix of the row's
existing `sku`.

## Safety

- **Read-only.** The scanner issues only `SELECT`. It never `UPDATE`/`INSERT`/
  `DELETE`s, never writes the DB. Its only output is a local summary/plan file.
- **No fabrication.** `classify.mjs` recovers codes by pure suffix-stripping; a unit
  test enforces that a proposed candidate is always a prefix of the row's existing
  `sku` — it never invents a code absent from a DB field. NOTE this is a
  *no-fabrication* guarantee, not an unconditional *no-mint* guarantee: a reverted
  Phase-4 mint WROTE its code into `sku`, so residue is trivially "already in a
  field." The **provenance guard** (greenfield-prefix routing) is what upgrades this
  to no-mint — so the no-mint property holds *provided the guard catches all residue
  prefixes*. Greenfield (Bucket B) is fully covered; a small Bucket-A subset (≤~668
  rows minted into already-in-use prefixes) needs the exact reverted-mint ledger to
  disambiguate. Those prefixes now fail closed into `PROVENANCE_REVIEW`; they are
  neither silently cleared nor incorrectly sent to re-scrape (see `evidence/ANALYSIS`).
- **dw_sku is intentionally non-unique.** A pattern's sellable + sample rows share one
  canonical `dw_sku`. The scanner reports `within_batch_shared_candidates` (benign
  same-pattern groups); an apply step MUST upsert, never assume per-row uniqueness.
- Any actual `dw_sku` write is canonical + customer-facing → **gated** (drafted to
  `~/.claude/yolo-queue/pending-approval/`, ledgered, reversible). Not this repo's job.

## Status — recoverable-now + content-match RECOVERY COMPLETE (2026-08-31)

Executed against **canonical Kamatera** (not the mirror — the mirror was stale, see
`evidence/ANALYSIS-2026-08-30.md`). Canonical blank `dw_sku` backlog **15,415 → 10,868**;
**4,547 codes recovered** (2,645 self-copy across 19 vendors + 2,000 Carnegie staging-link),
all EXISTING scraper-assigned codes, **zero mints**, every batch `preflight → apply →
verify GO`, reversible + ledgered. Remaining 10,868 = 10,671 re-scrape (TK-10900) +
145 `null-Sample` import-defect (TK-10900) + 42 provenance-review + 8 dedup (TK-10649).

After Steve's explicit `resume / go`, Program-A content-match recovered another **3,555**
existing codes across 10 vendors. Nine vendors totaling 810 rows ran as vendor tranches;
Carnegie's 2,745 rows ran as six fail-fast chunks (500/500/500/500/500/245) to honor
the approval package's <=500-row boundary. Every chunk verified `GO` with zero conflicts
or missing rows. The post-run canonical scan confirms **7,313** blanks remain;
those are primarily genuine vendor re-scrapes/private-label recovery, import defects,
provenance review, and dedup—not another safe blanket content-match batch.

## Files

- `classify.mjs` — pure classifier: `stripUnitSuffix()`, `classifyRow()`, `RECOVERY_GROUP`.
- `dwsku-backlog-scan.mjs` — read-only DB scanner (parameterized via `DWSKU_PSQL`) → segmentation + plan JSONL.
- `provenance-ledger-join.mjs` — local-only, fail-closed join of the SHA-256-verified
  reverted-mint undo ledger; ledger membership decides residue-vs-native for every DW-code row.
- `apply-plan-gen.mjs` — read-only generator of GATED per-vendor self-copy `apply.sql` /
  `undo.sql` / `restore-map.json`, keyed on `shopify_id` (stable cross-machine).
- `staging-link-gen.mjs` — read-only generator of GATED staging-link plans (recover
  `dw_sku` via `mfr_sku → <vendor>_catalog.dw_sku`, cross-machine: Kamatera rows × Mac2 catalog).
- `preflight-check.mjs` — read-only GO/NO-GO checker; `--mode preflight` (before) / `--mode verify` (after).
- `fire-self-copy.sh` / `fire-staging-link.sh` — Steve-run guarded `preflight→apply→verify→ledger` loops.
- `RUNBOOK.md` — the turnkey per-vendor gated firing runbook.
- `test/*.test.mjs` — 47 unit tests (`node --test`).
- `evidence/` — committed scan outputs (mirror + canonical) + the analysis writeup.

## Canonical target (all read-only tools + the gated writes)

```bash
export DWSKU_PSQL='ssh root@45.61.58.125 psql dw_unified'   # canonical Kamatera
node dwsku-backlog-scan.mjs                                  # SELECT-only segmentation
node preflight-check.mjs --plan-dir apply-plans-canonical --vendor <v>          # GO before firing
node preflight-check.mjs --plan-dir apply-plans-canonical --vendor <v> --mode verify   # GO after
# the apply itself is a canonical customer-facing write — HARD-GATED, fired by Steve via the fire-*.sh scripts.
```

## Usage

```bash
node --test                              # run the unit tests

# scan the local Mac2 mirror (default):
node dwsku-backlog-scan.mjs --out evidence/mac2-mirror.json --plan

# scan Kamatera (canonical) — set the psql prefix to a remote wrapper:
DWSKU_PSQL='ssh <kamatera-host> psql' node dwsku-backlog-scan.mjs \
    --out evidence/kamatera.json --plan
```

`--plan` also writes a `*.plan.jsonl` (one row → class → recovered candidate →
collision flag), suitable as the dry-run input to a *gated* apply step elsewhere.

### Join a separately authorized undo-ledger export

The importer never connects to PostgreSQL or Kamatera. A future read-only export
must be JSONL with one unique `assigned_dw_sku` per line and travel with a manifest:

```json
{"backup_table":"sku_repair_p4_20260826","complete":true,"row_count":11725,"sha256":"<sha256-of-ledger-file>"}
```

```bash
node provenance-ledger-join.mjs \
  --ledger phase4-ledger.jsonl --manifest phase4-ledger.manifest.json \
  --plan mirror.plan.jsonl --out mirror.joined.jsonl
```

Exact ledger matches remain mint residue. Only mixed-prefix codes excluded by the
attested-complete ledger become `SELF_COPY_DW_PROVEN_NATIVE`. Missing/incomplete,
wrong-table, count-mismatched, hash-mismatched, duplicate, or malformed ledgers fail.

## Classes → recovery group

| class | meaning | group |
|---|---|---|
| `SELF_COPY_DW` | `sku` holds a `DW…-` code | recoverable_now_self_copy |
| `SELF_COPY_SOURCE` | `sku` holds a non-DW source code | recoverable_now_self_copy |
| `SELF_COPY_CORK` | `sku` holds a `Cork-<n>` code | recoverable_now_self_copy |
| `STAGING_LINK` | no usable `sku`, real `mfr_sku` present | recoverable_now_staging_link |
| `SELF_COPY_*_COLLISION` | recovered code already live on another active product | dedup_gated_TK10649 |
| `MINT_RESIDUE_RESCRAPE` | code sits in a reverted Phase-4 **greenfield** mint prefix (`DWAG/DWAX/DWCX/DWST/DWSC/DWDX/DWWG`) — self-copy would re-mint | rescrape_program_TK10900 |
| `PROVENANCE_REVIEW` | code uses a mixed native/minted Phase-4 prefix (`DWKN/DWTT/DWRW/DWJS/DWRO/DWCC`) — exact undo ledger required | provenance_review_TK10896 |
| `RESCRAPE` | no `sku`, no `mfr_sku` — must re-scrape the vendor | rescrape_program_TK10900 |
| `IMPORT_DEFECT` | literal `null`/`null-Sample` sku (scraper bug) | rescrape_program_TK10900 |

Parent ticket: **TK-10896**. Related: TK-10649 (dedup), TK-10900/10901 (re-scrape).