← back to Dw Sku Integrity
TK-10900/A: unified CONCAT matcher — one mode for numeric + name colors
41e8df0c0528980e1a4e3d7b40fc96d09e4c457d · 2026-08-31 02:38:03 -0700 · codex-10896
Replaces the fragile trailing-number parser with a whole-identity concat key:
catalog norm(pattern_name+' '+color) [or title] vs vendor-stripped Shopify title.
Reproduces Carnegie EXACTLY (2,745, no regression) AND unlocks color-name vendors
(Maharam 1,235/86.4%, was 0). Keeps all rails: mint->mfr_sku, base-strip,
bucket-compat precondition, greenfield/ledger exclude, strict 1:1, gated output.
stripVendor handles prefix (Carnegie) + ' | Vendor' suffix (Maharam).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Files touched
Diff
commit 41e8df0c0528980e1a4e3d7b40fc96d09e4c457d
Author: codex-10896 <steve@designerwallcoverings.com>
Date: Mon Aug 31 02:38:03 2026 -0700
TK-10900/A: unified CONCAT matcher — one mode for numeric + name colors
Replaces the fragile trailing-number parser with a whole-identity concat key:
catalog norm(pattern_name+' '+color) [or title] vs vendor-stripped Shopify title.
Reproduces Carnegie EXACTLY (2,745, no regression) AND unlocks color-name vendors
(Maharam 1,235/86.4%, was 0). Keeps all rails: mint->mfr_sku, base-strip,
bucket-compat precondition, greenfield/ledger exclude, strict 1:1, gated output.
stripVendor handles prefix (Carnegie) + ' | Vendor' suffix (Maharam).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
content-match-gen.mjs | 96 +++++++++++++++++++++++++++------------------------
1 file changed, 51 insertions(+), 45 deletions(-)
diff --git a/content-match-gen.mjs b/content-match-gen.mjs
index 1b4b26e..b53e74c 100644
--- a/content-match-gen.mjs
+++ b/content-match-gen.mjs
@@ -4,19 +4,23 @@
// Recovers canonical dw_sku for blank Shopify rows that have NO sku and NO mfr_sku, by CONTENT-MATCHing
// the Shopify title -> a vendor staging-catalog row -> that row's EXISTING real code. Recovers an
// existing code — never mints. Cross-machine: blank rows from Kamatera (canonical shopify_products),
-// the pattern/color -> code map from the Mac2 <vendor>_catalog (staging is Mac2-canonical).
+// the map from the Mac2 <vendor>_catalog (staging is Mac2-canonical).
//
-// Algorithm (see evidence/TK-10900-rescrape-workplan.md):
-// 1. code source: MINT-CATALOG vendors (carnegie/maharam/cmo_paris/stout) have greenfield-mint dw_sku
-// -> use mfr_sku instead; others use dw_sku. Greenfield-mint + reverted-ledger codes are excluded.
-// 2. base-code collapse: strip the DW type-suffix ('655788-panels-museums' -> '655788').
-// 3. key = (normPattern, normColor). Multiple catalog rows usually collapse to ONE base code.
-// 4. strict emit: write ONLY when the key maps 1:1 to exactly one base code. If >1 base code (a real
-// fabric-vs-wallcovering variant), disambiguate by product_type BUCKET; still >1 -> review queue.
-// Parse-miss / no-map / still-ambiguous -> review queue, NEVER a best guess (codex-hardened).
+// UNIFIED CONCAT MATCHER (v2): the match key is the whole product-identity string, NOT a parsed
+// pattern/color split (which was fragile — it only worked for numeric colors and 100%-missed
+// color-name vendors). Both sides build a comparable key:
+// * catalog key = norm(pattern_name + ' ' + color) [color = color_number|color_name],
+// or norm(title) when the catalog has only a title column.
+// * shopify key = norm(title with the vendor name stripped off either end).
+// This reproduces Carnegie (numeric) AND recovers Maharam/Knoll/etc (color-name) with one mode.
//
-// Emits apply-plans-content-match/<vendor>/{apply.sql,undo.sql,restore-map.json,review-queue.json},
-// keyed on shopify_id + blank-guarded. NEVER writes a DB. Parent: TK-10900 (under TK-10896).
+// SAFETY RAILS (unchanged): mint-catalog vendors (carnegie/maharam/cmo paris/stout) have greenfield-mint
+// dw_sku -> use mfr_sku instead; base-code collapse strips only trailing '-<alpha>' type-suffixes so a
+// numeric-hyphenated real code survives; greenfield-mint + reverted-ledger codes excluded; a write is
+// emitted ONLY when the key resolves to exactly ONE base code whose product_type bucket is compatible
+// with the Shopify row's bucket (the Abbey-61 cross-class guard). Everything else -> review-queue, never
+// a best guess. Emits apply-plans-content-match/<vendor>/{apply,undo}.sql + restore-map + review-queue,
+// shopify_id-keyed + blank-guarded. NEVER writes a DB. Parent: TK-10900 (under TK-10896).
//
// Usage: node content-match-gen.mjs --vendor Carnegie --catalog carnegie_catalog \
// [--kam 'ssh root@45.61.58.125 psql dw_unified'] [--ledger /tmp/_ledcodes.txt]
@@ -36,22 +40,30 @@ const LEDGER = arg('--ledger', '/tmp/_ledcodes.txt');
const OUT = arg('--out', join(HERE, 'apply-plans-content-match'));
const CODE_SHAPE = /^[A-Za-z0-9][A-Za-z0-9._/-]{0,39}$/;
const GREENFIELD_MINT = /^DW(AG|AX|CX|ST|SC|DX|WG)/i;
-// Vendors whose catalog dw_sku is reverted greenfield-mint residue -> recover from mfr_sku instead.
const MINT_CATALOG = new Set(['carnegie', 'maharam', 'cmo paris', 'cmo_paris', 'stout', 'stout textiles']);
const sqlEscape = (v) => String(v).replace(/'/g, "''");
const norm = (s) => (s || '').trim().toLowerCase().replace(/\s+/g, ' ');
// Base code = strip trailing DW type-suffix groups ('-panels', '-panels-museums', '-dividers', ...).
-// Strips ONLY trailing '-<alpha>' tokens, so a genuinely hyphenated real code with a numeric segment
+// Strips ONLY trailing '-<alpha>' tokens so a genuinely hyphenated real code with a numeric segment
// ('91026-10') is preserved rather than truncated. Only applied to mint-catalog mfr_sku (dw_sku verbatim).
const baseCode = (c) => (c || '').trim().replace(/(-[A-Za-z][A-Za-z]*)+$/, '').toUpperCase();
-// product_type semantic bucket so Shopify 'Upholstery' can match catalog 'Upholstery'/'Fabric' etc.
+// product_type semantic bucket so Shopify 'Upholstery' matches catalog 'Upholstery'/'Fabric' etc.
const bucket = (pt) => {
const s = norm(pt);
if (/wallcover|wallpaper|mural|panel|museum|window|privacy|imo/.test(s)) return 'wall';
if (/upholst|fabric|textile|drapery|seat/.test(s)) return 'fabric';
return 'other';
};
+// Strip the vendor name off EITHER end of a Shopify title: leading "Vendor ..." OR trailing "... | Vendor"
+// / "... - Vendor". Handles Carnegie (prefix) and Maharam (" | Maharam" suffix).
+function stripVendor(title, vendor) {
+ let t = norm(title); const v = norm(vendor);
+ const esc = v.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+ t = t.replace(new RegExp('\\s*[|\\-–]\\s*' + esc + '\\s*$', 'i'), ''); // trailing " | Vendor" / " - Vendor"
+ if (t.startsWith(v + ' ')) t = t.slice(v.length + 1); // leading "Vendor "
+ return t.trim();
+}
function runPsql(cmd, sql) {
const out = execFileSync(cmd[0], [...cmd.slice(1), '-tA', '-F', US, '-R', RS], { input: sql, maxBuffer: 1 << 30, encoding: 'utf8' });
@@ -59,44 +71,44 @@ function runPsql(cmd, sql) {
}
const LOCAL = ['psql', '-h', '/tmp', '-d', 'dw_unified'];
-// 0. Resolve the catalog table + detect which columns it actually has (schema-adaptive).
+// 0. Resolve the catalog + detect columns (schema-adaptive).
const catalog = CATALOG || `${VENDOR.toLowerCase().replace(/\s+/g, '_')}_catalog`;
const cols = new Set(runPsql(LOCAL,
`SELECT column_name FROM information_schema.columns WHERE table_name='${sqlEscape(catalog)}';`).map((r) => r[0]));
const useMfr = MINT_CATALOG.has(VENDOR.toLowerCase());
const colColumn = cols.has('color_number') ? 'color_number' : (cols.has('color_name') ? 'color_name' : null);
const hasProductType = cols.has('product_type');
+const hasPatternColor = cols.has('pattern_name') && colColumn;
+const keyMode = hasPatternColor ? 'concat' : (cols.has('title') ? 'title' : null);
-// Graceful schema gate: this tool implements the pattern+color match mode. Catalogs that don't fit
-// (no code source, or no pattern_name+color) are DEFERRED to a fitted matcher (see the work-plan),
-// reported cleanly rather than crashed, so a fleet sweep stays informative.
function deferUnsupported(reason, detail) {
- const s = { ticket: 'TK-10900', program: 'A-content-match', vendor: VENDOR, catalog, status: 'deferred_unsupported_schema', reason, detail, columns: [...cols], matched: 0, statements: 0, note: 'NOTHING executed. Needs a schema-fitted matcher (color-name / title-match mode).' };
+ mkdirSync(OUT, { recursive: true });
+ const s = { ticket: 'TK-10900', program: 'A-content-match', vendor: VENDOR, catalog, status: 'deferred_unsupported_schema', reason, detail, columns: [...cols], matched: 0, statements: 0, note: 'NOTHING executed. No usable match key in this catalog.' };
writeFileSync(join(OUT, `SUMMARY-${VENDOR}.json`), JSON.stringify(s, null, 2) + '\n');
- console.log(JSON.stringify(s, null, 2));
- process.exit(0);
+ console.log(JSON.stringify(s, null, 2)); process.exit(0);
}
mkdirSync(OUT, { recursive: true });
const codeCol = useMfr ? 'mfr_sku' : 'dw_sku';
if (!cols.has(codeCol)) deferUnsupported('no_code_source', `needs '${codeCol}' (${useMfr ? 'mint-catalog vendor' : 'clean-catalog vendor'})`);
-if (!cols.has('pattern_name') || !colColumn) deferUnsupported('no_pattern_color_key', `pattern+color mode needs pattern_name + color_number|color_name (has color? ${colColumn || 'none'})`);
+if (!keyMode) deferUnsupported('no_match_key', 'needs pattern_name+color OR a title column');
// 1. Reverted-mint exclude list.
const minted = new Set(existsSync(LEDGER) ? readFileSync(LEDGER, 'utf8').split('\n').map((s) => s.trim().toUpperCase()).filter(Boolean) : []);
-// 2. Build the catalog map: key "(pattern|color)" -> Map(baseCode -> Set(bucket)).
-const catRows = runPsql(LOCAL,
- `SELECT coalesce(pattern_name,''), coalesce(${colColumn},''), ${hasProductType ? 'coalesce(product_type,\'\')' : "''"}, ` +
- `coalesce(dw_sku,''), coalesce(mfr_sku,'') FROM ${catalog};`);
+// 2. Build the catalog map: identity key -> Map(baseCode -> Set(bucket)).
+const selCols = keyMode === 'concat'
+ ? `coalesce(pattern_name,''), coalesce(${colColumn},''), ${hasProductType ? "coalesce(product_type,'')" : "''"}, coalesce(dw_sku,''), coalesce(mfr_sku,'')`
+ : `coalesce(title,''), '', ${hasProductType ? "coalesce(product_type,'')" : "''"}, coalesce(dw_sku,''), coalesce(mfr_sku,'')`;
+const catRows = runPsql(LOCAL, `SELECT ${selCols} FROM ${catalog};`);
const map = new Map();
let catUsable = 0, catExcluded = 0;
-for (const [pat, col, ptype, dw, mfr] of catRows) {
+for (const [a, b, ptype, dw, mfr] of catRows) {
const raw = useMfr ? mfr : dw;
- if (!raw || !raw.trim()) { continue; }
+ if (!raw || !raw.trim()) continue;
const code = useMfr ? baseCode(mfr) : dw.trim();
if (!CODE_SHAPE.test(code) || GREENFIELD_MINT.test(code) || minted.has(code.toUpperCase())) { catExcluded++; continue; }
- const key = norm(pat) + '|' + norm(col);
- if (!norm(pat) || !norm(col)) continue;
+ const key = keyMode === 'concat' ? (norm(a) + ' ' + norm(b)).trim() : norm(a);
+ if (!key) continue;
if (!map.has(key)) map.set(key, new Map());
const codes = map.get(key);
if (!codes.has(code)) codes.set(code, new Set());
@@ -104,33 +116,27 @@ for (const [pat, col, ptype, dw, mfr] of catRows) {
catUsable++;
}
-// 3. Shopify blank rows (canonical). Parse title -> pattern + trailing color token.
+// 3. Shopify blank rows (canonical). Key = vendor-stripped title.
const shopRows = runPsql(KAM,
`SELECT coalesce(shopify_id,''), coalesce(title,''), coalesce(product_type,'') FROM shopify_products ` +
`WHERE lower(coalesce(status,''))='active' AND (dw_sku IS NULL OR btrim(dw_sku)='') ` +
`AND vendor ILIKE '${sqlEscape(VENDOR)}%';`);
-const vname = norm(VENDOR);
const entries = [], review = [];
-const stat = { shopify_blank: shopRows.length, no_shopify_id: 0, parse_miss: 0, no_map: 0, ambiguous: 0, cross_class: 0, matched: 0 };
+const stat = { shopify_blank: shopRows.length, no_shopify_id: 0, no_map: 0, ambiguous: 0, cross_class: 0, matched: 0 };
for (const [sid, title, ptype] of shopRows) {
if (!sid) { stat.no_shopify_id++; continue; }
- let body = norm(title);
- if (body.startsWith(vname + ' ')) body = body.slice(vname.length + 1);
- const m = body.match(/^(.*\S)\s+([0-9]+[a-z]?)$/); // "<pattern...> <colortoken>"
- if (!m) { stat.parse_miss++; review.push({ shopify_id: sid, title, reason: 'parse_miss' }); continue; }
- const key = norm(m[1]) + '|' + norm(m[2]);
+ const key = stripVendor(title, VENDOR);
const codes = map.get(key);
if (!codes) { stat.no_map++; review.push({ shopify_id: sid, title, key, reason: 'no_catalog_match' }); continue; }
let cand = [...codes.keys()];
- if (cand.length > 1) { // real variant fork -> filter by product_type bucket
+ if (cand.length > 1) { // multi base code -> disambiguate by product_type bucket
const want = bucket(ptype);
const filtered = cand.filter((c) => codes.get(c).has(want));
if (filtered.length === 1) cand = filtered;
}
if (cand.length !== 1) { stat.ambiguous++; review.push({ shopify_id: sid, title, key, candidates: [...codes.keys()], reason: 'ambiguous_multi_code' }); continue; }
- // Bucket-compatibility precondition — applies to SINGLE-candidate keys too (the Abbey-61 class):
- // never stamp a wall-only code onto a fabric row or vice versa. Shopify/catalog 'other' -> allow.
+ // Bucket-compatibility precondition (single-candidate keys too): never write wall onto fabric / vice versa.
const shopBucket = bucket(ptype);
const catBuckets = codes.get(cand[0]);
if (shopBucket !== 'other' && !catBuckets.has(shopBucket) && !catBuckets.has('other')) {
@@ -145,7 +151,7 @@ for (const [sid, title, ptype] of shopRows) {
// 4. Emit GATED artifacts.
const HEADER =
'-- GATED -- canonical Kamatera dw_unified write. Do NOT run automatically.\n' +
- `-- Content-match recovery (title -> ${catalog} ${useMfr ? 'mfr_sku(base)' : 'dw_sku'}, existing code, no mint). TK-10900.\n`;
+ `-- Content-match recovery (${keyMode}-key -> ${catalog} ${useMfr ? 'mfr_sku(base)' : 'dw_sku'}, existing code, no mint). TK-10900.\n`;
const applySql = HEADER + entries.map((e) =>
`UPDATE shopify_products SET dw_sku='${sqlEscape(e.candidate)}' WHERE shopify_id='${sqlEscape(e.shopify_id)}' AND (dw_sku IS NULL OR btrim(dw_sku)='');`).join('\n') + '\n';
const undoSql = HEADER + entries.map((e) =>
@@ -161,9 +167,9 @@ writeFileSync(join(dir, 'restore-map.json'), JSON.stringify(restoreMap, null, 2)
writeFileSync(join(dir, 'review-queue.json'), JSON.stringify(review, null, 2) + '\n');
const summary = {
- ticket: 'TK-10900', program: 'A-content-match', vendor: VENDOR, catalog, status: 'supported', color_column: colColumn,
- code_source: useMfr ? 'mfr_sku(base-stripped)' : 'dw_sku', bucket_guard: hasProductType, catalog_usable: catUsable, catalog_excluded_mint: catExcluded,
- ...stat, review_queue: review.length, statements: entries.length,
+ ticket: 'TK-10900', program: 'A-content-match', vendor: VENDOR, catalog, status: 'supported', key_mode: keyMode,
+ color_column: colColumn, code_source: useMfr ? 'mfr_sku(base-stripped)' : 'dw_sku', bucket_guard: hasProductType,
+ catalog_usable: catUsable, catalog_excluded_mint: catExcluded, ...stat, review_queue: review.length, statements: entries.length,
match_pct: stat.shopify_blank ? (100 * stat.matched / stat.shopify_blank).toFixed(1) + '%' : '0%',
note: 'NOTHING executed. apply.sql is a GATED draft; review-queue.json is NOT written.',
};
← abac289 TK-10900/A2: matcher PROVEN — concat-key (pattern+color vs s
·
back to Dw Sku Integrity
·
auto-data-snapshot: 2026-08-31T02:39:25 (76 data files) — ap f5b40ce →