← back to Wolfgordon Crawl

backfill: dedup images by basename (catalog cdn2-optimize vs Shopify-rehosted same file)

fe335e52b2412d0c349f2dc843913b27961a2d77 · 2026-06-19 15:58:24 -0700 · Steve

Files touched

• M backfill-enrichment.js

Diff

commit fe335e52b2412d0c349f2dc843913b27961a2d77
Author: Steve <steve@designerwallcoverings.com>
Date:   Fri Jun 19 15:58:24 2026 -0700

    backfill: dedup images by basename (catalog cdn2-optimize vs Shopify-rehosted same file)
---
 backfill-enrichment.js | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/backfill-enrichment.js b/backfill-enrichment.js
index 3d195a6..a584387 100644
--- a/backfill-enrichment.js
+++ b/backfill-enrichment.js
@@ -211,8 +211,13 @@ async function main() {
       const curDesc = (live.descriptionHtml || '').replace(/<[^>]+>/g, '').trim();
       const wantDesc = curDesc.length < 20 ? buildDescription(row) : null;
 
-      const liveImgs = new Set((live.media.edges || []).map(m => m.node?.image?.url).filter(Boolean).map(u => u.split('?')[0]));
-      const wantImgs = catalogImages(row).filter(u => u && !liveImgs.has(u.split('?')[0]));
+      // Dedup by image BASENAME, not full URL: the catalog swatch
+      // (cdn2-optimize.wolfgordon.com/.../y47665sv.jpg) is re-hosted by Shopify
+      // as cdn.shopify.com/.../y47665sv.jpg — same file, different host. Comparing
+      // full URLs would re-upload a duplicate. basename() collapses both.
+      const basename = (u) => { try { return decodeURIComponent(u.split('?')[0].split('/').pop() || '').toLowerCase(); } catch { return (u.split('?')[0].split('/').pop() || '').toLowerCase(); } };
+      const liveImgs = new Set((live.media.edges || []).map(m => m.node?.image?.url).filter(Boolean).map(basename));
+      const wantImgs = catalogImages(row).filter(u => u && !liveImgs.has(basename(u)));
       const liveImgCount = live.mediaCount?.count || 0;
       const willHaveImg = liveImgCount > 0 || wantImgs.length > 0;
 

← 02b4906 WG data-quality: add live-state audits + enrichment backfill  ·  back to Wolfgordon Crawl  ·  add find-draft-candidates audit (8 WG rows fail width gate - e2e3062 →