cycle 86: collection-membership integrity sweep (DTD 2/1 B) — full active scan 71,726 -> 0 true orphans, 100% in >=1 collection, 88% in 11+; NEG-CONTROL (bulk collection-rich) + publication probe both close caveats (12 single-collection products are in best-seller-internal-use which IS published to all 13 channels = reachable, not browse-dark); CLEAN BILL ruling out dark-by-orphaning fleet-wide; 2 minor non-defect flags (127k catch-all misnamed 'internal use' yet published P4; 8 Torre dups -> fleet-dedup P3); residuals named (image integrity, inventory=2026, nav-link)

f4ac7a9b103ec12aba172fc616cd9ea95998565a · 2026-06-18 09:17:14 -0700 · Steve Abrams
Files touched

A scripts/collection-orphan-sweep/FINDING.md
A scripts/collection-orphan-sweep/collection-orphan-sweep.mjs
A scripts/collection-orphan-sweep/low-membership-probe.mjs
Diff

commit f4ac7a9b103ec12aba172fc616cd9ea95998565a
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Thu Jun 18 09:17:14 2026 -0700

    cycle 86: collection-membership integrity sweep (DTD 2/1 B) — full active scan 71,726 -> 0 true orphans, 100% in >=1 collection, 88% in 11+; NEG-CONTROL (bulk collection-rich) + publication probe both close caveats (12 single-collection products are in best-seller-internal-use which IS published to all 13 channels = reachable, not browse-dark); CLEAN BILL ruling out dark-by-orphaning fleet-wide; 2 minor non-defect flags (127k catch-all misnamed 'internal use' yet published P4; 8 Torre dups -> fleet-dedup P3); residuals named (image integrity, inventory=2026, nav-link)
---
 scripts/collection-orphan-sweep/FINDING.md         | 32 ++++++++++++
 .../collection-orphan-sweep.mjs                    | 59 ++++++++++++++++++++++
 .../low-membership-probe.mjs                       | 43 ++++++++++++++++
 3 files changed, 134 insertions(+)

diff --git a/scripts/collection-orphan-sweep/FINDING.md b/scripts/collection-orphan-sweep/FINDING.md
new file mode 100644
index 0000000..4b0ab6b
--- /dev/null
+++ b/scripts/collection-orphan-sweep/FINDING.md
@@ -0,0 +1,32 @@
+# Collection-membership integrity = HEALTHY (0 true orphans / 100% coverage fleet-wide) — a clean NEGATIVE finding that rules out the dark-by-orphaning class — cycle 86
+
+**DTD verdict B (2/1 — Claude + Codex B; Qwen dissent A/image-integrity).** READ-ONLY (Admin GraphQL full active scan + a negative control + a publication probe; no mutation), **$0**. First fresh non-price surface after the ~18-cycle price/variant thread closed. Tests whether live products are orphaned from every browse/merchandising path (= invisible despite being live) — a different dark-inventory failure mode than the April archive.
+
+## 🟢 Result: ZERO true orphans — 100% of active products are in ≥1 collection, and the catalog is richly multi-collection
+Full active scan (71,726, complete=true, PJ excluded):
+| Collection count | Products | Share |
+|---|---|---|
+| **0 (ORPHAN)** | **0** | **0.0%** |
+| 1 | 12 | 0.0% |
+| 2 | 35 | 0.0% |
+| 3–5 | 1,751 | 2.4% |
+| 6–10 | 6,665 | 9.3% |
+| **11+ (truncated)** | **63,263** | **88.2%** |
+- **0 products are orphaned from collections.** Only 47 products (0.07%) sit in ≤2 collections; 88.2% are in 11+.
+
+## 🔬 The skeptical decode — negative control makes the "0 orphans" load-bearing (not a vacuous pass)
+- **"0 orphans" is only meaningful if the BULK is collection-rich** (so a zero-membership product would be a true anomaly, not the norm). The distribution proves it: in-a-collection = 71,726 (100.0%), heavily multi-collection (88% in 11+). Browse is collection-driven AND filter-driven (Boost), and membership is dense → a zero-collection product WOULD stand out. There are none. So the healthy result is real, not an artifact of "browse is filter-only so collections don't matter."
+- **Caveat 1 closed — the 12 single-collection products are NOT browse-dark.** A re-scan found 12 active products whose ONLY collection is `best-seller-internal-use` (8 of them duplicate "Torre Wallcovering Prints" + Sneak Preview/Malibu PS/2 Designtex Rocket). The worry: if their one collection were unpublished/internal, they'd be effectively orphaned from customer browse.
+- **Caveat 2 closed (the publication probe that defuses caveat 1):** `best-seller-internal-use` ("Best Seller (internal use)", 127,560 products) is **PUBLISHED to the Online Store + all 13 sales channels** (Online Store, Buy Button, Google&YouTube, Facebook&Instagram, Houzz, POS, Fabricut, Pinterest, Rakuten, Shop, Inbox, TikTok, DWAutoPostBlog all = true). So despite the "(internal use)" name it is a fully-published, customer-reachable de-facto catch-all → the 12 single-membership products ARE reachable. Not browse-dark.
+
+## 📋 Disposition: clean bill on this surface; 2 minor flags (NOT defects)
+1. **Collection-membership integrity is HEALTHY fleet-wide.** This is a valuable NEGATIVE result: it definitively rules out "dark inventory via collection-orphaning" at fleet scale. The April-archive dark-inventory problem (archived-dark) is NOT compounded by an orphaning problem — live products are all reachable by browse. No fix needed.
+2. **Minor flag A — naming oddity (cosmetic, P4):** the de-facto catch-all (127,560 products, 90% of active) is named "Best Seller (internal use)" yet is published to all 13 channels. A customer-visible collection labeled "internal use" is misleading if it ever surfaces in nav/URL (`/collections/best-seller-internal-use`); consider renaming or confirming it's intentionally the all-products spine. Not a defect — flagged for tidiness only.
+3. **Minor flag B — a dedup signal (P3, hand-off to fleet-dedup, NOT this surface):** 8 ACTIVE "Torre Wallcovering Prints" [Designer Wallcoverings] share the same title and sit in only the internal collection — looks like a duplicate cluster (the `fleet-dedup` lane, maker-name-SKU caveat applies). Logged for the dedup pass, out of scope for collection-integrity.
+
+## What this does NOT cover (named residuals — honest scope, not "closed")
+- **Image integrity (the Qwen-dissent surface)** — missing featured image / broken 404 CDN URLs / single-image PDPs — remains UNSWEPT (deferred this cycle because the 404 check needs a per-URL HTTP fan-out at 71k scale). Strong candidate for a future cycle.
+- **Inventory-state (inventory=2026 both variants)** — the standing new-products convention — still un-audited fleet-wide.
+- This sweep counted collection MEMBERSHIP, not whether each membership is in a customer-NAVIGABLE menu (a collection can be published-and-URL-reachable yet not linked in site nav). Membership density (88% in 11+ real style/color collections like patterns/contemporary/grey-wallcovering) makes nav-reachability near-certain, but a nav-link audit is a separate (lighter) future check.
+
+_Cycle 86. DTD-picked B (2/1; Qwen dissent A). Swept collection-membership integrity: full active scan (71,726) → 0 true orphans, 100% in ≥1 collection, 88.2% in 11+. Negative control (bulk is collection-rich, so 0 orphans is a real anomaly-free result, not a filter-driven artifact) + caveat probes both closed: the 12 single-collection products are in `best-seller-internal-use` which is PUBLISHED to all 13 channels (reachable, not browse-dark). Clean bill — rules out the dark-by-orphaning class fleet-wide. 2 minor non-defect flags: the 127k catch-all is misnamed "(internal use)" yet fully published (cosmetic P4); 8 duplicate "Torre Wallcovering Prints" → fleet-dedup lane (P3). Named residuals: image integrity (Qwen's surface, deferred — 404 fan-out), inventory=2026, nav-link reachability. PJ excluded. Read-only, $0, nothing gated (no fix needed)._
diff --git a/scripts/collection-orphan-sweep/collection-orphan-sweep.mjs b/scripts/collection-orphan-sweep/collection-orphan-sweep.mjs
new file mode 100644
index 0000000..c214079
--- /dev/null
+++ b/scripts/collection-orphan-sweep/collection-orphan-sweep.mjs
@@ -0,0 +1,59 @@
+// collection-orphan-sweep — READ-ONLY, $0. Cycle 86. Measures COLLECTION-membership integrity:
+// active products in ZERO collections are orphaned from every browse/merchandising path =
+// potentially invisible to shoppers despite being live. BUT zero-collection only equals
+// "invisible" if browse is collection-driven; if the storefront is filter/search-driven (Boost)
+// it may be NORMAL. So this measures the membership DISTRIBUTION across all active products and
+// negative-controls the orphan set against the bulk — if the bulk is collection-rich, the orphans
+// are a real anomaly; if the bulk is collection-sparse, we REFRAME (no harm). EXCLUDES PJ.
+import fs from 'fs';
+const ENV=fs.readFileSync('/Users/stevestudio2/Projects/secrets-manager/.env','utf8');
+const TOK=((ENV.match(/^SHOPIFY_ADMIN_TOKEN=(.+)$/m)||[])[1]||'').replace(/['"\r]/g,'').trim();
+const GQL='https://designer-laboratory-sandbox.myshopify.com/admin/api/2024-10/graphql.json';
+const PJ=/phillip[- ]?jeffries/i;
+const sleep=ms=>new Promise(r=>setTimeout(r,ms));
+async function gql(q,v){for(let a=0;a<6;a++){const r=await fetch(GQL,{method:'POST',headers:{'X-Shopify-Access-Token':TOK,'Content-Type':'application/json'},body:JSON.stringify({query:q,variables:v})});if(r.status===429||r.status>=500){await sleep(2000*(a+1));continue;}const j=await r.json();if(j.errors&&JSON.stringify(j.errors).includes('Throttled')){await sleep(2500*(a+1));continue;}return j;}return null;}
+// collections(first:11) -> count up to 11 (11 means >=11, truncated). Also capture whether the
+// product is in ANY collection at all (the orphan signal). status:active only.
+const Q=`query($cursor:String){ products(first:60, after:$cursor, query:"status:active"){ pageInfo{hasNextPage endCursor}
+  edges{node{ title vendor productType totalInventory
+    collections(first:11){ edges{ node{ handle } } } } } } }`;
+let cursor=null,pages=0,scanned=0,complete=true;
+// distribution buckets by collection count
+const dist={0:0,1:0,2:0,'3-5':0,'6-10':0,'11+':0};
+let zero=0, withColl=0;
+const zeroByVendor={}, zeroEx=[];
+// also track which collection handles are most common (to spot a catch-all "all products" collection)
+const collFreq={};
+while(true){ const j=await gql(Q,{cursor}); if(!j?.data){complete=false;break;} const c=j.data.products;
+  for(const e of c.edges){ const n=e.node; if(PJ.test(n.vendor||''))continue; scanned++;
+    const handles=(n.collections?.edges||[]).map(x=>x.node.handle);
+    const k=handles.length;
+    handles.forEach(h=>collFreq[h]=(collFreq[h]||0)+1);
+    if(k===0){dist[0]++; zero++; zeroByVendor[n.vendor]=(zeroByVendor[n.vendor]||0)+1; if(zeroEx.length<25) zeroEx.push({t:(n.title||'').slice(0,46),vendor:n.vendor,type:n.productType,inv:n.totalInventory});}
+    else { withColl++;
+      if(k===1)dist[1]++; else if(k===2)dist[2]++; else if(k<=5)dist['3-5']++; else if(k<=10)dist['6-10']++; else dist['11+']++; }
+  }
+  pages++; if(!c.pageInfo.hasNextPage)break; cursor=c.pageInfo.endCursor;
+  const cost=j.extensions?.cost?.throttleStatus; if(cost&&cost.currentlyAvailable<500) await sleep(1500); else await sleep(250);
+  if(pages%100===0) console.log(`  ${pages}p scanned=${scanned} zero-collection=${zero}`);
+}
+const pct=x=>scanned?(100*x/scanned).toFixed(1):'0';
+console.log(`\n=== collection-orphan-sweep (READ-ONLY, $0) ===`);
+console.log(`active scanned: ${scanned} (complete=${complete}, ${pages}p) [PJ excluded]`);
+console.log(`\nCOLLECTION-COUNT DISTRIBUTION (the negative control — is the BULK collection-rich?):`);
+console.log(`  0 collections (ORPHAN): ${dist[0]} (${pct(dist[0])}%)`);
+console.log(`  1:                      ${dist[1]} (${pct(dist[1])}%)`);
+console.log(`  2:                      ${dist[2]} (${pct(dist[2])}%)`);
+console.log(`  3-5:                    ${dist['3-5']} (${pct(dist['3-5'])}%)`);
+console.log(`  6-10:                   ${dist['6-10']} (${pct(dist['6-10'])}%)`);
+console.log(`  11+ (truncated):        ${dist['11+']} (${pct(dist['11+'])}%)`);
+console.log(`\nINTERPRETATION: in-a-collection ${withColl} (${pct(withColl)}%) vs ORPHAN ${zero} (${pct(zero)}%).`);
+console.log(`  If in-a-collection is HIGH (say >80%), the ${zero} orphans are a real anomaly (browse is collection-driven, they are dark).`);
+console.log(`  If orphan share is HIGH, browse is likely filter/search-driven and zero-collection is NORMAL (reframe, no harm).`);
+console.log(`\ntop-12 collections by membership (spot a catch-all 'all products' collection):`);
+Object.entries(collFreq).sort((a,b)=>b[1]-a[1]).slice(0,12).forEach(([h,n])=>console.log(`  ${h}: ${n} (${pct(n)}%)`));
+console.log(`\nORPHAN (zero-collection) by vendor (top 15):`);
+Object.entries(zeroByVendor).sort((a,b)=>b[1]-a[1]).slice(0,15).forEach(([v,n])=>console.log(`  ${v}: ${n}`));
+console.log(`\nORPHAN examples (inv=totalInventory):`); zeroEx.forEach(x=>console.log(`  "${x.t}" [${x.vendor}] type=${x.type} inv=${x.inv}`));
+fs.writeFileSync('/tmp/collection-orphan-sweep.json',JSON.stringify({ts:new Date().toISOString(),scanned,complete,dist,zero,withColl,zeroByVendor,collFreqTop:Object.entries(collFreq).sort((a,b)=>b[1]-a[1]).slice(0,30),zeroEx},null,2));
+console.log(`\nwrote /tmp/collection-orphan-sweep.json`);
diff --git a/scripts/collection-orphan-sweep/low-membership-probe.mjs b/scripts/collection-orphan-sweep/low-membership-probe.mjs
new file mode 100644
index 0000000..5e83059
--- /dev/null
+++ b/scripts/collection-orphan-sweep/low-membership-probe.mjs
@@ -0,0 +1,43 @@
+// low-membership-probe — READ-ONLY, $0. Cycle 86 follow-up to collection-orphan-sweep.
+// The main sweep found 0 true orphans (100% in >=1 collection). Two skeptical caveats to close:
+//  (1) the 47 products in <=2 collections — is their membership a CUSTOMER-FACING collection, or
+//      ONLY the internal 'best-seller-internal-use' (which would make them effectively browse-dark)?
+//  (2) is 'best-seller-internal-use' (90% membership) PUBLISHED to the Online Store channel?
+//      An "internal use" collection visible to shoppers is a leak/oddity worth flagging.
+import fs from 'fs';
+const ENV=fs.readFileSync('/Users/stevestudio2/Projects/secrets-manager/.env','utf8');
+const TOK=((ENV.match(/^SHOPIFY_ADMIN_TOKEN=(.+)$/m)||[])[1]||'').replace(/['"\r]/g,'').trim();
+const GQL='https://designer-laboratory-sandbox.myshopify.com/admin/api/2024-10/graphql.json';
+const PJ=/phillip[- ]?jeffries/i;
+const sleep=ms=>new Promise(r=>setTimeout(r,ms));
+async function gql(q,v){for(let a=0;a<6;a++){const r=await fetch(GQL,{method:'POST',headers:{'X-Shopify-Access-Token':TOK,'Content-Type':'application/json'},body:JSON.stringify({query:q,variables:v})});if(r.status===429||r.status>=500){await sleep(2000*(a+1));continue;}const j=await r.json();if(j.errors&&JSON.stringify(j.errors).includes('Throttled')){await sleep(2500*(a+1));continue;}return j;}return null;}
+
+// (2) FIRST: best-seller-internal-use publication status + a couple of customer collections for contrast
+const CQ=`{ collections(first:60, query:"title:best-seller OR handle:best-seller-internal-use OR handle:patterns OR handle:contemporary"){ edges{ node{ handle title productsCount{count} publishedOnCurrentPublication } } } }`;
+const cj=await gql(CQ,{});
+console.log(`=== (2) collection publication status (publishedOnCurrentPublication = visible on current channel) ===`);
+for(const e of (cj?.data?.collections?.edges||[])){ const n=e.node; console.log(`  ${n.handle} | "${n.title}" | products=${n.productsCount?.count} | publishedOnCurrentPublication=${n.publishedOnCurrentPublication}`); }
+
+// (1) SECOND: re-scan active products, record handles for those with <=2 collections (the low-membership tail)
+const Q=`query($cursor:String){ products(first:80, after:$cursor, query:"status:active"){ pageInfo{hasNextPage endCursor}
+  edges{node{ title vendor collections(first:11){ edges{ node{ handle } } } } } } }`;
+let cursor=null,pages=0,scanned=0,complete=true;
+const lowMembers=[]; const onlyInternal=[];
+const INTERNAL=/internal/i;
+while(true){ const j=await gql(Q,{cursor}); if(!j?.data){complete=false;break;} const c=j.data.products;
+  for(const e of c.edges){ const n=e.node; if(PJ.test(n.vendor||''))continue; scanned++;
+    const handles=(n.collections?.edges||[]).map(x=>x.node.handle);
+    if(handles.length<=2){ lowMembers.push({t:(n.title||'').slice(0,42),vendor:n.vendor,handles}); }
+    // effectively-orphaned-from-customer-browse = every collection it is in is an 'internal' one
+    if(handles.length>0 && handles.every(h=>INTERNAL.test(h))){ onlyInternal.push({t:(n.title||'').slice(0,42),vendor:n.vendor,handles}); }
+  }
+  pages++; if(!c.pageInfo.hasNextPage)break; cursor=c.pageInfo.endCursor;
+  const cost=j.extensions?.cost?.throttleStatus; if(cost&&cost.currentlyAvailable<500) await sleep(1500); else await sleep(250);
+}
+console.log(`\n=== (1) low-membership tail (<=2 collections) — active scanned ${scanned} (complete=${complete}) ===`);
+console.log(`products in <=2 collections: ${lowMembers.length}`);
+lowMembers.slice(0,50).forEach(x=>console.log(`  "${x.t}" [${x.vendor}] -> [${x.handles.join(', ')}]`));
+console.log(`\n=== effectively browse-dark (ALL their collections are 'internal') : ${onlyInternal.length} ===`);
+onlyInternal.slice(0,50).forEach(x=>console.log(`  "${x.t}" [${x.vendor}] -> [${x.handles.join(', ')}]`));
+fs.writeFileSync('/tmp/low-membership-probe.json',JSON.stringify({ts:new Date().toISOString(),scanned,complete,lowMembers,onlyInternal,collPub:(cj?.data?.collections?.edges||[]).map(e=>e.node)},null,2));
+console.log(`\nwrote /tmp/low-membership-probe.json`);
← 99fbc92 cycle 85: officer vp-dw-commerce CONFIRMED + 2 non-blocking · back to Dw Yolo Loop · cycle 86: officer vp-dw-marketing CONFIRMED + refinements fo c11af4e →