← back to Tk10658 Sitemap Audit
TK-10658: sitemap health audit — product sitemap clean (250/250 200); 19/19 empty collections crawl-instructed (amplifies TK-10656); ~2% sample-products -> gmc-agent
e0f59b22daeac2697fbfdc22e8c700437da4e881 · 2026-08-18 01:29:13 -0700 · steve
Files touched
A .gitignoreA FINDINGS.mdA data/collection-urls.txtA data/product-sitemap-result.jsonA product-sitemap-sampler.mjs
Diff
commit e0f59b22daeac2697fbfdc22e8c700437da4e881
Author: steve <steve@designerwallcoverings.com>
Date: Tue Aug 18 01:29:13 2026 -0700
TK-10658: sitemap health audit — product sitemap clean (250/250 200); 19/19 empty collections crawl-instructed (amplifies TK-10656); ~2% sample-products -> gmc-agent
---
.gitignore | 4 +
FINDINGS.md | 16 ++
data/collection-urls.txt | 577 +++++++++++++++++++++++++++++++++++++++
data/product-sitemap-result.json | 32 +++
product-sitemap-sampler.mjs | 31 +++
5 files changed, 660 insertions(+)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..cc6b63a
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,4 @@
+node_modules/
+.env*
+*.log
+data/subsitemaps.txt
diff --git a/FINDINGS.md b/FINDINGS.md
new file mode 100644
index 0000000..06c2a4e
--- /dev/null
+++ b/FINDINGS.md
@@ -0,0 +1,16 @@
+# TK-10658 — DW Shopify sitemap health audit (read-only, $0)
+
+**Verdict: mostly ALL-CLEAR** — the sitemap itself is healthy. Value is in cross-amplifying TK-10656 + one hand-off to gmc-agent. No NEW independent gated action (fixes route through existing tickets).
+
+## Verified findings
+1. **Product sitemap: HEALTHY.** 84 product sub-sitemaps (default locale) × 3 locales. Random sample of 250 product URLs across 12 sub-sitemaps → **250/250 resolve HTTP 200** (the 4 initial "code 0" were transient fetch timeouts; all 200 on retry — VERIFY-BEFORE-ACTING). No stale 404s, no redirects. Shopify keeps it current.
+2. **Collections sitemap crawl-instructs the 19 empty pages (amplifies TK-10656).** The collections sitemap lists 577 URLs = all published collections. **19/19 of the TK-10656 empty-to-shopper collections ARE in the sitemap** (30/30 sampled collection URLs = 200). So Google is actively told to crawl all 19 dead/empty pages — INCLUDING the `versa-designed-surfaces-wallcoverings` private-label LEAK page. → Reinforces the TK-10656 fix: **unpublishing those collections auto-removes them from the sitemap** (Shopify regenerates it). No separate sitemap action needed.
+
+## Secondary (hand-offs, not my lane to fix)
+3. **~2% of sitemap products are `-sample-` pages (→ gmc-agent / TK-10635).** In a 17k-URL sample, 345 (~2%, est ~1,700 across ~84k published) are memo/`-sample-` products (e.g. `dwhf-70000-sample-…`) — thin/duplicate $4.25 sample pages Google indexes. Overlaps the known $4.25 GMC-leak class; handing to gmc-agent, not fixing here.
+4. **en-gb sitemap persists (→ TK-10406).** en-ca + en-gb locale sitemaps both exist though Steve dropped UK from Google targeting (TK-10406). Minor — the GB market/sitemap is still generated. Flag to crash-recovery/TK-10406.
+5. **Agentic-commerce discovery (informational).** New `sitemap_agentic_discovery.xml` → `/agents.md` (HTTP 200) — Shopify's AI-agent discovery surface. No action; noted for awareness.
+
+## Honesty / limits
+- Product-URL health is a 250-URL random sample (0 non-200 after retry) → high confidence the sitemap is clean, but not an exhaustive 84k crawl. The ~1,700 sample-product estimate is extrapolated from a 17k-URL sample (~2%).
+- Self-gated (no full contrarian pass) because this is a low-risk all-clear with NO new independent gated action — the two actionable items route to their owning tickets (TK-10656 unpublish, gmc-agent samples). Applied contrarian discipline inline (retried the 4 code-0s; verified 19/19 directly).
diff --git a/data/collection-urls.txt b/data/collection-urls.txt
new file mode 100644
index 0000000..024273e
--- /dev/null
+++ b/data/collection-urls.txt
@@ -0,0 +1,577 @@
+/collections/paper-backed-cotton-walls
+/collections/phillipe-romano-fine-textures
+/collections/chenille-fabric
+/collections/faux-animal-skin
+/collections/madagascar-cloth
+/collections/cork
+/collections/flocked-velvet-wallpaper-collection
+/collections/fantastic-frames
+/collections/stripe-plaid
+/collections/chinoiserie
+/collections/decoglass-wallpaper
+/collections/et-cie-wall-panels
+/collections/european-import-wallpaper-collection
+/collections/glam-by-phillipe-romano
+/collections/green-wall-textures
+/collections/hollywood-wallpaper-commercial-wallpaper
+/collections/lounge
+/collections/naturally-glamorous
+/collections/phillipe-romano-essential-textures
+/collections/phillipe-romano-grasscloth
+/collections/phillipe-romano-grasses-vol-3
+/collections/phillipe-romano-luxe-textures
+/collections/phillipe-romano-textures
+/collections/phillipe-romano-textures-vol-4
+/collections/phillipe-romano-textures-vol-5
+/collections/phillipe-romano-vol-4
+/collections/primo-european-leathers
+/collections/retro-walls
+/collections/trellis
+/collections/phillipe-romano-faux-leathers-upholstery-more
+/collections/paisleys-1
+/collections/checks-plaids-and-stripes-1
+/collections/toile-de-juoy-1
+/collections/lattice-and-trellis-patterns
+/collections/tropical-and-leaves
+/collections/trees-and-twigs-1
+/collections/roses-wallpaper
+/collections/wall-murals-for-kids
+/collections/real-scala-silk
+/collections/glitter-wallpaper
+/collections/faux-grasscloth-wallpaper-collection
+/collections/shiny-mylar-3d-foils
+/collections/sequin-wallpapers
+/collections/thick-natural-sisal
+/collections/surface-stick-3m-self-adhesive
+/collections/basketweave-paper
+/collections/designer-artist-series
+/collections/specialty-wall-textures-styles-1
+/collections/commercial-durable
+/collections/paint
+/collections/coordonne-spain-wallpaper-collection
+/collections/wall-murals-on-demand
+/collections/champs-elysees-chenille
+/collections/faux-gold-leaf-walls-54
+/collections/hollywood-type-2-wallpaper-collection
+/collections/vintage-patterns-on-demand
+/collections/whimsical-traditional
+/collections/regal-trellis-and-lattice
+/collections/juliet-travers-european-classics
+/collections/ron-redding-classic-walls
+/collections/modern-originals-1
+/collections/versace-wallpaper
+/collections/modern-hamptons-1
+/collections/fine-paintables-by-phillipe-romano
+/collections/the-gardens-by-phillipe-romano
+/collections/missoni-home
+/collections/phillipe-romano-faux-metals-collection
+/collections/phillipe-romano-renaissance-metal-leaf-collection
+/collections/traditional-whimsy-v-3
+/collections/stripe
+/collections/faux-silk
+/collections/embossed
+/collections/type-2-wallpaper-collection
+/collections/hand-crafted
+/collections/suede
+/collections/acoustical
+/collections/lattices
+/collections/tropical
+/collections/animal-wallpaper-collection
+/collections/chevron-herringbone-wallpaper-collection
+/collections/geometric
+/collections/phillipe-romano
+/collections/phillipe-romano-naturals
+/collections/phillipe-romano-vinyls
+/collections/leed-walls-1
+/collections/damasks-1
+/collections/british-walls
+/collections/traditional-wimsy
+/collections/dots
+/collections/leaves-wallpaper-collection
+/collections/medallions
+/collections/scroll
+/collections/pattern-design-lab-2
+/collections/exclusive
+/collections/innovative
+/collections/reflective
+/collections/commercial
+/collections/paintable
+/collections/metallic-vinyl
+/collections/ralph-lauren-home
+/collections/grasscloth
+/collections/fine-paper
+/collections/shiny-mylar-mirror
+/collections/abstract-durable-vinyl
+/collections/faux-leather
+/collections/geometric-durable-vinyl
+/collections/traditional-durable-vinyl
+/collections/performance-linen-by-phillipe-romano
+/collections/quick-ship-hospitality
+/collections/tropicana-durable-textures
+/collections/contemporary
+/collections/french-country-wallpaper-collection
+/collections/industrial
+/collections/minimalist
+/collections/modern
+/collections/rustic-charm-wallpaper-collection
+/collections/traditional
+/collections/featured
+/collections/red-wallpaper-collection
+/collections/blue-wallpaper
+/collections/commercial-styles
+/collections/commercial-specialty
+/collections/commercial-featured
+/collections/themes
+/collections/patterns
+/collections/hollywood-sunset-collection
+/collections/glitter-walls
+/collections/authentic-1950s-reproduction-vintage-wallcoverings
+/collections/authentic-1960s-reproduction-vintage-wallcoverings
+/collections/vintage
+/collections/luxuria-fine-textile-wallcoverings-by-phillipe-romano
+/collections/innovations-usa
+/collections/scalamandre-fabrics
+/collections/mind-the-gap
+/collections/maya-romanoff
+/collections/fabric-trim
+/collections/outdoor-fabric
+/collections/plaid-fabric
+/collections/suede-faux-suede-fabric
+/collections/upholstery-vinyl
+/collections/gold-wallpaper
+/collections/grey-silver
+/collections/hand-made-wallcoverings
+/collections/green-wallpaper-collection
+/collections/hollywood-wallcovering-collection
+/collections/mylar-wallpaper
+/collections/european-wallpapers
+/collections/faux-collection
+/collections/murals
+/collections/pink-wallcovering-2
+/collections/new-arrivals
+/collections/authentic-1940s-reproduction-vintage-wallcoverings
+/collections/authentic-1970s-reproduction-vintage-wallcoverings
+/collections/cole-sons
+/collections/kravet-coillections
+/collections/the-graduate-collection
+/collections/lee-jofa
+/collections/gaston-y-daniela
+/collections/candice-olson-collection
+/collections/blithfield
+/collections/cole-and-son
+/collections/lawalls-collection
+/collections/novelty
+/collections/braid-tape
+/collections/madeinitaly
+/collections/mini-moderns-wallpaper
+/collections/phillipe-romano-ultimo
+/collections/finished-pillows
+/collections/designers-guild-wallpaper
+/collections/graham-and-brown-exclusive-wallpaper
+/collections/newmor-wallcovering
+/collections/exclusive-pillows-at-dw
+/collections/kravet-design
+/collections/as-seen-in-showrooms
+/collections/restoration-wallpaper
+/collections/pantone-color-of-the-year-2022-veri-periwinkle
+/collections/bella-italiano
+/collections/dw-staff-picks
+/collections/best-seller-internal-use
+/collections/zeuxis-parrhasius-wallpaper-collection
+/collections/dw-bespoke-studios
+/collections/dw-bespoke-studio-wallpaper-collection
+/collections/ombre
+/collections/grand-palace-wallpaper-collection
+/collections/kismet
+/collections/eden
+/collections/scenics
+/collections/stripes
+/collections/thibaut-wallpaper-and-fabrics
+/collections/throw-blanket
+/collections/fentucci
+/collections/dark-green
+/collections/metallic-wallpaper-collections
+/collections/brand-mckenzie-wallpaper-collection
+/collections/texture-gallery
+/collections/living-with-art
+/collections/luxe-retreat
+/collections/kids-wallpaper
+/collections/lillian-august-grasscloth-binder
+/collections/even-more-textures
+/collections/coastal-haven
+/collections/dragon-motifs
+/collections/beverly-hills-stripe-wallpaper
+/collections/malibu-knits
+/collections/coral
+/collections/gaston-nuevo-mundo
+/collections/lorenzo-castillo-v
+/collections/lorenzo-castillo-iv
+/collections/elements-ii-naturals
+/collections/emma-shipley-wallpaper-collection
+/collections/gaston-libreria
+/collections/ronald-redding
+/collections/phillipe-romano-asian-essence
+/collections/phillipe-romano-elegante
+/collections/lizzo-1
+/collections/blithfield-1
+/collections/yellow-wallpaper-collection
+/collections/barclay-butera-collection-1
+/collections/candice-olson-collection-1
+/collections/paolo-moschino-collections
+/collections/lorenzo-castillo-collection
+/collections/baker-lifestyle
+/collections/kravet
+/collections/threads
+/collections/fornasetti
+/collections/botanicals-and-florals
+/collections/animals-and-insects
+/collections/black
+/collections/novelty-wallpaper-collection
+/collections/drapery-fabrics
+/collections/modern-1
+/collections/upholstery-fabrics
+/collections/durable-w-c
+/collections/beige-cream-wallpaper-collection
+/collections/purple
+/collections/turquoise
+/collections/brown-wallpaper-collection
+/collections/orange
+/collections/multi
+/collections/cork-wallpaper-collection
+/collections/string
+/collections/crypton
+/collections/type-2-commercial-wallcovering
+/collections/velvet-upholstery
+/collections/animal-skin
+/collections/kelly-wearstler-collections
+/collections/the-kelly-wearstler-collection
+/collections/paul-montgomery-murals
+/collections/rebel-walls-murals
+/collections/equestrian-collection
+/collections/phillipe-romano-screen-printed-wallpaper
+/collections/phillipe-romano-screen-printed-fabric
+/collections/paul-montgomery-all-products
+/collections/phillipe-romano-flocked-velvet-wallpaper
+/collections/arts-and-crafts-wallpaper
+/collections/birds
+/collections/collections
+/collections/ajiro-wallpapers-by-maya-romanoff
+/collections/candice-olson-wallpaper
+/collections/clarke-clarke
+/collections/cole-and-son-collections
+/collections/grasscloth-wallpapers
+/collections/metallic-wallpaper-collection
+/collections/malibu-walls-1
+/collections/jeffrey-stevens-wallpaper-collection
+/collections/dresses-phillipe-romano
+/collections/steve-abrams-whimsy
+/collections/grasscloth-6
+/collections/catchii-european-wallpaper
+/collections/colefax-and-fowler
+/collections/d-f-brands
+/collections/m-o-wallpaper-brands
+/collections/mark-alexander
+/collections/martyn-lawrence-bullard
+/collections/p-z-wallpapers-by-designer
+/collections/clarke-and-clarke-1
+/collections/brands-at-designer-wallcoverings-a-c
+/collections/kravet-couture-1
+/collections/florals-and-flowers
+/collections/sarah-rowland-wallpaper-collection
+/collections/grasscloth-wallpaper-collection
+/collections/papis-lovejoy
+/collections/novasuede%E2%84%A2-microfiber-suede
+/collections/phillipe-romano-enlightenment
+/collections/phillipe-romano-metro-collection
+/collections/phillipe-romano-monument-collection
+/collections/phillipe-romano-sante-collection
+/collections/phillipe-romano-wilderie-collection
+/collections/phillipe-romano-faux-leather
+/collections/phillipe-romano-multi-use-fabrics
+/collections/phillipe-romano-outdoor-faux-leather
+/collections/phillipe-romano-pattern-upholstery
+/collections/phillipe-romano-print-basecloths
+/collections/phillipe-romano-texture-upholstery
+/collections/christian-lacroix-wallcoverings
+/collections/andrew-martin-wallcoverings
+/collections/architectural-fabrics
+/collections/armani-casa-wallcoverings
+/collections/donghia-wallcoverings
+/collections/elitis-wallcoverings
+/collections/mulberry-wallcoverings
+/collections/nobilis-wallcoverings
+/collections/thibaut-wallcoverings
+/collections/zoffany-wallcoverings
+/collections/minimalist-1
+/collections/phillipe-romano-acoustical
+/collections/phillipe-romano-commercial
+/collections/phillipe-romano-european-prints
+/collections/phillipe-romano-exclusive
+/collections/phillipe-romano-fabrics
+/collections/phillipe-romano-vegan-leather
+/collections/phillipe-romano-flock
+/collections/phillipe-romano-metals-1
+/collections/phillipe-romano-prints-1
+/collections/phillipe-romano-screen-prints
+/collections/phillipe-romano-type-2-vinyls
+/collections/phillipe-romano-vinyls-1
+/collections/phillipe-romano-wide-textures
+/collections/phillipe-romano-velvet-walls
+/collections/bespoke
+/collections/brunschwig-fils
+/collections/jeffrey-stevens
+/collections/koroseal
+/collections/ralph-lauren
+/collections/rebel-walls
+/collections/china-seas
+/collections/coordonne
+/collections/retro-walls-1
+/collections/wolf-gordon
+/collections/sandberg
+/collections/surface-stick
+/collections/anna-french-1
+/collections/graham-brown
+/collections/glitter-walls-1
+/collections/british-walls-1
+/collections/innovations
+/collections/baker-lifestyle-1
+/collections/carlisle-co
+/collections/daisy-bennett
+/collections/catchii
+/collections/colefax-fowler
+/collections/roberto-cavalli
+/collections/glass-bead
+/collections/novasuede
+/collections/primo-leathers
+/collections/leed-walls
+/collections/marburg
+/collections/borastapeter
+/collections/arte-international
+/collections/cole-son
+/collections/et-cie
+/collections/gp-j-baker
+/collections/harlequin
+/collections/holly-hunt
+/collections/laura-ashley
+/collections/malibu-wallpaper
+/collections/missoni
+/collections/nina-campbell
+/collections/phillipe-romano-1
+/collections/romo
+/collections/ultrasuede
+/collections/william-morris-1
+/collections/transitional-wallpaper
+/collections/textured-wallpaper
+/collections/scenic-wallpaper
+/collections/lattice-wallpaper
+/collections/toile-wallpaper
+/collections/bohemian-wallpaper
+/collections/victorian-wallpaper
+/collections/scandinavian-wallpaper
+/collections/animal-print-wallpaper
+/collections/medallion-wallpaper
+/collections/neoclassical-wallpaper
+/collections/vine-pattern-wallpaper
+/collections/farmhouse-wallpaper
+/collections/ogee-wallpaper
+/collections/art-nouveau-wallpaper
+/collections/greek-key-wallpaper
+/collections/baroque-wallpaper
+/collections/fretwork-wallpaper
+/collections/japonisme-wallpaper
+/collections/maximalist-wallpaper
+/collections/mediterranean-wallpaper
+/collections/renaissance-wallpaper
+/collections/quatrefoil-wallpaper
+/collections/vegan-crocodile-wallpaper
+/collections/designer-pulse-2026-03-07
+/collections/alligator-skin-wallpaper-collection
+/collections/type-2-durable
+/collections/european-imports
+/collections/renaissance-metal-leaf
+/collections/vegan-leathers
+/collections/warm-taupe
+/collections/warm-whites-ivory-1
+/collections/terracotta-clay-1
+/collections/sage-olive-1
+/collections/plum-aubergine-1
+/collections/chinoiserie-collection
+/collections/art-deco-metallics-1
+/collections/moody-dark
+/collections/coastal-nautical-1
+/collections/powder-room-statements-1
+/collections/primary-suite-1
+/collections/camel-cognac-1
+/collections/slate-charcoal-1
+/collections/steel-blue-1
+/collections/study-library-1
+/collections/commercial-contract
+/collections/residential
+/collections/murals-scenics
+/collections/grasscloth-natural-fiber
+/collections/textures-solids
+/collections/peel-stick
+/collections/sustainable-eco
+/collections/stripes-geometrics
+/collections/florals-botanicals
+/collections/damask-traditional
+/collections/designer-pulse-2026-03-21
+/collections/phyllis-morris-classic-wallcovering
+/collections/versa-designed-surfaces-wallcoverings
+/collections/daisy-bennett-wallcoverings
+/collections/marimekko-exclusive-wallcoverings
+/collections/tres-tintas-wallcoverings
+/collections/dolce-gabbana-casa-wallcoverings
+/collections/franquemont-london
+/collections/natural-paint
+/collections/emiliana-parati-wallcoverings
+/collections/sugarboo
+/collections/backdrop
+/collections/as-creation-wallcoverings
+/collections/wallpaper-nyc
+/collections/vahallan-wallcoverings
+/collections/fabricut-fabrics
+/collections/peg-norriss-wallcoverings
+/collections/pierre-frey-wallcoverings
+/collections/sacha-walckhoff-wallcoverings
+/collections/osborne-little
+/collections/farrow-ball-wallcoverings
+/collections/milton-and-king
+/collections/stout-wallcoverings
+/collections/fentucci-murals
+/collections/le-embossed-croc-vinyl
+/collections/fentucci-wallpaper
+/collections/tables
+/collections/pink-wallcovering-collection
+/collections/purple-wallcovering-collection
+/collections/white-wallpaper-collection
+/collections/non-woven
+/collections/vinyl-type-ii
+/collections/1838-wallcoverings
+/collections/newmor-wallcoverings
+/collections/dedar
+/collections/hermes
+/collections/holland-and-sherry
+/collections/sancar
+/collections/villa-nova
+/collections/apartment-wallpaper
+/collections/blue-wallcoverings
+/collections/green-wallcoverings
+/collections/gold-wallcoverings
+/collections/pink-wallcoverings
+/collections/red-wallcoverings
+/collections/gray-wallcoverings
+/collections/black-wallcoverings
+/collections/white-wallcoverings
+/collections/beige-wallcoverings
+/collections/navy-wallcoverings
+/collections/teal-wallcoverings
+/collections/purple-wallcoverings
+/collections/orange-wallcoverings
+/collections/brown-wallcoverings
+/collections/yellow-wallcoverings
+/collections/silver-wallcoverings
+/collections/dedar-wallcovering
+/collections/knoll-textiles
+/collections/architectural-wallcoverings-commercial
+/collections/architectural-wallcoverings
+/collections/hygge-and-west
+/collections/kids-nursery
+/collections/anthology-wallcoverings
+/collections/arte
+/collections/borastapeter-scandinavian-wallpapers
+/collections/carnegie-wallcoverings
+/collections/designers-guild-wallcoverings
+/collections/fabricut-wallcoverings
+/collections/groundworks-wallcoverings
+/collections/harlequin-wallcoverings
+/collections/maharam-wallcoverings
+/collections/ralph-lauren-wallpaper
+/collections/romo-europe-wallpapaper-collections
+/collections/threads-wallcoverings
+/collections/wolf-gordon-wallcoverings
+/collections/art-deco
+/collections/coastal
+/collections/bohemian
+/collections/toile
+/collections/damask
+/collections/ikat
+/collections/plaid
+/collections/paisley
+/collections/lattice
+/collections/paper
+/collections/silk
+/collections/sisal
+/collections/jute
+/collections/linen
+/collections/mica
+/collections/foil
+/collections/circles
+/collections/berber
+/collections/brazil
+/collections/steve-abrams-studios
+/collections/solid-wallcoverings
+/collections/textured-wallcoverings
+/collections/scenic-wallcoverings
+/collections/trellis-wallcoverings
+/collections/marble-wallcoverings
+/collections/fentucci-naturals
+/collections/artmura
+/collections/showroom-lines
+/collections/fallingstar
+/collections/tres-tintas
+/collections/la-walls
+/collections/quadrille-wallcoverings
+/collections/phillip-jeffries
+/collections/abruzzo-natural-collection
+/collections/pr-abaca
+/collections/pr-arrowroot
+/collections/pr-hemp
+/collections/pr-jacquard
+/collections/pr-jute
+/collections/pr-linen
+/collections/pr-metal-leaf
+/collections/pr-mixed-media
+/collections/pr-paper-weave
+/collections/pr-woven
+/collections/pr-raffia
+/collections/pr-silk
+/collections/pr-sisal
+/collections/pr-signature
+/collections/pr-suede
+/collections/pr-velvet
+/collections/pr-water-hyacinth
+/collections/pr-wood-veneer
+/collections/pr-wool
+/collections/pr-hue-greige
+/collections/pr-hue-black
+/collections/pr-hue-blue
+/collections/pr-hue-charcoal
+/collections/pr-hue-orange
+/collections/pr-hue-green
+/collections/pr-hue-red
+/collections/pr-hue-yellow
+/collections/pr-hue-chartreuse
+/collections/pr-hue-white
+/collections/pr-hue-teal
+/collections/trending-wallpaper-collection-2026
+/collections/carnegie-wall-panels
+/collections/carnegie-textiles
+/collections/collezione-italia
+/collections/material-non-woven
+/collections/material-vinyl-type-ii
+/collections/material-grasscloth
+/collections/material-linen
+/collections/material-cork
+/collections/material-metallic-foil
+/collections/material-silk
+/collections/material-natural-fiber
+/collections/material-paperweave
+/collections/material-leather
+/collections/material-glass-bead
+/collections/material-flock-velvet
+/collections/material-wood-veneer
+/collections/whimsical-wallcoverings
+/collections/vahallan
+/collections/graduate-collection
+/collections/majilite-metallic-specialties-i
diff --git a/data/product-sitemap-result.json b/data/product-sitemap-result.json
new file mode 100644
index 0000000..e2ecc9a
--- /dev/null
+++ b/data/product-sitemap-result.json
@@ -0,0 +1,32 @@
+{
+ "product_sitemaps_total": 84,
+ "sampled_sitemaps": 12,
+ "unique_product_urls_gathered": 12000,
+ "urls_headed": 250,
+ "status_distribution": {
+ "0": 4,
+ "200": 246
+ },
+ "non200_sample": [
+ {
+ "url": "/products/hilton-head-cork-wallcovering-carafe",
+ "code": 0,
+ "loc": null
+ },
+ {
+ "url": "/products/ressac-blond-cmo-paris",
+ "code": 0,
+ "loc": null
+ },
+ {
+ "url": "/products/dwkk-130012",
+ "code": 0,
+ "loc": null
+ },
+ {
+ "url": "/products/luxuria-beige-memo-sample-phillipe-romano-242",
+ "code": 0,
+ "loc": null
+ }
+ ]
+}
diff --git a/product-sitemap-sampler.mjs b/product-sitemap-sampler.mjs
new file mode 100644
index 0000000..85a10ca
--- /dev/null
+++ b/product-sitemap-sampler.mjs
@@ -0,0 +1,31 @@
+// TK-10658 — READ-ONLY: sample product URLs from the DW sitemap (default locale) and HEAD
+// each to measure status distribution (200 good vs 301/404 = crawl-budget waste / stale sitemap).
+const BASE='https://www.designerwallcoverings.com';
+const sleep=ms=>new Promise(r=>setTimeout(r,ms));
+async function get(u){for(let a=0;a<3;a++){try{const r=await fetch(u,{headers:{'User-Agent':'Mozilla/5.0 dw-audit'}});if(r.status>=500||r.status===429){await sleep(1000*(a+1));continue;}return r;}catch(e){await sleep(800*(a+1));}}return null;}
+// get raw index, product sub-sitemaps (default locale only, with params)
+const idx=await (await get(`${BASE}/sitemap.xml`)).text();
+const subs=[...idx.matchAll(/<loc>([^<]*sitemap_products_[^<]*)<\/loc>/g)].map(m=>m[1].replace(/&/g,'&'))
+ .filter(u=>!/\/en-(ca|gb)\//.test(u)); // default locale only
+// pick ~12 spread across the range
+const pick=[]; const step=Math.max(1,Math.floor(subs.length/12));
+for(let i=0;i<subs.length;i+=step) pick.push(subs[i]);
+let allUrls=[];
+for(const s of pick){ const xml=await (await get(s)).text(); const us=[...xml.matchAll(/<loc>([^<]*\/products\/[^<]+)<\/loc>/g)].map(m=>m[1]); allUrls.push(...us); }
+// dedupe + random sample 250
+allUrls=[...new Set(allUrls)];
+for(let i=allUrls.length-1;i>0;i--){const j=Math.floor(Math.random()*(i+1));[allUrls[i],allUrls[j]]=[allUrls[j],allUrls[i]];}
+const sample=allUrls.slice(0,250);
+const status={}; const bad=[];
+for(const u of sample){
+ const r=await get(u); const code=r?r.status:0;
+ // follow to final for 3xx
+ let finalCode=code, loc=null;
+ if(code>=300&&code<400){ loc=r.headers.get('location'); const rf=await get(u); const rl=await fetch(u,{redirect:'manual'}).catch(()=>null); }
+ status[code]=(status[code]||0)+1;
+ if(code!==200){ if(bad.length<40) bad.push({url:u.replace(BASE,''),code,loc}); }
+ await sleep(60);
+}
+console.log(JSON.stringify({product_sitemaps_total:subs.length, sampled_sitemaps:pick.length,
+ unique_product_urls_gathered:allUrls.length, urls_headed:sample.length,
+ status_distribution:status, non200_sample:bad.slice(0,20)},null,2));
(oldest)
·
back to Tk10658 Sitemap Audit
·
(newest)