New Import Viewer

Search the build

42 commits indexed

1. 8f6d032 2026-07-13 auto-save: 2026-07-13T14:55:38 (1 files) — package.json
2. 3479a52 2026-07-13 similar-pattern search: per-card '≈ Similar' button → /api/similar proxy to all.dw (creds server-side) → scored live substitutes in an in-card panel
3. 0250f13 2026-07-09 chore: lint (node --check ✓), v1.0.1 (session close)
4. ff72920 2026-07-08 new.: fill primary image from Shopify + full gallery (product_images) with hover-cycle
5. 7582d03 2026-07-03 chore: macstudio3 migration — reconcile from mac2 + repoint paths (stevestudio2→macstudio3, node/npm/npx→/opt/homebrew)
6. e34ec31 2026-07-02 auto-save: 2026-07-02T17:51:29 (2 files) — public/index.html server.js
7. f65d9a6 2026-07-02 Grid UI upgrade: sortable list-view column headers (server-side sorts), density max 20, Safari user:pass@ fetch fix
8. d85bde7 2026-06-19 Retire new-import-viewer duplicate cadence plists — consolidating on Designer-Wallcoverings/shopify cadence-import.js (already publishes + uses shared variant budget)
9. 4573e7c 2026-06-19 Add one-shot auto first-run (429-gated validated 42-batch + CNCP report + self-disable)
10. b4ec23e 2026-06-19 phase8-fix: real create+publish module (mirrors consumer.js 96cd954 publish-gap fix) + wiring patch for Kamatera full-monte
11. c345dc3 2026-06-19 cadence: hourly wrapper (run-cadence.sh, --limit 42 live+publish) + launchd plist (staged, not bootstrapped)
12. fb73edf 2026-06-19 cadence: manifest generator (42/batch, oldest-first round-robin, consumer-eligible, 3-source dedup-aware)
13. 96cd954 2026-06-19 consumer.js: fix publish gap (publishablePublish to Online Store) + add required Sample variant
14. 5a2715d 2026-06-11 snapshot: 9790 stats fix + required-indexes note
15. 6e38f8b 2026-06-11 9790: document load-bearing dw_sku_registry functional index (replay on DB rebuild)
16. 0d81ad3 2026-06-11 9790: fix /api/stats death-spiral — statement_timeout guard + non-blocking warmer
17. 1fb9644 2026-06-10 new-import-viewer: enforce 3-source NEVER-DUPLICATE in net-new predicate + hard server-side stage gate
18. 7be6207 2026-06-10 index.html review fixes: map Shopify lowercase status (active/draft/archived) → app vocabulary after live action (chip styled + Publish button disables correctly); show staged-so-far count in bulk error; only emit vendor link for http(s) urls (block javascript:/data: from scraped p.url)
19. 01ee7be 2026-06-10 review fixes (3-agent panel): consumer atomic ledger-before-UPDATE (no orphaned creates→dup), images.length gate (no imageless products via non-http urls), dedup keyed on mfr-alone + sibling-row check (dup2) + self-sealing mirror insert + NULL-mfr skip, command54 HOLD pending brand ruling, private-label-no-mask skip, payload-wide command54 leak scan, consec-failure circuit breaker, skipped no longer blocks reprocess, per-manifest parse guard + action=launch assert, single-number parsePrice, .env-first precedence; server localhost bind + SSRF redirect:manual host-revalidation + numeric-host block, NETNEW upper(trim()) match, /api/ids real 250k cap; functional indexes idx_sp/vc_mfr_upper_trim (stats 60s→1s)
20. 24a8d83 2026-06-10 launch-queue CONSUMER: manifested batches → Shopify draft products. DRY-RUN default; live needs LAUNCH_CONSUMER_LIVE=1 + --live (Steve gate). Per-product execution gates: image required, dw_sku+pattern required, fresh NEVER-DUPLICATE (mfr_sku vs shopify_products + in-run dedup + linked check), vendor_registry skip_shopify + private-label display names, cowtan denylist, status forced draft, 450/run hard cap (daily variant limit), 600ms throttle + 429 retry, idempotent ledger resume; writes linkage back to vendor_catalog on create
21. 863ce65 2026-06-10 staging manifests: every batch persists full id/SKU manifest to data/launch-batches/ (queue lines were counts-only = un-actionable); superseded pre-gate count-only batches via queue marker; restaged ALL netnew gated: 120,991 staged with manifests, 8,221 no-image BLOCKED, cowtan_tout (8,312) excluded per Steve 'never'
22. 4fb4e88 2026-06-10 HARD GATE: all products must have images (Steve 2026-06-10) — launch staging filters no-image rows server-side (single, bulk, raw API); launch buttons disabled on no-image cards with reason; bulk UI reports blocked count; non-launch actions unaffected
23. 01963b9 2026-06-10 harden: noimage boot-trap defused (persisted diagnostic filter falls back to netnew) + /img local thumbnail proxy with disk cache + SSRF guard — grid no longer depends on 100+ vendor hosts hotlink policy
24. cd99729 2026-06-10 stage net-new for ALL vendors (101 vendors, 111,234 skus) to launch queue — draft, stage-only (no Shopify writes). Excluded: cowtan_tout (Steve 'never'), newwall+command54 (already staged). With newwall+command54 = 125,950 total net-new queued. NEVER-DUPLICATE: all pre-filtered against Shopify mfr_sku
25. a4c7ca5 2026-06-09 stage net-new for newwall (8485) + command54 (6231) to launch queue — draft mode, stage-only (no Shopify writes). command54 carries private-label rule (Phillipe Romano relabel at import)
26. 58e24b2 2026-06-09 NET NEW by vendor panel: per-vendor breakdown (count + image coverage withImg/noImg) shown for netnew/new status, click-to-filter, image-% bar per vendor. /api/vendors now returns image coverage. Surfaces which vendors have importable net-new WITH images (e.g. hollywood 8997 100%, brewster_york 7772 94%)
27. bde393a 2026-06-09 NET NEW view: new MINUS already-on-Shopify (Steve: 'new - on shopify = new not on shopify'). NETNEW_PRED excludes any flagged-new row whose mfr pattern number already exists in shopify_products (index-friendly equality, 240ms). New default status + header pill + health-strip chip + dup-overlap warning. Found 62,030 'new' rows are actually already on Shopify -> true net-new 134,247 not 196,277
28. 1833735 2026-06-09 Full action suite per chip + select 1→all: every card renders all 5 actions (disabled-with-reason; launch hard-disabled on linked rows = dup guard) + copy SKU/DW + vendor/admin/storefront links; visible 🕓 crawl date+time on grid cards; /api/ids endpoint + first-N / shown / ALL-matching selection; bulk staging chunks through the 5k cap (includes parallel-session netnew predicate)
29. b8c32b4 2026-06-09 Panel items B+D: No-image filter (12,156 rows) + structured empty state; threshold-colored catalog-health strip with click-to-filter quick chips
30. fe11c1f 2026-06-09 Panel item A: split details/actions into separate triggers — ⓘ details (read-only) vs ⚙ actions (write surface); browsing metadata never reveals a live button; actions chip reddens when armed
31. e4e0ced 2026-06-09 DTD-C: per-session arm toggle defaults OFF every load; live write now needs .env ceiling AND session-armed (arm:true). .env stays hard ceiling, bulk still stage-only
32. 0d6de3d 2026-06-09 Apply design-agent critique: red danger treatment for live action buttons + armed top strip + pulsing LIVE badge, de-shout vendor labels, title hierarchy, two-column spec drawer
33. 90782f6 2026-06-09 Refine: bulk forced stage-only (live single-card only), in-place chip+action refresh after live write, accurate live-vs-staged confirms, live result detail
34. 16d4ca8 2026-06-09 Force live-write gate OFF by default (SHOPIFY_LIVE_WRITES master switch, stage-only); restore aligned CSS-grid list view; clear test launch-queue lines
35. b2cb29b 2026-06-09 Wire live Shopify unpublish behind double gate (master switch + per-action allowlist); fix admin-link store handle to designer-laboratory-sandbox
36. d24c5ab 2026-06-09 Show full Shopify state (new/published/unpublished/archived) + staged action suite under collapsed per-card chips; indexed shopify_products join
37. af2b077 2026-06-09 snapshot before shopify-status + collapsed-chips rebuild
38. 186111c 2026-06-09 viewer: add List view toggle (compact horizontal rows, persisted) alongside the card grid
39. 039ba4a 2026-06-09 viewer: select-and-launch — per-product checkboxes, select-all, launch bar (draft/active) → /api/launch stages a batch (live Shopify push stays a separate gated step)
40. a368e81 2026-06-09 viewer: clean 'no image' placeholder on broken vendor image URLs (e.g. timorous logo-404 bug)
41. f6b1300 2026-06-09 fix: strip psql trailing newline from final field (stats newestCrawl + per-vendor crawled)
42. 1a10b71 2026-06-09 new-import-viewer: agent + web viewer for products new-to-import (sync_status=new / not on Shopify)

Authors

• Steve41
• steve1

Creative ideas + design notes

Commits with substantial prose (≥120 chars) — the rationale behind each move.

- Density slider max 10 → 20.
- List view: sticky header row of 12 sortable column labels (vendor, pattern,
  color, status, vendor SKU, DW SKU, collection, type, sync, shopify id,
  first seen, last crawl) sharing ONE grid-template-columns rule with the row
  cards so labels align with cells by construction. Click = asc, click again =
  desc, ▲/▼ indicator; new list-only cells (.lcell) per field.
- Sorting stays SERVER-side (grid paginates limit/offset — client-sorting the
  loaded page would lie): extended the injection-safe SORTS whitelist in
  server.js with asc/desc pairs for dwsku/collection/type/sync/shopid/status
  (derived SHOP_STATUS CASE) + vendor/pattern/color/sku desc + firstseen_asc.
- Safari creds fix: all fetch('/api/…') → fetch(location.origin+'/api/…') —
  relative fetches break in WebKit on pages opened with user:pass@ URLs.
- Presentational + sort wiring only; NO changes to arm-toggle / live-write /
  stage-action machinery.
- E2E-verified with Playwright on local :9790 against dw_unified: slider
  max=20, header row renders + computed grid template matches row cards,
  vendor asc→desc reorders, DW-SKU header reorders again, sticky on scroll,
  0 console/page errors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

REST status:'active' never publishes to a sales channel — products landed on
zero channels (publishedOnPublication=false). Add GraphQL publishablePublish to
the Online Store publication after each active create, plus the mandatory
{DW_SKU}-Sample $4.25 variant. Drafts stay unpublished by design.

Root cause: netnew count's NOT EXISTS vs dw_sku_registry had no functional
index (upper(trim(mfr_sku))) → seq scan → 75s+/never. Every page-load + the
boot warmer fired it; runs piled up (13 stacked 18-43min), saturating PG into
a death spiral that froze the whole single-threaded server ('and loading').

Fix (this commit): PGOPTIONS statement_timeout=30000 on q()+qAsync so no query
can ever run away/pile up again; background stats warmer + STATS_CACHE serve
/api/stats instantly.
DB side (not in git): CREATE INDEX idx_registry_mfr_upper_trim ON
dw_sku_registry (upper(trim(mfr_sku))) — drops the stats query to ~1.6s.

NETNEW_PRED now NOT-EXISTS against dw_sku_registry as well as shopify_products
(was shopify-only — 49,386 distinct mfr_skus were shopify-clean but already in
the 84k registry → would have imported as dups). Added dedupFilterIds() that
re-validates every staged/live candidate row's mfr_sku against shopify_products
AND dw_sku_registry on upper(trim()), dropping dups + un-dedupable rows, fail-safe
(query error aborts the whole batch). Applied in /api/action before BOTH staging
and live writes so a stale id list / raw curl can never push a known dup. /dtd-A 2026-06-10.

- /api/action stages to data/launch-queue.jsonl and never writes the store unless SHOPIFY_LIVE_WRITES=1 (default 0); .env ships gate=0, no live actions allow-listed
- List view rebuilt as aligned CSS grid (display:contents on card body) so checkbox/thumb/vendor/pattern/status-chip/SKU/crawl columns line up across rows; full crawl date+time, ~54px rows; details drawer drops full-width below
- indexed shopify_products join (gid built on vc side) keeps /api/stats sub-second

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Zero-dep node+psql server over dw_unified.vendor_catalog. /api/stats, /api/vendors,
/api/new-products (vendor filter + search + 7 sort modes + pagination). Viewer has
the mandatory sort select + density slider (localStorage-persisted) and the admin
crawl date+time chip per standing rules. Read-only; surfaces what the latest vendor
crawl flagged as NEW before any Phase-8 push. Port 9790.

File tree

16 files tracked. Click any to browse the source at HEAD.

• .gitignore
• auto-first-run.sh
• com.steve.dw-cadence.plist.RETIRED-dup-of-cadence-import
• com.steve.dw-first-run.plist.RETIRED-dup-of-cadence-import
• consumer.js
• data/REQUIRED-INDEXES.sql
• data/cleared-links-backup-20260609-183447.csv
• data/launch-queue.jsonl
• data/shopify-link-mismatches.csv
• generate-manifests.js
• package.json
• phase8-fix/phase8-create-publish.patch
• phase8-fix/shopify-push.js
• public/index.html
• run-cadence.sh
• server.js

Other build journals

← New Arrivals Content Engine  ·  all 4 projects  ·  New Items Dashboard →

Export

commits.csv · feed.atom · project.json · commits.json