New Import Viewer

Search the build

53 commits indexed

1. bc6681c 2026-09-03 TK-11046 WS-C: exclude DELETED_FROM_SHOPIFY from net-new anti-join + dedup set
2. 4bb29d4 2026-08-25 creds-safe fetch guard: resolve relative fetch vs credential-free location (creds-in-URL trap)
3. 6ea7371 2026-08-20 nav-agent: sync v1.3 (host left-sidebar collapse on load)
4. a752b73 2026-08-19 nav-agent: List data full-width tight columns (grid) — layout fix
5. f9175fb 2026-08-19 Refresh nav-agent to v1.2 (adds kv-list compact List view, default-off)
6. f03b673 2026-08-13 chore: v1.1.1 (nav-agent static-serving fix, session close)
7. d0de553 2026-08-13 Serve /nav-agent/* static assets — raw http routed index/drill.js but not the nav-agent drop-in, so /nav-agent/* 404'd; add traversal-guarded route
8. a665308 2026-08-13 nav-agent: universal grid-controls drop-in on internal viewer pages (STRONG-only)
9. c4b93a3 2026-08-01 new-import-viewer: adopt href-to-deeper-data primitives (TK-10093)
10. fae9d94 2026-07-29 TK-10039: phase8-fix/shopify-push.js migrates width to metafield + prose-only body
11. fed1f23 2026-07-29 TK-10039: consumer.js (new-import-viewer cadence) migrates specs to metafields + prose-only body
12. 8f6d032 2026-07-13 auto-save: 2026-07-13T14:55:38 (1 files) — package.json
13. 3479a52 2026-07-13 similar-pattern search: per-card '≈ Similar' button → /api/similar proxy to all.dw (creds server-side) → scored live substitutes in an in-card panel
14. 0250f13 2026-07-09 chore: lint (node --check ✓), v1.0.1 (session close)
15. ff72920 2026-07-08 new.: fill primary image from Shopify + full gallery (product_images) with hover-cycle
16. 7582d03 2026-07-03 chore: macstudio3 migration — reconcile from mac2 + repoint paths (stevestudio2→macstudio3, node/npm/npx→/opt/homebrew)
17. e34ec31 2026-07-02 auto-save: 2026-07-02T17:51:29 (2 files) — public/index.html server.js
18. f65d9a6 2026-07-02 Grid UI upgrade: sortable list-view column headers (server-side sorts), density max 20, Safari user:pass@ fetch fix
19. d85bde7 2026-06-19 Retire new-import-viewer duplicate cadence plists — consolidating on Designer-Wallcoverings/shopify cadence-import.js (already publishes + uses shared variant budget)
20. 4573e7c 2026-06-19 Add one-shot auto first-run (429-gated validated 42-batch + CNCP report + self-disable)
21. b4ec23e 2026-06-19 phase8-fix: real create+publish module (mirrors consumer.js 96cd954 publish-gap fix) + wiring patch for Kamatera full-monte
22. c345dc3 2026-06-19 cadence: hourly wrapper (run-cadence.sh, --limit 42 live+publish) + launchd plist (staged, not bootstrapped)
23. fb73edf 2026-06-19 cadence: manifest generator (42/batch, oldest-first round-robin, consumer-eligible, 3-source dedup-aware)
24. 96cd954 2026-06-19 consumer.js: fix publish gap (publishablePublish to Online Store) + add required Sample variant
25. 5a2715d 2026-06-11 snapshot: 9790 stats fix + required-indexes note
26. 6e38f8b 2026-06-11 9790: document load-bearing dw_sku_registry functional index (replay on DB rebuild)
27. 0d81ad3 2026-06-11 9790: fix /api/stats death-spiral — statement_timeout guard + non-blocking warmer
28. 1fb9644 2026-06-10 new-import-viewer: enforce 3-source NEVER-DUPLICATE in net-new predicate + hard server-side stage gate
29. 7be6207 2026-06-10 index.html review fixes: map Shopify lowercase status (active/draft/archived) → app vocabulary after live action (chip styled + Publish button disables correctly); show staged-so-far count in bulk error; only emit vendor link for http(s) urls (block javascript:/data: from scraped p.url)
30. 01ee7be 2026-06-10 review fixes (3-agent panel): consumer atomic ledger-before-UPDATE (no orphaned creates→dup), images.length gate (no imageless products via non-http urls), dedup keyed on mfr-alone + sibling-row check (dup2) + self-sealing mirror insert + NULL-mfr skip, command54 HOLD pending brand ruling, private-label-no-mask skip, payload-wide command54 leak scan, consec-failure circuit breaker, skipped no longer blocks reprocess, per-manifest parse guard + action=launch assert, single-number parsePrice, .env-first precedence; server localhost bind + SSRF redirect:manual host-revalidation + numeric-host block, NETNEW upper(trim()) match, /api/ids real 250k cap; functional indexes idx_sp/vc_mfr_upper_trim (stats 60s→1s)
31. 24a8d83 2026-06-10 launch-queue CONSUMER: manifested batches → Shopify draft products. DRY-RUN default; live needs LAUNCH_CONSUMER_LIVE=1 + --live (Steve gate). Per-product execution gates: image required, dw_sku+pattern required, fresh NEVER-DUPLICATE (mfr_sku vs shopify_products + in-run dedup + linked check), vendor_registry skip_shopify + private-label display names, cowtan denylist, status forced draft, 450/run hard cap (daily variant limit), 600ms throttle + 429 retry, idempotent ledger resume; writes linkage back to vendor_catalog on create
32. 863ce65 2026-06-10 staging manifests: every batch persists full id/SKU manifest to data/launch-batches/ (queue lines were counts-only = un-actionable); superseded pre-gate count-only batches via queue marker; restaged ALL netnew gated: 120,991 staged with manifests, 8,221 no-image BLOCKED, cowtan_tout (8,312) excluded per Steve 'never'
33. 4fb4e88 2026-06-10 HARD GATE: all products must have images (Steve 2026-06-10) — launch staging filters no-image rows server-side (single, bulk, raw API); launch buttons disabled on no-image cards with reason; bulk UI reports blocked count; non-launch actions unaffected
34. 01963b9 2026-06-10 harden: noimage boot-trap defused (persisted diagnostic filter falls back to netnew) + /img local thumbnail proxy with disk cache + SSRF guard — grid no longer depends on 100+ vendor hosts hotlink policy
35. cd99729 2026-06-10 stage net-new for ALL vendors (101 vendors, 111,234 skus) to launch queue — draft, stage-only (no Shopify writes). Excluded: cowtan_tout (Steve 'never'), newwall+command54 (already staged). With newwall+command54 = 125,950 total net-new queued. NEVER-DUPLICATE: all pre-filtered against Shopify mfr_sku
36. a4c7ca5 2026-06-09 stage net-new for newwall (8485) + command54 (6231) to launch queue — draft mode, stage-only (no Shopify writes). command54 carries private-label rule (Phillipe Romano relabel at import)
37. 58e24b2 2026-06-09 NET NEW by vendor panel: per-vendor breakdown (count + image coverage withImg/noImg) shown for netnew/new status, click-to-filter, image-% bar per vendor. /api/vendors now returns image coverage. Surfaces which vendors have importable net-new WITH images (e.g. hollywood 8997 100%, brewster_york 7772 94%)
38. bde393a 2026-06-09 NET NEW view: new MINUS already-on-Shopify (Steve: 'new - on shopify = new not on shopify'). NETNEW_PRED excludes any flagged-new row whose mfr pattern number already exists in shopify_products (index-friendly equality, 240ms). New default status + header pill + health-strip chip + dup-overlap warning. Found 62,030 'new' rows are actually already on Shopify -> true net-new 134,247 not 196,277
39. 1833735 2026-06-09 Full action suite per chip + select 1→all: every card renders all 5 actions (disabled-with-reason; launch hard-disabled on linked rows = dup guard) + copy SKU/DW + vendor/admin/storefront links; visible 🕓 crawl date+time on grid cards; /api/ids endpoint + first-N / shown / ALL-matching selection; bulk staging chunks through the 5k cap (includes parallel-session netnew predicate)
40. b8c32b4 2026-06-09 Panel items B+D: No-image filter (12,156 rows) + structured empty state; threshold-colored catalog-health strip with click-to-filter quick chips
41. fe11c1f 2026-06-09 Panel item A: split details/actions into separate triggers — ⓘ details (read-only) vs ⚙ actions (write surface); browsing metadata never reveals a live button; actions chip reddens when armed
42. e4e0ced 2026-06-09 DTD-C: per-session arm toggle defaults OFF every load; live write now needs .env ceiling AND session-armed (arm:true). .env stays hard ceiling, bulk still stage-only
43. 0d6de3d 2026-06-09 Apply design-agent critique: red danger treatment for live action buttons + armed top strip + pulsing LIVE badge, de-shout vendor labels, title hierarchy, two-column spec drawer
44. 90782f6 2026-06-09 Refine: bulk forced stage-only (live single-card only), in-place chip+action refresh after live write, accurate live-vs-staged confirms, live result detail
45. 16d4ca8 2026-06-09 Force live-write gate OFF by default (SHOPIFY_LIVE_WRITES master switch, stage-only); restore aligned CSS-grid list view; clear test launch-queue lines
46. b2cb29b 2026-06-09 Wire live Shopify unpublish behind double gate (master switch + per-action allowlist); fix admin-link store handle to designer-laboratory-sandbox
47. d24c5ab 2026-06-09 Show full Shopify state (new/published/unpublished/archived) + staged action suite under collapsed per-card chips; indexed shopify_products join
48. af2b077 2026-06-09 snapshot before shopify-status + collapsed-chips rebuild
49. 186111c 2026-06-09 viewer: add List view toggle (compact horizontal rows, persisted) alongside the card grid
50. 039ba4a 2026-06-09 viewer: select-and-launch — per-product checkboxes, select-all, launch bar (draft/active) → /api/launch stages a batch (live Shopify push stays a separate gated step)
51. a368e81 2026-06-09 viewer: clean 'no image' placeholder on broken vendor image URLs (e.g. timorous logo-404 bug)
52. f6b1300 2026-06-09 fix: strip psql trailing newline from final field (stats newestCrawl + per-vendor crawled)
53. 1a10b71 2026-06-09 new-import-viewer: agent + web viewer for products new-to-import (sync_status=new / not on Shopify)

Authors

• Steve52
• steve1

Creative ideas + design notes

Commits with substantial prose (≥120 chars) — the rationale behind each move.

A deleted product's tombstone row (same mfr_sku) must not suppress its own legit
re-import. Both the NETNEW anti-join and the stage dedup CTE now exclude
DELETED_FROM_SHOPIFY (null-safe).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BbqKZLbd8emwVkNrxwtG2X

Fleet inoculation — opening this basic-auth app with credentials in the URL
poisoned document.baseURI and made relative fetch('/api/…') throw. Guard resolves
non-absolute request URLs against location instead. Ref: creds-in-url-fetch-guard-fleet-pattern.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Density slider max 10 → 20.
- List view: sticky header row of 12 sortable column labels (vendor, pattern,
  color, status, vendor SKU, DW SKU, collection, type, sync, shopify id,
  first seen, last crawl) sharing ONE grid-template-columns rule with the row
  cards so labels align with cells by construction. Click = asc, click again =
  desc, ▲/▼ indicator; new list-only cells (.lcell) per field.
- Sorting stays SERVER-side (grid paginates limit/offset — client-sorting the
  loaded page would lie): extended the injection-safe SORTS whitelist in
  server.js with asc/desc pairs for dwsku/collection/type/sync/shopid/status
  (derived SHOP_STATUS CASE) + vendor/pattern/color/sku desc + firstseen_asc.
- Safari creds fix: all fetch('/api/…') → fetch(location.origin+'/api/…') —
  relative fetches break in WebKit on pages opened with user:pass@ URLs.
- Presentational + sort wiring only; NO changes to arm-toggle / live-write /
  stage-action machinery.
- E2E-verified with Playwright on local :9790 against dw_unified: slider
  max=20, header row renders + computed grid template matches row cards,
  vendor asc→desc reorders, DW-SKU header reorders again, sticky on scroll,
  0 console/page errors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

REST status:'active' never publishes to a sales channel — products landed on
zero channels (publishedOnPublication=false). Add GraphQL publishablePublish to
the Online Store publication after each active create, plus the mandatory
{DW_SKU}-Sample $4.25 variant. Drafts stay unpublished by design.

Root cause: netnew count's NOT EXISTS vs dw_sku_registry had no functional
index (upper(trim(mfr_sku))) → seq scan → 75s+/never. Every page-load + the
boot warmer fired it; runs piled up (13 stacked 18-43min), saturating PG into
a death spiral that froze the whole single-threaded server ('and loading').

Fix (this commit): PGOPTIONS statement_timeout=30000 on q()+qAsync so no query
can ever run away/pile up again; background stats warmer + STATS_CACHE serve
/api/stats instantly.
DB side (not in git): CREATE INDEX idx_registry_mfr_upper_trim ON
dw_sku_registry (upper(trim(mfr_sku))) — drops the stats query to ~1.6s.

NETNEW_PRED now NOT-EXISTS against dw_sku_registry as well as shopify_products
(was shopify-only — 49,386 distinct mfr_skus were shopify-clean but already in
the 84k registry → would have imported as dups). Added dedupFilterIds() that
re-validates every staged/live candidate row's mfr_sku against shopify_products
AND dw_sku_registry on upper(trim()), dropping dups + un-dedupable rows, fail-safe
(query error aborts the whole batch). Applied in /api/action before BOTH staging
and live writes so a stale id list / raw curl can never push a known dup. /dtd-A 2026-06-10.

- /api/action stages to data/launch-queue.jsonl and never writes the store unless SHOPIFY_LIVE_WRITES=1 (default 0); .env ships gate=0, no live actions allow-listed
- List view rebuilt as aligned CSS grid (display:contents on card body) so checkbox/thumb/vendor/pattern/status-chip/SKU/crawl columns line up across rows; full crawl date+time, ~54px rows; details drawer drops full-width below
- indexed shopify_products join (gid built on vc side) keeps /api/stats sub-second

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Zero-dep node+psql server over dw_unified.vendor_catalog. /api/stats, /api/vendors,
/api/new-products (vendor filter + search + 7 sort modes + pagination). Viewer has
the mandatory sort select + density slider (localStorage-persisted) and the admin
crawl date+time chip per standing rules. Read-only; surfaces what the latest vendor
crawl flagged as NEW before any Phase-8 push. Port 9790.

File tree

19 files tracked. Click any to browse the source at HEAD.

• .gitignore
• auto-first-run.sh
• com.steve.dw-cadence.plist.RETIRED-dup-of-cadence-import
• com.steve.dw-first-run.plist.RETIRED-dup-of-cadence-import
• consumer.js
• data/REQUIRED-INDEXES.sql
• data/cleared-links-backup-20260609-183447.csv
• data/launch-queue.jsonl
• data/shopify-link-mismatches.csv
• generate-manifests.js
• package.json
• phase8-fix/phase8-create-publish.patch
• phase8-fix/shopify-push.js
• public/drill.js
• public/index.html
• public/nav-agent/nav-agent.css
• public/nav-agent/nav-agent.js
• run-cadence.sh
• server.js

Other build journals

← New Engine  ·  all 4 projects  ·  New Items Dashboard →

Export

commits.csv · feed.atom · project.json · commits.json