Cycle 4: project README (pipeline, host-scoping, deploy topology, data-regen)

ba08fecb5587e3685eb38d17f8b2d5ebf81b8bd8 · 2026-07-01 14:11:36 -0700 · Steve Abrams
Files touched

A README.md
M out/officer-yolo-ledger.md
Diff

commit ba08fecb5587e3685eb38d17f8b2d5ebf81b8bd8
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Wed Jul 1 14:11:36 2026 -0700

    Cycle 4: project README (pipeline, host-scoping, deploy topology, data-regen)
---
 README.md                  | 91 ++++++++++++++++++++++++++++++++++++++++++++++
 out/officer-yolo-ledger.md |  4 ++
 2 files changed, 95 insertions(+)

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..0d521b7
--- /dev/null
+++ b/README.md
@@ -0,0 +1,91 @@
+# japan-enrich
+
+Offline **staging + enrichment + viewer** pipeline for three Asian wallcovering
+vendors, feeding two host-scoped Designer Wallcoverings microsites:
+
+| Host | Scope | Vendors | Patterns |
+|---|---|---|---|
+| `china.designerwallcoverings.com` | `source:'greenland'` **only** | Greenland | 880 |
+| `japan.designerwallcoverings.com` | Lilycolor + Sangetsu | Lilycolor / Sangetsu | 2,543 / 622 |
+
+The two hosts are hard-partitioned by `hostScope()` in the viewer — a `china.`
+request can only ever see Greenland rows, `japan.` (or localhost) only ever sees
+the Japanese catalog. They can't leak into each other.
+
+## Pipeline
+
+```
+scrape_*.py        →   staging/*.jsonl      →   enrich_*.py        →   out/*-enriched.jsonl   →   viewer
+(feed-first,           (one row per             ($0 local Pillow       (one row per colorway,     (ENRICH index
+ stdlib only)           PATTERN, holding         hex + color_family;     keyed by mfr_sku)          keyed by
+                        colorway_skus[])         opt-in vision)                                     normSku)
+```
+
+Each vendor row is a **pattern** holding N colorway SKUs (Sangetsu/Greenland
+model) or a single SKU (Lilycolor). Enrichment emits **one line per colorway**
+so the viewer's `ENRICH` map (keyed by `normSku(mfr_sku)`) lights up each
+colorway card individually.
+
+### Scrapers (feed-first — hit the platform's JSON feed, never the rendered DOM)
+
+| Script | Vendor | Source |
+|---|---|---|
+| `scrape_greenland.py` | Greenland | public Vue/JSON API `api.greenlandwallcoverings.com/api` (`/category/list`, `/product/page`, `/product/search`) |
+| `scrape_sangetsu_backfill.py` | Sangetsu | WooCommerce Store API `/wp-json/wc/store/v1/products` |
+| `scrape_sangetsu_specs.py` | Sangetsu | same WC API — parses the spec block (Width / Type-Fire / Repeat / Backing) out of the product `description` |
+| `fix_sangetsu_stragglers.py` | Sangetsu | targeted repair of blank/edge patterns |
+
+Lilycolor staging (`lilycolor-unified-staging.jsonl`) is produced upstream; see
+`out/lilycolor-audit.md` for its current data-quality gaps.
+
+### Enrichers ($0 local — Pillow for color, optional qwen2.5vl for style/material)
+
+- **`enrich.py`** — shared helpers: `hex_palette`, `dominant_family`,
+  `vision_tags`, `norm_sku`, `OLLAMA` endpoint.
+- **`enrich_sangetsu.py`** — per-colorway Pillow hex/color-family + one vision
+  call per pattern. Vision (style/material) is the slow leg (~90s/call) and is
+  **decoupled behind `SANGETSU_VISION` (default off)** — the color pass is the
+  goal, vision is an opt-in bonus. Writes are **atomic (temp+rename) checkpoint
+  every 25 patterns** and **carry-forward resumable**, so a mid-run crash/restart
+  can never truncate the baseline.
+- **`enrich_greenland.py`** — Pillow, downscale-on-download to 256px
+  (~15 KB/swatch) to stay disk-safe.
+
+## Viewer
+
+`viewer-local/server.js` — dependency-free Node HTTP server (`public/` static +
+JSON API). Run locally:
+
+```bash
+node viewer-local/server.js 9931      # port arg optional; defaults to 9931
+```
+
+- Host-scoped catalog via `hostScope(req)` (reads `req.headers.host`).
+- Local swatch routes: `/img/greenland/<sku>`, `/img/lily/<sku>` (fall back to
+  the vendor CDN / `preview_images` when a local swatch isn't present).
+- Hot-reloads `ROWS` + `ENRICH` when staging/enriched files change.
+
+## Deploy (Kamatera)
+
+pm2 process **`japan-viewer`** → `/root/Projects/japan-staging-viewer/viewer/server.js`
+on port **9934**. Both `china.` and `japan.` nginx vhosts host-scope-proxy to
+that single process; nginx applies Basic Auth (401 gate). There is **no
+`.deploy.conf`** — deploy = targeted `rsync` of the changed files + `pm2 reload
+japan-viewer`. Live deploys are Steve-gated. (See memory `japan-viewer-deploy`.)
+
+## Regenerating data
+
+Staging + enriched `*.jsonl` are **gitignored build artifacts** — the scrapers,
+enrichers, and `staging/` are the source of truth. To rebuild:
+
+```bash
+python3 scrape_greenland.py            # → staging/greenland-staging.jsonl
+python3 enrich_greenland.py            # → out/greenland-enriched.jsonl
+python3 enrich_sangetsu.py             # → out/sangetsu-enriched.jsonl (color pass; add SANGETSU_VISION=1 for style/material)
+```
+
+## Ledgers
+
+- `out/officer-yolo-ledger.md` — autonomous officer-yolo loop history (DTD
+  verdict + officer + gate check + result per cycle).
+- `out/lilycolor-audit.md` — Lilycolor data-quality gap report.
diff --git a/out/officer-yolo-ledger.md b/out/officer-yolo-ledger.md
index cc543be..455da2f 100644
--- a/out/officer-yolo-ledger.md
+++ b/out/officer-yolo-ledger.md
@@ -39,3 +39,7 @@
 - **DTD verdict: C** (Codex) — read-only Lilycolor data-quality audit. Officer: vp-dw-commerce. Gate: NONE (read-only). Codex cost ~$0.001.
 - **Result:** wrote out/lilycolor-audit.md. 2,543 Lilycolor rows. Top gaps: **title_en 100% missing**, title_ja 62% missing, image_bytes_fetched 100% false (manifest zipUrls not localized), repeat_tate 51% missing, functions 49% empty. Prices/width/fire-ratings largely complete. Pipeline flags (cost_confirmed/settlement_checked/deduped/activation_ready) 100% false = expected `staged-for-new` state.
 - **Surfaced next candidate:** Cycle 4 → synthesize `title_en` for 2,543 patterns ($0 local, from title_ja/attributes) — the biggest customer-facing gap.
+
+### Cycle 4 — EXECUTED (2026-07-01)
+- **DTD verdict: I** (Codex) — write project README. Officer: vp-engineering (docs). Gate: NONE. Codex cost ~$0.001.
+- **Result:** wrote README.md — pipeline diagram, per-vendor scraper/enricher table, host-scoping, viewer run cmd, Kamatera deploy topology, data-regen commands, ledger pointers.
← e4ec595 Cycle 3: Lilycolor data-quality audit (title_en 100% gap, im · back to Japan Enrich · officer-yolo loop CONVERGED (Cycle 5): all high-value SAFE j e9bfa45 →