← back to Japan Enrich

README.md

92 lines

# japan-enrich

Offline **staging + enrichment + viewer** pipeline for three Asian wallcovering
vendors, feeding two host-scoped Designer Wallcoverings microsites:

| Host | Scope | Vendors | Patterns |
|---|---|---|---|
| `china.designerwallcoverings.com` | `source:'greenland'` **only** | Greenland | 880 |
| `japan.designerwallcoverings.com` | Lilycolor + Sangetsu | Lilycolor / Sangetsu | 2,543 / 622 |

The two hosts are hard-partitioned by `hostScope()` in the viewer — a `china.`
request can only ever see Greenland rows, `japan.` (or localhost) only ever sees
the Japanese catalog. They can't leak into each other.

## Pipeline

```
scrape_*.py        →   staging/*.jsonl      →   enrich_*.py        →   out/*-enriched.jsonl   →   viewer
(feed-first,           (one row per             ($0 local Pillow       (one row per colorway,     (ENRICH index
 stdlib only)           PATTERN, holding         hex + color_family;     keyed by mfr_sku)          keyed by
                        colorway_skus[])         opt-in vision)                                     normSku)
```

Each vendor row is a **pattern** holding N colorway SKUs (Sangetsu/Greenland
model) or a single SKU (Lilycolor). Enrichment emits **one line per colorway**
so the viewer's `ENRICH` map (keyed by `normSku(mfr_sku)`) lights up each
colorway card individually.

### Scrapers (feed-first — hit the platform's JSON feed, never the rendered DOM)

| Script | Vendor | Source |
|---|---|---|
| `scrape_greenland.py` | Greenland | public Vue/JSON API `api.greenlandwallcoverings.com/api` (`/category/list`, `/product/page`, `/product/search`) |
| `scrape_sangetsu_backfill.py` | Sangetsu | WooCommerce Store API `/wp-json/wc/store/v1/products` |
| `scrape_sangetsu_specs.py` | Sangetsu | same WC API — parses the spec block (Width / Type-Fire / Repeat / Backing) out of the product `description` |
| `fix_sangetsu_stragglers.py` | Sangetsu | targeted repair of blank/edge patterns |

Lilycolor staging (`lilycolor-unified-staging.jsonl`) is produced upstream; see
`out/lilycolor-audit.md` for its current data-quality gaps.

### Enrichers ($0 local — Pillow for color, optional qwen2.5vl for style/material)

- **`enrich.py`** — shared helpers: `hex_palette`, `dominant_family`,
  `vision_tags`, `norm_sku`, `OLLAMA` endpoint.
- **`enrich_sangetsu.py`** — per-colorway Pillow hex/color-family + one vision
  call per pattern. Vision (style/material) is the slow leg (~90s/call) and is
  **decoupled behind `SANGETSU_VISION` (default off)** — the color pass is the
  goal, vision is an opt-in bonus. Writes are **atomic (temp+rename) checkpoint
  every 25 patterns** and **carry-forward resumable**, so a mid-run crash/restart
  can never truncate the baseline.
- **`enrich_greenland.py`** — Pillow, downscale-on-download to 256px
  (~15 KB/swatch) to stay disk-safe.

## Viewer

`viewer-local/server.js` — dependency-free Node HTTP server (`public/` static +
JSON API). Run locally:

```bash
node viewer-local/server.js 9931      # port arg optional; defaults to 9931
```

- Host-scoped catalog via `hostScope(req)` (reads `req.headers.host`).
- Local swatch routes: `/img/greenland/<sku>`, `/img/lily/<sku>` (fall back to
  the vendor CDN / `preview_images` when a local swatch isn't present).
- Hot-reloads `ROWS` + `ENRICH` when staging/enriched files change.

## Deploy (Kamatera)

pm2 process **`japan-viewer`** → `/root/Projects/japan-staging-viewer/viewer/server.js`
on port **9934**. Both `china.` and `japan.` nginx vhosts host-scope-proxy to
that single process; nginx applies Basic Auth (401 gate). There is **no
`.deploy.conf`** — deploy = targeted `rsync` of the changed files + `pm2 reload
japan-viewer`. Live deploys are Steve-gated. (See memory `japan-viewer-deploy`.)

## Regenerating data

Staging + enriched `*.jsonl` are **gitignored build artifacts** — the scrapers,
enrichers, and `staging/` are the source of truth. To rebuild:

```bash
python3 scrape_greenland.py            # → staging/greenland-staging.jsonl
python3 enrich_greenland.py            # → out/greenland-enriched.jsonl
python3 enrich_sangetsu.py             # → out/sangetsu-enriched.jsonl (color pass; add SANGETSU_VISION=1 for style/material)
```

## Ledgers

- `out/officer-yolo-ledger.md` — autonomous officer-yolo loop history (DTD
  verdict + officer + gate check + result per cycle).
- `out/lilycolor-audit.md` — Lilycolor data-quality gap report.