`README.md`

65 lines
# patternbank-archive

**Internal-only PG ingest of patternbank.com.** Read [LICENSE_NOTES.md](./LICENSE_NOTES.md) before running anything — this archive is gated by an explicit AI-train opt-out and a ClaudeBot disallow on the source site, and exists only as a locked-row local reference dataset for the wallco.ai / DW design pipeline.

## Quick start

```bash
# 1. Apply schema (already done at init)
npm run init-db

# 2. Smoke test — 1 small category, 2 pages
node scripts/discover.js --cat=/categories/1-womenswear/10-floral --max-pages=2

# 3. Process a few PDPs
node scripts/ingest-pdp.js --limit=10 --concurrency=2

# 4. Pull image bytes
node scripts/download-images.js --limit=25

# 5. Stats
npm run stats
```

## Long-running unattended ingest

```bash
nohup bash scripts/loop.sh > logs/loop.log 2>&1 &
tail -f logs/loop.log
# loop runs discover --auto then perpetually drains the queue at 20 PDPs +
# 40 images per tick, with RATE_LIMIT_MS between requests. Idle-sleeps 10 min
# when fully drained.
```

## Schema (PG `patternbank_archive`)

| Table            | Purpose                                          |
|------------------|--------------------------------------------------|
| `designers`      | Per-designer metadata keyed by URL slug          |
| `patterns`       | One row per design; `pattern_id` = numeric ID    |
| `pattern_images` | Per-image: remote URL, local path, sha256, bytes |
| `categories`     | Top × sub category tree (sparse)                 |
| `ingest_runs`    | Per-run counters and timestamps                  |
| `crawl_queue`    | URL queue with status / attempts                 |

## Provenance / compliance fields (every `patterns` row)

- `source = 'patternbank-headless'`
- `owned_by_steve = FALSE`
- `image_license = 'unknown_third_party'`
- `ok_for_commercial_use = FALSE`
- `ok_for_ai_training = FALSE`  (matches site `Content-Signal: ai-train=no`)

## Kill switch

```bash
bash scripts/halt.sh
# Truncates all PG tables, removes data/images/, no confirmation.
```

## Crawler conduct
- User-Agent: `patternbank-archive-internal/0.1 (research; contact: steve@designerwallcoverings.com)`
- Concurrency: 2 (max 3 hard cap)
- Inter-request rate limit: 2.5s default
- Routes through Browserbase (Cloudflare anti-bot is otherwise unsolvable from local IP)