`skills/onboard-source/SKILL.md`

65 lines
---
name: onboard-source
description: Creates a new price-source connector (API/scraper/manual import). Use when adding a new marketplace, auction house, dealer archive, or forum source to the Omega Watches 2.0 system.
disable-model-invocation: true
---
You are implementing a new data source connector for the Omega Watches 2.0 enterprise price history platform.

## Context
The system uses a dual-ledger architecture:
- **Ledger 1**: MSRP snapshots (daily immutable observations from manufacturers)
- **Ledger 2**: Secondary market events (auctions, marketplaces, dealers, forums)

Database: PostgreSQL `omega_watches_v2` on 127.0.0.1:5432
Project: /root/Projects/omega-watches-2

## When Adding a New Source, Produce:

### 1. Legal/Compliance Checklist
- Check robots.txt at the source URL
- Review Terms of Service for scraping/automation clauses
- Check if login gating exists
- Document rate limit requirements
- Determine if explicit permission is required (like eBay API access)
- Record findings in the `data_source` table with `tos_reviewed`, `tos_allows_collection`, `tos_notes`

### 2. Data Field Mapping
Map source-specific fields to the canonical schema:
- **Watch identity**: reference_number_raw, serial_number, year_text, case_material, movement
- **Condition**: condition_raw → condition_normalized (via condition_mapping table)
- **Price**: price_amount, currency, hammer_price, buyers_premium, total_to_buyer, fee_semantics
- **Sale**: sale_date, location_text, seller_type, event_type
- **Provenance**: provenance_raw, images_json
- **Source**: source_url, source_listing_id

### 3. Collector Approach (preference order)
1. **API** — Best: structured, rate-limited, documented
2. **Export/Tooling** — Good: official data exports, RSS feeds
3. **Scrape** — Acceptable: only where ToS permits, with robots.txt compliance
4. **Manual** — Fallback: for ToS-restricted sources (e.g., Hodinkee)

### 4. Test Plan
- Create fixture HTML/JSON files in `collectors/fixtures/{source_name}/`
- Write parser unit tests
- Define backfill strategy (date ranges, pagination)
- Document expected record volume

### 5. Database Registration
```sql
INSERT INTO data_source (name, display_name, source_type, access_method, base_url, reliability_score, tos_reviewed, tos_allows_collection, tos_notes, rate_limit_rpm)
VALUES ('new_source', 'New Source Display Name', 'auction|marketplace|dealer|forum', 'api|scrape|manual', 'https://...', 3, false, NULL, 'ToS review pending', 10);
```

### 6. Parser Version
```sql
INSERT INTO parser_version (source_id, version, selectors_json, is_current)
VALUES ((SELECT id FROM data_source WHERE name = 'new_source'), '1.0.0', '{"title": "...", "price": "...", "date": "..."}', true);
```

## Output Files
- `collectors/{source_name}/collector.mjs` — Collector module
- `collectors/{source_name}/parser.mjs` — Parser with versioned selectors
- `collectors/{source_name}/fixtures/` — Test fixtures
- `collectors/{source_name}/README.md` — Source documentation
- SQL migration for data_source + parser_version registration