`tasks/prd-historical-price-crawl.md`

123 lines
# PRD: Comprehensive Historical Price Crawl (2010-2025)

## Introduction

Crawl complete historical price data for all Omega watches from 2010 to 2025 using the Wayback Machine. This will build a comprehensive database of real market prices over 15 years.

## Goals

- Fetch historical prices for ALL 15 Omega watches from 2010-2025
- Store data in a persistent JSON database
- Create price trend analysis from 15 years of data
- Build data foundation for price prediction features
- Zero cost using Wayback Machine (free)

## User Stories

### US-020: Create Historical Data Schema
**Description:** Define the data structure for storing 15 years of price history.

**Acceptance Criteria:**
- [ ] Create data/historical-prices.json schema
- [ ] Include: watchId, date, price, source, condition
- [ ] Support multiple prices per date
- [ ] Include metadata (crawl date, source URL)

### US-021: Create Bulk Wayback Crawler
**Description:** Create a Python script that crawls Chrono24 archives from 2010-2025.

**Acceptance Criteria:**
- [ ] Create scripts/bulk-wayback-crawler.py
- [ ] Accept date range parameters (--from-year, --to-year)
- [ ] Crawl all 15 watch models
- [ ] Rate limiting (respect archive.org)
- [ ] Resume capability (skip already crawled dates)
- [ ] Progress logging
- [ ] Output to data/historical-prices.json

### US-022: Implement Multi-Source Crawling
**Description:** Crawl from multiple archived sources for better coverage.

**Acceptance Criteria:**
- [ ] Chrono24 archives (primary)
- [ ] WatchCharts archives (if available)
- [ ] Omega official site archives (MSRP)
- [ ] eBay archives (secondary market)
- [ ] Deduplicate overlapping data

### US-023: Create Crawl Orchestrator
**Description:** Node.js script to manage the crawl process.

**Acceptance Criteria:**
- [ ] Create scripts/crawl-orchestrator.js
- [ ] Queue all watches for crawling
- [ ] Track progress in data/crawl-status.json
- [ ] Handle failures gracefully
- [ ] Email/log notification on completion
- [ ] Estimated 8-12 hours for full crawl

### US-024: Create Data Aggregation Pipeline
**Description:** Process raw crawl data into usable format.

**Acceptance Criteria:**
- [ ] Aggregate prices by month
- [ ] Calculate min/max/avg per period
- [ ] Detect price anomalies/outliers
- [ ] Generate trend statistics
- [ ] Output to data/aggregated-prices.json

### US-025: Update API to Serve Historical Data
**Description:** Add API endpoint for historical price queries.

**Acceptance Criteria:**
- [ ] GET /api/historical-prices/:watchId
- [ ] Query params: from, to, granularity (day/month/year)
- [ ] Cache results
- [ ] Return aggregated statistics

### US-026: Create Historical Price Charts
**Description:** Frontend component to visualize 15-year price history.

**Acceptance Criteria:**
- [ ] Update PriceChart.jsx with historical data
- [ ] Show real data points (not simulated)
- [ ] Interactive zoom/pan
- [ ] Compare multiple watches

## Technical Approach

### Crawl Strategy
1. Query Wayback CDX API for all snapshots 2010-2025
2. Sample snapshots (1-2 per month to avoid overload)
3. Parse prices from archived HTML
4. Store raw data with metadata
5. Post-process to aggregate

### Rate Limiting
- 1 request per 2 seconds to archive.org
- Maximum 30 requests per minute
- Pause 5 minutes after every 100 requests
- Full crawl: ~8-12 hours

### Data Sources Priority
1. Chrono24 (best price data)
2. WatchCharts (market analytics)
3. eBay completed listings (secondary)
4. Omega.com (MSRP reference)

## Success Metrics

- 80%+ coverage of 2010-2025 timeframe
- At least 50 price points per watch
- Data quality: <5% outliers
- Crawl completes without errors

---

## Checklist

- [x] User stories are specific
- [x] Free solution using Wayback Machine
- [x] Technical approach outlined
- [x] Saved to tasks/prd-historical-price-crawl.md