`tasks/prd-real-api-integration.md`

172 lines
# PRD: Real Market Data via Chrono24 (Free GitHub Library)

## Introduction

Upgrade the market data service to fetch real Omega watch prices from Chrono24 using the free [irahorecka/chrono24](https://github.com/irahorecka/chrono24) Python library. Since our app is Node.js, we'll create a Python scraper script that Node.js calls via child process.

## Goals

- Fetch real watch prices from Chrono24 using free GitHub library
- Create Python scraper script callable from Node.js
- Cache results aggressively (1 hour) to minimize requests
- Graceful fallback to mock data when scraping fails
- Zero cost solution

## User Stories

### US-010: Install Chrono24 Python Library
**Description:** As a developer, I need the chrono24 Python package installed.

**Acceptance Criteria:**
- [ ] Install chrono24 package: `pip install chrono24`
- [ ] Verify import works: `python -c "import chrono24; print('OK')"`
- [ ] Document Python version requirement in README

### US-011: Create Python Scraper Script
**Description:** As a developer, I need a Python script that fetches Omega watch prices from Chrono24.

**Acceptance Criteria:**
- [ ] Create `scripts/chrono24-scraper.py`
- [ ] Accept args: --brand --model --reference --limit
- [ ] Output JSON to stdout with: prices[], avgPrice, minPrice, maxPrice
- [ ] Handle errors gracefully (output empty JSON, not crash)
- [ ] Include rate limiting (sleep between requests)
- [ ] Script runs without errors: `python scripts/chrono24-scraper.py --brand "Omega" --model "Speedmaster"`

### US-012: Create Watch Reference Mapping
**Description:** As a developer, I need a mapping from our watch IDs to Chrono24 search terms.

**Acceptance Criteria:**
- [ ] Create `data/watch-chrono24-map.json`
- [ ] Map watch IDs to search queries
- [ ] Include: brand, model, reference for each watch
- [ ] Cover all 32 watches in omega-watches.json
- [ ] File is valid JSON

### US-013: Create Backend API Endpoint
**Description:** As a developer, I need a server endpoint that calls the Python scraper.

**Acceptance Criteria:**
- [ ] Create `GET /api/market-data/:watchId` endpoint in server.js
- [ ] Call Python script via child_process.spawn
- [ ] Parse JSON output from Python script
- [ ] Cache results in memory (1 hour TTL)
- [ ] Return mock data on Python script failure
- [ ] Server starts without errors

### US-014: Update Frontend Service
**Description:** As a developer, I need to update the frontend to call our backend endpoint.

**Acceptance Criteria:**
- [ ] Update `src/services/marketDataService.js`
- [ ] Call `/api/market-data/:watchId` instead of using mock data directly
- [ ] Keep mock data as fallback
- [ ] Maintain same return shape
- [ ] npm run build passes

### US-015: Add Cache Warmup Script
**Description:** As a developer, I need a script to pre-populate the cache with real data.

**Acceptance Criteria:**
- [ ] Create `scripts/warmup-market-cache.js`
- [ ] Fetch data for all watches sequentially (with delays)
- [ ] Log progress and any failures
- [ ] Can be run manually or via cron
- [ ] Script completes without errors

## Technical Approach

### Python Scraper (`scripts/chrono24-scraper.py`)
```python
#!/usr/bin/env python3
import chrono24
import json
import sys
import argparse

def scrape_prices(brand, model, reference=None, limit=20):
    query_str = f"{brand} {model}"
    if reference:
        query_str += f" {reference}"

    try:
        results = []
        for listing in chrono24.query(query_str).search(limit=limit):
            results.append({
                'price': listing.get('price'),
                'condition': listing.get('condition'),
                'location': listing.get('location'),
                'url': listing.get('url')
            })

        prices = [r['price'] for r in results if r['price']]
        return {
            'listings': results,
            'count': len(results),
            'minPrice': min(prices) if prices else None,
            'maxPrice': max(prices) if prices else None,
            'avgPrice': sum(prices) / len(prices) if prices else None
        }
    except Exception as e:
        return {'error': str(e), 'listings': []}

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--brand', required=True)
    parser.add_argument('--model', required=True)
    parser.add_argument('--reference', default='')
    parser.add_argument('--limit', type=int, default=20)
    args = parser.parse_args()

    result = scrape_prices(args.brand, args.model, args.reference, args.limit)
    print(json.dumps(result))
```

### Node.js Endpoint
```javascript
app.get('/api/market-data/:watchId', async (req, res) => {
  const { watchId } = req.params;

  // Check cache first
  if (cache.has(watchId)) {
    return res.json(cache.get(watchId));
  }

  // Get search params from mapping
  const mapping = watchMap[watchId];
  if (!mapping) {
    return res.json(getMockData(watchId));
  }

  // Call Python scraper
  const result = await callPythonScraper(mapping);

  // Cache and return
  cache.set(watchId, result, 3600000); // 1 hour
  res.json(result);
});
```

## Non-Goals

- WatchCharts integration (paid)
- Real-time updates
- Database storage
- Paid APIs

## Success Metrics

- Real prices for 80%+ of watches
- Scraping success > 70%
- Cache hit rate > 90%
- Zero cost

---

## Checklist

- [x] User stories are small and specific
- [x] Free solution using GitHub library
- [x] Technical approach outlined
- [x] Saved to tasks/prd-real-api-integration.md