`AUTOMATION_GUIDE.md`

323 lines
# Omega Watch Price Automation System

Complete automation suite using **Puppeteer** and **Playwright** for price tracking, PDF generation, and screenshot capture.

## 🚀 Features

### 1. **Price Scraping (Playwright)**
Automated web scraping of watch prices from multiple marketplaces:
- **Chrono24** - Multi-source watch marketplace
- **WatchCharts** - Market index and trends
- **Bob's Watches** - Pre-owned luxury watches

**File**: `utils/price-scraper.js`

**Usage**:
```bash
# Scrape prices for a specific watch
node utils/price-scraper.js

# Import in code
import PriceScraper from './utils/price-scraper.js';
const scraper = new PriceScraper();
const results = await scraper.scrapeAllSources(watch);
```

**Features**:
- Stealth browser settings to avoid bot detection
- Rate limiting (2-5 second delays)
- Multi-source aggregation
- Automatic price averaging

---

### 2. **Automated Price Monitoring**
Daily cron job at **6:00 AM** that scrapes all watches and stores historical data.

**File**: `utils/price-monitor-simple.js`

**Cron Schedule**: `0 6 * * *` (6 AM daily)

**Data Storage**:
- **CSV Format**: Stock-like time series data (Date, Open, High, Low, Close, Volume)
- **JSON Summaries**: Detailed price breakdowns with sources
- **Master Index**: Quick access to all watches

**Storage Locations**:
```
/root/Projects/watches/data/price-history-csv/     ← Stock-like CSV files
/root/Projects/watches/data/price-summaries/       ← Detailed JSON data
```

**CSV Format** (compatible with stock charting libraries):
```csv
Date,Open,High,Low,Close,Volume,Model,Reference,Sources
2025-11-17,12500,15000,11000,13250,5,"Speedmaster Moonwatch","ST 105.012","Chrono24;WatchCharts;Bob's"
2025-11-18,13250,14500,12800,14000,6,"Speedmaster Moonwatch","ST 105.012","Chrono24;WatchCharts;Bob's"
```

**Manual Run**:
```bash
node utils/price-monitor-simple.js
```

**View Logs**:
```bash
tail -f /var/log/watch-price-monitor.log
```

---

### 3. **PDF Report Generation (Puppeteer)**
Convert HTML analytics reports to professional PDFs.

**File**: `utils/pdf-generator.js`

**Usage**:
```bash
# From HTML file
node utils/pdf-generator.js report.html output-name

# From URL
node utils/pdf-generator.js --url http://localhost:7600/analytics analytics-report

# Generate analytics PDF
node utils/pdf-generator.js --analytics daily
```

**Features**:
- A4 format with margins
- Full background/color printing
- Network idle waiting (ensures all content loaded)
- Automatic output directory creation

**Output**: `/root/Projects/watches/analytics/reports/pdf/`

---

### 4. **Screenshot Capture (Puppeteer)**
Automated screenshot capture for dashboards, watches, and reports.

**File**: `utils/screenshot-capture.js`

**Usage**:
```bash
# Full page screenshot
node utils/screenshot-capture.js http://localhost:7600/ homepage

# Capture all dashboards
node utils/screenshot-capture.js --dashboard

# Capture specific element
import ScreenshotCapture from './utils/screenshot-capture.js';
const capture = new ScreenshotCapture();
await capture.captureElement(url, '.watch-card', 'watch-detail');
```

**Features**:
- Full-page or element-specific capture
- 1920x1080 viewport (desktop)
- PNG format with transparency
- Automatic wait for selectors

**Output**: `/root/Projects/watches/public/screenshots/`

---

## 📊 Data Format for Charting

The CSV files use **stock market format**, making them compatible with financial charting libraries like:
- **Chart.js** with Financial plugin
- **TradingView** lightweight charts
- **Highcharts Stock**
- **D3.js** financial charts

### Example CSV Structure:
```csv
Date,Open,High,Low,Close,Volume,Model,Reference,Sources
2025-11-17,12500,15000,11000,13250,5,"Speedmaster","ST 105.012","Chrono24;WatchCharts"
```

### Reading CSV Data:
```javascript
import SimplePriceMonitor from './utils/price-monitor-simple.js';

const monitor = new SimplePriceMonitor();
const history = monitor.getPriceHistory('speedmaster-moonwatch-1969');

// Returns array: [{date, open, high, low, close, volume, model, reference}, ...]
```

### Chart.js Integration Example:
```javascript
const history = monitor.getPriceHistory('speedmaster-moonwatch-1969');

const chartData = {
  labels: history.map(d => d.date),
  datasets: [{
    label: 'Watch Price',
    data: history.map(d => d.close),
    borderColor: '#C41E3A',
    backgroundColor: 'rgba(196, 30, 58, 0.1)'
  }]
};
```

---

## 🛠️ Setup & Installation

### Initial Setup:
```bash
# Install dependencies
npm install puppeteer playwright

# Setup automated monitoring
./setup-cron-price-monitor.sh
```

### Verify Installation:
```bash
# Test all features
./test-automation.sh

# Check cron job
crontab -l | grep price-monitor

# View created files
ls -lh /root/Projects/watches/data/price-history-csv/
ls -lh /root/Projects/watches/public/screenshots/
ls -lh /root/Projects/watches/analytics/reports/pdf/
```

---

## 📁 File Structure

```
/root/Projects/watches/
├── utils/
│   ├── price-scraper.js           ← Playwright web scraper
│   ├── price-monitor-simple.js    ← Daily monitoring (CSV)
│   ├── pdf-generator.js           ← PDF report generation
│   └── screenshot-capture.js      ← Screenshot automation
├── data/
│   ├── price-history-csv/         ← Stock-like CSV files
│   │   ├── speedmaster-moonwatch-1969.csv
│   │   ├── seamaster-300-1957.csv
│   │   └── index.json             ← Master index
│   └── price-summaries/           ← Detailed JSON data
│       └── speedmaster-moonwatch-1969-2025-11-17.json
├── public/screenshots/            ← Screenshots
├── analytics/reports/pdf/         ← Generated PDFs
└── setup-cron-price-monitor.sh    ← Setup script
```

---

## ⏰ Cron Job Details

**Schedule**: Every day at 6:00 AM
**Command**:
```bash
cd /root/Projects/watches && node utils/price-monitor-simple.js
```

**Log File**: `/var/log/watch-price-monitor.log`

**Edit Cron**:
```bash
crontab -e
```

**Disable Monitoring**:
```bash
crontab -l | grep -v "price-monitor" | crontab -
```

---

## 🔍 Troubleshooting

### Scraper Returns No Results:
- **Cause**: Website changed HTML structure or blocking bots
- **Fix**: Update selectors in `price-scraper.js`
- **Test**: Run manually with extended timeout

### PDF Generation Fails:
- **Cause**: Page not loading or network timeout
- **Fix**: Increase timeout in `pdf-generator.js`
- **Check**: Verify URL is accessible

### Screenshots Are Blank:
- **Cause**: Page rendering not complete
- **Fix**: Add `waitForSelector` option
- **Example**: `captureFullPage(url, name, {waitForSelector: '.grid'})`

### Cron Job Not Running:
- **Check Logs**: `tail -f /var/log/watch-price-monitor.log`
- **Verify Cron**: `crontab -l`
- **Test Manually**: `node utils/price-monitor-simple.js`

---

## 📈 Future Enhancements

1. **PostgreSQL Integration** (optional)
   - Enable database storage in `price-monitor.js`
   - Better querying and aggregation
   - 30-day moving averages

2. **Anti-Bot Bypass**
   - Residential proxies
   - Browser fingerprint randomization
   - CAPTCHA solving services

3. **Real-time Monitoring**
   - WebSocket connections
   - Live price updates
   - Price alerts

4. **Advanced Analytics**
   - Price prediction models
   - Seasonal trends
   - Market correlation analysis

---

## 🎯 Quick Commands

```bash
# Run price monitoring now
node utils/price-monitor-simple.js

# Generate PDF report
node utils/pdf-generator.js --analytics daily

# Capture dashboard screenshots
node utils/screenshot-capture.js --dashboard

# Test all features
./test-automation.sh

# View CSV data
cat data/price-history-csv/speedmaster-moonwatch-1969.csv

# View logs
tail -f /var/log/watch-price-monitor.log
```

---

## ✅ Summary

- ✅ **Puppeteer** installed and working (PDFs, Screenshots)
- ✅ **Playwright** installed and working (Web scraping)
- ✅ **Daily monitoring** at 6 AM via cron
- ✅ **CSV storage** in stock market format
- ✅ **JSON summaries** with detailed breakdowns
- ✅ **Automated PDFs** from HTML reports
- ✅ **Screenshot capture** for dashboards

**Status**: Production-ready automated price tracking system ✨