`auction-viewer/ANALYTICS_DOCUMENTATION.md`

674 lines
# LUXVAULT Analytics Engine - Documentation

## Overview

The LUXVAULT Analytics Engine provides advanced statistical analysis, machine learning insights, and predictive analytics for auction data. It combines Python-based data science algorithms with Node.js API endpoints to deliver actionable intelligence.

---

## Architecture

### Components

1. **Python Analytics Engine** (`analytics-engine.py`)
   - Statistical analysis using Python's scientific stack
   - Deal scoring algorithm
   - Price prediction models
   - Anomaly detection
   - Trend analysis

2. **API Endpoints** (`server-secured.js`)
   - RESTful API for analytics data
   - Input validation and security
   - Response caching
   - Rate limiting

3. **SQLite Database**
   - Historical auction data
   - Efficient indexing for analytics queries

---

## Deal Score Algorithm

### Methodology

The Deal Score is a composite metric (0-100) that ranks auction opportunities based on multiple factors:

#### Components (with weights):

1. **Savings Percentage (40%)**: How much below estimate the current price is
   - Formula: `((estimate_low - current_price) / estimate_low) * 100`
   - Linear scaling: 0-100% savings = 0-40 points

2. **Price Point Relative to Brand Average (25%)**:
   - Compares current price to historical brand average
   - Lower relative prices score higher
   - Scoring:
     - ≤50% of average: 25 points
     - 50-100% of average: Linear scale 25→0 points
     - >100% of average: 0 points

3. **Time Urgency (20%)**:
   - Rewards auctions ending soon
   - Scoring:
     - ≤1 day: 20 points
     - ≤3 days: 15 points
     - ≤7 days: 10 points
     - ≤14 days: 5 points
     - >14 days: 2 points

4. **Historical Price Performance (15%)**:
   - Price percentile within brand (lower = better)
   - Formula: `(100 - percentile) * 0.15`

### Example Calculation

```
Auction: Chanel Boy @ $1,587 (estimate: $2,877)
- Savings: 44.84% → 17.94 points (40% weight)
- Price vs Average: 0.52x → 24 points (25% weight)
- Time: 10 days → 5 points (20% weight)
- Percentile: 30th → 10.5 points (15% weight)

Total Deal Score: 57.44 / 100
```

---

## API Endpoints

### Base URL
```
http://45.61.58.125:7500/api/insights
```

### 1. Deal Scores

**GET** `/api/insights/deal-scores`

Returns top auction deals ranked by Deal Score algorithm.

**Parameters:**
- `limit` (optional): Number of results (1-100, default: 50)

**Response:**
```json
{
  "success": true,
  "count": 50,
  "data": [
    {
      "id": 2062,
      "auction_id": "LA-104A9D16A3F1",
      "title": "Chanel Boy Gray Canvas 26cm",
      "current_price": 1587,
      "estimate_low": 2877,
      "deal_score": 57.67,
      "savings_percent": 44.84,
      "savings_amount": 1290,
      "end_date": "2025-10-19",
      ...
    }
  ],
  "generated_at": "2025-11-17T09:33:00.727Z"
}
```

**Cache:** 5 minutes

---

### 2. Price Trends

**GET** `/api/insights/trends`

Analyzes price trends by brand over time, including trend direction and monthly statistics.

**Response:**
```json
{
  "success": true,
  "data": {
    "Hermes Birkin": {
      "trend": "increasing",
      "change_percent": 12.5,
      "monthly_data": [
        {
          "month": "2025-11",
          "avg_price": 15234.50,
          "min_price": 8900,
          "max_price": 45000,
          "auction_count": 15,
          "avg_savings_pct": 2.3
        }
      ]
    }
  },
  "generated_at": "2025-11-17T09:33:00.727Z"
}
```

**Cache:** 10 minutes

---

### 3. Price Anomalies

**GET** `/api/insights/anomalies`

Detects statistical outliers using z-score analysis (>2 standard deviations).

**Response:**
```json
{
  "success": true,
  "count": 50,
  "data": [
    {
      "id": 2138,
      "auction_id": "LA-7B657F539C8D",
      "title": "Louis Vuitton Neverfull Green Patent Leather 26cm",
      "current_price": 72045,
      "anomaly_type": "unusually_high",
      "z_score": 6.83,
      "brand_avg_price": 3843.01,
      "deviation_pct": 1774.7,
      ...
    }
  ]
}
```

**Use Cases:**
- Identify potentially rare/special items (high outliers)
- Find data entry errors
- Spot exceptional deals (low outliers)

**Cache:** 10 minutes

---

### 4. Price Prediction

**GET** `/api/insights/predict/:auction_id`

Predicts final auction price using historical brand data and median ratios.

**Parameters:**
- `auction_id` (required): Auction ID to predict

**Response:**
```json
{
  "success": true,
  "data": {
    "auction_id": "LA-104A9D16A3F1",
    "current_price": 1587,
    "estimate_low": 2877,
    "estimate_high": 4765,
    "predicted_final_price": 2667.28,
    "prediction_range_low": 2184.46,
    "prediction_range_high": 3150.09,
    "confidence": "high",
    "sample_size": 115
  }
}
```

**Confidence Levels:**
- `high`: ≥20 historical samples
- `medium`: 5-19 samples
- `low`: <5 samples (falls back to estimate)

**Cache:** None (real-time)

---

### 5. Recommendations

**GET** `/api/insights/recommendations`

Provides personalized auction recommendations based on Deal Score and optional filters.

**Parameters:**
- `budget` (optional): Maximum budget filter (float)

**Response:**
```json
{
  "success": true,
  "count": 20,
  "data": [
    {
      "id": 2062,
      "deal_score": 57.67,
      "savings_percent": 44.84,
      "recommendation_reasons": [
        "45% savings",
        "Ends in 3 days"
      ],
      ...
    }
  ],
  "filters": {
    "budget": "5000"
  }
}
```

**Cache:** 5 minutes

---

### 6. Market Insights

**GET** `/api/insights/market`

Comprehensive market overview with summary statistics and top value brands.

**Response:**
```json
{
  "success": true,
  "data": {
    "market_summary": {
      "total_auctions": 2250,
      "average_price": 5471.54,
      "average_savings_percent": 3.4,
      "unique_brands": 22,
      "unique_auction_houses": 6
    },
    "price_distribution": [
      {"range": "$1K-$5K", "count": 1314},
      {"range": "$5K-$10K", "count": 178}
    ],
    "top_value_brands": [
      {
        "brand": "Bottega Veneta Cassette",
        "auction_count": 73,
        "avg_price": 9763.75,
        "avg_savings_percent": 7.22
      }
    ]
  }
}
```

**Cache:** 10 minutes

---

### 7. Brand-Specific Analytics

**GET** `/api/insights/brand/:brand`

Detailed analytics for a specific brand including statistics, recent auctions, and price history.

**Parameters:**
- `brand` (required): Brand name (URL-encoded)

**Example:**
```
/api/insights/brand/Hermes%20Birkin
```

**Response:**
```json
{
  "success": true,
  "brand": "Hermes Birkin",
  "statistics": {
    "total_auctions": 82,
    "avg_price": 14510.86,
    "min_price": 2015.8,
    "max_price": 62422,
    "avg_savings_pct": 1.93
  },
  "recent_auctions": [...],
  "price_history": [
    {"month": "2025-11", "avg_price": 15234, "count": 15}
  ]
}
```

**Cache:** 5 minutes

---

### 8. Time-Series Data

**GET** `/api/insights/time-series`

Time-series data optimized for chart visualization (Chart.js compatible).

**Parameters:**
- `brand` (optional): Filter by brand
- `interval` (optional): `day`, `week`, or `month` (default: `month`)

**Response:**
```json
{
  "success": true,
  "interval": "month",
  "brand": "Chanel Boy",
  "data": [
    {
      "period": "2024-01",
      "auction_count": 25,
      "avg_price": 3245.50,
      "min_price": 1200,
      "max_price": 8900,
      "avg_savings_pct": 5.2
    }
  ]
}
```

**Cache:** 10 minutes

---

### 9. Brand Comparison

**GET** `/api/insights/compare-brands`

Comparative analysis across all brands with sufficient data (≥5 auctions).

**Response:**
```json
{
  "success": true,
  "count": 22,
  "data": [
    {
      "brand": "Bottega Veneta Cassette",
      "auction_count": 73,
      "avg_price": 9763.75,
      "min_price": 651.9,
      "max_price": 59168,
      "avg_savings_pct": 7.22,
      "deals_count": 45
    }
  ]
}
```

**Use Cases:**
- Identify best value brands
- Compare market positions
- Analyze price ranges

**Cache:** 10 minutes

---

### 10. Auction Velocity

**GET** `/api/insights/velocity`

Tracks auction velocity by brand (auctions ending soon).

**Response:**
```json
{
  "success": true,
  "data": [
    {
      "brand": "Louis Vuitton Neverfull",
      "total": 181,
      "ending_today": 5,
      "ending_3days": 23,
      "ending_week": 47,
      "avg_price": 3843.01
    }
  ]
}
```

**Use Cases:**
- Identify urgent opportunities
- Track market activity
- Plan bidding strategies

**Cache:** 5 minutes

---

## Data Science Techniques

### Statistical Methods

1. **Descriptive Statistics**
   - Mean, median, standard deviation
   - Percentile analysis
   - Distribution analysis

2. **Outlier Detection**
   - Z-score method (threshold: |z| > 2)
   - Used for anomaly detection

3. **Time-Series Analysis**
   - Trend detection (increasing/decreasing/stable)
   - Moving averages
   - Seasonal patterns

4. **Predictive Modeling**
   - Historical ratio analysis
   - Median-based predictions (robust to outliers)
   - Confidence interval calculation

---

## Performance Optimization

### Caching Strategy

| Endpoint | Cache Duration | Reason |
|----------|---------------|--------|
| Deal Scores | 5 minutes | Frequently updated |
| Trends | 10 minutes | Computationally expensive |
| Anomalies | 10 minutes | Stable results |
| Market Insights | 10 minutes | Aggregate data |
| Time-Series | 10 minutes | Large datasets |
| Predictions | No cache | Real-time analysis |

### Database Optimization

- **Indexes**: Created on `search_term`, `current_price`, `end_date`, `created_at`
- **Read-only connections**: All analytics use readonly mode
- **Connection pooling**: Connections closed immediately after queries
- **Query limits**: Maximum 1000 rows per query

---

## Error Handling

### Error Response Format

```json
{
  "success": false,
  "error": "Error message",
  "data": null
}
```

### Common Errors

1. **Invalid Parameters**: HTTP 400
   - Missing required parameters
   - Out-of-range values
   - Invalid data types

2. **Not Found**: HTTP 404
   - Auction ID doesn't exist
   - Brand not found

3. **Server Error**: HTTP 500
   - Database connection issues
   - Python script execution errors
   - Calculation errors

---

## Security Features

1. **Input Validation**
   - express-validator for all parameters
   - Type checking and sanitization
   - SQL injection prevention

2. **Rate Limiting**
   - General API: 100 requests/15 min
   - Analytics endpoints: Shared with general limit
   - Per-IP tracking

3. **Path Safety**
   - Python script path validation
   - No user-controllable paths
   - Timeout protection (30s)

4. **Error Sanitization**
   - Production mode hides stack traces
   - Generic error messages
   - Detailed logging for debugging

---

## Usage Examples

### JavaScript/Fetch

```javascript
// Get top 10 deals
const response = await fetch('http://45.61.58.125:7500/api/insights/deal-scores?limit=10');
const data = await response.json();

data.data.forEach(deal => {
  console.log(`${deal.title}: Score ${deal.deal_score}, Save ${deal.savings_percent}%`);
});
```

### Python

```python
import requests

# Get market insights
response = requests.get('http://45.61.58.125:7500/api/insights/market')
data = response.json()

if data['success']:
    summary = data['data']['market_summary']
    print(f"Total Auctions: {summary['total_auctions']}")
    print(f"Average Price: ${summary['average_price']:.2f}")
```

### cURL

```bash
# Get recommendations under $5000 budget
curl -s "http://45.61.58.125:7500/api/insights/recommendations?budget=5000" | jq '.data[0]'

# Get price prediction
curl -s "http://45.61.58.125:7500/api/insights/predict/LA-104A9D16A3F1" | jq '.data'

# Get brand analytics
curl -s "http://45.61.58.125:7500/api/insights/brand/Hermes%20Birkin" | jq '.statistics'
```

---

## Roadmap

### Future Enhancements

1. **Machine Learning Models**
   - Random Forest for price prediction
   - Gradient Boosting for deal scoring
   - Clustering for similar items

2. **Advanced Features**
   - User preference learning
   - Bid timing optimization
   - Competitor analysis

3. **Real-Time Analytics**
   - WebSocket support
   - Live deal alerts
   - Price movement tracking

4. **Visualization**
   - Built-in Chart.js integration
   - Interactive dashboards
   - Export to PDF/Excel

---

## Support

### Logs

Analytics errors are logged to:
```
/root/Projects/handbag-auth-nextjs/auction-viewer/logs/error.log
/root/Projects/handbag-auth-nextjs/auction-viewer/logs/combined.log
```

### Monitoring

Check endpoint health:
```bash
curl http://45.61.58.125:7500/health
curl http://45.61.58.125:7500/api/status
```

### Python Script Direct Usage

```bash
# Run analytics directly
cd /root/Projects/handbag-auth-nextjs/auction-viewer
python3 analytics-engine.py ../data/auction-history/auctions.db deal-scores 10
python3 analytics-engine.py ../data/auction-history/auctions.db market-insights
python3 analytics-engine.py ../data/auction-history/auctions.db trends
```

---

## Version History

**v1.0.0** (2025-11-17)
- Initial release
- 10 analytics endpoints
- Deal Score algorithm
- Price prediction
- Anomaly detection
- Market insights
- Time-series analysis
- Brand comparison
- Velocity tracking

---

## Credits

**Data Science Agent**
- Algorithm design and implementation
- Statistical analysis
- Documentation

**Technologies Used**
- Python 3.10 (analytics engine)
- Node.js + Express (API server)
- SQLite (data storage)
- express-validator (input validation)
- Winston (logging)
- better-sqlite3 (database access)

---

## License

Proprietary - LUXVAULT Project