← back to Watches

analytics/ANALYTICS_COMPLETE.md

743 lines

# Complete Analytics System - Omega Watches

Comprehensive analytics implementation with GA4 integration, event tracking, user funnels, cohort analysis, A/B testing, custom dashboards, and automated reports.

## System Overview

This analytics system provides TWO distinct but complementary analytics approaches:

### 1. Business Intelligence Analytics (Python - Existing)
- **Purpose**: Watch price predictions, market analysis, investment insights
- **Location**: `/root/Projects/watches/analytics/` (Python files)
- **Tools**: scikit-learn, pandas, numpy
- **Use Case**: Investment research, price forecasting, risk analysis

### 2. User Behavior Analytics (JavaScript/Node - NEW)
- **Purpose**: User tracking, engagement, conversion optimization
- **Location**: `/root/Projects/watches/analytics/` (JavaScript files)
- **Tools**: GA4, custom analytics server
- **Use Case**: Product optimization, user experience, A/B testing

---

## Part 1: Business Intelligence (Existing System)

See `README.md` for full documentation.

**Quick Commands:**
```bash
# Run all analytics
python3 /root/Projects/watches/analytics/run_all_analytics.py

# Access via API
curl http://45.61.58.125:7600/api/analytics/investment-opportunities
```

---

## Part 2: User Behavior Analytics (NEW SYSTEM)

### Components

#### Frontend Tracking
1. **GA4 Integration** (`ga4-integration.js`)
   - Google Analytics 4 tracking
   - Custom events, user properties
   - Scroll depth, engagement time
   - Auto-tracking

2. **Funnel Tracking** (`funnel-tracker.js`)
   - User journey tracking
   - Conversion funnels
   - Step tracking, abandonment

3. **Cohort Analysis** (`cohort-analysis.js`)
   - User cohorts by first visit
   - Retention metrics
   - Engagement scoring
   - Lifecycle stages

4. **A/B Testing** (`ab-testing.js`)
   - Variant assignment
   - Goal tracking
   - Metric collection
   - Statistical analysis

#### Backend Services
5. **Analytics Server** (`analytics-server-complete.js`)
   - Event collection API
   - Data storage
   - Real-time processing
   - **Port: 7650**

6. **Report Generator** (`report-generator.js`)
   - Daily/weekly/monthly reports
   - HTML/JSON/PDF export
   - Automated insights

7. **Scheduler** (`scheduler.js`)
   - Cron-based automation
   - Report scheduling
   - Data backups

#### Dashboards
8. **Analytics Dashboard** (`dashboard.html`)
   - Real-time metrics
   - Visual charts
   - Event tracking
   - Data export

---

## Installation & Setup

### 1. Start Analytics Server

```bash
cd /root/Projects/watches/analytics

# Start with PM2
pm2 start analytics-server-complete.js --name "analytics-watches"

# Or start with Node
node analytics-server-complete.js
```

### 2. Verify Server Running

```bash
# Check PM2
pm2 list | grep analytics

# Test health endpoint
curl http://45.61.58.125:7650/api/analytics/health

# Open firewall if needed
sudo ufw allow 7650/tcp
```

### 3. Integrate with Main App

Add to `/root/Projects/watches/public/index.html` (before closing `</body>`):

```html
<!-- Analytics Scripts -->
<script src="/analytics/ga4-integration.js"></script>
<script src="/analytics/funnel-tracker.js"></script>
<script src="/analytics/cohort-analysis.js"></script>
<script src="/analytics/ab-testing.js"></script>
<script src="/analytics/analytics-integration.js"></script>
```

---

## Accessing Dashboards

### User Behavior Analytics Dashboard
**URL**: http://45.61.58.125:7650/dashboard.html

Features:
- Real-time event metrics
- User engagement stats
- Funnel conversion rates
- Cohort retention
- A/B test results
- Auto-refresh every 30 seconds

### Main Watches Application
**URL**: http://45.61.58.125:7600

Now includes integrated analytics tracking.

---

## Tracking User Events

### Automatic Tracking
These events are tracked automatically:
- Page views
- Scroll depth (25%, 50%, 75%, 100%)
- Time on page
- Outbound clicks
- JavaScript errors
- Page visibility changes

### Manual Event Tracking

Add to your app's JavaScript:

```javascript
// Track watch view
trackWatchView({
  id: 'speedmaster-001',
  model: 'Speedmaster Professional',
  series: 'Speedmaster',
  priceHistory: [...]
});

// Track watch comparison
trackWatchComparison([watch1, watch2, watch3]);

// Track search
trackSearch('speedmaster', 15); // term, resultsCount

// Track filter
trackFilter('series', 'Speedmaster');

// Track chart interactions
trackChartInteraction('zoom', { watchId: 'speedmaster-001' });

// Track data export
trackDataExport('csv');

// Track price predictions
trackPricePredictionView('speedmaster-001');

// Track trending watches
trackTrendingView();
```

---

## API Endpoints

### Base URL
`http://45.61.58.125:7650/api/analytics`

### Core Analytics
- `POST /events` - Track custom event
- `POST /funnels` - Track funnel step
- `POST /cohorts` - Update cohort data
- `POST /ab-tests/assignments` - Track A/B assignment
- `POST /ab-tests/goals` - Track A/B goal
- `POST /ab-tests/metrics` - Track A/B metric

### Data Retrieval
- `GET /dashboard?hours=24` - Dashboard data
- `GET /funnels/:funnelId/conversions?days=30` - Funnel conversion rates
- `GET /cohorts/summary` - Cohort summary
- `GET /ab-tests/:experimentId/results` - A/B test results
- `GET /export` - Export all analytics data

### System
- `GET /health` - Health check
- `GET /scheduler/status` - Scheduler status
- `POST /reports/generate` - Generate report manually

---

## User Funnels

### 1. Watch Discovery Funnel
```
Landing → Browse → View Details → Compare → Export
```

### 2. Price Research Funnel
```
Search → View History → View Predictions → Export
```

### 3. Collection Analysis Funnel
```
Filter Series → View Stats → Compare → Deep Analysis
```

### 4. Investment Research Funnel
```
View Trending → Check Appreciation → View Predictions → Export
```

---

## A/B Tests Currently Running

### 1. Price Chart Layout
- **Control**: Original layout
- **Variant A**: Side-by-side layout
- **Metrics**: chart_interaction_rate, time_on_page, comparison_rate

### 2. Watch Card Design
- **Control**: Original design
- **Variant A**: Enhanced details
- **Metrics**: watch_view_rate, engagement_time

### 3. CTA Button Color
- **Control**: Blue button
- **Variant A**: Green button
- **Variant B**: Orange button
- **Metrics**: cta_click_rate, conversion_rate

### 4. Search Algorithm
- **Control**: Basic search
- **Variant A**: Fuzzy search
- **Metrics**: search_success_rate, results_clicked

---

## Cohort Metrics

### Tracked Data
- First visit date (cohort month/week)
- Total visits
- Total actions
- Days/weeks/months since first visit
- Engagement score (0-100)
- Lifecycle stage
- Cohort segment

### Lifecycle Stages
- **Acquisition**: First visit
- **Activation**: 2-5 visits in first week
- **Active**: 5+ visits, last visit <30 days
- **At Risk**: Last visit 30-60 days ago
- **Churned**: Last visit >60 days ago
- **Retention**: Regular return visitor

### Cohort Segments
- New User
- Active New (high engagement, <7 days)
- Passive New (low engagement, <7 days)
- Power User (very high engagement)
- Regular User (moderate engagement)
- Occasional User (low engagement)
- Loyal User (long-term, high engagement)
- Returning User (long-term, moderate)
- At Risk (declining engagement)
- Dormant (inactive)

---

## Automated Reporting

### Schedule

| Report Type | Frequency | Time | Format |
|------------|-----------|------|--------|
| Daily | Every day | 2:00 AM | JSON, HTML |
| Weekly | Mondays | 3:00 AM | JSON, HTML |
| Monthly | 1st of month | 4:00 AM | JSON, HTML, PDF |

### Report Contents

**Daily Report:**
- Summary metrics (events, users, sessions)
- Top events
- User acquisition (new vs returning)
- Funnel performance
- Cohort insights
- A/B test updates
- Recommendations

**Weekly Report:**
- All daily metrics
- Trend analysis (week-over-week)
- Top performing content
- User engagement deep-dive
- Conversion funnel analysis
- Cohort retention curves
- Experiment results
- Action items

**Monthly Report:**
- Executive summary
- KPI dashboard
- Growth analysis (MoM, YoY)
- User behavior patterns
- Funnel optimization insights
- Cohort lifetime analysis
- Experiment impact assessment
- Strategic recommendations

### Report Location
```
/root/Projects/watches/analytics/reports/
├── daily-report-2025-11-17.json
├── daily-report-2025-11-17.html
├── weekly-report-2025-11-17.json
├── weekly-report-2025-11-17.html
├── monthly-report-2025-11-17.json
├── monthly-report-2025-11-17.html
└── monthly-report-2025-11-17.pdf
```

### Manual Report Generation

```bash
# Via API
curl -X POST http://45.61.58.125:7650/api/analytics/reports/generate \
  -H "Content-Type: application/json" \
  -d '{"type":"daily"}'

# Via browser
# Navigate to dashboard and click "Generate Report"
```

---

## Data Export

### Export User Analytics Data

```javascript
// Export from browser console
window.exportAnalyticsData();

// Or via API
curl http://45.61.58.125:7650/api/analytics/export > analytics-export.json
```

### Export Format
```json
{
  "exportedAt": "2025-11-17T12:00:00.000Z",
  "events": [...],
  "funnels": [...],
  "cohorts": [...],
  "abTests": {
    "assignments": [...],
    "goals": [...],
    "metrics": [...]
  }
}
```

---

## Monitoring & Health Checks

### Check System Health

```bash
# Analytics server health
curl http://45.61.58.125:7650/api/analytics/health

# Response
{
  "status": "healthy",
  "uptime": 86400,
  "storage": {
    "events": 5000,
    "funnels": 1200,
    "cohorts": 150
  },
  "scheduler": {
    "running": true,
    "jobs": ["daily_report", "weekly_report", "monthly_report"]
  }
}
```

### PM2 Monitoring

```bash
# Check status
pm2 list

# View logs
pm2 logs analytics-watches

# Monitor resources
pm2 monit
```

---

## Testing Analytics

### Test Event Tracking

Open browser console on http://45.61.58.125:7600:

```javascript
// Test GA4 event
window.analyticsIntegration.analytics.trackEvent('test_event', {
  test_param: 'test_value'
});

// Test funnel
const tracker = window.analyticsIntegration.funnelTracker;
tracker.trackStep('watch_discovery', 'landing');
tracker.trackStep('watch_discovery', 'browse');
tracker.trackStep('watch_discovery', 'view_details');

// Check funnel state
console.log(tracker.getActiveFunnels());

// Test A/B assignment
const ab = window.analyticsIntegration.abTesting;
console.log(ab.getActiveExperiments());

// Force variant (for testing)
ab.forceVariant('price_chart_layout', 'variant_a');

// Test cohort tracking
const cohort = window.analyticsIntegration.cohortAnalyzer;
console.log(cohort.getCohortMetrics());
```

### Verify in Dashboard

1. Visit http://45.61.58.125:7650/dashboard.html
2. Check "Recent Events" table
3. Switch to relevant tabs (Funnels, Cohorts, A/B Tests)
4. Verify data appears

---

## Troubleshooting

### Analytics Not Tracking

1. **Check if server is running**
   ```bash
   pm2 list | grep analytics
   ```

2. **Check firewall**
   ```bash
   sudo ufw status
   sudo ufw allow 7650/tcp
   ```

3. **Check browser console for errors**
   - Open DevTools (F12)
   - Look for `[Analytics]` log messages

4. **Verify scripts loaded**
   ```javascript
   console.log(window.analyticsIntegration);
   // Should show object with analytics, funnelTracker, etc.
   ```

### Dashboard Not Loading

1. **Test endpoint directly**
   ```bash
   curl http://localhost:7650/api/analytics/health
   ```

2. **Check CORS settings**
   - Analytics server has CORS enabled for all origins

3. **Check browser network tab**
   - Look for failed requests to port 7650

### Reports Not Generating

1. **Check scheduler status**
   ```bash
   curl http://45.61.58.125:7650/api/analytics/scheduler/status
   ```

2. **Check PM2 logs**
   ```bash
   pm2 logs analytics-watches --lines 100
   ```

3. **Manually trigger report**
   ```bash
   curl -X POST http://45.61.58.125:7650/api/analytics/reports/generate \
     -H "Content-Type: application/json" \
     -d '{"type":"daily"}'
   ```

### Missing Events

1. **Check event queue**
   ```javascript
   // Browser console
   console.log(window.analyticsIntegration.analytics.eventQueue);
   ```

2. **Verify backend receiving events**
   ```bash
   # Check recent events
   curl http://45.61.58.125:7650/api/analytics/dashboard | jq '.recentEvents'
   ```

3. **Check data directory**
   ```bash
   ls -lh /root/Projects/watches/analytics/data/
   ```

---

## Performance Considerations

### Event Batching
- Events queued if analytics not initialized
- Sent in batches to reduce requests
- Maximum queue size: 100 events

### Caching
- 5-minute TTL on aggregated queries
- Session storage for user session data
- LocalStorage for persistent user data

### Data Retention
- **In-memory**: Last 10,000 events, 5,000 funnel steps
- **Disk**: Daily exports, 30-day retention
- **Aggregated**: Monthly summaries, indefinite retention

### Optimization Tips
- Use event batching for high-frequency tracking
- Leverage browser-side caching
- Compress large payloads
- Use pagination for large result sets

---

## Privacy & GDPR Compliance

### User Data Handling
- **Anonymous by default**: User IDs are random, no PII
- **LocalStorage**: User preferences, cohort data
- **Opt-out support**: Users can disable tracking
- **Data export**: Users can export their data
- **Data deletion**: Users can request deletion

### Implementation

```javascript
// Disable analytics
window.analyticsIntegration.disable();

// Export user data
window.exportAnalyticsData();

// Clear user data
localStorage.removeItem('ga4_user_id');
localStorage.removeItem('cohort_data');
localStorage.removeItem('ab_assignments');
sessionStorage.clear();
```

### GDPR Compliance Checklist
- [ ] Privacy policy updated
- [ ] Cookie consent banner implemented
- [ ] Opt-out mechanism available
- [ ] Data export functionality
- [ ] Data deletion process
- [ ] Retention policy documented
- [ ] Third-party processors listed (GA4)

---

## File Structure

```
/root/Projects/watches/analytics/
├── # User Behavior Analytics (NEW)
├── ga4-integration.js              # GA4 tracking
├── funnel-tracker.js               # Funnel analysis
├── cohort-analysis.js              # Cohort tracking
├── ab-testing.js                   # A/B testing
├── analytics-integration.js        # Main app integration
├── analytics-server.js             # Basic server (legacy)
├── analytics-server-complete.js    # Complete server with scheduler
├── report-generator.js             # Report generation
├── scheduler.js                    # Automation
├── dashboard.html                  # Analytics dashboard
├── ANALYTICS_COMPLETE.md           # This file
│
├── # Business Intelligence (EXISTING)
├── price_prediction.py             # Price forecasting
├── market_analysis.py              # Market clustering
├── statistical_insights.py         # Risk analysis
├── report_generator.py             # BI reports
├── run_all_analytics.py            # BI runner
├── utils.py                        # Helpers
├── README.md                       # BI documentation
│
├── # Data & Reports
├── data/
│   └── analytics-2025-11-17.json
├── reports/
│   ├── daily-report-2025-11-17.json
│   ├── daily-report-2025-11-17.html
│   └── ...
│
└── # Generated BI Files
    ├── price_predictions.json
    ├── market_analysis.json
    ├── statistical_insights.json
    ├── comprehensive_report.json
    ├── INVESTMENT_REPORT.md
    └── ANALYTICS_SUMMARY.md
```

---

## Quick Reference

### URLs
- **Main App**: http://45.61.58.125:7600
- **Analytics Dashboard**: http://45.61.58.125:7650/dashboard.html
- **Analytics API**: http://45.61.58.125:7650/api/analytics/
- **BI API**: http://45.61.58.125:7600/api/analytics/

### Ports
- **7600**: Main watches application
- **7650**: Analytics server (NEW)

### PM2 Commands
```bash
pm2 start analytics-server-complete.js --name "analytics-watches"
pm2 stop analytics-watches
pm2 restart analytics-watches
pm2 logs analytics-watches
pm2 delete analytics-watches
```

### Key Files to Edit
- Add tracking: `/root/Projects/watches/public/app.js`
- Custom events: `/root/Projects/watches/analytics/analytics-integration.js`
- Funnels: `/root/Projects/watches/analytics/funnel-tracker.js`
- A/B tests: `/root/Projects/watches/analytics/ab-testing.js`

---

## Next Steps

1. **Start Analytics Server**
   ```bash
   cd /root/Projects/watches/analytics
   pm2 start analytics-server-complete.js --name "analytics-watches"
   ```

2. **Integrate Scripts** (add to main app HTML)

3. **Open Dashboard** → http://45.61.58.125:7650/dashboard.html

4. **Test Event Tracking** → Browse watches site, check dashboard

5. **Review First Report** → Wait for scheduled report or generate manually

6. **Configure GA4** → Replace `G-XXXXXXXXXX` with real measurement ID

7. **Customize Tracking** → Add app-specific events

8. **Setup Alerts** → Configure Slack/email notifications

---

## Support

**Issues?**
1. Check PM2 logs: `pm2 logs analytics-watches`
2. Test health: `curl http://localhost:7650/api/analytics/health`
3. Check browser console for errors
4. Review this documentation

**Questions?**
- All code is documented with inline comments
- Check `/root/Projects/watches/analytics/README.md` for BI analytics
- Review dashboard source for examples

---

## License

Part of Omega Watches Price History system.