← back to Wine Finder Next

OPTIMIZATION_SETUP_GUIDE.md

356 lines

# Backend Optimization Setup Guide

## Quick Start

### 1. Review the Optimized Files

The following files have been created with production-ready optimizations:

```bash
/root/Projects/wine-finder-next/
├── lib/
│   ├── membershipDatabase.optimized.ts    # Optimized database layer
│   ├── apiOptimization.ts                  # API response utilities
│   └── connectionPool.ts                   # Connection pool manager
├── app/api/
│   ├── membership/bottles/route.optimized.ts  # Example optimized route
│   └── monitoring/performance/route.ts        # Performance dashboard
└── BACKEND_OPTIMIZATION_REPORT.md         # Full documentation
```

### 2. Enable Optimizations (Step-by-Step)

#### Option A: Gradual Migration (Recommended)

Test the optimizations alongside existing code:

```bash
# 1. Keep existing files, import optimized database
# In any API route, change:
# FROM:
import { bottleDb } from '@/lib/membershipDatabase';

# TO:
import { bottleDb } from '@/lib/membershipDatabase.optimized';
```

#### Option B: Full Replacement

Replace all files at once (after testing):

```bash
cd /root/Projects/wine-finder-next

# Backup originals
cp lib/membershipDatabase.ts lib/membershipDatabase.backup.ts
cp app/api/membership/bottles/route.ts app/api/membership/bottles/route.backup.ts

# Replace with optimized versions
mv lib/membershipDatabase.optimized.ts lib/membershipDatabase.ts
mv app/api/membership/bottles/route.optimized.ts app/api/membership/bottles/route.ts

# Restart the application
pm2 restart wine-finder-next
```

### 3. Test the Optimizations

#### Performance Monitoring

Access the monitoring dashboard:
```bash
# Start the app if not running
cd /root/Projects/wine-finder-next
npm run dev  # or pm2 start

# View performance metrics
curl http://localhost:7250/api/monitoring/performance | jq

# Or visit in browser:
# http://45.61.58.125:7250/api/monitoring/performance
```

#### Test API Endpoints

```bash
# Test bottles endpoint with pagination
curl "http://localhost:7250/api/membership/bottles?page=1&pageSize=10" | jq

# Test with field filtering (smaller payload)
curl "http://localhost:7250/api/membership/bottles?fields=id,name,vintage" | jq

# Test ETag caching
curl -I "http://localhost:7250/api/membership/bottles"
# Copy the ETag header, then:
curl -H "If-None-Match: <etag-value>" "http://localhost:7250/api/membership/bottles"
# Should return 304 Not Modified

# Test rate limiting
for i in {1..150}; do
  curl -s "http://localhost:7250/api/membership/bottles" > /dev/null
  echo "Request $i"
done
# Should see 429 after ~100 requests
```

### 4. Benchmark Performance

#### Before Optimization
```bash
# Install Apache Bench if not installed
apt-get install -y apache2-utils

# Test current performance
ab -n 1000 -c 10 http://localhost:7250/api/membership/bottles
```

#### After Optimization
```bash
# Should see:
# - Lower avg response time (60-80% reduction)
# - Higher requests/sec (2-3x improvement)
# - Better consistency (lower stddev)
```

### 5. Migration to Production Database

When ready to replace in-memory storage:

#### PostgreSQL Setup

```bash
# 1. Install PostgreSQL
apt-get install -y postgresql postgresql-contrib

# 2. Create database
sudo -u postgres psql -c "CREATE DATABASE wine_membership;"
sudo -u postgres psql -c "CREATE USER wine_app WITH PASSWORD 'your_secure_password';"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE wine_membership TO wine_app;"

# 3. Run migrations
cd /root/Projects/wine-finder-next
npm install pg  # PostgreSQL driver

# Create migration file (see schema in BACKEND_OPTIMIZATION_REPORT.md)
psql -U wine_app -d wine_membership -f migrations/001_initial_schema.sql

# 4. Update environment variables
echo "DATABASE_URL=postgresql://wine_app:your_secure_password@localhost:5432/wine_membership" >> .env
echo "DATABASE_POOL_MIN=2" >> .env
echo "DATABASE_POOL_MAX=10" >> .env

# 5. Update code to use connection pool
# Replace Map operations with actual SQL queries in membershipDatabase.ts
```

#### Redis Cache Setup

```bash
# 1. Install Redis
apt-get install -y redis-server

# 2. Configure Redis
systemctl enable redis-server
systemctl start redis-server

# 3. Update environment
echo "REDIS_URL=redis://localhost:6379" >> .env

# 4. Install Redis client
npm install redis

# 5. Update cache layer to use Redis instead of in-memory Map
```

### 6. Monitoring Setup

#### Application Performance

The monitoring endpoint provides real-time metrics:

```javascript
// Example: Fetch and display metrics every 10 seconds
setInterval(async () => {
  const response = await fetch('http://localhost:7250/api/monitoring/performance');
  const metrics = await response.json();

  console.log('Database Performance:');
  console.log(metrics.database.operations);

  console.log('Cache Hit Rate:', metrics.cache.hitRate);
  console.log('Memory Usage:', metrics.memory.heapUsed, 'MB');
}, 10000);
```

#### Alerts Setup

Add alerting for critical metrics:

```typescript
// In /lib/monitoring/alerts.ts
export function checkAlerts(metrics) {
  const alerts = [];

  // Slow queries
  if (metrics.database.operations['vote.cast'] > 100) {
    alerts.push({
      severity: 'warning',
      message: 'Vote casting is slow (>100ms)',
      action: 'Check database indices'
    });
  }

  // Low cache hit rate
  if (parseFloat(metrics.cache.hitRate) < 50) {
    alerts.push({
      severity: 'warning',
      message: 'Cache hit rate below 50%',
      action: 'Increase cache TTL or size'
    });
  }

  // High memory usage
  if (metrics.memory.heapUsed > 500) {
    alerts.push({
      severity: 'critical',
      message: 'Memory usage above 500MB',
      action: 'Investigate memory leaks'
    });
  }

  return alerts;
}
```

### 7. Load Testing

#### Simple Load Test

```bash
# Install k6 for load testing
wget https://github.com/grafana/k6/releases/download/v0.45.0/k6-v0.45.0-linux-amd64.tar.gz
tar -xzf k6-v0.45.0-linux-amd64.tar.gz
mv k6-v0.45.0-linux-amd64/k6 /usr/local/bin/

# Create load test script
cat > load_test.js << 'EOF'
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '30s', target: 50 },   // Ramp up to 50 users
    { duration: '1m', target: 100 },   // Ramp up to 100 users
    { duration: '2m', target: 100 },   // Stay at 100 users
    { duration: '30s', target: 0 },    // Ramp down
  ],
};

export default function () {
  const response = http.get('http://localhost:7250/api/membership/bottles');

  check(response, {
    'status is 200': (r) => r.status === 200,
    'response time < 200ms': (r) => r.timings.duration < 200,
  });

  sleep(1);
}
EOF

# Run load test
k6 run load_test.js
```

Expected results:
- 95% of requests < 200ms
- 0% error rate
- Throughput: 500+ req/s

### 8. Production Deployment Checklist

Before deploying to production:

- [ ] All tests passing
- [ ] Load testing completed
- [ ] Database migrated from in-memory to PostgreSQL
- [ ] Redis cache configured
- [ ] Environment variables set
- [ ] SSL certificates installed
- [ ] Firewall configured
- [ ] Backups scheduled
- [ ] Monitoring dashboards created
- [ ] Alerting configured
- [ ] Error tracking enabled (Sentry)
- [ ] Log aggregation set up
- [ ] Documentation updated
- [ ] Team trained on new system
- [ ] Rollback plan prepared

### 9. Troubleshooting

#### High Response Times

```bash
# Check performance metrics
curl http://localhost:7250/api/monitoring/performance | jq '.database.slowQueries'

# Common causes:
# - Missing database indices (add them)
# - Cache not working (check Redis connection)
# - Too many concurrent connections (increase pool size)
```

#### Memory Leaks

```bash
# Monitor memory over time
watch -n 5 'curl -s http://localhost:7250/api/monitoring/performance | jq ".memory"'

# If heapUsed keeps growing:
# - Check for unclosed database connections
# - Verify cache eviction is working
# - Look for event listener leaks
```

#### Rate Limit Issues

```bash
# Check rate limit headers
curl -I http://localhost:7250/api/membership/bottles

# Adjust limits in lib/rateLimit.ts if needed
```

### 10. Performance Targets

**Response Times:**
- p50 (median): < 50ms
- p95: < 100ms
- p99: < 200ms

**Throughput:**
- Minimum: 500 req/s
- Target: 1000 req/s
- Peak: 2000 req/s

**Availability:**
- Target: 99.9% (8.76 hours/year downtime)
- Error rate: < 0.1%

**Resource Usage:**
- Memory: < 500MB per instance
- CPU: < 70% average
- Database connections: < 80% of pool

---

## Support

For questions or issues:
1. Check the full documentation: `BACKEND_OPTIMIZATION_REPORT.md`
2. Review code comments in optimized files
3. Test in development before production deployment

**Last Updated:** 2025-11-17