← back to Watches

DEPLOYMENT_RUNBOOK.md

674 lines

# Omega Watch Price History - Production Deployment Runbook

## Table of Contents
1. [System Overview](#system-overview)
2. [Deployment Procedures](#deployment-procedures)
3. [Monitoring and Alerting](#monitoring-and-alerting)
4. [Backup and Recovery](#backup-and-recovery)
5. [Troubleshooting Guide](#troubleshooting-guide)
6. [Incident Response](#incident-response)
7. [Maintenance Procedures](#maintenance-procedures)

---

## System Overview

### Architecture
- **Platform**: Node.js 22.x with Express
- **Process Manager**: PM2 (cluster mode)
- **Reverse Proxy**: Nginx
- **Port**: 7600 (internal), 80/443 (external via Nginx)
- **Server**: Kamatera VPS (45.61.58.125)

### Key Components
```
┌─────────────────────────────────────────┐
│          Nginx Reverse Proxy            │
│         (80/443 → localhost:7600)       │
└─────────────┬───────────────────────────┘
              │
┌─────────────▼───────────────────────────┐
│         PM2 Process Manager             │
│  ┌──────────────┐  ┌─────────────────┐ │
│  │ omega-watches│  │ health-monitor  │ │
│  │  (cluster)   │  │    (fork)       │ │
│  └──────────────┘  └─────────────────┘ │
└─────────────────────────────────────────┘
              │
┌─────────────▼───────────────────────────┐
│      Application Components             │
│  - Express API Server                   │
│  - WebSocket Server                     │
│  - In-memory Cache                      │
│  - JSON Data Store                      │
└─────────────────────────────────────────┘
```

### File Structure
```
/root/Projects/watches/
├── server.js                    # Main application
├── ecosystem.config.js          # PM2 configuration
├── data/
│   ├── watches.json            # Main database
│   └── backups/                # Database backups
├── dist/                       # Built frontend
├── logs/                       # Application logs
├── monitoring/
│   ├── logger.js               # Winston logger
│   └── health-monitor.js       # Health monitoring
├── scripts/
│   ├── deploy.sh               # Deployment script
│   ├── backup-system.sh        # Backup automation
│   ├── setup-nginx.sh          # Nginx configuration
│   └── setup-cron-jobs.sh      # Cron job setup
└── nginx/
    └── omega-watches.conf      # Nginx config
```

---

## Deployment Procedures

### Standard Deployment

#### 1. Pre-Deployment Checklist
- [ ] All tests passing in CI/CD pipeline
- [ ] Code reviewed and approved
- [ ] Database backup completed
- [ ] Change notification sent to stakeholders
- [ ] Rollback plan prepared

#### 2. Deploy to Production
```bash
cd /root/Projects/watches

# Run deployment script (includes automatic backup)
./scripts/deploy.sh deploy

# Deployment script will:
# - Run pre-deployment checks
# - Create backup
# - Pull latest code (if git configured)
# - Install dependencies
# - Build frontend
# - Reload PM2 process
# - Run health checks
# - Rollback automatically if health checks fail
```

#### 3. Post-Deployment Verification
```bash
# Check service status
pm2 status
pm2 logs omega-watches --lines 50

# Verify endpoints
curl http://localhost:7600/api/health
curl http://45.61.58.125/api/health

# Check response times
time curl -s http://localhost:7600/api/watches?limit=10

# Monitor for 15 minutes
pm2 monit
```

### Emergency Deployment (Hot Fix)

```bash
# 1. Create backup first
./scripts/backup-system.sh backup

# 2. Apply fix directly to production
vim server.js  # or relevant file

# 3. Quick reload
pm2 reload omega-watches

# 4. Monitor immediately
pm2 logs omega-watches --lines 100
```

### Rollback Procedure

#### Automatic Rollback
The deployment script automatically rolls back if health checks fail.

#### Manual Rollback
```bash
# Option 1: Using deployment script
./scripts/deploy.sh rollback

# Option 2: Restore from specific backup
./scripts/backup-system.sh list
./scripts/backup-system.sh restore /path/to/backup.tar.gz

# Option 3: PM2 previous version (if available)
pm2 reload omega-watches --update-env
```

---

## Monitoring and Alerting

### Health Monitoring

#### Real-time Monitoring
```bash
# PM2 monitoring dashboard
pm2 monit

# Live logs
pm2 logs omega-watches --lines 100 -f

# System metrics
pm2 show omega-watches

# Custom health monitor
pm2 logs omega-health-monitor --lines 50
```

#### Health Check Endpoints
```bash
# Basic health
curl http://localhost:7600/api/health

# Expected response:
{
  "status": "healthy",
  "uptime": 3600,
  "memory": { "heapUsed": 50000000, "heapTotal": 100000000 },
  "cache": { "keys": 10, "size": 5000 },
  "database": { "watches": 500, "collections": 10 },
  "websocket": { "connections": 5 }
}
```

### Log Locations

```bash
# Application logs
/root/Projects/watches/logs/
├── combined-YYYY-MM-DD.log      # All logs
├── error-YYYY-MM-DD.log         # Error logs only
├── access-YYYY-MM-DD.log        # HTTP access logs
├── pm2-error.log                # PM2 error logs
├── pm2-out.log                  # PM2 output logs
└── deploy_TIMESTAMP.log         # Deployment logs

# Nginx logs
/var/log/nginx/
├── omega-watches-access.log
└── omega-watches-error.log

# System logs
journalctl -u nginx
journalctl -f  # Follow system logs
```

### Key Metrics to Monitor

#### Application Metrics
- **Response Time**: < 500ms (p95)
- **Error Rate**: < 1% of requests
- **Memory Usage**: < 400MB
- **CPU Usage**: < 60%
- **Cache Hit Rate**: > 80%

#### System Metrics
- **Disk Space**: > 20% free
- **Network**: < 80% bandwidth utilization
- **Open Files**: < 80% of limit
- **Process Count**: PM2 processes online

#### Business Metrics
- **API Requests/min**: Track trends
- **Top Endpoints**: Most accessed APIs
- **Watch Views**: Popular watches
- **Search Queries**: User behavior

### Alerting Rules

#### Critical Alerts (Immediate Action)
- Service down (health check fails 3 consecutive times)
- Error rate > 5% for 5 minutes
- Memory usage > 90%
- Disk space < 10%
- Backup failure

#### Warning Alerts (Monitor Closely)
- Response time > 1s (p95)
- Memory usage > 80%
- Error rate > 2%
- Cache misses > 30%
- Slow queries > 3s

### Alert Channels

Configure in `.env`:
```bash
# Slack webhook for alerts
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

# Email alerts
EMAIL_ALERTS_TO=admin@example.com

# UptimeRobot for external monitoring
UPTIME_ROBOT_API_KEY=your_key
```

---

## Backup and Recovery

### Backup Schedule

| Type | Frequency | Retention | Storage |
|------|-----------|-----------|---------|
| Daily | 2:00 AM | 30 days | Local |
| Weekly | Sunday 3:00 AM | 90 days | Local + S3 |
| Monthly | 1st, 4:00 AM | 365 days | Local + S3 |

### Manual Backup

```bash
# Full backup
./scripts/backup-system.sh backup

# Verify backup
./scripts/backup-system.sh verify /path/to/backup.tar.gz

# List all backups
./scripts/backup-system.sh list

# Backup statistics
./scripts/backup-system.sh stats
```

### Restore Procedures

#### Restore from Latest Backup
```bash
# 1. Stop service
pm2 stop omega-watches

# 2. List available backups
./scripts/backup-system.sh list

# 3. Restore
./scripts/backup-system.sh restore /path/to/backup.tar.gz

# 4. Verify data integrity
jq empty /root/Projects/watches/data/watches.json

# 5. Restart service
pm2 restart omega-watches

# 6. Health check
curl http://localhost:7600/api/health
```

#### Point-in-Time Recovery
```bash
# Find backup from specific date/time
ls -lh /root/Projects/watches/backups/daily/ | grep "20241117"

# Restore that backup
./scripts/backup-system.sh restore /path/to/specific/backup.tar.gz
```

#### Disaster Recovery (Complete System Loss)

```bash
# 1. Provision new server
# 2. Install dependencies
apt-get update
apt-get install -y nodejs npm nginx

# 3. Install PM2
npm install -g pm2

# 4. Clone/copy project files
cd /root/Projects
# Copy from backup or git repository

# 5. Restore data from S3 (if configured)
aws s3 cp s3://bucket/omega-watches/backups/latest.tar.gz ./
./scripts/backup-system.sh restore latest.tar.gz

# 6. Setup environment
cp .env.example .env
# Edit .env with production values

# 7. Install dependencies
npm ci --production

# 8. Setup Nginx
./scripts/setup-nginx.sh

# 9. Setup cron jobs
./scripts/setup-cron-jobs.sh

# 10. Start services
pm2 start ecosystem.config.js
pm2 save

# 11. Verify
curl http://localhost:7600/api/health
```

---

## Troubleshooting Guide

### Service Not Responding

```bash
# 1. Check if process is running
pm2 status

# 2. Check logs for errors
pm2 logs omega-watches --err --lines 100

# 3. Check port availability
netstat -tuln | grep 7600
lsof -i :7600

# 4. Check system resources
free -h
df -h
top -bn1 | head -20

# 5. Restart service
pm2 restart omega-watches

# 6. If still failing, restore from backup
./scripts/deploy.sh rollback
```

### High Memory Usage

```bash
# 1. Check memory consumption
pm2 show omega-watches

# 2. Clear application cache
curl -X POST http://localhost:7600/api/admin/clear-cache

# 3. Check for memory leaks
pm2 logs omega-watches | grep "memory"

# 4. Restart with memory limit
pm2 restart omega-watches --max-memory-restart 400M

# 5. Monitor after restart
pm2 monit
```

### Slow Response Times

```bash
# 1. Check slow query log
grep "SLOW REQUEST" /root/Projects/watches/logs/combined-*.log

# 2. Check system load
uptime
iostat -x 1 5

# 3. Clear cache to force reload
curl -X POST http://localhost:7600/api/admin/clear-cache

# 4. Check Nginx logs
tail -f /var/log/nginx/omega-watches-access.log

# 5. Optimize if needed
# - Increase cache TTL
# - Add database indexes (if migrated to DB)
# - Enable Nginx caching
```

### Database Corruption

```bash
# 1. Verify JSON integrity
jq empty /root/Projects/watches/data/watches.json

# 2. If corrupted, restore from backup
./scripts/backup-system.sh list
./scripts/backup-system.sh restore /path/to/good/backup.tar.gz

# 3. Verify restored data
jq '.watches | length' /root/Projects/watches/data/watches.json

# 4. Restart service
pm2 restart omega-watches
```

### Nginx Issues

```bash
# 1. Test configuration
nginx -t

# 2. Check Nginx logs
tail -f /var/log/nginx/omega-watches-error.log

# 3. Check if Nginx is running
systemctl status nginx

# 4. Restart Nginx
systemctl restart nginx

# 5. Check backend connectivity
curl http://localhost:7600/api/health
```

---

## Incident Response

### Severity Levels

#### P0 - Critical (Response: Immediate)
- Complete service outage
- Data loss
- Security breach
- System compromise

#### P1 - High (Response: < 30 minutes)
- Major functionality broken
- High error rates (> 10%)
- Performance degradation (> 5s response)

#### P2 - Medium (Response: < 2 hours)
- Minor functionality issues
- Elevated error rates (2-10%)
- Slow response times (1-5s)

#### P3 - Low (Response: Next business day)
- Cosmetic issues
- Feature requests
- Performance optimization

### Incident Response Procedure

#### 1. Detection and Alert
```bash
# Verify incident
curl http://45.61.58.125/api/health
pm2 status
pm2 logs omega-watches --err --lines 50
```

#### 2. Immediate Response
```bash
# For P0/P1 incidents:

# Create incident log
echo "$(date): INCIDENT - Service down" >> /root/Projects/watches/logs/incidents.log

# Check status
./scripts/deploy.sh health-check

# Attempt auto-recovery
pm2 restart omega-watches

# If that fails, rollback
./scripts/deploy.sh rollback
```

#### 3. Investigation
```bash
# Collect diagnostic information
pm2 logs omega-watches --lines 500 > /tmp/incident-logs.txt
pm2 show omega-watches > /tmp/incident-pm2.txt
df -h > /tmp/incident-disk.txt
free -h > /tmp/incident-memory.txt
```

#### 4. Resolution
- Apply fix
- Test thoroughly
- Deploy using standard procedure
- Monitor for 30 minutes

#### 5. Post-Incident Review
- Document root cause
- Update runbook
- Implement preventive measures
- Update monitoring/alerts

---

## Maintenance Procedures

### Regular Maintenance Schedule

#### Daily
- Review error logs
- Check backup status
- Monitor resource usage

#### Weekly
- Review performance metrics
- Check disk space
- Review security logs
- Update dependencies (if needed)

#### Monthly
- Full system backup verification
- Update Node.js/npm (minor versions)
- Review and optimize database
- Security audit

### Dependency Updates

```bash
# Check for outdated packages
npm outdated

# Update dependencies (test in staging first!)
npm update

# Security audit
npm audit
npm audit fix

# Test after updates
npm run build
./scripts/deploy.sh deploy
```

### Log Rotation

Automatic log rotation is configured via Winston. Manual cleanup:

```bash
# Clean logs older than 30 days
find /root/Projects/watches/logs -name "*.log" -mtime +30 -delete

# Compress old logs
find /root/Projects/watches/logs -name "*.log" -mtime +7 -exec gzip {} \;
```

### Database Optimization

```bash
# Verify data integrity
jq empty /root/Projects/watches/data/watches.json

# Compact/optimize JSON (pretty print)
jq '.' /root/Projects/watches/data/watches.json > /tmp/watches-clean.json
mv /tmp/watches-clean.json /root/Projects/watches/data/watches.json

# Clear old backups
./scripts/backup-system.sh rotate
```

---

## Quick Reference

### Essential Commands

```bash
# Service Management
pm2 start ecosystem.config.js      # Start all services
pm2 restart omega-watches           # Restart main service
pm2 stop omega-watches              # Stop service
pm2 delete omega-watches            # Remove service
pm2 reload omega-watches            # Zero-downtime reload

# Logs
pm2 logs omega-watches              # View logs
pm2 logs omega-watches --err        # View errors only
pm2 logs --lines 100                # View last 100 lines
pm2 flush                           # Clear logs

# Monitoring
pm2 monit                           # Real-time monitor
pm2 status                          # Service status
pm2 show omega-watches              # Detailed info

# Deployment
./scripts/deploy.sh deploy          # Deploy
./scripts/deploy.sh rollback        # Rollback
./scripts/deploy.sh health-check    # Health check

# Backup/Restore
./scripts/backup-system.sh backup   # Create backup
./scripts/backup-system.sh list     # List backups
./scripts/backup-system.sh restore <file>  # Restore

# Health Checks
curl http://localhost:7600/api/health
curl http://45.61.58.125/api/health
```

### Emergency Contacts

- **On-Call Engineer**: [Contact Info]
- **System Administrator**: [Contact Info]
- **Escalation**: [Contact Info]

### External Resources

- **Application URL**: http://45.61.58.125:7600
- **Health Endpoint**: http://45.61.58.125:7600/api/health
- **API Documentation**: http://45.61.58.125:7600/api/docs
- **Server IP**: 45.61.58.125

---

## Revision History

| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0 | 2024-11-17 | Claude | Initial runbook creation |

---

**Last Updated**: 2024-11-17
**Next Review Date**: 2024-12-17