`DEVOPS_RUNBOOK.md`

474 lines
# LUXVAULT Auction Viewer - DevOps Runbook

## Service Information
- **Service Name**: auction-viewer
- **Port**: 7500
- **Server IP**: 45.61.58.125
- **Process Manager**: PM2
- **Database**: SQLite (`/root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db`)
- **Platform**: Ubuntu on Kamatera VPS

---

## Quick Reference

### Service URLs
- **Main Dashboard**: http://45.61.58.125:7500
- **Health Check**: http://45.61.58.125:7500/health
- **Readiness Check**: http://45.61.58.125:7500/ready
- **Metrics**: http://45.61.58.125:7500/metrics
- **API Stats**: http://45.61.58.125:7500/api/stats
- **Monitoring Dashboard**: http://45.61.58.125:7500/monitoring-dashboard.html

### PM2 Commands
```bash
pm2 status auction-viewer          # Check status
pm2 logs auction-viewer            # View logs
pm2 restart auction-viewer         # Restart service
pm2 reload auction-viewer          # Zero-downtime reload
pm2 stop auction-viewer            # Stop service
pm2 start auction-viewer           # Start service
pm2 monit                          # Real-time monitoring
```

---

## Common Operations

### 1. Deployment

**Standard Deployment** (Zero-downtime):
```bash
cd /root/Projects/handbag-auth-nextjs
./scripts/devops/deploy.sh
```

The deployment script:
- Creates automatic backup
- Installs dependencies
- Validates configuration
- Performs graceful reload
- Runs health checks
- Auto-rollback on failure

**Manual Deployment**:
```bash
cd /root/Projects/handbag-auth-nextjs/auction-viewer
git pull origin main  # If using git
npm install --production
pm2 reload auction-viewer
```

### 2. Rollback

**Automatic Rollback** (to latest backup):
```bash
/root/Projects/handbag-auth-nextjs/scripts/devops/rollback.sh latest
```

**Manual Rollback**:
```bash
# List available backups
ls -lh /root/Projects/handbag-auth-nextjs/backups/deployments/

# Rollback to specific version
/root/Projects/handbag-auth-nextjs/scripts/devops/rollback.sh <timestamp>
```

### 3. Database Backup

**Manual Backup**:
```bash
/root/Projects/handbag-auth-nextjs/scripts/devops/backup-database.sh
```

**Automated Backups**:
- Scheduled via cron: Daily at 2:00 AM
- Retention: 7 days
- Location: `/root/Projects/handbag-auth-nextjs/backups/database/`
- Backups are compressed with gzip
- Latest backup symlinked as `latest.db.gz`

**View Backups**:
```bash
ls -lh /root/Projects/handbag-auth-nextjs/backups/database/
```

### 4. Database Restore

**Restore from Latest Backup**:
```bash
/root/Projects/handbag-auth-nextjs/scripts/devops/restore-database.sh latest
```

**Restore from Specific Backup**:
```bash
/root/Projects/handbag-auth-nextjs/scripts/devops/restore-database.sh /path/to/backup.db.gz
```

The restore script:
- Stops the service
- Creates safety backup
- Verifies database integrity
- Restores database
- Restarts service
- Validates operation

---

## Monitoring & Health Checks

### Health Check (Automated every 5 minutes)
```bash
# Manual health check
/root/Projects/handbag-auth-nextjs/scripts/devops/health-check.sh
```

The health check monitors:
- PM2 process status
- HTTP endpoint availability
- Database connectivity
- Response times
- Disk space usage
- Memory usage

**Auto-restart**: If service is down, health check automatically attempts restart.

### Real-time Monitoring
```bash
# PM2 monitoring
pm2 monit

# Service logs (live)
pm2 logs auction-viewer --lines 100

# Health endpoint
curl http://45.61.58.125:7500/health | jq

# Metrics endpoint
curl http://45.61.58.125:7500/metrics
```

### Monitoring Dashboard
Access web-based monitoring at:
```
http://45.61.58.125:7500/monitoring-dashboard.html
```

Features:
- Real-time service status
- Performance metrics
- Memory usage graphs
- Database connectivity
- Live logs
- Auto-refresh every 10 seconds

---

## Troubleshooting

### Service Won't Start

1. **Check PM2 status**:
```bash
pm2 status
pm2 logs auction-viewer --lines 50
```

2. **Check port availability**:
```bash
lsof -ti:7500
netstat -tulnp | grep 7500
```

3. **Kill process on port**:
```bash
lsof -ti:7500 | xargs kill -9
pm2 restart auction-viewer
```

4. **Check database**:
```bash
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "PRAGMA integrity_check;"
```

### High Memory Usage

1. **Check current usage**:
```bash
pm2 show auction-viewer | grep memory
```

2. **Restart service** (clears memory):
```bash
pm2 restart auction-viewer
```

3. **Update memory limit** in PM2:
```bash
pm2 delete auction-viewer
pm2 start /root/Projects/handbag-auth-nextjs/auction-viewer/config/pm2.ecosystem.config.js
```

### Slow Performance

1. **Run performance test**:
```bash
/root/Projects/handbag-auth-nextjs/scripts/devops/performance-test.sh
```

2. **Check database size**:
```bash
du -sh /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db
```

3. **Vacuum database** (optimize):
```bash
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "VACUUM;"
```

### Database Corruption

1. **Check integrity**:
```bash
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "PRAGMA integrity_check;"
```

2. **Restore from backup**:
```bash
/root/Projects/handbag-auth-nextjs/scripts/devops/restore-database.sh latest
```

3. **If backup fails**, rebuild from CSV:
```bash
cd /root/Projects/handbag-auth-nextjs/auction-viewer
node migrations.js
```

### Service Health Check Failing

1. **Manual health check**:
```bash
curl -v http://45.61.58.125:7500/health
```

2. **Check firewall**:
```bash
sudo ufw status | grep 7500
```

3. **Check service logs**:
```bash
pm2 logs auction-viewer --err --lines 50
```

4. **Restart service**:
```bash
pm2 restart auction-viewer
sleep 5
curl http://45.61.58.125:7500/health
```

---

## Maintenance Tasks

### Daily
- Automated database backup (2:00 AM)
- Automated health checks (every 5 minutes)
- Log rotation (PM2 handles automatically)

### Weekly
```bash
# Check disk space
df -h /root/Projects/handbag-auth-nextjs

# Review error logs
pm2 logs auction-viewer --err --lines 100

# Check backup integrity
ls -lh /root/Projects/handbag-auth-nextjs/backups/database/
```

### Monthly
```bash
# Optimize database
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "VACUUM;"

# Review performance metrics
/root/Projects/handbag-auth-nextjs/scripts/devops/performance-test.sh

# Update dependencies
cd /root/Projects/handbag-auth-nextjs/auction-viewer
npm update
npm audit fix
```

---

## Emergency Procedures

### Complete Service Failure

1. **Check PM2 daemon**:
```bash
pm2 status
pm2 ping
```

2. **Restart PM2 daemon** (if needed):
```bash
pm2 kill
pm2 resurrect
```

3. **Start from PM2 config**:
```bash
pm2 start /root/Projects/handbag-auth-nextjs/auction-viewer/config/pm2.ecosystem.config.js
```

### Database Lost/Corrupted

1. **Restore from latest backup**:
```bash
/root/Projects/handbag-auth-nextjs/scripts/devops/restore-database.sh latest
```

2. **If no backup available**, re-scrape data:
```bash
cd /root/Projects/handbag-auth-nextjs
python3 scripts/scrape-auction-data.py
```

### Server Reboot

PM2 automatically restarts services on reboot if startup script is configured:

```bash
# Save PM2 configuration
pm2 save

# Setup startup script (one-time)
pm2 startup
# Copy and run the generated command
```

### Disk Space Full

1. **Identify large files**:
```bash
du -sh /root/Projects/handbag-auth-nextjs/* | sort -h
```

2. **Clean old backups**:
```bash
find /root/Projects/handbag-auth-nextjs/backups -mtime +30 -delete
```

3. **Clean logs**:
```bash
pm2 flush
```

4. **Vacuum database**:
```bash
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "VACUUM;"
```

---

## Cron Jobs

### Current Cron Schedule

```bash
# Daily database backup at 2:00 AM
0 2 * * * /root/Projects/handbag-auth-nextjs/scripts/devops/backup-database.sh >> /root/Projects/handbag-auth-nextjs/logs/backup-cron.log 2>&1

# Health check every 5 minutes
*/5 * * * * /root/Projects/handbag-auth-nextjs/scripts/devops/health-check.sh >> /root/Projects/handbag-auth-nextjs/logs/health-check-cron.log 2>&1

# Daily auction data scrape at 6:00 AM
0 6 * * * /usr/bin/python3 /root/Projects/handbag-auth-nextjs/scripts/scrape-auction-data.py >> /root/Projects/handbag-auth-nextjs/data/auction-history/cron.log 2>&1
```

### View Cron Jobs
```bash
crontab -l | grep auction
```

### Edit Cron Jobs
```bash
crontab -e
```

---

## Performance Metrics

### Target Metrics
- **Uptime**: > 99.9%
- **Response Time**: < 500ms (avg)
- **Error Rate**: < 1%
- **Memory Usage**: < 500MB
- **Database Size**: Monitor for growth

### Monitoring
- Real-time: http://45.61.58.125:7500/metrics
- Dashboard: http://45.61.58.125:7500/monitoring-dashboard.html
- PM2: `pm2 monit`

---

## Contact & Escalation

### Logs Location
- **Application Logs**: `/root/Projects/handbag-auth-nextjs/logs/`
- **PM2 Logs**: `~/.pm2/logs/`
- **Backup Logs**: `/root/Projects/handbag-auth-nextjs/logs/backup.log`
- **Deployment Logs**: `/root/Projects/handbag-auth-nextjs/logs/deploy.log`
- **Health Check Logs**: `/root/Projects/handbag-auth-nextjs/logs/health-check.log`

### Support Resources
- Application dashboard: http://45.61.58.125:7500
- API documentation: Available at `/api/*` endpoints
- This runbook: `/root/Projects/handbag-auth-nextjs/DEVOPS_RUNBOOK.md`

---

## Quick Command Reference

```bash
# Check service status
pm2 status auction-viewer

# View live logs
pm2 logs auction-viewer

# Restart service
pm2 restart auction-viewer

# Deploy new version
/root/Projects/handbag-auth-nextjs/scripts/devops/deploy.sh

# Rollback
/root/Projects/handbag-auth-nextjs/scripts/devops/rollback.sh latest

# Manual backup
/root/Projects/handbag-auth-nextjs/scripts/devops/backup-database.sh

# Health check
curl http://45.61.58.125:7500/health | jq

# Performance test
/root/Projects/handbag-auth-nextjs/scripts/devops/performance-test.sh

# Database integrity check
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "PRAGMA integrity_check;"
```

---

*Last Updated: 2025-11-17*
*Version: 1.0*