← back to Watches

RUNBOOKS/DEPLOYMENT_RUNBOOK.md

433 lines

# Omega Watches - Deployment Runbook

## Table of Contents
- [Overview](#overview)
- [Pre-Deployment Checklist](#pre-deployment-checklist)
- [Deployment Procedures](#deployment-procedures)
- [Rollback Procedures](#rollback-procedures)
- [Post-Deployment Verification](#post-deployment-verification)
- [Troubleshooting](#troubleshooting)

---

## Overview

This runbook covers all deployment procedures for the Omega Watches application running on production server at **45.61.58.125:7500**.

### Deployment Strategy
- **Type**: Blue-Green Deployment
- **Downtime**: Zero downtime
- **Rollback Time**: < 2 minutes
- **Automated**: Yes (via script)

---

## Pre-Deployment Checklist

### 1. Environment Verification
```bash
# Verify server access
ssh root@45.61.58.125

# Check current application status
pm2 list | grep omega-watches
curl -f http://localhost:7500/api/health

# Check disk space (need at least 2GB free)
df -h /root/Projects/watches

# Check memory availability
free -h
```

### 2. Code Preparation
```bash
# Ensure latest code is committed
cd /root/Projects/watches
git status

# Run local tests (if applicable)
npm test

# Build locally to verify
npm run build
```

### 3. Backup Verification
```bash
# Check recent backup exists
ls -lh /root/Projects/watches/backups/daily/

# Verify backup integrity
./scripts/backup-automation.sh verify
```

### 4. Dependencies Check
```bash
# Verify PM2 is running
pm2 status

# Check Node.js version
node --version  # Should be v18.x or higher

# Verify npm packages
npm audit --production
```

---

## Deployment Procedures

### Automated Deployment (Recommended)

#### 1. Using Deployment Script
```bash
cd /root/Projects/watches

# Run automated deployment
./scripts/production-deploy.sh
```

**Expected Output:**
```
╔════════════════════════════════════════════════════════════════╗
║        OMEGA WATCHES - PRODUCTION DEPLOYMENT                   ║
╚════════════════════════════════════════════════════════════════╝

[INFO] Checking prerequisites...
[INFO] Creating backup...
[INFO] Installing dependencies...
[INFO] Building application...
[INFO] Deploying to green environment...
[INFO] Starting green instance...
[INFO] Running health check...
[INFO] Switching traffic to green deployment...

╔════════════════════════════════════════════════════════════════╗
║           DEPLOYMENT SUCCESSFUL!                               ║
╚════════════════════════════════════════════════════════════════╝
```

#### 2. Monitor Deployment
```bash
# Watch PM2 logs during deployment
pm2 logs omega-watches --lines 50

# Monitor system resources
htop

# Check deployment logs
tail -f /root/Projects/watches/logs/deployment-*.log
```

### Manual Deployment (Emergency Only)

If automated deployment fails, follow these manual steps:

#### Step 1: Backup Current State
```bash
cd /root/Projects/watches
./scripts/backup-automation.sh daily
```

#### Step 2: Install Dependencies
```bash
npm ci --production
```

#### Step 3: Build Application
```bash
npm run build
```

#### Step 4: Restart Application
```bash
pm2 restart omega-watches
```

#### Step 5: Verify Health
```bash
sleep 5
curl -f http://localhost:7500/api/health
```

---

## Rollback Procedures

### Quick Rollback (< 2 minutes)

#### Option 1: Rollback via Script
```bash
cd /root/Projects/watches

# Find latest backup
ls -lht backups/deployments/ | head -n 5

# Restore from backup
./scripts/backup-automation.sh restore backups/deployments/backup-YYYYMMDD_HHMMSS.tar.gz
```

#### Option 2: Manual Rollback
```bash
# Stop current application
pm2 stop omega-watches

# Extract latest backup
cd /root/Projects/watches
LATEST_BACKUP=$(ls -t backups/deployments/backup-*.tar.gz | head -1)
tar -xzf $LATEST_BACKUP

# Restart application
pm2 restart omega-watches

# Verify
curl -f http://localhost:7500/api/health
```

### Verify Rollback Success
```bash
# Check application is running
pm2 list | grep omega-watches

# Test API endpoints
curl http://localhost:7500/api/watches | jq '.total'
curl http://localhost:7500/api/health | jq '.'

# Check error logs
pm2 logs omega-watches --lines 20 --err
```

---

## Post-Deployment Verification

### 1. Health Checks
```bash
# Application health
curl -f http://localhost:7500/api/health
curl -f http://45.61.58.125:7500/api/health

# API functionality
curl http://localhost:7500/api/watches?limit=5
curl http://localhost:7500/api/statistics
```

### 2. Performance Verification
```bash
# Response time check
time curl -s http://localhost:7500/api/watches > /dev/null

# Load test (optional)
ab -n 1000 -c 10 http://localhost:7500/api/health
```

### 3. Monitoring Dashboard
```bash
# Access Grafana
http://45.61.58.125:7511
# Login: admin / omega-watches-admin

# Access Prometheus
http://45.61.58.125:7510
```

### 4. Log Inspection
```bash
# Check for errors in last 100 lines
pm2 logs omega-watches --lines 100 --err

# Check access logs
tail -f /var/log/nginx/omega-watches-access.log

# Check application logs
tail -f /root/Projects/watches/logs/*.log
```

### 5. Functional Testing

#### Test Critical Endpoints:
```bash
# Homepage
curl -I http://45.61.58.125:7500/

# Watch list
curl http://45.61.58.125:7500/api/watches | jq '.total'

# Specific watch
curl http://45.61.58.125:7500/api/watches/seamaster-300m | jq '.watch.model'

# Statistics
curl http://45.61.58.125:7500/api/statistics | jq '.totalWatches'

# WebSocket (optional)
wscat -c ws://45.61.58.125:7500
```

---

## Troubleshooting

### Issue: Application Won't Start

#### Symptoms
```bash
pm2 list
# Shows: omega-watches | errored
```

#### Diagnosis
```bash
# Check error logs
pm2 logs omega-watches --err --lines 50

# Check port availability
lsof -i :7500

# Check Node.js process
ps aux | grep node
```

#### Resolution
```bash
# Kill processes on port 7500
lsof -ti:7500 | xargs kill -9

# Restart application
pm2 restart omega-watches

# If still failing, rollback
./scripts/backup-automation.sh restore <latest-backup>
```

### Issue: High Response Times

#### Diagnosis
```bash
# Check system resources
top
free -h
df -h

# Check PM2 metrics
pm2 monit

# Check Node.js heap
pm2 logs omega-watches | grep "heap"
```

#### Resolution
```bash
# Restart application to clear memory
pm2 restart omega-watches

# Clear cache (if using Redis)
redis-cli FLUSHALL

# Scale horizontally (if needed)
pm2 scale omega-watches +2
```

### Issue: Database Connection Errors

#### Symptoms
```
Error: Connection refused to database
```

#### Resolution
```bash
# This app uses JSON files, not database
# Check file permissions
ls -l /root/Projects/watches/data/

# Verify data files exist
cat /root/Projects/watches/data/watches.json | jq '.watches | length'

# Restore data from backup if corrupted
tar -xzf backups/daily/backup-*.tar.gz data/
```

### Issue: Nginx Not Proxying

#### Diagnosis
```bash
# Check Nginx status
systemctl status nginx

# Check Nginx configuration
nginx -t

# Check Nginx logs
tail -f /var/log/nginx/error.log
```

#### Resolution
```bash
# Reload Nginx configuration
nginx -s reload

# Restart Nginx
systemctl restart nginx

# Test direct application access
curl http://localhost:7500/api/health
```

### Issue: Memory Leak

#### Symptoms
```bash
pm2 monit
# Shows continuously increasing memory
```

#### Resolution
```bash
# Restart application
pm2 restart omega-watches

# Enable memory limit (if needed)
pm2 restart omega-watches --max-memory-restart 500M

# Monitor for recurrence
watch -n 5 'pm2 list | grep omega-watches'
```

---

## Emergency Contacts

- **Server**: 45.61.58.125
- **Application Port**: 7500
- **PM2 App Name**: omega-watches
- **Backup Location**: /root/Projects/watches/backups/

## Quick Commands Reference

```bash
# Status check
pm2 list
curl http://localhost:7500/api/health

# View logs
pm2 logs omega-watches

# Restart
pm2 restart omega-watches

# Deploy
./scripts/production-deploy.sh

# Backup
./scripts/backup-automation.sh daily

# Rollback
./scripts/backup-automation.sh restore <backup-file>
```

---

**Last Updated**: 2025-11-17
**Version**: 1.0