← back to Watches

RUNBOOKS/DISASTER_RECOVERY.md

578 lines

# Omega Watches - Disaster Recovery Plan

## Table of Contents
- [Overview](#overview)
- [Recovery Time Objectives](#recovery-time-objectives)
- [Disaster Scenarios](#disaster-scenarios)
- [Recovery Procedures](#recovery-procedures)
- [Data Backup Strategy](#data-backup-strategy)
- [Testing Procedures](#testing-procedures)

---

## Overview

This document outlines disaster recovery procedures for the Omega Watches application infrastructure.

### Critical Information
- **Server IP**: 45.61.58.125
- **Application Port**: 7500
- **Application Directory**: /root/Projects/watches
- **Backup Location**: /root/Projects/watches/backups/
- **PM2 App Name**: omega-watches

---

## Recovery Time Objectives

| Component | RTO | RPO | Priority |
|-----------|-----|-----|----------|
| Application | 15 minutes | 24 hours | P0 - Critical |
| Data Files | 30 minutes | 24 hours | P0 - Critical |
| Configuration | 15 minutes | 7 days | P1 - High |
| Monitoring | 1 hour | N/A | P2 - Medium |

**Legend:**
- **RTO** (Recovery Time Objective): Maximum acceptable downtime
- **RPO** (Recovery Point Objective): Maximum acceptable data loss
- **P0**: Critical - Immediate recovery required
- **P1**: High - Recovery within hours
- **P2**: Medium - Recovery within 24 hours

---

## Disaster Scenarios

### Scenario 1: Application Crash

#### Detection
```bash
# Application not responding
curl -f http://45.61.58.125:7500/api/health
# Returns: Connection refused or 502

# PM2 shows error
pm2 list | grep omega-watches
# Shows: errored or stopped
```

#### Recovery Steps
```bash
# 1. Check application logs
pm2 logs omega-watches --err --lines 50

# 2. Attempt restart
pm2 restart omega-watches

# 3. Verify health
sleep 5
curl -f http://localhost:7500/api/health

# 4. If restart fails, rollback
cd /root/Projects/watches
./scripts/backup-automation.sh restore $(ls -t backups/daily/*.tar.gz | head -1)

# 5. Final verification
curl -f http://45.61.58.125:7500/api/watches
```

**Recovery Time**: 5-10 minutes

---

### Scenario 2: Data Corruption

#### Detection
```bash
# API returns errors
curl http://45.61.58.125:7500/api/watches
# Returns: 500 Internal Server Error

# Data file is corrupted
cat /root/Projects/watches/data/watches.json | jq '.'
# Returns: parse error
```

#### Recovery Steps
```bash
# 1. Stop application
pm2 stop omega-watches

# 2. Backup corrupted files
cd /root/Projects/watches
mkdir -p /tmp/corrupted-data-$(date +%Y%m%d)
cp -r data/ /tmp/corrupted-data-$(date +%Y%m%d)/

# 3. Find latest valid backup
LATEST_BACKUP=$(ls -t backups/daily/backup-*.tar.gz | head -1)

# 4. Extract data files only
tar -xzf $LATEST_BACKUP data/

# 5. Verify data integrity
cat data/watches.json | jq '.watches | length'

# 6. Restart application
pm2 restart omega-watches

# 7. Verify functionality
curl http://localhost:7500/api/watches | jq '.total'
```

**Recovery Time**: 10-15 minutes

---

### Scenario 3: Server Failure

#### Detection
```bash
# Cannot SSH to server
ssh root@45.61.58.125
# Returns: Connection timeout

# Application unreachable
curl -f http://45.61.58.125:7500
# Returns: Connection timeout
```

#### Recovery Steps (New Server)

##### Step 1: Provision New Server
```bash
# Requirements:
# - Ubuntu 20.04 or newer
# - 2GB RAM minimum
# - 20GB disk space
# - Same IP (45.61.58.125) or update DNS
```

##### Step 2: Install Dependencies
```bash
# Update system
apt update && apt upgrade -y

# Install Node.js 18.x
curl -fsSL https://deb.nodesource.com/setup_18.x | bash -
apt install -y nodejs

# Install PM2
npm install -g pm2

# Install Nginx
apt install -y nginx

# Install Docker (for monitoring)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose
```

##### Step 3: Restore Application
```bash
# Create application directory
mkdir -p /root/Projects/watches
cd /root/Projects/watches

# Restore from backup (if available from remote storage)
# Option 1: Download from remote backup
# scp user@backup-server:/backups/omega-watches/latest.tar.gz .

# Option 2: Restore from local backup (if available)
# tar -xzf backup-latest.tar.gz

# If no backup available, clone from repository
git clone <repository-url> .

# Install dependencies
npm ci --production

# Build application
npm run build
```

##### Step 4: Configure and Start
```bash
# Start application with PM2
pm2 start ecosystem.config.cjs --name omega-watches

# Save PM2 configuration
pm2 save

# Configure PM2 startup
pm2 startup
# Run the command it outputs

# Configure Nginx
cp infrastructure/nginx-omega-watches.conf /etc/nginx/sites-available/omega-watches
ln -s /etc/nginx/sites-available/omega-watches /etc/nginx/sites-enabled/
nginx -t
systemctl restart nginx

# Open firewall ports
ufw allow 7500/tcp
ufw allow 80/tcp
ufw allow 443/tcp
```

##### Step 5: Start Monitoring Stack
```bash
# Create monitoring directories
mkdir -p monitoring/prometheus monitoring/grafana monitoring/alertmanager

# Start monitoring containers
docker-compose -f docker-compose.monitoring.yml up -d
```

##### Step 6: Verify Recovery
```bash
# Check application health
curl -f http://localhost:7500/api/health
curl -f http://45.61.58.125:7500/api/health

# Check all endpoints
curl http://localhost:7500/api/watches | jq '.total'
curl http://localhost:7500/api/statistics

# Check monitoring
curl http://localhost:7510  # Prometheus
curl http://localhost:7511  # Grafana

# Check PM2
pm2 list
pm2 logs omega-watches --lines 20
```

**Recovery Time**: 2-4 hours (depending on backup availability)

---

### Scenario 4: Disk Full

#### Detection
```bash
# Disk space check
df -h
# Shows: 100% or 95%+ usage

# Application logs show write errors
pm2 logs omega-watches --err
# Shows: ENOSPC: no space left on device
```

#### Recovery Steps
```bash
# 1. Find large files
du -ah /root/Projects/watches | sort -rh | head -20

# 2. Clean up old logs
find /root/Projects/watches/logs -name "*.log" -mtime +7 -delete
pm2 flush  # Clear PM2 logs

# 3. Clean old backups (keep last 5 of each type)
cd /root/Projects/watches/backups
ls -t daily/*.tar.gz | tail -n +6 | xargs rm -f
ls -t weekly/*.tar.gz | tail -n +6 | xargs rm -f
ls -t monthly/*.tar.gz | tail -n +6 | xargs rm -f

# 4. Clean npm cache
npm cache clean --force

# 5. Clean Docker (if using)
docker system prune -af

# 6. Restart application if needed
pm2 restart omega-watches

# 7. Verify disk space
df -h
```

**Recovery Time**: 5-10 minutes

---

### Scenario 5: Database/Data File Loss

#### Detection
```bash
# Missing data files
ls -l /root/Projects/watches/data/
# Shows: No such file or directory

# Application errors
pm2 logs omega-watches --err
# Shows: Cannot find module './data/watches.json'
```

#### Recovery Steps
```bash
# 1. Stop application
pm2 stop omega-watches

# 2. Restore from latest backup
cd /root/Projects/watches
LATEST_BACKUP=$(ls -t backups/daily/backup-*.tar.gz | head -1)
tar -xzf $LATEST_BACKUP data/

# 3. Verify data integrity
cat data/watches.json | jq '.watches | length'

# 4. If no backup available, restore default data
mkdir -p data
# Contact administrator for data restoration

# 5. Restart application
pm2 restart omega-watches

# 6. Verify
curl http://localhost:7500/api/watches
```

**Recovery Time**: 10-20 minutes

---

### Scenario 6: Security Breach

#### Detection
```bash
# Unusual traffic patterns
tail -f /var/log/nginx/omega-watches-access.log

# Suspicious processes
ps aux | grep -v "grep" | grep -E "omega|node"

# Unauthorized file modifications
find /root/Projects/watches -mtime -1 -type f
```

#### Recovery Steps
```bash
# 1. IMMEDIATE: Isolate the server
ufw deny from any to any

# 2. Stop application
pm2 stop omega-watches

# 3. Analyze compromise
# - Check system logs: journalctl -xe
# - Check access logs: /var/log/nginx/
# - Check auth logs: /var/log/auth.log

# 4. Backup potentially compromised system
cd /root/Projects/watches
tar -czf /tmp/compromised-backup-$(date +%Y%m%d_%H%M%S).tar.gz .

# 5. Restore from known-good backup
LAST_KNOWN_GOOD="backups/daily/backup-YYYYMMDD_HHMMSS.tar.gz"
rm -rf /root/Projects/watches/*
tar -xzf $LAST_KNOWN_GOOD

# 6. Update all dependencies
npm audit fix
npm update

# 7. Change all credentials
# - Update environment variables
# - Rotate API keys
# - Change system passwords

# 8. Restore firewall rules
ufw --force reset
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 7500/tcp
ufw enable

# 9. Restart application
pm2 restart omega-watches

# 10. Monitor closely
pm2 logs omega-watches --lines 100
tail -f /var/log/nginx/omega-watches-access.log
```

**Recovery Time**: 1-4 hours (plus investigation time)

---

## Data Backup Strategy

### Automated Backups

#### Daily Backups
```bash
# Configured via cron
0 2 * * * /root/Projects/watches/scripts/backup-automation.sh daily

# Retention: 7 days
# Location: /root/Projects/watches/backups/daily/
```

#### Weekly Backups
```bash
# Configured via cron
0 3 * * 0 /root/Projects/watches/scripts/backup-automation.sh weekly

# Retention: 30 days
# Location: /root/Projects/watches/backups/weekly/
```

#### Monthly Backups
```bash
# Configured via cron
0 4 1 * * /root/Projects/watches/scripts/backup-automation.sh monthly

# Retention: 365 days
# Location: /root/Projects/watches/backups/monthly/
```

### Manual Backup
```bash
# Create immediate backup
cd /root/Projects/watches
./scripts/backup-automation.sh daily
```

### Backup Verification
```bash
# Verify all backups
./scripts/backup-automation.sh verify

# List all backups
./scripts/backup-automation.sh list

# Show backup statistics
./scripts/backup-automation.sh stats
```

### Off-site Backup (Recommended)

#### Setup Remote Backup
```bash
# Edit backup script to enable remote backup
nano /root/Projects/watches/scripts/backup-automation.sh

# Set these variables:
# REMOTE_BACKUP_ENABLED=true
# REMOTE_HOST="backup-server.example.com"
# REMOTE_DIR="/backups/omega-watches"
# REMOTE_USER="backup-user"

# Setup SSH key authentication
ssh-keygen -t rsa -b 4096
ssh-copy-id backup-user@backup-server.example.com
```

---

## Testing Procedures

### Monthly DR Test

#### Test 1: Backup Restoration
```bash
# 1. Create test environment
mkdir -p /tmp/dr-test
cd /tmp/dr-test

# 2. Restore latest backup
cp /root/Projects/watches/backups/daily/backup-*.tar.gz .
tar -xzf backup-*.tar.gz

# 3. Verify all files
ls -la
cat data/watches.json | jq '.watches | length'

# 4. Cleanup
cd /
rm -rf /tmp/dr-test
```

#### Test 2: Application Recovery
```bash
# 1. Stop application
pm2 stop omega-watches

# 2. Wait 1 minute

# 3. Restart application
pm2 restart omega-watches

# 4. Verify health
sleep 5
curl -f http://localhost:7500/api/health

# 5. Document recovery time
```

#### Test 3: Data Corruption Simulation
```bash
# 1. Backup current data
cp /root/Projects/watches/data/watches.json /tmp/watches.json.bak

# 2. Corrupt data file
echo "corrupted" > /root/Projects/watches/data/watches.json

# 3. Detect issue
curl http://localhost:7500/api/watches
# Should return error

# 4. Restore data
./scripts/backup-automation.sh restore <latest-backup>

# 5. Verify restoration
curl http://localhost:7500/api/watches
```

---

## Recovery Checklist

### Post-Recovery Verification

- [ ] Application is running (`pm2 list`)
- [ ] Health endpoint responds (`curl http://localhost:7500/api/health`)
- [ ] All API endpoints functional
- [ ] Data integrity verified
- [ ] Monitoring is operational
- [ ] Backups are running
- [ ] Logs are being written
- [ ] Performance is normal
- [ ] Security measures are in place
- [ ] Documentation is updated

---

## Emergency Contact Information

### Server Access
```
Host: 45.61.58.125
User: root
Port: 22
```

### Application URLs
```
Main: http://45.61.58.125:7500
Health: http://45.61.58.125:7500/api/health
Grafana: http://45.61.58.125:7511
Prometheus: http://45.61.58.125:7510
```

### Critical Paths
```
Application: /root/Projects/watches
Backups: /root/Projects/watches/backups
Logs: /root/Projects/watches/logs
Data: /root/Projects/watches/data
```

---

**Last Updated**: 2025-11-17
**Version**: 1.0
**Review Schedule**: Monthly