← back to Watches

INFRASTRUCTURE_GUIDE.md

609 lines

# Omega Watches - Production Infrastructure Guide

## Overview

Complete production-grade infrastructure for Omega Watches application with CI/CD, monitoring, automated backups, disaster recovery, and zero-downtime deployments.

**Server**: 45.61.58.125
**Application**: http://45.61.58.125:7500
**Environment**: Production

---

## Quick Start

### One-Command Setup
```bash
cd /root/Projects/watches
./PRODUCTION_SETUP.sh --full
```

This will install and configure:
- ✓ Prometheus + Grafana monitoring
- ✓ Nginx reverse proxy with caching
- ✓ Automated backup system
- ✓ Cron jobs for maintenance
- ✓ Firewall rules
- ✓ Health monitoring

---

## Infrastructure Components

### 1. Application Stack

#### Primary Application
- **Port**: 7500
- **Process Manager**: PM2
- **App Name**: omega-watches
- **Health Check**: http://localhost:7500/api/health

#### Staging Environment (Optional)
- **Port**: 7501
- **App Name**: omega-watches-staging

#### Blue-Green Deployment
- **Green Port**: 7502
- **Used during deployments only**

### 2. Monitoring Stack (Docker Compose)

All monitoring components run in Docker containers:

#### Prometheus
- **Port**: 7510
- **URL**: http://45.61.58.125:7510
- **Purpose**: Metrics collection and alerting
- **Retention**: 30 days
- **Scrape Interval**: 15 seconds

#### Grafana
- **Port**: 7511
- **URL**: http://45.61.58.125:7511
- **Username**: admin
- **Password**: omega-watches-admin
- **Purpose**: Visualization dashboards

#### Node Exporter
- **Port**: 7512
- **Purpose**: System metrics (CPU, memory, disk, network)

#### Alertmanager
- **Port**: 7513
- **URL**: http://45.61.58.125:7513
- **Purpose**: Alert routing and notifications

#### Redis
- **Port**: 7514
- **Purpose**: Caching layer
- **Password**: omega-redis-pass

### 3. Nginx Reverse Proxy

#### Configuration
- **Config File**: /etc/nginx/sites-available/omega-watches
- **Cache Path**: /var/cache/nginx/omega-watches
- **Access Log**: /var/log/nginx/omega-watches-access.log
- **Error Log**: /var/log/nginx/omega-watches-error.log

#### Features
- Load balancing (least connections)
- Rate limiting (10 req/s general, 30 req/s API)
- Static file caching (7 days)
- API response caching (5 minutes)
- Gzip compression
- WebSocket support

### 4. Backup System

#### Backup Types
```
Daily Backups:   2:00 AM daily    (Retention: 7 days)
Weekly Backups:  3:00 AM Sunday   (Retention: 30 days)
Monthly Backups: 4:00 AM 1st      (Retention: 365 days)
```

#### Backup Locations
```
/root/Projects/watches/backups/
├── daily/          - Daily backups
├── weekly/         - Weekly backups
├── monthly/        - Monthly backups
└── deployments/    - Pre-deployment backups
```

#### Backup Contents
- Application code (dist/, server.js)
- Data files (data/)
- Configuration files
- Public assets
- Monitoring configs
- Nginx configs

---

## Deployment Procedures

### Automated Deployment (Blue-Green)

```bash
cd /root/Projects/watches
./scripts/production-deploy.sh
```

**Process**:
1. Create backup of current deployment
2. Install dependencies
3. Build application
4. Deploy to green environment (port 7502)
5. Start green instance
6. Run health checks
7. Switch traffic to green
8. Promote green to production
9. Cleanup

**Rollback**: Automatic if health checks fail

### Manual Deployment

```bash
# 1. Backup
./scripts/backup-automation.sh daily

# 2. Install dependencies
npm ci --production

# 3. Build
npm run build

# 4. Restart
pm2 restart omega-watches

# 5. Verify
curl http://localhost:7500/api/health
```

---

## Monitoring & Alerts

### Prometheus Metrics

Access: http://45.61.58.125:7510

**Application Metrics**:
- `up{job="omega-watches"}` - Application availability
- `http_requests_total` - Request count
- `http_request_duration_seconds` - Response times
- `nodejs_heap_size_used_bytes` - Memory usage
- `nodejs_eventloop_lag_seconds` - Event loop lag

**System Metrics**:
- `node_cpu_seconds_total` - CPU usage
- `node_memory_MemAvailable_bytes` - Available memory
- `node_filesystem_avail_bytes` - Disk space
- `node_disk_io_time_seconds_total` - Disk I/O

### Grafana Dashboards

Access: http://45.61.58.125:7511
Login: admin / omega-watches-admin

**Available Dashboards**:
- Omega Watches Production Monitoring
- Node Exporter Full
- Kubernetes Cluster (if needed)

### Alert Rules

**Critical Alerts**:
- Application down for 2+ minutes
- Disk space < 15%
- Memory usage > 85%

**Warning Alerts**:
- High error rate (> 5% requests)
- Slow response times (> 2s at 95th percentile)
- CPU usage > 80%
- Redis connection failures

**Alert Destinations**:
- Webhook: http://45.61.58.125:7500/api/webhook/alerts
- Alertmanager: http://45.61.58.125:7513

---

## Backup & Recovery

### Create Manual Backup

```bash
cd /root/Projects/watches

# Daily backup
./scripts/backup-automation.sh daily

# Weekly backup
./scripts/backup-automation.sh weekly

# Monthly backup
./scripts/backup-automation.sh monthly
```

### List Backups

```bash
./scripts/backup-automation.sh list
```

### Show Backup Statistics

```bash
./scripts/backup-automation.sh stats
```

### Verify Backup Integrity

```bash
./scripts/backup-automation.sh verify
```

### Restore from Backup

```bash
# Find backup
ls -lh backups/daily/

# Restore
./scripts/backup-automation.sh restore backups/daily/backup-YYYYMMDD_HHMMSS.tar.gz
```

---

## Automated Tasks (Cron Jobs)

### Configured Jobs

```bash
# Daily backup at 2:00 AM
0 2 * * * /root/Projects/watches/scripts/backup-automation.sh daily

# Weekly backup every Sunday at 3:00 AM
0 3 * * 0 /root/Projects/watches/scripts/backup-automation.sh weekly

# Monthly backup on 1st at 4:00 AM
0 4 1 * * /root/Projects/watches/scripts/backup-automation.sh monthly

# Health check every 5 minutes
*/5 * * * * curl -f http://localhost:7500/api/health || log failure

# Clean old logs daily at 1:00 AM
0 1 * * * find /root/Projects/watches/logs -name "*.log" -mtime +30 -delete

# PM2 save every hour
0 * * * * pm2 save --force

# Verify backups every Monday at 5:00 AM
0 5 * * 1 /root/Projects/watches/scripts/backup-automation.sh verify
```

### Manage Cron Jobs

```bash
# List all cron jobs
crontab -l

# Edit cron jobs
crontab -e

# View cron logs
tail -f /root/Projects/watches/logs/cron-*.log
```

---

## Disaster Recovery

### RTO/RPO Targets

| Component | RTO | RPO |
|-----------|-----|-----|
| Application | 15 min | 24 hours |
| Data Files | 30 min | 24 hours |
| Configuration | 15 min | 7 days |

### Recovery Scenarios

#### 1. Application Crash
```bash
# Restart
pm2 restart omega-watches

# Or rollback
./scripts/backup-automation.sh restore <latest-backup>
```

#### 2. Data Corruption
```bash
# Stop app
pm2 stop omega-watches

# Restore data only
tar -xzf backups/daily/backup-*.tar.gz data/

# Restart
pm2 restart omega-watches
```

#### 3. Server Failure
See [RUNBOOKS/DISASTER_RECOVERY.md](./RUNBOOKS/DISASTER_RECOVERY.md) for complete server rebuild procedure.

---

## Security

### Firewall Rules

```bash
# View current rules
ufw status numbered

# Application ports
7500/tcp - Main application
7501/tcp - Staging
7502/tcp - Green deployment

# Monitoring ports
7510/tcp - Prometheus
7511/tcp - Grafana
7512/tcp - Node Exporter
7513/tcp - Alertmanager
7514/tcp - Redis

# Web server ports
80/tcp - HTTP
443/tcp - HTTPS
```

### SSL/TLS (Future)

To enable HTTPS:
1. Obtain SSL certificate (Let's Encrypt recommended)
2. Uncomment SSL configuration in Nginx config
3. Update certificate paths
4. Reload Nginx: `systemctl reload nginx`

---

## Performance Optimization

### Nginx Caching

- **Static files**: Cached for 7 days
- **API responses**: Cached for 5 minutes
- **Cache location**: /var/cache/nginx/omega-watches
- **Max cache size**: 1GB

### Application Scaling

#### Horizontal Scaling (PM2 Cluster Mode)
```bash
# Scale to 4 instances
pm2 scale omega-watches 4

# Scale up/down
pm2 scale omega-watches +2
pm2 scale omega-watches -1
```

#### Load Balancing (Nginx)
Uncomment additional upstream servers in Nginx config:
```nginx
upstream omega_watches_backend {
    server 127.0.0.1:7500 weight=3;
    server 127.0.0.1:7501 weight=2;
    server 127.0.0.1:7502 weight=2;
}
```

---

## Troubleshooting

### Common Issues

#### Application not responding
```bash
# Check status
pm2 list

# Check logs
pm2 logs omega-watches --lines 50

# Restart
pm2 restart omega-watches

# Check health
curl http://localhost:7500/api/health
```

#### High memory usage
```bash
# Check memory
pm2 monit

# Restart with memory limit
pm2 restart omega-watches --max-memory-restart 500M
```

#### Nginx not proxying
```bash
# Check Nginx status
systemctl status nginx

# Test configuration
nginx -t

# Reload
systemctl reload nginx

# Check logs
tail -f /var/log/nginx/error.log
```

#### Monitoring stack down
```bash
cd /root/Projects/watches

# Check containers
docker-compose -f docker-compose.monitoring.yml ps

# View logs
docker-compose -f docker-compose.monitoring.yml logs

# Restart
docker-compose -f docker-compose.monitoring.yml restart
```

---

## Useful Commands

### Application Management
```bash
# PM2 commands
pm2 list                    # List all processes
pm2 logs omega-watches      # View logs
pm2 monit                   # Monitor resources
pm2 restart omega-watches   # Restart app
pm2 stop omega-watches      # Stop app
pm2 save                    # Save process list

# Health checks
curl http://localhost:7500/api/health | jq
curl http://localhost:7500/api/statistics | jq
```

### Monitoring
```bash
# Docker Compose
docker-compose -f docker-compose.monitoring.yml ps
docker-compose -f docker-compose.monitoring.yml logs -f
docker-compose -f docker-compose.monitoring.yml restart

# Prometheus queries
curl http://localhost:7510/api/v1/query?query=up

# Grafana
curl http://localhost:7511/api/health
```

### Nginx
```bash
nginx -t                    # Test configuration
systemctl reload nginx      # Reload config
systemctl restart nginx     # Restart Nginx
tail -f /var/log/nginx/omega-watches-access.log
tail -f /var/log/nginx/omega-watches-error.log
```

### System
```bash
# Resource usage
htop
free -h
df -h
netstat -tulnp | grep node

# Logs
journalctl -u nginx -f
tail -f /root/Projects/watches/logs/*.log
```

---

## CI/CD Pipeline

### GitHub Actions Workflow

File: `.github/workflows/production-ci-cd.yml`

**Triggers**:
- Push to `main` branch → Production deployment
- Push to `develop` branch → Staging deployment
- Pull requests → Tests only

**Pipeline Stages**:
1. **Test**: Run security audits and code quality checks
2. **Security**: Trivy vulnerability scanning
3. **Build**: Install dependencies and build application
4. **Deploy**: Blue-green deployment with health checks
5. **Monitor**: Post-deployment verification

**Required Secrets** (configure in GitHub):
- `SSH_USERNAME`: Server SSH username
- `SSH_PRIVATE_KEY`: SSH private key for deployment

---

## Maintenance Schedule

### Daily
- Automated backups (2:00 AM)
- Log rotation (1:00 AM)
- Health checks (every 5 minutes)

### Weekly
- Weekly backups (Sunday 3:00 AM)
- Backup verification (Monday 5:00 AM)

### Monthly
- Monthly backups (1st day, 4:00 AM)
- Review monitoring dashboards
- Check disk space usage
- Review and update dependencies

### Quarterly
- Test disaster recovery procedures
- Review and update documentation
- Security audit
- Performance optimization review

---

## Support & Documentation

### Runbooks
- [Deployment Runbook](./RUNBOOKS/DEPLOYMENT_RUNBOOK.md)
- [Disaster Recovery Plan](./RUNBOOKS/DISASTER_RECOVERY.md)

### Scripts
- `PRODUCTION_SETUP.sh` - Master setup script
- `scripts/production-deploy.sh` - Deployment automation
- `scripts/backup-automation.sh` - Backup management
- `scripts/setup-monitoring.sh` - Monitoring stack setup
- `scripts/nginx-setup.sh` - Nginx configuration
- `scripts/cron-setup.sh` - Cron job setup
- `scripts/health-monitor.sh` - Health monitoring

### Access Information
```
Server IP:      45.61.58.125
SSH:            ssh root@45.61.58.125

Application:    http://45.61.58.125:7500
Grafana:        http://45.61.58.125:7511 (admin/omega-watches-admin)
Prometheus:     http://45.61.58.125:7510
Alertmanager:   http://45.61.58.125:7513

App Directory:  /root/Projects/watches
Backups:        /root/Projects/watches/backups
Logs:           /root/Projects/watches/logs
```

---

**Last Updated**: 2025-11-17
**Version**: 1.0
**Maintained By**: DevOps Team