← back to Watches

INFRASTRUCTURE.md

767 lines

# Omega Watch Price History - Infrastructure Documentation

## Production Infrastructure Overview

This document describes the complete production infrastructure, monitoring, and DevOps automation for the Omega Watch Price History application.

---

## Table of Contents
1. [Infrastructure Stack](#infrastructure-stack)
2. [CI/CD Pipeline](#cicd-pipeline)
3. [Monitoring & Logging](#monitoring--logging)
4. [Backup & Disaster Recovery](#backup--disaster-recovery)
5. [Security](#security)
6. [Scalability](#scalability)
7. [Quick Start Guide](#quick-start-guide)

---

## Infrastructure Stack

### Core Components

```
Production Server: 45.61.58.125 (Kamatera VPS)
├── Operating System: Ubuntu Linux
├── Runtime: Node.js 22.x
├── Process Manager: PM2 (cluster mode)
├── Reverse Proxy: Nginx
├── Monitoring: Winston + Custom Health Monitor
└── Backup: Automated daily/weekly/monthly backups
```

### Network Architecture

```
Internet
    │
    ▼
[Firewall - UFW]
    │
    ▼
[Nginx Reverse Proxy :80/443]
    │
    ├─ Rate Limiting
    ├─ SSL/TLS Termination
    ├─ Static File Caching
    └─ Load Balancing (ready)
    │
    ▼
[PM2 Cluster Manager]
    │
    ├─ omega-watches (main app) :7600
    │   ├─ Express HTTP Server
    │   ├─ WebSocket Server
    │   └─ In-memory Cache
    │
    └─ health-monitor (monitoring)
        ├─ Health Checks
        ├─ Alert System
        └─ Auto-recovery
```

### File System Layout

```
/root/Projects/watches/
├── server.js                      # Main application server
├── ecosystem.config.js            # PM2 configuration
├── package.json                   # Dependencies & scripts
├── .env                          # Environment variables (gitignored)
├── .env.example                  # Environment template
├── data/
│   ├── watches.json              # Main database
│   └── backups/                  # Local backups
│       ├── daily/                # Daily backups (30 days)
│       ├── weekly/               # Weekly backups (90 days)
│       └── monthly/              # Monthly backups (365 days)
├── dist/                         # Built frontend (production)
├── logs/                         # Application logs
│   ├── combined-YYYY-MM-DD.log   # All logs
│   ├── error-YYYY-MM-DD.log      # Errors only
│   ├── access-YYYY-MM-DD.log     # HTTP access
│   ├── pm2-error.log             # PM2 errors
│   └── pm2-out.log               # PM2 output
├── monitoring/
│   ├── logger.js                 # Winston logger config
│   └── health-monitor.js         # Health monitoring system
├── scripts/
│   ├── deploy.sh                 # Deployment automation
│   ├── backup-system.sh          # Backup automation
│   ├── setup-nginx.sh            # Nginx setup
│   └── setup-cron-jobs.sh        # Cron job configuration
├── nginx/
│   └── omega-watches.conf        # Nginx configuration
└── .github/
    └── workflows/
        └── ci-cd-production.yml  # GitHub Actions pipeline
```

---

## CI/CD Pipeline

### GitHub Actions Workflow

The CI/CD pipeline automatically tests, builds, and deploys code on every push to the main branch.

#### Pipeline Stages

1. **Test & Quality** (runs on every push/PR)
   - Security audit (`npm audit`)
   - Code quality checks
   - API endpoint testing
   - Frontend build verification

2. **Build** (after tests pass)
   - Frontend compilation
   - Asset optimization
   - Artifact creation

3. **Deploy** (on main branch only)
   - Pre-deployment backup
   - File sync to production server
   - Dependency installation
   - PM2 process reload
   - Health check verification

4. **Rollback** (on deployment failure)
   - Automatic rollback triggered on health check failure
   - Restore from latest backup
   - Alert team of failure

#### Setup GitHub Actions

```bash
# 1. Create GitHub repository
git init
git add .
git commit -m "Initial commit"
git remote add origin git@github.com:username/omega-watches.git
git push -u origin main

# 2. Configure GitHub Secrets
# Go to: GitHub Repo → Settings → Secrets → Actions
# Add secret: SSH_PRIVATE_KEY (your server's SSH private key)

# 3. Generate SSH key pair on server
ssh-keygen -t ed25519 -C "github-actions@omega-watches"
# Add public key to ~/.ssh/authorized_keys
# Add private key to GitHub secrets

# 4. Test pipeline
git push origin main
# Check: GitHub → Actions tab
```

### Manual Deployment

```bash
# Standard deployment (includes backup + rollback on failure)
npm run deploy

# Quick deployment (no backup)
pm2 reload omega-watches

# Rollback to previous version
npm run rollback
```

---

## Monitoring & Logging

### Real-time Monitoring

#### PM2 Monitoring
```bash
# Live monitoring dashboard
pm2 monit

# Service status
pm2 status

# Detailed process info
pm2 show omega-watches

# Live logs
pm2 logs omega-watches --lines 100 -f
```

#### Health Monitoring System

The application includes a custom health monitoring system that:
- Checks service health every 5 minutes
- Monitors response times, memory, and error rates
- Sends alerts to Slack (if configured)
- Attempts auto-recovery on repeated failures
- Tracks metrics history for 24 hours

```bash
# View health monitor logs
pm2 logs omega-health-monitor

# Check current health status
curl http://localhost:7600/api/health | jq

# View historical metrics
cat logs/metrics.json | jq '.history[-10:]'
```

### Logging Infrastructure

#### Winston Logger Features
- **Structured Logging**: JSON format with timestamps
- **Log Rotation**: Daily rotation with 30-day retention
- **Multiple Transports**: Console, file, error-only
- **Performance Tracking**: Automatic slow query logging
- **Log Levels**: error, warn, info, http, debug

#### Log Files

| Log File | Purpose | Retention | Location |
|----------|---------|-----------|----------|
| `combined-YYYY-MM-DD.log` | All application logs | 30 days | `logs/` |
| `error-YYYY-MM-DD.log` | Errors only | 30 days | `logs/` |
| `access-YYYY-MM-DD.log` | HTTP requests | 14 days | `logs/` |
| `pm2-error.log` | PM2 errors | Manual cleanup | `logs/` |
| `deploy_TIMESTAMP.log` | Deployment logs | Permanent | `logs/` |

#### Viewing Logs

```bash
# All logs (last 100 lines)
tail -100 logs/combined-$(date +%Y-%m-%d).log

# Errors only (follow mode)
tail -f logs/error-$(date +%Y-%m-%d).log

# Search for specific term
grep "ERROR" logs/combined-*.log

# Analyze response times
grep "SLOW REQUEST" logs/combined-*.log | tail -20

# View deployment history
cat logs/deployments.log
```

### Alert Configuration

Configure alerts in `.env`:

```bash
# Slack webhook for alerts
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00/B00/XX

# Email alerts
EMAIL_ALERTS_TO=admin@example.com

# Alert thresholds
MEMORY_THRESHOLD_PERCENT=85
CPU_THRESHOLD_PERCENT=80
RESPONSE_TIME_THRESHOLD_MS=3000
ERROR_RATE_THRESHOLD=5
```

Alert triggers:
- Service health check fails 3 consecutive times
- Response time > 3 seconds
- Memory usage > 85%
- Error rate > 5 errors/minute

---

## Backup & Disaster Recovery

### Automated Backup System

#### Backup Schedule

| Type | Frequency | Retention | Command |
|------|-----------|-----------|---------|
| **Daily** | 2:00 AM (cron) | 30 days | `npm run backup` |
| **Weekly** | Sunday 3:00 AM | 90 days | `npm run backup:weekly` |
| **Monthly** | 1st, 4:00 AM | 365 days | `npm run backup:monthly` |

#### What Gets Backed Up

1. **Database**: All JSON data files
2. **Logs**: Application logs (last 30 days)
3. **Configuration**: package.json, .env, ecosystem.config.js
4. **Checksums**: SHA-256 hashes for integrity verification

#### Backup Commands

```bash
# Create manual backup
npm run backup

# List all backups
npm run backup:list

# View backup statistics
npm run backup:stats

# Verify backup integrity
./scripts/backup-system.sh verify /path/to/backup.tar.gz

# Restore from specific backup
./scripts/backup-system.sh restore /path/to/backup.tar.gz

# Restore latest backup
npm run restore-latest-backup
```

### Disaster Recovery Procedure

#### Complete System Recovery

```bash
# 1. Provision new server with same specs
# 2. Install base dependencies
apt-get update && apt-get install -y nodejs npm nginx git

# 3. Install PM2 globally
npm install -g pm2

# 4. Clone project or restore from backup
cd /root/Projects
git clone <repository> watches
# OR
# Copy from backup/S3

# 5. Restore data
cd watches
./scripts/backup-system.sh restore /path/to/latest/backup.tar.gz

# 6. Configure environment
cp .env.example .env
# Edit .env with production values

# 7. Install dependencies
npm ci --production

# 8. Build frontend
npm run build

# 9. Setup infrastructure
npm run setup:nginx
npm run setup:cron

# 10. Start services
pm2 start ecosystem.config.js --env production
pm2 save
pm2 startup  # Configure auto-start

# 11. Verify
curl http://localhost:7600/api/health
curl http://45.61.58.125/api/health
```

#### Recovery Time Objectives (RTO)

- **Application Recovery**: < 30 minutes
- **Data Recovery**: < 15 minutes (from local backup)
- **Complete Infrastructure**: < 2 hours

#### Recovery Point Objectives (RPO)

- **Data Loss**: < 24 hours (daily backups)
- **Critical Changes**: < 1 hour (if deployed immediately)

### Off-site Backup (S3)

Configure AWS S3 for off-site backup storage:

```bash
# Install AWS CLI
apt-get install -y awscli

# Configure AWS credentials
aws configure

# Set S3 bucket in .env
S3_BACKUP_BUCKET=my-omega-watches-backups
AWS_REGION=us-east-1

# Backups will now automatically upload to S3
npm run backup
```

---

## Security

### Application Security

#### Implemented Security Measures

1. **HTTP Security Headers** (Helmet.js)
   - X-Frame-Options: SAMEORIGIN
   - X-Content-Type-Options: nosniff
   - X-XSS-Protection: 1; mode=block

2. **Rate Limiting** (Nginx + Express)
   - API endpoints: 100 requests/minute
   - General endpoints: 200 requests/minute
   - Connection limit: 10 per IP

3. **Input Validation**
   - Express Validator for API inputs
   - XSS clean for user inputs
   - MongoDB sanitization (if migrated)

4. **CORS Configuration**
   - Configured allowed origins
   - Credential handling

5. **Process Isolation**
   - PM2 cluster mode
   - Non-root user execution (recommended)

### Network Security

```bash
# UFW Firewall Configuration
sudo ufw status

# Essential ports only
sudo ufw allow 22/tcp    # SSH
sudo ufw allow 80/tcp    # HTTP
sudo ufw allow 443/tcp   # HTTPS (when SSL configured)
sudo ufw allow 7600/tcp  # Application (direct access)

# Deny all other incoming
sudo ufw default deny incoming
sudo ufw default allow outgoing
```

### SSL/TLS Configuration

```bash
# Install Certbot for Let's Encrypt
apt-get install -y certbot python3-certbot-nginx

# Obtain SSL certificate
certbot --nginx -d omega-watches.yourdomain.com

# Auto-renewal (already configured by certbot)
certbot renew --dry-run

# Update Nginx config for HTTPS
# Uncomment HTTPS server block in nginx/omega-watches.conf
```

### Secrets Management

```bash
# Never commit .env to git
# Use .env.example as template

# Required secrets in .env:
JWT_SECRET=<generate-with: openssl rand -base64 32>
DATABASE_PASSWORD=<strong-password>
REDIS_PASSWORD=<strong-password>
```

### Security Audit

```bash
# NPM audit
npm audit
npm audit fix

# Check for vulnerabilities
npm audit --production --audit-level=high

# Update dependencies
npm update
npm outdated
```

---

## Scalability

### Current Capacity

- **Concurrent Users**: ~1,000
- **Requests/second**: ~100
- **Memory**: 400MB (single instance)
- **CPU**: 60% peak usage

### Horizontal Scaling

#### Multi-instance Setup (PM2 Cluster)

```javascript
// ecosystem.config.js
{
  instances: 4,  // or "max" for CPU count
  exec_mode: "cluster",
  max_memory_restart: "400M"
}
```

```bash
# Scale up
pm2 scale omega-watches +2

# Scale down
pm2 scale omega-watches -1

# Set specific count
pm2 scale omega-watches 4
```

#### Multi-server Setup (Nginx Load Balancing)

```nginx
# nginx/omega-watches.conf
upstream omega_watches_backend {
    least_conn;  # Load balancing algorithm

    server 127.0.0.1:7600 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:7601 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:7602 max_fails=3 fail_timeout=30s;

    keepalive 32;
}
```

### Vertical Scaling

```bash
# Upgrade server resources
# - Increase RAM
# - Add CPU cores
# - Upgrade to SSD storage

# Adjust PM2 configuration
pm2 restart omega-watches --max-memory-restart 800M
pm2 scale omega-watches max  # Use all CPU cores
```

### Performance Optimization

#### Caching Strategy

```javascript
// Implement Redis caching
// Install: npm install ioredis

// Configure in .env
REDIS_URL=redis://localhost:6379

// Cache expensive queries
// - Watch list (5 min TTL)
// - Statistics (5 min TTL)
// - Search results (10 min TTL)
```

#### Database Optimization (Future)

```bash
# Migrate from JSON to PostgreSQL for better performance

# Benefits:
# - Indexed queries
# - Concurrent access
# - ACID transactions
# - Better caching
# - Replication support
```

### CDN Integration

```bash
# Use CDN for static assets
# - CloudFlare (free tier)
# - AWS CloudFront
# - Fastly

# Update Nginx to set proper cache headers
# Already configured in nginx/omega-watches.conf
```

---

## Quick Start Guide

### Initial Setup (First Time)

```bash
# 1. Clone repository
cd /root/Projects
git clone <repository-url> watches
cd watches

# 2. Install dependencies
npm ci --production

# 3. Configure environment
cp .env.example .env
vim .env  # Edit with your values

# 4. Build frontend
npm run build

# 5. Setup Nginx reverse proxy
npm run setup:nginx

# 6. Setup automated backups
npm run setup:cron

# 7. Start application
pm2 start ecosystem.config.js --env production
pm2 save
pm2 startup  # Enable auto-start on reboot

# 8. Verify everything works
npm run health-check
curl http://45.61.58.125/api/health

# 9. Create first backup
npm run backup

# 10. Monitor
pm2 monit
```

### Daily Operations

```bash
# View service status
pm2 status

# View logs
npm run logs

# Create manual backup
npm run backup

# Deploy updates
npm run deploy

# Check health
npm run health-check
```

### Common Tasks

```bash
# Restart service
pm2 restart omega-watches

# View detailed metrics
pm2 show omega-watches

# Clear application cache
curl -X POST http://localhost:7600/api/admin/clear-cache

# List backups
npm run backup:list

# Rotate logs manually
find logs -name "*.log" -mtime +30 -delete
```

---

## Monitoring Checklist

### Daily
- [ ] Check PM2 status
- [ ] Review error logs
- [ ] Verify backup completed
- [ ] Check disk space (`df -h`)

### Weekly
- [ ] Review performance metrics
- [ ] Check security logs
- [ ] Update dependencies (if needed)
- [ ] Test backup restore
- [ ] Review alert history

### Monthly
- [ ] Full system backup verification
- [ ] Security audit (`npm audit`)
- [ ] Review and optimize queries
- [ ] Check log rotation
- [ ] Update documentation

---

## Support and Troubleshooting

### Health Check Fails

```bash
# Check service
pm2 status
pm2 logs omega-watches --err --lines 50

# Check port
netstat -tuln | grep 7600

# Restart if needed
pm2 restart omega-watches

# Rollback if broken
npm run rollback
```

### High Memory Usage

```bash
# Check current usage
pm2 show omega-watches

# Clear cache
curl -X POST http://localhost:7600/api/admin/clear-cache

# Restart with memory limit
pm2 restart omega-watches --max-memory-restart 400M
```

### Backup Failed

```bash
# Check disk space
df -h

# Check backup logs
cat logs/backup-cron.log

# Manual backup
npm run backup

# Verify backup
./scripts/backup-system.sh verify /path/to/backup.tar.gz
```

---

## Additional Resources

- **Deployment Runbook**: See `DEPLOYMENT_RUNBOOK.md` for detailed procedures
- **API Documentation**: http://45.61.58.125:7600/api/docs
- **PM2 Documentation**: https://pm2.keymetrics.io/
- **Nginx Documentation**: https://nginx.org/en/docs/

---

## Maintenance Windows

**Scheduled Maintenance**: First Sunday of each month, 2:00 AM - 4:00 AM UTC

**Emergency Maintenance**: As needed with 15-minute notice

---

**Document Version**: 1.0
**Last Updated**: 2024-11-17
**Next Review**: 2024-12-17