← back to Handbag Auth Nextjs
DEVOPS_GUIDE.md
567 lines
# LUXVAULT DevOps Guide
Complete operational guide for the LUXVAULT Auction Viewer system.
## Table of Contents
1. [System Overview](#system-overview)
2. [Deployment](#deployment)
3. [Monitoring](#monitoring)
4. [Backup & Recovery](#backup--recovery)
5. [Troubleshooting](#troubleshooting)
6. [Runbook](#runbook)
---
## System Overview
### Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ LUXVAULT System │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Cron Job │ │ PM2 Process │ │
│ │ Python │ │ Node/Express│ │
│ │ Scraper │─────▶│ API Server │ │
│ │ Daily 6am │ │ Port 7500 │ │
│ └──────────────┘ └──────┬───────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ SQLite DB │ │
│ │ auctions.db │ │
│ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
```
### Components
- **Express Server**: REST API on port 7500 (PM2 managed)
- **Python Scraper**: Daily cron job at 6am UTC
- **SQLite Database**: `/root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db`
- **Monitoring**: Health check scripts
- **Backups**: Automated daily backups at 2am UTC
### URLs
- **Dashboard**: http://45.61.58.125:7500
- **API**: http://45.61.58.125:7500/api/auctions
- **Health Check**: http://45.61.58.125:7500/health
- **Metrics**: http://45.61.58.125:7500/metrics
---
## Deployment
### Quick Deployment
```bash
cd /root/Projects/handbag-auth-nextjs
./scripts/devops/deploy.sh
```
### Deployment Process
The deployment script performs:
1. ✅ Pre-deployment health checks
2. 💾 Creates backup of current version
3. 📦 Installs/updates dependencies
4. ⚙️ Validates configuration
5. 🔄 Zero-downtime PM2 reload
6. ❤️ Health verification (10 retries)
7. 🧪 API endpoint testing
8. 📊 Database connectivity check
9. 🧹 Cleanup old backups
### Manual Deployment Steps
If automatic deployment fails:
```bash
cd /root/Projects/handbag-auth-nextjs/auction-viewer
# Install dependencies
npm ci --production
# Test configuration
node -c server.js
# Reload PM2
pm2 reload auction-viewer
# Verify health
curl http://45.61.58.125:7500/health
```
### Rollback Procedure
If deployment fails, automatic rollback occurs. Manual rollback:
```bash
# Find latest backup
ls -lt /root/Projects/handbag-auth-nextjs/backups/deployments/
# Stop service
pm2 stop auction-viewer
# Restore from backup
BACKUP="/root/Projects/handbag-auth-nextjs/backups/deployments/deployment_backup_YYYYMMDD_HHMMSS"
rm -rf /root/Projects/handbag-auth-nextjs/auction-viewer
cp -r $BACKUP/auction-viewer /root/Projects/handbag-auth-nextjs/
# Restart service
pm2 restart auction-viewer
# Verify
curl http://45.61.58.125:7500/health
```
---
## Monitoring
### Health Monitoring
Automated health checks run every 5 minutes:
```bash
# Add to crontab (already configured)
*/5 * * * * /root/Projects/handbag-auth-nextjs/scripts/devops/health-monitor.sh
```
#### What It Monitors
- ✅ PM2 process status
- ✅ HTTP health endpoint
- ✅ API functionality
- ✅ Database size
- ✅ Scraper execution
- ✅ Disk space usage
#### Manual Health Check
```bash
# Run health monitor manually
/root/Projects/handbag-auth-nextjs/scripts/devops/health-monitor.sh
# Check health endpoint
curl http://45.61.58.125:7500/health | jq
# Check metrics
curl http://45.61.58.125:7500/metrics | jq
```
### Viewing Logs
```bash
# PM2 logs (real-time)
pm2 logs auction-viewer
# PM2 logs (last 100 lines)
pm2 logs auction-viewer --lines 100
# Health monitor log
tail -f /root/Projects/handbag-auth-nextjs/logs/health-monitor.log
# Scraper log
tail -f /root/Projects/handbag-auth-nextjs/data/auction-history/scraper.log
# Deployment log
tail -f /root/Projects/handbag-auth-nextjs/logs/deployment.log
# All logs
ls -lh /root/Projects/handbag-auth-nextjs/logs/
```
### Metrics Endpoints
```bash
# Health status
curl http://45.61.58.125:7500/health
# Application metrics
curl http://45.61.58.125:7500/metrics
# Database stats
curl http://45.61.58.125:7500/api/stats
# Scraper status
curl http://45.61.58.125:7500/api/scraper/status
```
### Alert Configuration
Configure Slack webhooks in `.env`:
```bash
# Edit .env file
nano /root/Projects/handbag-auth-nextjs/auction-viewer/.env
# Add Slack webhook
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
```
---
## Backup & Recovery
### Automated Backups
Daily backups run at 2am UTC:
```bash
# Cron configuration
0 2 * * * /root/Projects/handbag-auth-nextjs/scripts/devops/backup-database.sh
```
### Manual Backup
```bash
# Create immediate backup
/root/Projects/handbag-auth-nextjs/scripts/devops/backup-database.sh
# View backups
ls -lh /root/Projects/handbag-auth-nextjs/backups/
```
### Restore from Backup
```bash
# List available backups
ls -lh /root/Projects/handbag-auth-nextjs/backups/*.tar.gz
# Restore specific backup
/root/Projects/handbag-auth-nextjs/scripts/devops/restore-backup.sh \
/root/Projects/handbag-auth-nextjs/backups/luxvault_backup_20251117_020000.tar.gz
```
### Backup Strategy
- **Retention**: 30 days
- **Frequency**: Daily at 2am UTC
- **Contents**:
- SQLite database
- CSV exports
- Scraper logs
- Metadata
- **Compression**: gzip
- **Verification**: Automatic integrity checks
---
## Troubleshooting
### Service Not Responding
```bash
# Check PM2 status
pm2 status auction-viewer
# Check if port is in use
lsof -ti:7500
# Check for errors
pm2 logs auction-viewer --lines 50 --err
# Restart service
pm2 restart auction-viewer
# Full restart if needed
pm2 delete auction-viewer
pm2 start /root/Projects/handbag-auth-nextjs/scripts/devops/pm2-config.json
```
### Database Issues
```bash
# Check database integrity
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db \
"PRAGMA integrity_check;"
# Check database size
ls -lh /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db
# Check record count
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db \
"SELECT COUNT(*) FROM auctions;"
# Vacuum database (optimize)
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db \
"VACUUM;"
```
### Scraper Not Running
```bash
# Check cron job
crontab -l | grep scrape-auction-data
# Check scraper log
tail -100 /root/Projects/handbag-auth-nextjs/data/auction-history/scraper.log
# Manual scraper test
cd /root/Projects/handbag-auth-nextjs
python3 scripts/scrape-auction-data.py
# Check last scrape time
curl http://45.61.58.125:7500/api/scraper/status | jq
```
### High Memory Usage
```bash
# Check PM2 memory usage
pm2 status
# Check system memory
free -h
# Restart with memory limit
pm2 restart auction-viewer --max-memory-restart 500M
# Clear application cache
curl -X POST http://45.61.58.125:7500/api/cache/clear
```
### Port Already in Use
```bash
# Find process using port 7500
lsof -ti:7500
# Kill process
lsof -ti:7500 | xargs kill -9
# Restart PM2
pm2 restart auction-viewer
```
---
## Runbook
### Common Operations
#### Restart Service
```bash
pm2 restart auction-viewer
```
#### View Real-time Logs
```bash
pm2 logs auction-viewer --lines 100
```
#### Check Service Health
```bash
curl http://45.61.58.125:7500/health | jq
```
#### Clear Cache
```bash
curl -X POST http://45.61.58.125:7500/api/cache/clear
```
#### Deploy New Version
```bash
cd /root/Projects/handbag-auth-nextjs
./scripts/devops/deploy.sh
```
#### Create Backup
```bash
/root/Projects/handbag-auth-nextjs/scripts/devops/backup-database.sh
```
#### Rotate Logs
```bash
/root/Projects/handbag-auth-nextjs/scripts/devops/log-rotation.sh
```
### Emergency Procedures
#### Service Down Emergency
1. Check PM2 status
2. Check logs for errors
3. Attempt restart
4. If restart fails, restore from backup
5. Notify team
6. Create incident report
```bash
# Quick recovery
pm2 restart auction-viewer --update-env
# If that fails
pm2 delete auction-viewer
pm2 start /root/Projects/handbag-auth-nextjs/scripts/devops/pm2-config.json
```
#### Database Corruption
1. Stop service immediately
2. Check database integrity
3. Restore from most recent backup
4. Verify data integrity
5. Restart service
6. Monitor for issues
```bash
# Stop service
pm2 stop auction-viewer
# Check integrity
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "PRAGMA integrity_check;"
# If corrupted, restore
/root/Projects/handbag-auth-nextjs/scripts/devops/restore-backup.sh [latest_backup]
```
#### Disk Space Full
1. Check disk usage
2. Archive old logs
3. Remove old backups
4. Vacuum database
5. Monitor space usage
```bash
# Check disk space
df -h /root
# Clean logs
/root/Projects/handbag-auth-nextjs/scripts/devops/log-rotation.sh
# Remove old backups (keep 7 days)
find /root/Projects/handbag-auth-nextjs/backups -name "*.tar.gz" -mtime +7 -delete
# Vacuum database
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "VACUUM;"
```
---
## Configuration Files
### Environment Variables
Location: `/root/Projects/handbag-auth-nextjs/auction-viewer/.env`
```bash
PORT=7500
NODE_ENV=production
DB_PATH=/root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db
API_RATE_LIMIT=100
BACKUP_ENABLED=true
SLACK_WEBHOOK_URL=
```
### PM2 Configuration
Location: `/root/Projects/handbag-auth-nextjs/scripts/devops/pm2-config.json`
```json
{
"apps": [{
"name": "auction-viewer",
"script": "./server.js",
"max_memory_restart": "500M",
"autorestart": true
}]
}
```
### Cron Jobs
```bash
# View crontab
crontab -l | grep handbag-auth
# Edit crontab
crontab -e
```
Current schedule:
- **6:00 AM**: Python scraper runs
- **2:00 AM**: Database backup
- **1:00 AM**: Log rotation
- **Every 5 minutes**: Health monitoring
---
## Maintenance Tasks
### Daily
- ✅ Automated: Database backup (2am)
- ✅ Automated: Log rotation (1am)
- ✅ Automated: Scraper execution (6am)
- 👤 Manual: Review alerts log
### Weekly
- 👤 Check disk space usage
- 👤 Review error logs
- 👤 Verify backup integrity
- 👤 Test restore procedure
### Monthly
- 👤 Review and optimize database
- 👤 Update dependencies
- 👤 Security audit
- 👤 Disaster recovery drill
---
## Contact & Support
### Key Files
- **Project**: `/root/Projects/handbag-auth-nextjs`
- **Server**: `/root/Projects/handbag-auth-nextjs/auction-viewer`
- **Database**: `/root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db`
- **Logs**: `/root/Projects/handbag-auth-nextjs/logs/`
- **Backups**: `/root/Projects/handbag-auth-nextjs/backups/`
- **Scripts**: `/root/Projects/handbag-auth-nextjs/scripts/devops/`
### Quick Reference
```bash
# Service status
pm2 status auction-viewer
# Health check
curl http://45.61.58.125:7500/health
# View logs
pm2 logs auction-viewer
# Restart
pm2 restart auction-viewer
# Deploy
./scripts/devops/deploy.sh
# Backup
./scripts/devops/backup-database.sh
# Restore
./scripts/devops/restore-backup.sh [backup_file]
```
---
**Last Updated**: 2025-11-17
**Version**: 1.0