← back to Designer Wallcoverings

DW-Agents/dw-agents/DEPLOYMENT-GUIDE.md

563 lines

# DW-AGENTS PERMANENT RELIABILITY SOLUTION
## Complete Deployment Guide

This guide provides step-by-step instructions for deploying the permanent reliability solution for all DW-Agents.

---

## Table of Contents
1. [Overview](#overview)
2. [Components](#components)
3. [Pre-Deployment Checklist](#pre-deployment-checklist)
4. [Deployment Steps](#deployment-steps)
5. [Verification](#verification)
6. [Monitoring](#monitoring)
7. [Troubleshooting](#troubleshooting)

---

## Overview

The permanent reliability solution consists of:
- **Enhanced PM2 Ecosystem Configuration** - Complete agent lifecycle management
- **Watchdog Service** - Advanced health monitoring with auto-recovery
- **Recovery Service** - Intelligent crash pattern detection and resolution
- **Startup Scripts** - Automated deployment and initialization
- **Cron Jobs** - Continuous monitoring and maintenance

---

## Components

### 1. PM2 Ecosystem Configuration
**File:** `/root/DW-Agents/ecosystem-complete.config.js`

Features:
- All 27+ agents configured
- Auto-restart on crash
- Memory limits (500MB per agent)
- Exponential backoff for crash loops
- Centralized logging
- Environment variable management

### 2. Watchdog Service
**File:** `/root/DW-Agents/scripts/watchdog-service.sh`

Features:
- Port monitoring with HTTP health checks
- Automatic PM2 restart on failure
- Firewall port verification
- Memory usage monitoring
- Slack notifications (optional)
- Comprehensive logging

### 3. Recovery Service
**File:** `/root/DW-Agents/scripts/agent-recovery-service.sh`

Features:
- Crash pattern detection
- Smart restart with exponential backoff
- Port conflict resolution
- Persistent failure tracking
- Automatic dependency resolution

### 4. Startup Script
**File:** `/root/DW-Agents/scripts/startup-all-agents.sh`

Features:
- Clean PM2 slate
- Environment variable loading
- All agents startup
- Firewall configuration
- Health verification
- PM2 persistence

---

## Pre-Deployment Checklist

### 1. Verify Environment File
```bash
cd /root/DW-Agents
cat .env
```

Ensure these variables are set:
- `ANTHROPIC_API_KEY`
- `SHOPIFY_STORE_DOMAIN`
- `SHOPIFY_ACCESS_TOKEN`
- `SHOPIFY_PRODUCTS_TOKEN`
- `SLACK_BOT_TOKEN`
- `GOOGLE_CLIENT_ID`
- `GOOGLE_CLIENT_SECRET`
- `GOOGLE_REFRESH_TOKEN`

### 2. Check System Dependencies
```bash
# Node.js
node --version  # Should be v18+

# PM2
pm2 --version  # Should be 5.x+

# jq (for JSON parsing)
jq --version

# netstat
which netstat

# UFW firewall
sudo ufw status
```

### 3. Ensure Directory Permissions
```bash
sudo chown -R root:root /root/DW-Agents
chmod +x /root/DW-Agents/scripts/*.sh
```

---

## Deployment Steps

### Step 1: Backup Current Configuration
```bash
cd /root/DW-Agents

# Backup current PM2 processes
pm2 save
cp /root/.pm2/dump.pm2 /root/.pm2/dump.pm2.backup.$(date +%Y%m%d)

# Backup crontab
crontab -l > /root/crontab.backup.$(date +%Y%m%d)
```

### Step 2: Make Scripts Executable
```bash
chmod +x /root/DW-Agents/scripts/startup-all-agents.sh
chmod +x /root/DW-Agents/scripts/watchdog-service.sh
chmod +x /root/DW-Agents/scripts/agent-recovery-service.sh
```

### Step 3: Deploy All Agents
```bash
cd /root/DW-Agents
sudo ./scripts/startup-all-agents.sh
```

This script will:
1. Load environment variables
2. Stop all existing PM2 processes
3. Start all agents from ecosystem config
4. Open required firewall ports
5. Verify all agents are running
6. Save PM2 configuration
7. Configure PM2 auto-start on boot

**Expected Output:**
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ All DW-Agents Started Successfully
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Next steps:
  • View agent status:  pm2 list
  • View logs:          pm2 logs
  • Monitor agents:     pm2 monit
  • Access Master Hub:  http://45.61.58.125:9893
```

### Step 4: Configure Cron Jobs
```bash
# Edit crontab
crontab -e

# Add these lines (if not already present):

# Watchdog service - runs every 2 minutes
*/2 * * * * /root/DW-Agents/scripts/watchdog-service.sh >> /root/DW-Agents/logs/watchdog-cron.log 2>&1

# Recovery service - runs every minute
* * * * * /root/DW-Agents/scripts/agent-recovery-service.sh >> /root/DW-Agents/logs/recovery-cron.log 2>&1

# Daily log cleanup - runs at 2 AM
0 2 * * * find /root/DW-Agents/logs -name "*.log" -mtime +7 -delete
```

Save and exit. Verify cron jobs:
```bash
crontab -l
```

### Step 5: Ensure PM2 Starts on Boot
```bash
pm2 startup systemd -u root --hp /root
pm2 save
```

**Expected Output:**
```
[PM2] Init System found: systemd
[PM2] Writing init configuration in /etc/systemd/system/pm2-root.service
[PM2] [v] Command successfully executed.
```

Verify systemd service:
```bash
systemctl status pm2-root
```

### Step 6: Enable Systemd Service (if not already enabled)
```bash
sudo systemctl enable pm2-root
sudo systemctl start pm2-root
```

---

## Verification

### 1. Verify All Agents Are Running
```bash
pm2 list
```

**Expected:** All agents should show status `online` with low restart counts.

### 2. Check Port Accessibility
```bash
# Test a few key ports
curl -I http://localhost:9893  # Master Hub
curl -I http://localhost:9881  # Marketing
curl -I http://localhost:7238  # Shopify Store
curl -I http://localhost:9900  # Task Orchestrator
```

**Expected:** HTTP 200 or 302 responses.

### 3. Verify External Access
```bash
# From another machine or browser
curl -I http://45.61.58.125:9893
```

### 4. Check Firewall Rules
```bash
sudo ufw status numbered | grep -E "9893|9881|7238|9900"
```

**Expected:** All agent ports should be ALLOW.

### 5. Test Watchdog Service
```bash
# Run watchdog manually
sudo /root/DW-Agents/scripts/watchdog-service.sh

# Check log
tail -n 50 /root/DW-Agents/logs/watchdog.log
```

**Expected:** Should show health check summary with all agents healthy.

### 6. Test Recovery Service
```bash
# Stop one agent
pm2 stop dw-marketing

# Wait 1 minute, then check
pm2 list

# The recovery service should auto-restart it
tail -n 50 /root/DW-Agents/logs/recovery.log
```

### 7. Test Auto-Restart on Crash
```bash
# Kill an agent process
pm2 stop dw-accounting

# PM2 should auto-restart it
sleep 5
pm2 list
```

**Expected:** Agent should be back online with restart count incremented.

---

## Monitoring

### Real-Time Monitoring
```bash
# PM2 monitoring dashboard
pm2 monit

# View all logs
pm2 logs

# View specific agent logs
pm2 logs dw-master-hub

# View watchdog logs
tail -f /root/DW-Agents/logs/watchdog.log

# View recovery logs
tail -f /root/DW-Agents/logs/recovery.log
```

### Check Agent Health
```bash
# Quick status
pm2 jlist | jq '.[] | {name: .name, status: .pm2_env.status, restarts: .pm2_env.restart_time, memory: .monit.memory}'

# Memory usage
pm2 jlist | jq '.[] | {name: .name, memory_mb: (.monit.memory / 1024 / 1024 | floor)}'

# Restart counts
pm2 jlist | jq '.[] | {name: .name, restarts: .pm2_env.restart_time}' | grep -v '"restarts": 0'
```

### Access Dashboards
- **Master Hub:** http://45.61.58.125:9893
- **CEO Dashboard:** http://45.61.58.125:7120
- **CFO Dashboard:** http://45.61.58.125:7121
- **COO Dashboard:** http://45.61.58.125:7122
- **All Agents Viewer:** http://45.61.58.125:9111
- **Task Orchestrator:** http://45.61.58.125:9900

---

## Troubleshooting

### Problem: Agent Won't Start

**Diagnosis:**
```bash
# Check PM2 logs
pm2 logs dw-AGENT-NAME --lines 100

# Check error logs
tail -n 100 /root/DW-Agents/logs/AGENT-NAME-error.log

# Check if port is occupied
lsof -ti:PORT
```

**Solution:**
```bash
# Kill process on port
lsof -ti:PORT | xargs kill -9

# Restart agent
pm2 restart dw-AGENT-NAME

# Or restart all
pm2 restart all
```

### Problem: Port Not Accessible Externally

**Diagnosis:**
```bash
# Check if port is listening
netstat -tlnp | grep :PORT

# Check firewall
sudo ufw status | grep PORT
```

**Solution:**
```bash
# Open firewall port
sudo ufw allow PORT/tcp

# Reload firewall
sudo ufw reload
```

### Problem: Agent Keeps Crashing

**Diagnosis:**
```bash
# Check crash database
cat /root/DW-Agents/.crash-database.json

# Check restart count
pm2 jlist | jq '.[] | select(.name == "dw-AGENT-NAME") | .pm2_env.restart_time'

# Check memory usage
pm2 jlist | jq '.[] | select(.name == "dw-AGENT-NAME") | .monit'
```

**Solution:**
```bash
# Check for port conflicts
lsof -ti:PORT

# Increase memory limit (edit ecosystem config)
# max_memory_restart: '1G'

# Check dependencies
cd /root/DW-Agents
npm install

# Reload ecosystem
pm2 reload ecosystem-complete.config.js
```

### Problem: Watchdog Not Running

**Diagnosis:**
```bash
# Check cron jobs
crontab -l | grep watchdog

# Check cron log
tail -n 50 /root/DW-Agents/logs/watchdog-cron.log

# Test manually
sudo /root/DW-Agents/scripts/watchdog-service.sh
```

**Solution:**
```bash
# Make executable
chmod +x /root/DW-Agents/scripts/watchdog-service.sh

# Re-add to crontab
crontab -e

# Restart cron service
sudo systemctl restart cron
```

### Problem: PM2 Not Starting on Boot

**Diagnosis:**
```bash
# Check systemd service
systemctl status pm2-root

# Check PM2 dump file
cat /root/.pm2/dump.pm2
```

**Solution:**
```bash
# Reconfigure startup
pm2 unstartup
pm2 startup systemd -u root --hp /root

# Save current processes
pm2 save

# Enable and start service
sudo systemctl enable pm2-root
sudo systemctl start pm2-root

# Verify
systemctl status pm2-root
```

### Problem: High Memory Usage

**Diagnosis:**
```bash
# Check memory usage by agent
pm2 jlist | jq '.[] | {name: .name, memory_mb: (.monit.memory / 1024 / 1024 | floor)}' | sort -k2 -n

# Check system memory
free -m
```

**Solution:**
```bash
# Restart memory-heavy agents
pm2 restart dw-AGENT-NAME

# Reduce memory limits in ecosystem config
# max_memory_restart: '300M'

# Reload configuration
pm2 delete all
pm2 start ecosystem-complete.config.js
pm2 save
```

---

## Maintenance

### Daily Tasks
- Check PM2 status: `pm2 list`
- Review watchdog logs: `tail /root/DW-Agents/logs/watchdog.log`
- Check for persistent failures: `cat /root/DW-Agents/.crash-database.json`

### Weekly Tasks
- Review agent restart counts: `pm2 jlist | jq '.[] | {name, restarts: .pm2_env.restart_time}'`
- Clean old logs: `find /root/DW-Agents/logs -name "*.log" -mtime +7 -delete`
- Update dependencies: `cd /root/DW-Agents && npm update`

### Monthly Tasks
- Review memory usage trends
- Optimize slow-starting agents
- Update PM2: `npm install -g pm2@latest`
- Backup PM2 configuration: `pm2 save`

---

## Emergency Procedures

### Complete System Restart
```bash
# Stop all agents
pm2 stop all

# Clear PM2 processes
pm2 delete all

# Full restart
cd /root/DW-Agents
sudo ./scripts/startup-all-agents.sh
```

### Disaster Recovery
```bash
# Restore from backup
pm2 resurrect /root/.pm2/dump.pm2.backup.YYYYMMDD

# Or full redeploy
cd /root/DW-Agents
sudo ./scripts/startup-all-agents.sh
```

---

## Success Metrics

Your deployment is successful when:
1. ✅ All agents show `online` status in `pm2 list`
2. ✅ All agent ports are accessible via HTTP
3. ✅ Watchdog service runs every 2 minutes without errors
4. ✅ Recovery service runs every minute without errors
5. ✅ PM2 systemd service is active and enabled
6. ✅ Agent restart counts remain stable (< 5 per hour)
7. ✅ All dashboards are accessible via browser
8. ✅ Agents survive server reboot

---

## Support

For issues or questions:
1. Check logs: `/root/DW-Agents/logs/`
2. Review PM2 status: `pm2 list` and `pm2 logs`
3. Run watchdog manually: `sudo /root/DW-Agents/scripts/watchdog-service.sh`
4. Check this guide's troubleshooting section

---

**Deployment Date:** $(date)
**Version:** 1.0.0
**Status:** Production Ready