← back to Designer Wallcoverings

DW-Agents/dw-agents/CRITICAL_AGENTS_AUTO_RESTART.md

377 lines

# Critical DW-Agents - Auto-Restart Configuration

**Last Updated:** November 17, 2025
**Status:** ✅ Active and Configured

---

## Overview

This system ensures critical DW-Agents automatically restart on server reboot and recover from crashes without manual intervention.

---

## Critical Agents (Auto-Managed)

| Agent | Port | Purpose |
|-------|------|---------|
| **dw-control-panel** | 7200 | Main control panel for all DW-Agents |
| **dw-server-uptime** | 9888 | Server uptime monitoring and reporting |
| **dw-completed-tasks** | 9889 | Task completion tracking and history |
| **dw-in-parallel** | 9891 | Parallel process execution manager |
| **dw-skills-manager** | 9894 | Skills and capabilities management |

---

## Auto-Restart Architecture

### 1. PM2 Systemd Service
- **Service:** `pm2-root.service`
- **Status:** Enabled
- **Function:** Resurrects all PM2 processes on server boot
- **Config:** `/etc/systemd/system/pm2-root.service`

### 2. PM2 Ecosystem Configuration
- **File:** `/root/DW-Agents/ecosystem-critical-agents.config.js`
- **Features:**
  - Auto-restart on crash
  - Exponential backoff (prevents crash loops)
  - Memory limit monitoring (500MB)
  - Proper logging to files
  - 10 max restarts before giving up

### 3. Cron-Based Health Checks
- **Schedule:** Every 5 minutes
- **Script:** `/root/DW-Agents/scripts/check-critical-agents.sh`
- **Function:** Detects down agents and auto-restarts them
- **Log:** `/root/DW-Agents/logs/health-check.log`

---

## How It Works

```
Server Boots
    ↓
systemd starts pm2-root.service
    ↓
PM2 resurrects saved process list
    ↓
All agents start automatically
    ↓
Cron runs health check every 5 minutes
    ↓
If agent down → Auto-restart via ecosystem config
    ↓
Agents stay running 24/7
```

---

## Manual Operations

### Restart All Critical Agents
```bash
/root/DW-Agents/scripts/restart-critical-agents.sh
```

### Check Agent Health
```bash
/root/DW-Agents/scripts/check-critical-agents.sh
```

### View Health Check Logs
```bash
tail -f /root/DW-Agents/logs/health-check.log
```

### Check PM2 Status
```bash
pm2 list | grep "dw-"
```

### Manually Start from Ecosystem Config
```bash
cd /root/DW-Agents
pm2 start ecosystem-critical-agents.config.js
pm2 save
```

### Stop All Critical Agents
```bash
pm2 delete dw-control-panel dw-server-uptime dw-completed-tasks dw-skills-manager dw-in-parallel
pm2 save
```

---

## Verification Commands

### 1. Verify PM2 Startup Enabled
```bash
systemctl is-enabled pm2-root.service
# Should output: enabled
```

### 2. Verify Cron Job Installed
```bash
crontab -l | grep check-critical-agents
# Should show: */5 * * * * /root/DW-Agents/scripts/check-critical-agents.sh
```

### 3. Verify All Agents Running
```bash
pm2 list | grep -E "(dw-control-panel|dw-server-uptime|dw-completed-tasks|dw-skills-manager|dw-in-parallel)"
# All should show status: online
```

### 4. Test Auto-Restart (Safe Test)
```bash
# Kill one agent (it will auto-restart)
pm2 delete dw-control-panel

# Wait 10 seconds
sleep 10

# Check if it restarted (cron will catch it within 5 min)
pm2 list | grep dw-control-panel
```

---

## File Locations

### Configuration Files
```
/root/DW-Agents/ecosystem-critical-agents.config.js    # PM2 ecosystem config
/etc/systemd/system/pm2-root.service                   # Systemd service
/root/.pm2/dump.pm2                                     # PM2 saved processes
```

### Scripts
```
/root/DW-Agents/scripts/restart-critical-agents.sh     # Manual restart script
/root/DW-Agents/scripts/check-critical-agents.sh       # Health check script
/root/DW-Agents/scripts/setup-auto-restart.sh          # Setup script
```

### Logs
```
/root/DW-Agents/logs/health-check.log                  # Health check logs
/root/DW-Agents/logs/control-panel-error.log           # Control panel errors
/root/DW-Agents/logs/control-panel-out.log             # Control panel output
/root/DW-Agents/logs/server-uptime-error.log           # Server uptime errors
/root/DW-Agents/logs/server-uptime-out.log             # Server uptime output
/root/DW-Agents/logs/completed-tasks-error.log         # Completed tasks errors
/root/DW-Agents/logs/completed-tasks-out.log           # Completed tasks output
/root/DW-Agents/logs/skills-manager-error.log          # Skills manager errors
/root/DW-Agents/logs/skills-manager-out.log            # Skills manager output
/root/DW-Agents/logs/in-parallel-error.log             # In parallel errors
/root/DW-Agents/logs/in-parallel-out.log               # In parallel output
```

---

## Troubleshooting

### Problem: Agent won't stay up

**Solution:**
```bash
# Check logs for errors
pm2 logs dw-AGENT-NAME --lines 50

# Check health check log
tail -50 /root/DW-Agents/logs/health-check.log

# Manually restart with ecosystem config
cd /root/DW-Agents
pm2 delete dw-AGENT-NAME
pm2 start ecosystem-critical-agents.config.js --only dw-AGENT-NAME
pm2 save
```

### Problem: Agents don't start on boot

**Solution:**
```bash
# Re-run setup script
/root/DW-Agents/scripts/setup-auto-restart.sh

# Verify systemd service
systemctl status pm2-root.service

# Manually enable if needed
pm2 startup systemd -u root --hp /root
pm2 save
```

### Problem: Cron health check not running

**Solution:**
```bash
# Verify cron job exists
crontab -l | grep check-critical-agents

# Re-add if missing
(crontab -l 2>/dev/null; echo "*/5 * * * * /root/DW-Agents/scripts/check-critical-agents.sh") | crontab -

# Test manually
/root/DW-Agents/scripts/check-critical-agents.sh
```

### Problem: tsx not found

**Solution:**
```bash
# Install tsx globally
npm install -g tsx

# Verify installation
which tsx

# Restart agents
/root/DW-Agents/scripts/restart-critical-agents.sh
```

---

## Recovery Scenarios

### Scenario 1: Server Reboots
1. Systemd starts `pm2-root.service`
2. PM2 resurrects all saved processes
3. All agents start automatically
4. Cron health check confirms status every 5 minutes

### Scenario 2: Agent Crashes
1. PM2 detects crash
2. PM2 auto-restarts with exponential backoff
3. If agent keeps crashing (10 times), PM2 gives up
4. Cron health check (within 5 min) detects failure
5. Cron restarts agent via ecosystem config

### Scenario 3: PM2 Process Manager Crashes
1. Systemd detects PM2 service down
2. Systemd restarts `pm2-root.service`
3. PM2 resurrects all processes
4. All agents back online

### Scenario 4: Manual Stop
1. User runs `pm2 delete dw-AGENT-NAME`
2. Agent stops
3. Within 5 minutes, cron health check detects missing agent
4. Cron restarts agent automatically
5. PM2 configuration auto-saves

---

## Performance & Limits

### Auto-Restart Limits
- **Max Restarts:** 10 attempts per agent
- **Min Uptime:** 10 seconds (must run this long to count as "started")
- **Max Memory:** 500MB (restart if exceeded)
- **Restart Delay:** 1 second base + exponential backoff

### Health Check Frequency
- **Interval:** Every 5 minutes
- **Log Rotation:** Automatic (keeps last 1000 lines when >10MB)

### Systemd Service
- **Restart Policy:** `on-failure`
- **File Limits:** Unlimited
- **Process Limits:** Unlimited

---

## Testing the System

### Test 1: Crash Recovery
```bash
# Kill an agent
pm2 delete dw-control-panel

# Check if PM2 auto-restarted it
sleep 3
pm2 list | grep dw-control-panel

# If not, wait for cron (max 5 min)
sleep 300
pm2 list | grep dw-control-panel
```

### Test 2: Boot Recovery
```bash
# Save current state
pm2 save

# Simulate reboot (don't actually reboot, just test resurrection)
pm2 kill
pm2 resurrect

# Verify all agents came back
pm2 list
```

### Test 3: Health Check
```bash
# Run health check manually
/root/DW-Agents/scripts/check-critical-agents.sh

# View results
tail -20 /root/DW-Agents/logs/health-check.log
```

---

## Monitoring & Alerts

### Check Agent Status
```bash
# Quick status
pm2 list | grep "dw-"

# Detailed status
pm2 describe dw-control-panel

# Live monitoring
pm2 monit
```

### View Logs
```bash
# Real-time logs for all agents
pm2 logs

# Logs for specific agent
pm2 logs dw-control-panel

# Last 100 lines of logs
pm2 logs dw-control-panel --lines 100
```

---

## Summary

✅ **PM2 Systemd Service:** Enabled
✅ **Auto-Restart on Boot:** Configured
✅ **Crash Recovery:** Enabled
✅ **Health Checks:** Every 5 minutes
✅ **Logging:** Configured
✅ **Ecosystem Config:** Active

All critical DW-Agents will now:
1. Start automatically on server boot
2. Auto-restart on crash (up to 10 times)
3. Self-heal via cron health checks every 5 minutes
4. Log all activity for debugging

**No manual intervention required!**

---

Generated with Claude Code
https://claude.com/claude-code