`DW-Agents/dw-agents/agent-monitor-dashboard/README.md`

114 lines
# DW-Agents Monitor Dashboard

Real-time monitoring and health management system for all DW-Agents.

## Access

**Dashboard URL:** http://45.61.58.125:9899

## Features

### 1. Real-Time Monitoring
- Live status of all DW agents
- Auto-refresh every 10 seconds
- Overall system health score
- Color-coded alerts (Green/Yellow/Red)

### 2. Agent Categorization
- **Executive:** CEO, CFO, COO, VP-Ops agents
- **Operations:** Master Hub, Control Panel, Dashboard Monitor, Crash Loop Resolver
- **Customer Service:** Zendesk Chat, Shopify Store, Digital Samples, Needs Attention
- **Business Intelligence:** BigQuery Gov, Trend Research, Marketing, Accounting
- **Infrastructure:** Server Uptime, Log Monitor, UI Manager, Legal Team
- **Support:** Purchasing Office

### 3. Metrics Tracked
- **Status:** online, stopped, errored, launching
- **Restart Count:** Highlights when >10 (warning), >20 (critical), >50 (severe)
- **Memory Usage:** Warning at 200MB, Critical at 400MB
- **CPU Usage:** Warning at 50%, Critical at 80%
- **Uptime:** Formatted as days/hours/minutes/seconds
- **Error Logs:** Last 3 error messages for problematic agents

### 4. Quick Actions
- **Restart Individual Agent:** Click restart button on any agent card
- **Stop Agent:** Stop any running agent
- **View Logs:** Full log viewer for each agent
- **Restart Problematic:** Batch restart all agents with issues
- **Export Report:** Download JSON health report

## Thresholds

```javascript
RESTART_WARNING: 10     // Yellow alert
RESTART_CRITICAL: 20    // Red alert
RESTART_SEVERE: 50      // Severe red alert
MEMORY_WARNING: 200MB   // Yellow alert
MEMORY_CRITICAL: 400MB  // Red alert
CPU_WARNING: 50%        // Yellow alert
CPU_CRITICAL: 80%       // Red alert
```

## API Endpoints

- `GET /` - Dashboard UI
- `GET /api/stats` - Get all agent statistics
- `POST /api/restart/:name` - Restart specific agent
- `POST /api/stop/:name` - Stop specific agent
- `POST /api/delete/:name` - Delete agent from PM2
- `GET /api/logs/:name` - View agent logs
- `GET /api/export` - Export health report
- `POST /api/restart-problematic` - Restart all problematic agents

## PM2 Management

```bash
# Check status
pm2 status dw-monitor-dashboard

# View logs
pm2 logs dw-monitor-dashboard

# Restart
pm2 restart dw-monitor-dashboard

# Stop
pm2 stop dw-monitor-dashboard
```

## Current Issues Detected

As of deployment, the dashboard has identified these critical issues:
- **dw-digital-samples:** 407 restarts
- **dw-marketing:** 415 restarts
- **dw-purchasing-office:** 397 restarts
- **dw-zendesk-chat:** 376 restarts
- **dw-bigquery-gov:** 334 restarts

These agents are experiencing crash loops and need investigation.

## Health Score Calculation

The system health score (0-100) is calculated based on:
- Agent status (online vs errored/stopped)
- Restart counts
- Memory usage
- CPU usage
- Uptime duration

A score below 70 indicates system issues requiring attention.

## Auto-Recovery

The dashboard can automatically restart problematic agents using the "Restart Problematic" button, which will:
1. Identify agents with >20 restarts or errored/launching status
2. Batch restart them
3. Monitor for improvements

## Files

- `monitor-dashboard.ts` - Main TypeScript server
- `public/index.html` - Dashboard UI
- `ecosystem.config.js` - PM2 configuration
- `package.json` - Dependencies
- `tsconfig.json` - TypeScript configuration