`DW-Agents/dw-agents/agent-restart-analyzer/README.md`

129 lines
# Restart Analyzer Agent

## Purpose
Automated analysis of agent restart patterns to identify stability issues, predict potential failures, and provide proactive system health insights.

## Core Responsibilities
- Monitor and analyze PM2 process restart patterns
- Identify agents with frequent restarts
- Detect crash patterns and root causes
- Predict potential system failures
- Generate stability reports
- Recommend preventive actions
- Alert on critical stability issues

## Technical Details
- **Port**: 7241
- **Main File**: restart-analyzer-agent.ts
- **Data Source**: PM2 process management logs
- **Analysis Engine**: Pattern recognition and anomaly detection

## Key Features
- **Restart Pattern Analysis**:
  - Track restart frequency per agent
  - Identify time-based restart patterns
  - Correlate restarts across agents
  - Detect cascade failure patterns

- **Root Cause Identification**:
  - Memory leak detection
  - Port conflict identification
  - Configuration issue detection
  - Dependency failure analysis
  - Error pattern correlation

- **Predictive Analytics**:
  - Failure prediction based on patterns
  - Stability trend analysis
  - Risk scoring for agents
  - Proactive alert generation

- **AI-Powered Analysis**:
  - Claude AI integration for pattern analysis
  - Intelligent root cause suggestions
  - Automated remediation recommendations
  - Learning from historical data

## Monitored Metrics
- **Restart Frequency**: Number of restarts per agent per time period
- **Uptime Percentage**: Actual uptime vs. expected uptime
- **Memory Usage Trends**: Memory consumption before crashes
- **Error Patterns**: Common errors leading to restarts
- **Restart Duration**: Time to recovery after restart
- **Cascade Failures**: Related agent failures

## Analysis Categories
1. **Normal Restarts**: Planned maintenance or updates
2. **Memory Issues**: Memory leaks or OOM errors
3. **Crash Patterns**: Unhandled exceptions or errors
4. **Configuration Issues**: Port conflicts, missing env vars
5. **Dependency Failures**: External service unavailability
6. **Resource Exhaustion**: CPU, disk, network issues

## Alert Thresholds
- **Critical**: >5 restarts in 1 hour
- **High**: >10 restarts in 24 hours
- **Medium**: >3 restarts in same hour pattern
- **Low**: Isolated restarts with no pattern

## Data Collection
- PM2 process events (start, stop, restart, crash)
- System resource metrics at restart time
- Error logs from crashed processes
- Configuration snapshots
- Dependency health status

## Integration Points
- **PM2**: Process management events
- **Log Monitor Agent (7239)**: Error log correlation
- **Server Uptime Agent (9888)**: System health data
- **Needs Attention Agent (9886)**: Critical alert routing
- **Task Orchestrator (9900)**: Remediation task creation

## Reports Generated
- **Daily Stability Report**: Overall system stability
- **Agent Health Cards**: Individual agent stability scores
- **Trend Analysis**: Historical stability trends
- **Root Cause Analysis**: Detailed failure investigations
- **Remediation Recommendations**: Actionable improvements

## Use Cases
- "Which agents are restarting most frequently?"
- "Analyze restart patterns for digital-samples agent"
- "Predict potential failures in the next 24 hours"
- "What's causing the purchasing agent to crash?"
- "Show me agents with declining stability"
- "Generate weekly stability report"

## Remediation Workflows
1. **Detection**: Identify restart pattern
2. **Analysis**: Determine root cause
3. **Alert**: Notify appropriate stakeholders
4. **Task Creation**: Generate remediation task in orchestrator
5. **Monitoring**: Track remediation effectiveness
6. **Learning**: Update pattern database

## Stability Scoring
- **Score Range**: 0-100 (100 = perfect stability)
- **Factors**:
  - Restart frequency (40%)
  - Error severity (30%)
  - Recovery time (15%)
  - Pattern consistency (15%)

## Automated Actions
- Create tasks for agents with critical stability scores
- Alert Needs Attention agent for urgent issues
- Recommend memory limit increases
- Suggest configuration changes
- Trigger preventive restarts during low-traffic periods

## Authentication
Uses shared-auth.ts SSO system

## Security
- Read-only access to PM2 process data
- Audit trail of all analyses
- Secure alert delivery
- Session-based authentication