← back to Designer Wallcoverings

DW-Agents/dw-agents/archives/README.md

389 lines

# Archives Directory

## Purpose
Long-term storage for historical data, old reports, backups, and archived agent data that needs to be retained but is not actively used.

## Directory Structure
```
/root/DW-Agents/archives/
├── highlights/              # Archived daily highlights
│   └── YYYY-MM-DD/         # Date-based organization
├── reports/                 # Archived reports
│   ├── daily/
│   ├── weekly/
│   └── monthly/
├── logs/                    # Archived log files
│   └── YYYY-MM/            # Month-based organization
├── databases/               # Database snapshots
│   └── tasks-database-TIMESTAMP.json
├── backups/                 # System backups
│   ├── agents/             # Agent configuration backups
│   ├── data/               # Data file backups
│   └── configs/            # Configuration backups
└── versions/                # Historical code versions
```

## Archive Types

### Daily Highlights Archives

**Location**: `archives/highlights/YYYY-MM-DD/`
**Retention**: 2 years
**Format**: Markdown + JSON

**Archived Daily**:
- Today's highlights summaries
- Daily operational reports
- Key metrics snapshots

**Archive Process**:
```bash
# Automated via archive-highlights.sh (runs daily at 12:01 AM)
/root/DW-Agents/archive-highlights.sh
```

**Structure**:
```
archives/highlights/
├── 2025-11-10/
│   ├── highlights.md
│   ├── highlights.json
│   └── metrics.json
├── 2025-11-09/
└── 2025-11-08/
```

### Report Archives

**Location**: `archives/reports/`
**Retention**:
- Daily: 90 days active, 2 years archived
- Weekly: 1 year active, 5 years archived
- Monthly: 3 years active, permanent archived
- Annual: Permanent retention

**Compression**:
- After 90 days: gzip compression
- After 1 year: tar.gz archives by quarter
- After 3 years: deep archive storage

**Example**:
```
archives/reports/
├── daily/
│   ├── 2025-Q3.tar.gz
│   └── 2025-Q2.tar.gz
├── weekly/
│   └── 2024-weekly-reports.tar.gz
└── monthly/
    └── 2023-monthly-reports.tar.gz
```

### Log Archives

**Location**: `archives/logs/YYYY-MM/`
**Retention**: 90 days
**Format**: gzip compressed log files

**Archived When**:
- Log rotation (daily)
- File size exceeds 10MB
- Agent restart/redeployment

**Structure**:
```
archives/logs/
├── 2025-11/
│   ├── digital-samples-2025-11-01.log.gz
│   ├── digital-samples-2025-11-02.log.gz
│   ├── purchasing-2025-11-01.log.gz
│   └── ...
└── 2025-10/
    └── ...
```

### Database Archives

**Location**: `archives/databases/`
**Retention**: 30 daily, 12 weekly, 24 monthly
**Format**: JSON (compressed)

**Backup Schedule**:
- **Hourly**: tasks-database.json (rotating 24-hour window)
- **Daily**: End of day snapshot (30 days)
- **Weekly**: Sunday snapshot (12 weeks)
- **Monthly**: First of month (24 months)

**Naming Convention**:
```
tasks-database-daily-YYYY-MM-DD.json.gz
tasks-database-weekly-YYYY-WW.json.gz
tasks-database-monthly-YYYY-MM.json.gz
```

**Example**:
```
archives/databases/
├── tasks-database-daily-2025-11-10.json.gz
├── tasks-database-daily-2025-11-09.json.gz
├── tasks-database-weekly-2025-45.json.gz
└── tasks-database-monthly-2025-11.json.gz
```

### Configuration Backups

**Location**: `archives/backups/configs/`
**Retention**: All significant changes (permanent)

**Backed Up**:
- ecosystem.config.js changes
- .env file changes (credentials redacted)
- Agent configuration changes
- Authentication changes

**Naming Convention**:
```
config-name-TIMESTAMP.backup
ecosystem.config.js.backup-1762549988
.env.backup-1762615744
```

### Agent Backups

**Location**: `archives/backups/agents/`
**Retention**: Before major updates

**Backed Up**:
- Agent TypeScript files before updates
- Memory files before reset
- Configuration files

**Example**:
```
archives/backups/agents/
├── accounting-agent.ts.backup-1762549988
├── digital-samples-agent.ts.backup-1762615744
└── marketing-agent.ts.backup-1762617052
```

## Archive Management

### Automated Archival

**Daily Archival (12:01 AM)**:
```bash
# Archive highlights
/root/DW-Agents/archive-highlights.sh

# Rotate logs
pm2 flush

# Backup tasks database
cp tasks-database.json archives/databases/tasks-database-daily-$(date +%Y-%m-%d).json
gzip archives/databases/tasks-database-daily-$(date +%Y-%m-%d).json
```

**Weekly Archival (Sunday 2 AM)**:
```bash
# Weekly database snapshot
cp tasks-database.json archives/databases/tasks-database-weekly-$(date +%Y-%W).json.gz

# Archive logs older than 30 days
find logs/ -name "*.log" -mtime +30 -exec gzip {} \; -exec mv {}.gz archives/logs/ \;
```

**Monthly Archival (1st of month 3 AM)**:
```bash
# Monthly database snapshot
cp tasks-database.json archives/databases/tasks-database-monthly-$(date +%Y-%m).json.gz

# Compress old reports
tar -czf archives/reports/monthly/$(date -d "last month" +%Y-%m)-reports.tar.gz reports/monthly/$(date -d "last month" +%Y-%m)*.md
```

### Manual Archival

**Archive Specific Agent**:
```bash
cd /root/DW-Agents
cp agent-purchasing-office/purchasing-office-agent.ts archives/backups/agents/purchasing-office-agent-$(date +%s).ts.backup
```

**Archive Reports**:
```bash
tar -czf archives/reports/Q3-2025-reports.tar.gz reports/daily/2025-0[789]* reports/weekly/2025-W[23][0-9]* reports/monthly/2025-0[789]*
```

**Archive Database**:
```bash
cp tasks-database.json archives/databases/tasks-database-manual-$(date +%Y-%m-%d-%H%M%S).json
gzip archives/databases/tasks-database-manual-*.json
```

## Data Retention Policy

### By Data Type

| Data Type | Active Retention | Archive Retention | Total |
|-----------|-----------------|-------------------|--------|
| Daily Highlights | 7 days | 2 years | 2 years |
| Daily Reports | 90 days | 2 years | ~2.25 years |
| Weekly Reports | 1 year | 5 years | 6 years |
| Monthly Reports | 3 years | Permanent | Permanent |
| Logs | 30 days | 90 days | 120 days |
| Database Snapshots | 30 daily | 12 weeks, 24 months | Variable |
| Config Backups | All | All | Permanent |

### Cleanup Schedule

**Daily Cleanup (3 AM)**:
```bash
# Remove highlights older than 2 years
find archives/highlights/ -type d -mtime +730 -exec rm -rf {} \;

# Remove daily logs older than 120 days
find archives/logs/ -name "*.log.gz" -mtime +120 -delete
```

**Weekly Cleanup (Sunday 4 AM)**:
```bash
# Remove old daily database backups (keep 30)
ls -t archives/databases/tasks-database-daily-* | tail -n +31 | xargs rm -f
```

**Monthly Cleanup (1st 5 AM)**:
```bash
# Remove weekly backups older than 12 weeks
ls -t archives/databases/tasks-database-weekly-* | tail -n +13 | xargs rm -f
```

## Restoration Procedures

### Restore Daily Highlights
```bash
# Restore specific date
cp archives/highlights/2025-11-05/highlights.json data/todays-real-highlights.json
```

### Restore Database
```bash
# Stop all agents
pm2 stop all

# Backup current database
cp tasks-database.json tasks-database.json.pre-restore-$(date +%s)

# Restore from archive
gunzip -c archives/databases/tasks-database-daily-2025-11-09.json.gz > tasks-database.json

# Restart agents
pm2 start all
```

### Restore Agent Configuration
```bash
# Find backup
ls -lh archives/backups/agents/purchasing-office-agent*

# Restore
cp archives/backups/agents/purchasing-office-agent-1762615744.ts.backup agent-purchasing-office/purchasing-office-agent.ts

# Restart agent
pm2 restart purchasing
```

### Restore Logs
```bash
# Extract archived log
gunzip -c archives/logs/2025-10/digital-samples-2025-10-15.log.gz > logs/digital-samples-restored.log
```

## Storage Management

### Current Usage
```bash
# Check archive directory size
du -sh /root/DW-Agents/archives

# Breakdown by subdirectory
du -h --max-depth=1 /root/DW-Agents/archives
```

### Storage Limits
- **Total Archive Space**: 10GB allocated
- **Alerts**:
  - Warning: 7GB used (70%)
  - Critical: 9GB used (90%)
  - Auto-cleanup: >9.5GB

### Compression Ratios
- **Logs**: ~90% compression (10:1 ratio)
- **JSON Databases**: ~80% compression (5:1 ratio)
- **Markdown Reports**: ~60% compression (2.5:1 ratio)

## Access Control

### Permissions
```bash
# Archive directory
drwxr-x--- 2 root dw-agents 4096 /root/DW-Agents/archives

# Files
-rw-r----- 1 root dw-agents [size] archive-file.gz
```

### Access Rights
- **Root**: Full read/write access
- **dw-agents group**: Read-only access
- **Other**: No access

## Disaster Recovery

### Recovery Point Objectives (RPO)
- **Task Database**: 1 hour (hourly backups)
- **Agent Configurations**: Last change (versioned)
- **Reports**: 1 day (daily archival)

### Recovery Time Objectives (RTO)
- **Critical Data**: < 15 minutes
- **Standard Data**: < 1 hour
- **Historical Data**: < 4 hours

### Backup Verification
```bash
# Weekly verification (Sunday 6 AM)
/root/DW-Agents/scripts/verify-backups.sh
```

**Checks**:
1. File integrity (checksums)
2. Compression validity
3. Restoration test
4. Completeness verification

## Monitoring

### Archive Health Checks
- Daily: Storage usage monitoring
- Weekly: Integrity verification
- Monthly: Full audit

### Alerts
- Storage >70%: Email notification
- Backup failure: Immediate alert
- Corruption detected: Critical alert

## Related Components
- **archive-highlights.sh**: Daily archival script
- **Reports Directory**: Source of archived reports
- **Logs Directory**: Source of archived logs
- **Task Database**: Source of database archives

## Best Practices
1. **Verify Before Delete**: Always verify archive before deleting source
2. **Test Restores**: Regularly test restoration procedures
3. **Document Changes**: Log all archive-related changes
4. **Monitor Space**: Keep archive space under 80%
5. **Encrypt Sensitive**: Encrypt archives with sensitive data
6. **Off-site Backup**: Maintain off-site copies of critical archives