`DW-Agents/dw-agents/data/README.md`

207 lines
# Data Directory

## Purpose
Central data storage for cached information, order data, inventory tracking, price history, and other persistent data used across multiple agents.

## Directory Structure
```
/root/DW-Agents/data/
├── amazon-cache/              # Amazon data caching
│   ├── orders.json           # Cached Amazon order data
│   ├── products.json         # Cached product information
│   ├── price-history.json    # Historical price tracking
│   └── searches.json         # Cached search results
├── amazon-orders.json        # Real Amazon order data
├── office-inventory.json     # Office supply inventory
├── price-tracking.json       # Multi-vendor price tracking
├── price-history.json        # Historical price database
└── amazon-orders-*.html/png  # Cached web pages and screenshots
```

## Data Files

### amazon-orders.json
**Purpose**: Real-time Amazon order data from both Personal and Business accounts

**Structure**:
```json
{
  "orders": [
    {
      "orderId": "xxx-xxxxxxx-xxxxxxx",
      "date": "YYYY-MM-DD",
      "total": "$XXX.XX",
      "items": [...],
      "status": "Delivered/Shipped/Processing",
      "account": "Personal/Business",
      "trackingNumber": "...",
      "vendor": "Amazon"
    }
  ]
}
```

**Used By**:
- Purchasing Office Agent (9880) - Order history and tracking
- Accounting Agent (9882) - Expense tracking
- Today's Highlights (9885) - Order summaries

### office-inventory.json
**Purpose**: Office supply inventory tracking with reorder levels

**Structure**:
```json
{
  "inventory": [
    {
      "itemId": "unique-id",
      "name": "Item Name",
      "sku": "SKU-CODE",
      "quantity": 0,
      "reorderLevel": 10,
      "reorderQuantity": 50,
      "lastOrdered": "YYYY-MM-DD",
      "supplier": "Supplier Name",
      "cost": 0.00,
      "location": "Storage Location"
    }
  ]
}
```

**Used By**:
- Purchasing Office Agent (9880) - Inventory management
- Accounting Agent (9882) - Asset tracking

### price-tracking.json / price-history.json
**Purpose**: Multi-vendor price comparison and historical price data

**Structure**:
```json
{
  "products": [
    {
      "asin": "ASIN-CODE",
      "name": "Product Name",
      "priceHistory": [
        {
          "date": "YYYY-MM-DD",
          "vendor": "Amazon/Staples/Office Depot",
          "price": 0.00,
          "url": "product-url",
          "inStock": true
        }
      ],
      "lowestPrice": 0.00,
      "currentBestVendor": "Vendor Name"
    }
  ]
}
```

**Used By**:
- Purchasing Office Agent (9880) - Price comparison
- office-competitors-search.ts - Competitive analysis
- daily-price-tracker.ts - Price monitoring

## Amazon Cache Directory

### amazon-cache/orders.json
Cached Amazon order data to reduce API calls and improve performance

### amazon-cache/products.json
Product details cache including:
- ASIN lookups
- Product descriptions
- Current prices
- Image URLs
- Availability status

### amazon-cache/price-history.json
Historical price data for trend analysis and price drop alerts

### amazon-cache/searches.json
Cached search results with timestamps to avoid redundant searches

**Cache Invalidation**:
- Orders: 6 hours
- Products: 24 hours
- Prices: 1 hour (for active monitoring)
- Searches: 1 hour

## Data Management

### Backup Strategy
- Automated backups before major updates
- Manual backup files with timestamps
- Version control for critical data

### Data Retention
- **Order Data**: Indefinite (financial records)
- **Inventory Data**: Current + 1 year history
- **Price Data**: 2 years rolling window
- **Cache Data**: 30 days maximum

### Data Integrity
- JSON validation on read/write
- Atomic file updates
- Error logging for data corruption
- Recovery procedures documented

## Access Patterns

### Read Access
- All agents have read access to relevant data files
- Cached data reduces external API calls
- Fast local file access for queries

### Write Access
- Purchasing Office Agent: inventory, orders
- Accounting Agent: financial data validation
- Price tracking utilities: price history
- Amazon integration scripts: order sync

## Performance Optimization

### Caching Strategy
1. **Check Cache First**: Always check local cache before external API
2. **TTL Management**: Time-to-live based on data volatility
3. **Lazy Loading**: Load data only when needed
4. **Incremental Updates**: Update only changed data

### File Size Management
- Regular cleanup of old cache entries
- Archival of historical data
- Compression for large datasets
- Pagination for large result sets

## Integration Points
- **Purchasing Office Agent (9880)**: Primary data consumer
- **Accounting Agent (9882)**: Financial data validation
- **Amazon Integration Suite**: Data synchronization
- **Price Tracking Utilities**: Automated updates

## Monitoring
- File size monitoring
- Cache hit rate tracking
- Data freshness validation
- Error rate monitoring

## Security
- File permissions: Owner read/write only
- No sensitive credentials in data files
- Data sanitization on import
- Access logging

## Maintenance Tasks
- Weekly cache cleanup
- Monthly data validation
- Quarterly archival of old data
- Annual data retention review

## Related Files
- `/root/DW-Agents/amazon-data-cache.ts` - Cache management utilities
- `/root/DW-Agents/amazon-real-orders.ts` - Order fetching
- `/root/DW-Agents/daily-price-tracker.ts` - Price monitoring
- `/root/DW-Agents/office-competitors-search.ts` - Price comparison