`TROUBLESHOOTING.md`

547 lines
# LUXVAULT Troubleshooting Guide

Quick reference for diagnosing and fixing common issues.

## Quick Diagnostics

Run this one-liner to check system health:

```bash
echo "=== PM2 Status ===" && pm2 status auction-viewer && \
echo -e "\n=== Health Check ===" && curl -s http://45.61.58.125:7500/health | jq && \
echo -e "\n=== Database ===" && ls -lh /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db && \
echo -e "\n=== Disk Space ===" && df -h /root && \
echo -e "\n=== Recent Errors ===" && pm2 logs auction-viewer --lines 20 --nostream 2>&1 | grep -i error | tail -5
```

---

## Issue Categories

### 🔴 Service Not Running

**Symptoms:**
- HTTP connection refused
- PM2 shows "stopped" or "errored"
- Port 7500 not responding

**Diagnosis:**

```bash
# Check PM2 status
pm2 status auction-viewer

# Check process
lsof -ti:7500

# Check recent errors
pm2 logs auction-viewer --err --lines 50
```

**Solutions:**

```bash
# Solution 1: Simple restart
pm2 restart auction-viewer

# Solution 2: Kill port and restart
lsof -ti:7500 | xargs kill -9
pm2 restart auction-viewer

# Solution 3: Full PM2 reset
pm2 delete auction-viewer
cd /root/Projects/handbag-auth-nextjs/auction-viewer
pm2 start server.js --name auction-viewer

# Solution 4: Use PM2 config
pm2 delete auction-viewer
pm2 start /root/Projects/handbag-auth-nextjs/scripts/devops/pm2-config.json
```

---

### 🟡 Service Running But Unhealthy

**Symptoms:**
- Health endpoint returns 503
- API returns errors
- Slow response times

**Diagnosis:**

```bash
# Check health details
curl -v http://45.61.58.125:7500/health | jq

# Check metrics
curl http://45.61.58.125:7500/metrics | jq

# Check database connectivity
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db \
  "SELECT COUNT(*) FROM auctions;"

# Check memory usage
pm2 status
free -h
```

**Solutions:**

```bash
# Clear application cache
curl -X POST http://45.61.58.125:7500/api/cache/clear

# Restart with memory limit
pm2 restart auction-viewer --max-memory-restart 500M

# Check for database locks
lsof /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db

# Optimize database
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "VACUUM;"
```

---

### 🔵 Database Issues

**Symptoms:**
- "Database locked" errors
- Slow queries
- Integrity check failures

**Diagnosis:**

```bash
# Check database integrity
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db \
  "PRAGMA integrity_check;"

# Check for locks
lsof /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db

# Check file permissions
ls -l /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db

# Check database size
du -h /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db
```

**Solutions:**

```bash
# Kill processes holding locks
lsof /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db | \
  awk 'NR>1 {print $2}' | xargs kill

# Fix permissions
chmod 644 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db

# Optimize database
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "VACUUM;"

# Restore from backup if corrupted
/root/Projects/handbag-auth-nextjs/scripts/devops/restore-backup.sh \
  /root/Projects/handbag-auth-nextjs/backups/luxvault_backup_latest.tar.gz
```

---

### 🟢 Scraper Issues

**Symptoms:**
- No new data today
- Scraper errors in logs
- Cron job not running

**Diagnosis:**

```bash
# Check last scraper run
tail -50 /root/Projects/handbag-auth-nextjs/data/auction-history/scraper.log | \
  grep "DAILY SCRAPE COMPLETED"

# Check scraper status via API
curl http://45.61.58.125:7500/api/scraper/status | jq

# Check cron job
crontab -l | grep scrape-auction-data

# Check cron log
grep scrape-auction /var/log/syslog | tail -20
```

**Solutions:**

```bash
# Run scraper manually
cd /root/Projects/handbag-auth-nextjs
python3 scripts/scrape-auction-data.py

# Check Python dependencies
python3 -c "import requests, bs4, sqlite3; print('OK')"

# Verify cron job
crontab -e
# Ensure this line exists:
# 0 6 * * * /usr/bin/python3 /root/Projects/handbag-auth-nextjs/scripts/scrape-auction-data.py >> /root/Projects/handbag-auth-nextjs/data/auction-history/cron.log 2>&1

# Check cron service
systemctl status cron
```

---

### 🟠 Performance Issues

**Symptoms:**
- Slow API responses
- High memory usage
- Response timeout errors

**Diagnosis:**

```bash
# Check response times
time curl http://45.61.58.125:7500/api/auctions

# Check memory usage
pm2 status
ps aux | grep node

# Check metrics
curl http://45.61.58.125:7500/metrics | jq '.metrics'

# Check database query performance
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db \
  ".timer on" \
  "SELECT * FROM auctions LIMIT 100;"
```

**Solutions:**

```bash
# Clear cache
curl -X POST http://45.61.58.125:7500/api/cache/clear

# Restart with memory optimization
pm2 restart auction-viewer --max-memory-restart 300M

# Optimize database
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db <<EOF
VACUUM;
ANALYZE;
REINDEX;
EOF

# Add database indexes (if needed)
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db <<EOF
CREATE INDEX IF NOT EXISTS idx_search_term ON auctions(search_term);
CREATE INDEX IF NOT EXISTS idx_created_at ON auctions(created_at);
CREATE INDEX IF NOT EXISTS idx_current_price ON auctions(current_price);
EOF
```

---

### 💾 Disk Space Issues

**Symptoms:**
- "No space left on device"
- Backup failures
- Log write errors

**Diagnosis:**

```bash
# Check disk usage
df -h /root

# Check project directory size
du -sh /root/Projects/handbag-auth-nextjs/*

# Check log sizes
du -sh /root/Projects/handbag-auth-nextjs/logs/*

# Check backup sizes
du -sh /root/Projects/handbag-auth-nextjs/backups/*

# Find largest files
find /root/Projects/handbag-auth-nextjs -type f -size +10M -exec ls -lh {} \;
```

**Solutions:**

```bash
# Rotate logs immediately
/root/Projects/handbag-auth-nextjs/scripts/devops/log-rotation.sh

# Clean old backups (keep last 7)
find /root/Projects/handbag-auth-nextjs/backups -name "*.tar.gz" -mtime +7 -delete

# Clean PM2 logs
pm2 flush auction-viewer

# Vacuum database
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "VACUUM;"

# Clean npm cache
npm cache clean --force

# Remove old deployment backups
find /root/Projects/handbag-auth-nextjs/backups/deployments -mtime +7 -delete
```

---

### 🔐 Permission Issues

**Symptoms:**
- "Permission denied" errors
- Cannot write to database
- Cannot create backups

**Diagnosis:**

```bash
# Check file ownership
ls -l /root/Projects/handbag-auth-nextjs/data/auction-history/

# Check directory permissions
ls -ld /root/Projects/handbag-auth-nextjs/backups

# Check current user
whoami
```

**Solutions:**

```bash
# Fix data directory permissions
chown -R root:root /root/Projects/handbag-auth-nextjs/data
chmod -R 755 /root/Projects/handbag-auth-nextjs/data

# Fix backup directory
mkdir -p /root/Projects/handbag-auth-nextjs/backups
chmod 755 /root/Projects/handbag-auth-nextjs/backups

# Fix log directory
mkdir -p /root/Projects/handbag-auth-nextjs/logs
chmod 755 /root/Projects/handbag-auth-nextjs/logs
```

---

### 🌐 Network Issues

**Symptoms:**
- Cannot reach external services
- Scraper cannot fetch data
- Health checks timeout

**Diagnosis:**

```bash
# Check port is open
lsof -ti:7500

# Check firewall
sudo ufw status | grep 7500

# Test external connectivity
curl -I https://www.google.com

# Check DNS
nslookup google.com

# Test local connectivity
curl -v http://localhost:7500/health
curl -v http://45.61.58.125:7500/health
```

**Solutions:**

```bash
# Open firewall port
sudo ufw allow 7500/tcp

# Restart firewall
sudo ufw reload

# Check if service is listening
netstat -tulpn | grep 7500

# Restart service
pm2 restart auction-viewer
```

---

## Error Messages Reference

### "EADDRINUSE: address already in use"

**Cause:** Port 7500 is already in use

**Fix:**
```bash
lsof -ti:7500 | xargs kill -9
pm2 restart auction-viewer
```

---

### "Database is locked"

**Cause:** Another process has the database locked

**Fix:**
```bash
lsof /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db
# Kill the process if safe
pm2 restart auction-viewer
```

---

### "Cannot read property of undefined"

**Cause:** Database query returned no results

**Fix:**
```bash
# Check if database has data
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db \
  "SELECT COUNT(*) FROM auctions;"

# If empty, run scraper
python3 /root/Projects/handbag-auth-nextjs/scripts/scrape-auction-data.py
```

---

### "Maximum call stack size exceeded"

**Cause:** Memory leak or infinite loop

**Fix:**
```bash
pm2 restart auction-viewer --max-memory-restart 300M
pm2 logs auction-viewer --err --lines 100
```

---

## Emergency Recovery Procedures

### Complete System Reset

Use only as last resort:

```bash
# 1. Stop service
pm2 delete auction-viewer

# 2. Backup current data
mkdir -p /tmp/luxvault_emergency_backup
cp -r /root/Projects/handbag-auth-nextjs/data /tmp/luxvault_emergency_backup/

# 3. Restore from last good backup
LATEST_BACKUP=$(ls -t /root/Projects/handbag-auth-nextjs/backups/luxvault_backup_*.tar.gz | head -1)
/root/Projects/handbag-auth-nextjs/scripts/devops/restore-backup.sh "$LATEST_BACKUP"

# 4. Restart service
pm2 start /root/Projects/handbag-auth-nextjs/scripts/devops/pm2-config.json

# 5. Verify
curl http://45.61.58.125:7500/health | jq
```

---

## Preventive Maintenance

### Weekly Checks

```bash
# Health check
curl http://45.61.58.125:7500/health | jq

# Database integrity
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db \
  "PRAGMA integrity_check;"

# Disk space
df -h /root

# Error logs
pm2 logs auction-viewer --err --lines 100 | grep -i error | tail -20

# Backup verification
ls -lh /root/Projects/handbag-auth-nextjs/backups/*.tar.gz
```

### Monthly Tasks

```bash
# Database optimization
sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db \
  "VACUUM; ANALYZE; REINDEX;"

# Test backup restore
/root/Projects/handbag-auth-nextjs/scripts/devops/restore-backup.sh \
  $(ls -t /root/Projects/handbag-auth-nextjs/backups/*.tar.gz | head -1)

# Update dependencies
cd /root/Projects/handbag-auth-nextjs/auction-viewer
npm audit
npm update

# Review logs
tail -1000 /root/Projects/handbag-auth-nextjs/logs/*.log
```

---

## Getting Help

### Collect Diagnostic Information

Before asking for help, collect this information:

```bash
# Create diagnostic report
cat > /tmp/luxvault_diagnostic.txt <<'EOF'
=== System Info ===
$(uname -a)

=== PM2 Status ===
$(pm2 status auction-viewer)

=== Health Check ===
$(curl -s http://45.61.58.125:7500/health)

=== Metrics ===
$(curl -s http://45.61.58.125:7500/metrics)

=== Database Info ===
$(ls -lh /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db)
$(sqlite3 /root/Projects/handbag-auth-nextjs/data/auction-history/auctions.db "SELECT COUNT(*) FROM auctions;")

=== Disk Space ===
$(df -h /root)

=== Recent Errors ===
$(pm2 logs auction-viewer --err --lines 50 --nostream)

=== Recent Health Logs ===
$(tail -50 /root/Projects/handbag-auth-nextjs/logs/health-monitor.log)
EOF

cat /tmp/luxvault_diagnostic.txt
```

---

**Last Updated**: 2025-11-17