`SETUP.md`

387 lines
# Setup Guide - Japanese Handbag Arbitrage Platform

## Quick Start

### 1. Install Dependencies
```bash
npm install
```

### 2. Configure Environment
```bash
# .env is already created, just add your OpenAI API key
nano .env
```

Required: Add your OpenAI API key:
```
OPENAI_API_KEY=sk-your-key-here
```

Optional: Add Slack webhook for deal notifications:
```
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
```

### 3. Initialize Database
```bash
node scripts/init-db.js
```

### 4. Configure Firewall (Port 7989)
```bash
sudo bash scripts/configure-firewall.sh
```

### 5. Start the Server
```bash
npm start
```

Access at: `http://your-server-ip:7989`

---

## Complete Setup Steps

### Option A: Manual Crawl First (Recommended)

1. **Run Initial Crawl** (takes 5-10 minutes)
```bash
npm run crawler
```

2. **Run AI Analysis**
```bash
node ai/analyzer.js
```

3. **Start Web Server**
```bash
npm start
```

4. **Check for Deals & Send Slack Notifications**
```bash
node scripts/deal-notifier.js check
```

### Option B: Automated with Cron

1. **Start Cron Job** (crawls every 6 hours)
```bash
node scripts/setup-cron.js
```
Keep this process running in the background (use PM2 or systemd)

2. **Start Web Server** (in another terminal/process)
```bash
npm start
```

---

## Production Deployment

### Using PM2 (Recommended)

```bash
# Install PM2
npm install -g pm2

# Start server
pm2 start server.js --name handbags-server

# Start cron job
pm2 start scripts/setup-cron.js --name handbags-cron

# Save PM2 configuration
pm2 save

# Setup PM2 to start on boot
pm2 startup
```

### Using systemd

Create `/etc/systemd/system/handbags.service`:

```ini
[Unit]
Description=Japanese Handbag Arbitrage Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/root/WebsitesMisc/handbags
Environment=NODE_ENV=production
ExecStart=/usr/bin/node server.js
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

Create `/etc/systemd/system/handbags-cron.service`:

```ini
[Unit]
Description=Japanese Handbag Arbitrage Cron
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/root/WebsitesMisc/handbags
ExecStart=/usr/bin/node scripts/setup-cron.js
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

Enable and start:
```bash
systemctl enable handbags handbags-cron
systemctl start handbags handbags-cron
systemctl status handbags handbags-cron
```

---

## Scheduled Tasks

### Cron Schedule (6-hour intervals)
- 00:00 - Midnight crawl
- 06:00 - Morning crawl
- 12:00 - Noon crawl
- 18:00 - Evening crawl

After each crawl:
1. Scrape all Japanese sites
2. Run AI price analysis
3. Check for new deals
4. Send Slack notifications (if configured)

### Daily Summary (Optional)
Add to crontab for daily summary at 8 AM:
```bash
crontab -e
```

Add:
```
0 8 * * * cd /root/WebsitesMisc/handbags && node scripts/deal-notifier.js summary
```

---

## Manual Commands

### Crawling
```bash
# Full crawl
npm run crawler

# Crawl via API (server must be running)
curl -X POST http://localhost:7989/api/crawl
```

### AI Analysis
```bash
node ai/analyzer.js
```

### Exports
```bash
# Export all listings
node scripts/export-spreadsheet.js all

# Export deals only (20%+ off)
node scripts/export-spreadsheet.js deals 20

# Export deals 40%+ off
node scripts/export-spreadsheet.js deals 40

# Export by brand
node scripts/export-spreadsheet.js brand "Gucci"

# Export today's deals
node scripts/export-spreadsheet.js daily
```

### Notifications
```bash
# Check and notify new deals
node scripts/deal-notifier.js check

# Send daily summary
node scripts/deal-notifier.js summary
```

---

## API Endpoints

Access via browser or curl:

```bash
# Get all listings
curl http://localhost:7989/api/listings

# Get deals only
curl http://localhost:7989/api/deals?minDeal=30

# Get statistics
curl http://localhost:7989/api/stats

# Export CSV
curl http://localhost:7989/api/export/csv > deals.csv

# Trigger crawl
curl -X POST http://localhost:7989/api/crawl
```

---

## Slack Integration

### Setup Slack Webhook

1. Go to https://api.slack.com/messaging/webhooks
2. Create a new webhook for your workspace
3. Copy the webhook URL
4. Add to `.env`:
```
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
```

### Test Slack Notification

```bash
# After configuring webhook, check for deals
node scripts/deal-notifier.js check
```

You'll receive notifications for:
- 🔥 Deals 20-30% below market
- 🔥🔥 Deals 30-50% below market
- 🔥🔥🔥 Deals 50%+ below market

---

## Troubleshooting

### Port 7989 not accessible
```bash
# Check if server is running
curl http://localhost:7989

# Check firewall
sudo ufw status

# Reconfigure firewall
sudo bash scripts/configure-firewall.sh

# Check if port is in use
sudo lsof -i :7989
```

### Crawl fails
```bash
# Check puppeteer dependencies (Ubuntu/Debian)
sudo apt-get install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 \
libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 \
libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 \
libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 \
libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates \
fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
```

### Database locked
```bash
# Stop all processes
pm2 stop all

# Or kill node processes
pkill node

# Restart
pm2 restart all
```

### OpenAI API errors
- Check API key in `.env`
- Verify API key has credits: https://platform.openai.com/account/usage
- Check rate limits

---

## Monitoring

### View Logs (PM2)
```bash
pm2 logs handbags-server
pm2 logs handbags-cron
```

### Check Database
```bash
sqlite3 data/handbags.db

# Run queries
sqlite> SELECT COUNT(*) FROM listings;
sqlite> SELECT COUNT(*) FROM deal_analysis WHERE is_deal = 1;
sqlite> SELECT brand, COUNT(*) FROM listings GROUP BY brand;
```

### Server Health
```bash
curl http://localhost:7989/api/stats
```

---

## Updating Configuration

### Change Crawl Interval
Edit `.env`:
```
CRAWL_INTERVAL_HOURS=12  # Change from 6 to 12 hours
```
Restart cron job.

### Change Deal Threshold
Edit `.env`:
```
DEAL_THRESHOLD=30  # Only flag deals 30%+ off instead of 20%
```
Restart analyzer.

### Enable/Disable Sites
Edit `.env`:
```
ENABLE_KOMEHYO=true
ENABLE_YAHOO_AUCTIONS=false  # Disable Yahoo Auctions
ENABLE_MERCARI_JP=true
```

---

## Security Notes

- Server runs on `0.0.0.0:7989` (publicly accessible)
- No authentication implemented (add nginx reverse proxy with auth for production)
- Keep `.env` file secure (contains API keys)
- Consider restricting firewall to specific IPs

---

## Support

For issues or questions:
1. Check logs: `pm2 logs`
2. Review this guide
3. Check database: `sqlite3 data/handbags.db`