← back to Wine Finder

CLAUDE.md

269 lines

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

Red Thunder Wine Tracker is a secure, session-based web application for tracking wine prices from multiple sources (Vivino, Total Wine & More, and K&L Wine Merchants). It automatically logs all searches to Google Sheets, monitors for price spikes, and sends alerts via Slack webhooks.

## Data Sources

**CRITICAL: Always use real, live data from the wine scraping services.** NEVER use sample/demo data unless all scrapers fail. The app has multiple wine data sources:

### Live Price Data (Current):
1. **Vivino** (Primary, most reliable) - Uses Vivino's explore API with client-side filtering
2. **Total Wine & More** - HTML scraping (may be blocked)
3. **K&L Wine Merchants** - HTML scraping (currently blocked with 403 errors)

### Historical/Academic Data (Available):
4. **Learning to Taste Dataset** (UC Davis/NeurIPS 2023) - 824k wine reviews, 350k+ unique vintages (1950-2021), with historical pricing from Vivino collected in May 2023
   - Dataset URL: https://thoranna.github.io/learning_to_taste/
   - HuggingFace: https://huggingface.co/datasets/Dakhoo/L2T-NeurIPS-2023
   - Fields: vintage_id, wine, year, winery_id, country, region, price (USD), rating, grape composition, alcohol %, reviews
   - License: CC BY-NC-ND 4.0
   - Service: `services/historicalWineData.js` (framework ready, dataset integration pending)

When scrapers fail or are blocked, the app falls back to demo data only as a last resort. Always attempt to use real wine data first.

## Commands

### Development
```bash
npm run dev          # Start with nodemon (auto-restart on changes)
npm start            # Start production server
```

### Process Management
```bash
# PM2 (recommended for production)
pm2 start server.js --name wine-tracker
pm2 logs wine-tracker
pm2 restart wine-tracker
pm2 stop wine-tracker

# Or use nohup
nohup npm start > wine-tracker.log 2>&1 &
tail -f wine-tracker.log
```

### Port and Firewall
```bash
# Default port: 9222 (configurable via PORT env var)
sudo ufw allow 9222/tcp
sudo netstat -tulpn | grep 9222
```

## Architecture

### Request Flow
1. **Authentication Layer** - All routes except `/login` and `/auth/*` require session authentication via `requireAuth` middleware
2. **Route Layer** - Express routes delegate to services for business logic
3. **Service Layer** - Three independent services handle core functionality:
   - `klWineScraper.js` - Web scraping with Cheerio
   - `googleSheets.js` - Async logging to Google Sheets
   - `slackNotifier.js` - Alert webhook calls
4. **Response** - Services return success/error; routes handle HTTP responses

### Key Architectural Patterns

**Service Independence**: Services don't call each other. The route layer orchestrates them:
```javascript
// routes/scraper.js search endpoint (multi-source):
const result = await wineAggregator.searchAll(query, {
  sources: ['kl', 'vivino', 'totalwine'],      // 1. Search all sources in parallel
  deduplicate: true,                            // 2. Remove duplicate wines across sources
  sortBy: 'price'                               // 3. Sort by price
});
await googleSheets.logMultipleWines(result.wines);    // 4. Log to sheets
const alerts = await detectPriceSpikes(result.wines); // 5. Detect spikes
await slackNotifier.sendAlerts(alerts);                // 6. Alert (async)
```

**Multi-Source Aggregation**: The `wineAggregator` service searches all enabled sources in parallel, deduplicates results using fuzzy name matching, and combines prices to find best deals. Individual scrapers can be accessed directly or through the aggregator.

**Demo Data Fallback**: When K&L Wine scraper fails (403 blocks, HTML changes), routes generate demo data via `generateDemoWines()` so the app continues functioning.

**Session-Based Auth**: Uses express-session with in-memory store (MemoryStore). Sessions expire after 24 hours. No JWT/tokens.

**Aggressive Cache Busting**: Static files served with `no-store, no-cache` headers and `etag: false` to prevent browser caching issues during development.

## Critical Implementation Details

### Content Security Policy
The CSP in `server.js` has `upgradeInsecureRequests: null` to allow HTTP access (since deployed without HTTPS). If deploying with HTTPS, remove this override.

### Price Spike Detection
Implemented in `routes/scraper.js` and `routes/history.js`:
- Fetches historical prices from Google Sheets for each wine
- Compares current price to most recent historical price
- Triggers if `(currentPrice - lastPrice) / lastPrice * 100 > PRICE_SPIKE_THRESHOLD`
- Default threshold: 20% (set via `PRICE_SPIKE_THRESHOLD` env var)

### Google Sheets Schema
Sheet name: "Red Thunder Wine Tracker"
Columns (in order):
1. Timestamp
2. Wine Name
3. Vintage
4. Price
5. Rating
6. Source
7. Availability
8. URL
9. Search Query

### Environment Variables
Required:
- `GOOGLE_SHEET_ID` - Google Sheets document ID from URL
- `GOOGLE_APPLICATION_CREDENTIALS` - Path to service account JSON (default: `./google-credentials.json`)
- `SLACK_WEBHOOK_URL` - Slack incoming webhook URL
- `ADMIN_USER` / `ADMIN_PASS` - Login credentials

Optional:
- `PORT` - Server port (default: 9222)
- `SESSION_SECRET` - Session encryption key
- `PRICE_SPIKE_THRESHOLD` - Alert threshold percentage (default: 20)
- `NODE_ENV` - Environment mode

## Common Development Scenarios

### Web Scraper Blocked/Failing
If K&L changes HTML structure or blocks requests:
1. Check `services/klWineScraper.js` selectors (lines 38-80)
2. Update CSS selectors to match K&L's current DOM
3. Test with: `curl -A "Mozilla/5.0..." https://www.klwines.com/Products?searchText=cabernet`
4. Alternatively, app falls back to demo data automatically

### Adding New Wine Retailers
1. Create new scraper service in `services/` (e.g., `wineComScraper.js`)
2. Implement same interface: `search(query)` returns array of wine objects
3. Add to `routes/scraper.js` search endpoint
4. Update `generateDemoWines()` to include new source

### Modifying UI Without Cache Issues
Static files aggressively bypass cache. However, for guaranteed fresh load:
1. Change filename (e.g., `styles.css` → `styles-v2.css`)
2. Or use embedded styles in HTML (like `test.html` / `wine-search.html`)
3. Or change port number in `.env` to force new origin

### Testing Google Sheets Integration
```javascript
// In routes/scraper.js, check these logs:
console.log('Logging to Google Sheets...');  // Before logging
console.log('Successfully logged to sheets');  // After success
// If no logs appear, googleSheets.logWines() silently failed
```

Check service account has Editor access to the sheet. Email is in `google-credentials.json` under `client_email`.

### Testing Slack Alerts
```bash
# Manual test
curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Test alert"}' \
  $SLACK_WEBHOOK_URL

# Trigger in app: manually edit historical price in Google Sheets
# to be >20% lower than current price, then search that wine
```

## File Structure

```
server.js               # Main Express app, middleware, session config
routes/
  auth.js              # Login/logout endpoints
  scraper.js           # Wine search orchestration, demo data fallback, price spike logic
  history.js           # Historical price data from Google Sheets
services/
  wineAggregator.js    # Multi-source search orchestration, deduplication, best price finder
  vivinoScraper.js     # Vivino API scraper (explore endpoint with client-side filtering)
  totalWineScraper.js  # Total Wine & More HTML scraper (Cheerio-based)
  klWineScraper.js     # K&L Wine Merchants HTML scraper (Cheerio-based, currently blocked)
  googleSheets.js      # Google Sheets API wrapper
  slackNotifier.js     # Slack webhook poster
public/
  index.html           # Main app with source filter checkboxes (requires auth)
  login.html           # Login page
  test.html            # Old test page
  wine-search.html     # New test page with sorting/links
  test-search.js       # Puppeteer automated testing script
  css/                 # Pink/purple gradient theme
  js/app.js            # Client-side app logic with multi-source support
```

## Vivino API Implementation

**Endpoint**: `https://www.vivino.com/api/explore/explore`

**Important Notes**:
- The explore API does NOT support free-text search via a `q` parameter
- It requires at least one filter parameter (e.g., `wine_type_ids[]`, `min_rating`, etc.)
- Text search is implemented client-side by filtering results after API response
- Default filter: `wine_type_ids[]=1` (red wines) - can be made dynamic in future

**Search Flow**:
1. Call explore API with filters (country, price range, wine type, min rating)
2. API returns wines sorted by price (not by search relevance)
3. Client-side filter: match query terms against wine name, winery, or region
4. Return only matching wines

**Why this approach**: Vivino doesn't provide a public search API that filters by text. The explore endpoint is designed for browsing with structured filters, not keyword search. Client-side filtering ensures users get relevant results.

## Known Issues & Limitations

1. **MemoryStore Warning**: Express session uses in-memory storage. Not suitable for multi-process deployments. Sessions lost on restart.

2. **Vivino Text Search Limitation**: Vivino's explore API doesn't support free-text search. Results are filtered client-side, which may miss some wines if they're not in the first page of results.

3. **K&L Wine Scraper Blocked**: Website returns 403 errors. App handles gracefully with fallback to other sources.

4. **Total Wine Scraper Fragility**: May break if Total Wine changes HTML structure or implements blocking.

5. **No HTTPS**: CSP configured for HTTP deployment. Production should use reverse proxy (nginx) with SSL.

6. **Single User**: No user management. One shared admin account in environment variables.

7. **Browser Cache Stubbornness**: Despite aggressive no-cache headers, some browsers (especially with proxies) cache aggressively. Port changes or new filenames may be needed.

## Wine Image Policy - Affiliate Programs

**CRITICAL: Only use images from affiliate partners or official winery sources**

### ✅ Approved Image Sources (Affiliate Programs):

1. **Vivino** - Vivino Affiliates Program
   - Best source for real wine bottle photos
   - Official wine database with 350k+ vintages
   - Images: `https://images.vivino.com/thumbs/`

2. **Total Wine & More** - Affiliate Program via Impact
   - Product images OK to use as affiliate
   - Images include real bottle photography

3. **K&L Wine Merchants** - Affiliate Program Available
   - Product images OK to use as affiliate
   - Real bottle photos from their catalog

4. **Wine.com** - CJ Affiliate & ShareASale
   - Affiliate partner images allowed
   - Professional wine photography

5. **Wine-Searcher** - Affiliate Program Available
   - Database of wine images across retailers
   - Aggregate pricing data

### Image Priority Order:
1. Vivino (best database, most complete)
2. Retailer affiliate partners (Total Wine, K&L, Wine.com)
3. Official winery images (when available)

### Implementation:
- `lib/services/wineImageService.js` - Validates affiliate sources
- All scrapers preserve affiliate partner images
- Vivino fallback for missing images

## Contact

Steve Abrams - steve@designerwallcoverings.com