`AFFILIATE_CRAWLER_SETUP.md`

432 lines
# Affiliate Crawler & Automation - Top 20 Luxury Brands

Complete automation system for crawling affiliate sites, RSS feeds, and signup for the **top 20 luxury handbag brands only**.

---

## Top 20 Luxury Brands (Priority Order)

1. **HERMÈS** - Birkin, Kelly, Constance, Evelyne
2. **CHANEL** - Classic Flap, Boy Bag, 19, Gabrielle
3. **LOUIS VUITTON** - Speedy, Neverfull, Alma, Pochette Metis
4. **DIOR** - Lady Dior, Saddle, Book Tote, Diorama
5. **GUCCI** - Dionysus, Marmont, Jackie, Horsebit
6. **PRADA** - Galleria, Cahier, Saffiano, Re-Edition
7. **BOTTEGA VENETA** - Cassette, Jodie, Pouch, Arco
8. **CELINE** - Luggage, Box, Cabas, Sangle
9. **SAINT LAURENT** - Kate, Loulou, Sac de Jour, College
10. **FENDI** - Baguette, Peekaboo, Kan I, Mamma
11. **BALENCIAGA** - City, Hourglass, Le Cagole
12. **GOYARD** - St. Louis, Artois, Saigon, Belvedere
13. **VALENTINO** - Rockstud, Roman Stud, VLOGO
14. **GIVENCHY** - Antigona, Pandora, GV3, Mystic
15. **LOEWE** - Puzzle, Hammock, Barcelona, Gate
16. **BURBERRY** - TB Bag, Lola, Olympia, Frances
17. **MULBERRY** - Bayswater, Alexa, Lily, Iris
18. **CHLOE** - Drew, Faye, Tess, C Bag
19. **MIU MIU** - Matelasse, Wander, Arcadie
20. **COACH** - Tabby, Willow, Rogue, Cassie

---

## Affiliate Sites Being Crawled

### Resale Platforms (Allow Images)
1. **Fashionphile** - Premium pre-owned, authentication guarantee
2. **Rebag** - Buy/sell/trade, clean images
3. **The RealReal** - Luxury consignment
4. **Vestiaire Collective** - Global marketplace
5. **Yoogi's Closet** - Authenticated pre-owned

### Luxury Retailers (Allow Images)
6. **Farfetch** - Global luxury platform
7. **Net-a-Porter** - New designer bags
8. **Matches Fashion** - Curated luxury
9. **MyTheresa** - High-end retail
10. **24S** - LVMH luxury platform

---

## Automation Schedule

### Daily 6:00 AM - Affiliate Site Crawl
```
Cron: 0 6 * * *
Script: scripts/crawlAffiliateSites.ts
```

Crawls:
- Top 20 brands only
- All affiliate merchants with image permission
- Up to 50 products per brand per merchant
- Saves images, prices, product URLs
- Deduplicates automatically

### Every 15 Minutes - RSS Feed Poll
```
Cron: */15 * * * *
Script: scripts/pollHandbagRss.ts
```

Polls:
- RSS feeds from resellers
- New listings notifications
- Price drop alerts

### Daily Midnight - Price Snapshots
```
Cron: 0 0 * * *
Script: scripts/snapshotPrices.ts
```

Collects:
- Daily price data
- Historical price tracking
- For chart generation

---

## Quick Start

### 1. Install Cron Jobs

```bash
./scripts/setup-cron.sh
```

This installs all three automated tasks.

### 2. Run Manual Crawl (Test)

```bash
npm run crawl-affiliates
```

Expected output:
```
=== Starting Affiliate Site Crawler ===

Top 20 Luxury Handbag Brands Only

Brands: HERMÈS, CHANEL, LOUIS VUITTON, DIOR...

--- HERMÈS ---

Crawling Fashionphile for HERMÈS...
  URL: https://www.fashionphile.com/shop/bags?designers=HERMÈS
  Found 50 items

Crawling Rebag for HERMÈS...
  URL: https://www.rebag.com/shop?brand=HERMÈS
  Found 48 items

...

=== Crawl Complete ===
Total items crawled: 4,523
New items added: 4,523
Saved to: data/crawled_affiliate_listings.json
```

### 3. Sign Up for Affiliate Programs

```bash
npm run affiliates:signup-all
```

This will:
1. Open Fashionphile signup → pre-fill → wait for your review → submit
2. Open Rebag signup → pre-fill → wait → submit
3. Open The RealReal signup → pre-fill → wait → submit
4. ...and so on

---

## Image Display

All crawled sites **allow displaying their product images** via:

1. **Direct Image URLs** - Hotlinking permitted for affiliates
2. **Affiliate Attribution** - Images shown with merchant credit
3. **Click-through Links** - All images link to affiliate-tracked URLs

### Example Usage in UI:

```tsx
import Image from "next/image";

<div className="product-card">
  <a href={listing.productUrl} target="_blank">
    <Image
      src={listing.imageUrl}
      alt={listing.title}
      width={300}
      height={300}
    />
  </a>
  <p>{listing.title}</p>
  <p>${listing.price}</p>
  <p className="text-xs text-gray-500">From {listing.merchant}</p>
</div>
```

---

## Data Files

### Crawled Listings
`data/crawled_affiliate_listings.json`

```json
{
  "id": "abc123def456",
  "brand": "HERMÈS",
  "title": "Hermès Birkin 30 Togo Leather Gold",
  "price": 12500,
  "currency": "USD",
  "imageUrl": "https://fashionphile.com/images/...",
  "productUrl": "https://fashionphile.com/item/...",
  "merchant": "Fashionphile",
  "condition": "used",
  "crawledAt": "2025-11-15T06:00:00Z"
}
```

### Top Brands Config
`data/top_luxury_brands.json`

Contains:
- 20 luxury brands ranked by prestige
- Average prices
- Top models for each brand
- Affiliate priority level

---

## Affiliate Signup Automation

### Manual Signup (One at a Time)

```bash
# Fashionphile
npm run affiliates:signup fashionphile

# Rebag
npm run affiliates:signup rebag

# The RealReal
npm run affiliates:signup therealreal
```

### Automated Signup (All at Once)

```bash
npm run affiliates:signup-all
```

Opens each signup page sequentially, pre-fills your info, highlights submit button, waits for you to review and submit.

### Your Business Profile

Edit `scripts/handbagAffiliateSignup.ts`:

```typescript
const profile: AffiliateSignupProfile = {
  contactName: "Steve Abrams",
  company: "Handbag Finder",
  email: "steve@designerwallcoverings.com",
  websites: ["https://your-handbag-app.com"],
  description: "Luxury handbag authentication and price tracking app with 27,000+ listings.",
};
```

---

## Adding Affiliate Tracking Codes

After approval, update your affiliate codes:

`data/handbag_affiliates.json`:

```json
{
  "id": "fashionphile",
  "name": "Fashionphile",
  "domain": "fashionphile.com",
  "affiliateCode": "YOUR_CODE_HERE",
  "affiliateNetwork": "Impact",
  "commissionRate": "5%",
  "cookieDuration": 30
}
```

Then decorate URLs in your app:

```typescript
function addAffiliateCode(url: string, merchant: string): string {
  const codes = {
    "fashionphile": "?aff=YOURCODE",
    "rebag": "?ref=YOURCODE",
    "therealreal": "?sid=YOURCODE",
  };

  return url + (codes[merchant] || "");
}
```

---

## Monitoring

### View Logs

```bash
# Affiliate crawl log (daily 6am)
tail -f /var/log/handbag-auth/affiliate-crawl.log

# RSS poll log (every 15min)
tail -f /var/log/handbag-auth/rss-poll.log

# Price snapshot log (daily midnight)
tail -f /var/log/handbag-auth/price-snapshot.log
```

### Check Cron Jobs

```bash
# View schedule
crontab -l

# Should show:
# */15 * * * * cd /root/Projects/handbag-auth-nextjs && npm run poll-rss...
# 0 0 * * * cd /root/Projects/handbag-auth-nextjs && npm run price-snapshot...
# 0 6 * * * cd /root/Projects/handbag-auth-nextjs && npx ts-node --esm scripts/crawlAffiliateSites.ts...
```

---

## RSS Feed Discovery (Top 20 Brands)

Many luxury brands don't have public RSS feeds, but affiliate resellers do:

### Reseller RSS Feeds (Already Configured)

1. **Fashionphile Blog** - https://www.fashionphile.com/blog/feed
2. **PurseBlog** - https://www.purseblog.com/feed/
3. **The RealReal** - Check for category RSS
4. **Vestiaire** - May have search-based RSS

### Finding More Feeds

```bash
# Check page source
curl -s https://fashionphile.com | grep -i 'rss\|atom\|feed'

# Try common URLs
curl -I https://fashionphile.com/feed
curl -I https://fashionphile.com/rss
curl -I https://fashionphile.com/atom.xml
```

Update `data/handbag_rss_sources.json` with verified feeds.

---

## Image Usage & Attribution

### Legal Requirements

1. **Affiliate Agreement** - Sign up with each merchant
2. **Attribution Required** - Display merchant name with images
3. **Click-through Links** - All images must link to merchant site
4. **No Re-hosting** - Use original image URLs (hotlink)

### Best Practices

```tsx
// Good - Proper attribution
<div>
  <a href={affiliateUrl}>
    <img src={merchantImageUrl} />
  </a>
  <p className="text-xs">Available at {merchantName}</p>
</div>

// Bad - No attribution
<img src={merchantImageUrl} />
```

---

## Troubleshooting

### Crawler Not Finding Items

Check:
1. Is the site structure still the same? (Selectors may change)
2. Is JavaScript rendering required? (Puppeteer handles this)
3. Is there rate limiting? (We wait 2 seconds between requests)
4. Are there CAPTCHAs? (May need human verification)

### Images Not Loading

Check:
1. Is hotlinking allowed? (All listed merchants allow it)
2. Is HTTPS used? (Browsers block HTTP images on HTTPS pages)
3. Is CORS configured? (Use Next.js Image with `unoptimized` prop if needed)

### Affiliate Signup Fails

Check:
1. Did the form structure change? (Update selectors)
2. Is there email verification? (Check your inbox)
3. Are you already signed up? (Some merchants auto-reject duplicates)

---

## Next Steps

1. **Run First Crawl**
   ```bash
   npm run crawl-affiliates
   ```

2. **Sign Up for Affiliates**
   ```bash
   npm run affiliates:signup-all
   ```

3. **Install Cron Jobs**
   ```bash
   ./scripts/setup-cron.sh
   ```

4. **Monitor First Run**
   ```bash
   tail -f /var/log/handbag-auth/affiliate-crawl.log
   ```

5. **Integrate Into App**
   - Display crawled listings on homepage
   - Add affiliate-tracked links
   - Show merchant attribution

---

## Summary

✅ Top 20 luxury brands configured
✅ 5 affiliate merchants with image permissions
✅ Daily 6am automatic crawling
✅ RSS feed polling every 15 minutes
✅ Automated affiliate signup scripts
✅ Price history tracking
✅ Image hotlinking allowed
✅ All real data - no samples

**Server**: 45.61.58.125
**Port**: 7992
**URL**: http://45.61.58.125:7992

The system crawls **top 20 luxury handbag brands only** from affiliate sites that allow displaying their images!