`CORRECT_SELECTORS.md`

565 lines
# CORRECT CSS SELECTORS AND DATA EXTRACTION METHODS

**CRITICAL:** Most sites don't use traditional CSS selectors because they render with JavaScript frameworks. The "selectors" below are actually JSON paths and API endpoints.

---

## 1. Fashionphile - USE JSON, NOT CSS

**Wrong Approach:** CSS selectors on HTML
**Correct Approach:** Parse `__NEXT_DATA__` script tag

### Method 1: Extract JSON (RECOMMENDED)

```javascript
// Selector for the data
const selector = 'script#__NEXT_DATA__';

// How to use it
const html = await page.content();
const $ = cheerio.load(html);
const jsonText = $('script#__NEXT_DATA__').html();
const data = JSON.parse(jsonText);

// Path to products
const products = data.props.pageProps.serverState
  .initialResults.prod_ecom_products_date_desc.results;
```

### JSON Structure

```json
{
  "props": {
    "pageProps": {
      "serverState": {
        "initialResults": {
          "prod_ecom_products_date_desc": {
            "results": [
              {
                "objectID": "12345",
                "name": "Chanel Classic Flap Bag",
                "price": 5200,
                "original_price": 8500,
                "brand": "Chanel",
                "condition": "Excellent",
                "image_url": "https://...",
                "url": "/products/chanel-classic-flap-bag-12345",
                "category": "Handbags",
                "color": "Black",
                "material": "Caviar Leather"
              }
            ],
            "nbPages": 42
          }
        }
      }
    }
  }
}
```

### Fallback CSS Selectors (if JSON fails)

Only use these if `__NEXT_DATA__` is not available:

| Element | Selector | Notes |
|---------|----------|-------|
| Container | `article[data-product-id]` | May not exist until JS renders |
| Title | `h2.product-name` | |
| Price | `span[data-testid='product-price']` | |
| Image | `img[data-testid='product-image']` | Check for lazy loading |
| Link | `a[data-testid='product-link']` | |

**Pagination:** `?page=N` in URL

---

## 2. Rebag - USE SHOPIFY API, NOT CSS

**Wrong Approach:** CSS selectors on HTML
**Correct Approach:** Call `/products.json` API endpoint

### API Endpoint

```
https://shop.rebag.com/products.json?limit=250&page=1
```

### No Puppeteer Needed!

```javascript
const axios = require('axios');

const response = await axios.get('https://shop.rebag.com/products.json?limit=250&page=1');
const products = response.data.products;

products.forEach(product => {
  console.log({
    id: product.id,
    title: product.title,
    price: product.variants[0].price,
    vendor: product.vendor,
    image: product.images[0].src,
    url: `https://shop.rebag.com/products/${product.handle}`
  });
});
```

### JSON Response Structure

```json
{
  "products": [
    {
      "id": 8137413689521,
      "title": "Chanel So Black Reissue 2.55 Flap Bag",
      "vendor": "Chanel",
      "product_type": "Handbag",
      "handle": "handbags-chanel-so-black-reissue",
      "tags": ["Chanel", "Black", "Leather"],
      "variants": [
        {
          "id": 44670519640241,
          "price": "3115.00",
          "compare_at_price": null,
          "sku": "297714/3"
        }
      ],
      "images": [
        {
          "src": "//shop.rebag.com/cdn/shop/files/image.jpg"
        }
      ]
    }
  ]
}
```

### Alternative: Extract from Page JSON

If the API is disabled, extract from embedded data:

```javascript
// Find script tag with product data
const scriptContent = $('script:contains("productVariants")').html();
const match = scriptContent.match(/"productVariants":\s*(\[.*?\])/s);
const variants = JSON.parse(match[1]);
```

### Pagination

```javascript
let page = 1;
while (true) {
  const url = `https://shop.rebag.com/products.json?limit=250&page=${page}`;
  const response = await axios.get(url);

  if (response.data.products.length === 0) break;

  // Process products...

  page++;
  await sleep(1000); // Rate limiting
}
```

---

## 3. The RealReal - REQUIRES ADVANCED BOT BYPASS

**Status:** BLOCKED by Cloudflare (403 errors)
**Approach:** puppeteer-real-browser with Turnstile auto-solve

### Likely Selectors (Cannot Verify Due to Blocking)

| Element | Likely Selector | Alternatives |
|---------|----------------|--------------|
| Container | `article[data-testid="product-card"]` | `.product-tile`, `article[class*="product"]` |
| Title | `h3[data-testid="product-title"]` | `h3`, `a.product-link` |
| Price | `span[data-testid="product-price"]` | `.price`, `[class*="price"]` |
| Image | `img[data-testid="product-image"]` | `img` (first in container) |
| Link | `a[data-testid="product-link"]` | `a` (first in container) |

### Required Configuration

```javascript
const { connect } = require('puppeteer-real-browser');

const { page, browser } = await connect({
  headless: 'auto',
  fingerprint: true,      // Generate unique fingerprint
  turnstile: true,        // Auto-solve Cloudflare Turnstile
  tf: true,
  args: [
    '--no-sandbox',
    '--disable-setuid-sandbox',
    '--disable-blink-features=AutomationControlled'
  ]
});

await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36');
await page.setViewport({ width: 1920, height: 1080 });

await page.goto(url, {
  waitUntil: 'networkidle2',
  timeout: 120000  // 2 minutes for Cloudflare
});

// Wait extra time for challenge
await sleep(10000);
```

### Detection Strategy

Since selectors are unknown, try multiple:

```javascript
const possibleSelectors = [
  'article[data-testid="product-card"]',
  '.product-tile',
  '[data-testid="product"]',
  'article[class*="product"]',
  'div[class*="ProductCard"]',
  '[class*="ProductGrid"] > div'
];

let foundSelector = null;
for (const selector of possibleSelectors) {
  try {
    await page.waitForSelector(selector, { timeout: 5000 });
    foundSelector = selector;
    break;
  } catch (e) {
    continue;
  }
}
```

### Extraction

```javascript
const products = await page.evaluate((selector) => {
  const items = document.querySelectorAll(selector);

  return Array.from(items).map(item => {
    const title = item.querySelector('h3')?.textContent.trim() ||
                  item.querySelector('[data-testid="product-title"]')?.textContent.trim();

    const price = item.querySelector('[data-testid="product-price"]')?.textContent.trim() ||
                  item.querySelector('.price')?.textContent.trim();

    const image = item.querySelector('img')?.src ||
                  item.querySelector('img')?.getAttribute('data-src');

    const url = item.querySelector('a')?.href;

    return { title, price, image, url };
  });
}, foundSelector);
```

### Additional Requirements

- Residential proxies (datacenter IPs often blocked)
- 5-10 second delays between requests
- Random mouse movements and scrolling
- Success rate: 60-70% even with perfect setup

---

## 4. Vestiaire Collective - USE JSON + PROXIES

**Status:** BLOCKED by Cloudflare (403 errors)
**Approach:** Parse `__NEXT_DATA__` with HTTP/2 client and residential proxies

### JSON Selector

```
script#__NEXT_DATA__::text
```

### JSON Path (Product Page)

```
data.props.pageProps.product
```

### Product Fields

```json
{
  "productId": "12345",
  "name": "Chanel Classic Flap Bag",
  "brand": "Chanel",
  "pricing": {
    "price": 3500,
    "currency": "USD"
  },
  "seller": {
    "username": "luxuryseller",
    "rating": 4.8
  },
  "condition": "Very Good",
  "size": "Medium",
  "material": "Lambskin Leather",
  "color": "Black",
  "images": [
    {
      "url": "https://...",
      "position": 1
    }
  ]
}
```

### Required Setup (Python is better for this)

```python
import httpx
import json
from parsel import Selector

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive'
}

# MUST use residential proxy
proxy = "http://user:pass@residential-proxy.com:8080"

async with httpx.AsyncClient(
    headers=headers,
    http2=True,  # Required for Cloudflare
    proxies=proxy
) as client:
    response = await client.get(product_url)

    selector = Selector(response.text)
    json_text = selector.css('script#__NEXT_DATA__::text').get()
    data = json.loads(json_text)

    product = data['props']['pageProps']['product']
```

### Better Approach: Sitemap Scraping

1. Get product URLs from sitemap: `https://us.vestiairecollective.com/sitemaps/https_sitemap-en.xml`
2. Extract URLs with CSS selector: `url > loc::text`
3. Scrape each product page individually
4. Handle 308 redirects (sold items)

### JavaScript Alternative (Less Reliable)

```javascript
const axios = require('axios');
const cheerio = require('cheerio');

// Requires HTTP/2 proxy with residential IPs
const response = await axios.get(url, {
  headers: {
    'User-Agent': 'Mozilla/5.0...',
    'Accept-Language': 'en-US,en;q=0.9',
  },
  proxy: {
    host: 'residential-proxy.com',
    port: 8080,
    auth: { username: 'user', password: 'pass' }
  },
  httpAgent: new require('http2-wrapper').Agent()
});

const $ = cheerio.load(response.data);
const jsonData = JSON.parse($('script#__NEXT_DATA__').html());
```

---

## 5. Farfetch - NO RELIABLE SELECTORS (Use API Interception or Commercial API)

**Status:** CONNECTION TIMEOUT (Akamai Bot Manager)
**Success Rate DIY:** 15-20%
**Recommended:** Commercial API

### Approach 1: Find Internal API

Farfetch likely uses GraphQL or REST API. Intercept network requests:

```javascript
const puppeteer = require('puppeteer');

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();

await page.setRequestInterception(true);

page.on('request', request => {
  const url = request.url();
  if (url.includes('api') || url.includes('graphql')) {
    console.log('API FOUND:', url);
    console.log('Method:', request.method());
    console.log('POST:', request.postData());
  }
  request.continue();
});

await page.goto('https://www.farfetch.com/shopping/women/bags-1/items.aspx');
```

### Possible API Patterns

- `/api/products`
- `/graphql` (with POST query)
- `/api/search?query=chanel`
- `/api/catalog/products`

### Approach 2: Commercial APIs

**Recommended Options:**

1. **Apify Farfetch Scraper**
   - URL: https://apify.com/autofacts/farfetch
   - Pricing: Pay per use
   - Handles all anti-bot measures

2. **Retailed.io**
   - URL: https://www.retailed.io/datasources/api/farfetch-product
   - 50 free requests
   - JSON/CSV output

3. **Bright Data Scraping Browser**
   - Full browser automation with Akamai bypass
   - $100-500/month

### Why DIY is Not Worth It

- Akamai Bot Manager is military-grade
- Requires advanced fingerprinting evasion
- Even with perfect setup: <20% success rate
- Time spent debugging > API cost

---

## Summary Table: Best Extraction Method by Site

| Site | Best Method | Tools Needed | Difficulty | Success Rate |
|------|-------------|--------------|-----------|--------------|
| **Fashionphile** | Parse `__NEXT_DATA__` JSON | cheerio + puppeteer-real-browser | Easy | 85-90% |
| **Rebag** | `/products.json` API | axios only | Easiest | 95%+ |
| **The RealReal** | HTML scraping with bot bypass | puppeteer-real-browser + proxies | Hard | 60-70% |
| **Vestiaire** | Parse `__NEXT_DATA__` or sitemap | httpx (Python) + proxies | Hard | 70-80% |
| **Farfetch** | Commercial API | Apify/Retailed.io | N/A | 90%+ |

---

## Installation Requirements

### For Rebag + Fashionphile

```bash
npm install axios cheerio puppeteer-real-browser
```

### For The RealReal

```bash
npm install puppeteer-real-browser
sudo apt-get install xvfb  # Linux only
```

### For Vestiaire (Python recommended)

```bash
pip install httpx parsel
```

### Proxy Services (Residential IPs Required)

- Bright Data: https://brightdata.com
- Smartproxy: https://smartproxy.com
- Oxylabs: https://oxylabs.io

Cost: $50-200/month for residential proxies

---

## Testing Your Selectors

Use this script to test if selectors work:

```javascript
const { connect } = require('puppeteer-real-browser');

async function testSelectors(url, selectors) {
  const { page, browser } = await connect({
    headless: 'auto',
    fingerprint: true,
    turnstile: true
  });

  await page.goto(url, { waitUntil: 'networkidle2' });
  await sleep(5000);

  for (const [name, selector] of Object.entries(selectors)) {
    try {
      const count = await page.$$eval(selector, els => els.length);
      const sample = await page.$eval(selector, el => el.textContent?.trim());

      console.log(`✓ ${name}: ${selector}`);
      console.log(`  Count: ${count}`);
      console.log(`  Sample: ${sample}`);
    } catch (e) {
      console.log(`✗ ${name}: ${selector} NOT FOUND`);
    }
  }

  await browser.close();
}

// Test it
testSelectors('https://www.fashionphile.com/shop/bags?designers=CHANEL', {
  'JSON Data': 'script#__NEXT_DATA__',
  'Container': 'article[data-product-id]',
  'Title': 'h2.product-name',
  'Price': 'span[data-testid="product-price"]'
});
```

---

## Common Mistakes to Avoid

1. **Using CSS selectors on JavaScript-rendered content**
   - Solution: Always check for `__NEXT_DATA__` or JSON APIs first

2. **Not waiting for JavaScript to render**
   - Solution: Use `waitUntil: 'networkidle2'` and add 3-5 second delays

3. **Using datacenter proxy IPs**
   - Solution: Only residential proxies work on luxury sites

4. **Ignoring Cloudflare challenges**
   - Solution: Use puppeteer-real-browser with `turnstile: true`

5. **Scraping too fast**
   - Solution: 5-15 second delays between requests

6. **Not handling lazy-loaded images**
   - Solution: Scroll page before extracting data

7. **Using standard Puppeteer**
   - Solution: Use puppeteer-real-browser or puppeteer-extra-stealth

---

## Need Help?

1. Start with **Rebag** (easiest, no anti-bot issues)
2. Then **Fashionphile** (easy JSON extraction)
3. Skip or use commercial API for **Farfetch**
4. Only tackle **The RealReal** if you have time to set up proxies
5. **Vestiaire** is best scraped via sitemap