← back to Handbag Auth Nextjs

python-matcher/README.md

258 lines

# Handbag Visual Matching System

Vector-based image similarity search for exact handbag model/pattern identification using CLIP embeddings and pgvector.

## Architecture

```
┌─────────────┐
│ User Image  │
└──────┬──────┘
       │
       ▼
┌─────────────────────┐
│  CLIP Embedder      │  ← openai/clip-vit-large-patch14
│  (768-dim vector)   │
└──────┬──────────────┘
       │
       ▼
┌─────────────────────┐
│  PostgreSQL +       │
│  pgvector           │  ← Vector similarity search
│  (IVFFlat index)    │
└──────┬──────────────┘
       │
       ▼
┌─────────────────────┐
│  Top K Matches      │  ← Ranked by cosine similarity
│  with Scores        │
└─────────────────────┘
```

## Setup

### 1. Install Dependencies

```bash
cd /root/Projects/handbag-auth-nextjs/python-matcher
python3 -m pip install -r requirements.txt
```

### 2. Setup PostgreSQL Database

```bash
# Create database
createdb handbags

# Enable pgvector and create tables
psql handbags < scripts/setup_db.sql
```

### 3. Configure Environment

```bash
export DATABASE_URL="postgresql://localhost/handbags"
export API_PORT=8000
```

## Usage

### Embed Handbag Catalog

Process all handbags and generate vector embeddings:

```bash
cd /root/Projects/handbag-auth-nextjs/python-matcher
python3 scripts/embed_handbags.py
```

This will:
- Fetch all handbags with `embedding IS NULL`
- Download images
- Generate 768-dim CLIP vectors
- Update database with embeddings
- Process in batches of 100

### Start Matching API

```bash
python3 api/server.py
```

Or with uvicorn:

```bash
uvicorn api.server:app --host 0.0.0.0 --port 8000
```

### API Endpoints

**Health Check:**
```bash
GET /
```

**Statistics:**
```bash
GET /stats
```

**Match by URL:**
```bash
POST /handbags/match/url
Content-Type: application/json

{
  "image_url": "https://example.com/handbag.jpg",
  "top_k": 5,
  "min_similarity": 0.7
}
```

**Match by Upload:**
```bash
POST /handbags/match/upload
Content-Type: multipart/form-data

file: <image file>
top_k: 5
min_similarity: 0.7
```

**Get by SKU:**
```bash
GET /handbags/{sku}
```

## Response Format

```json
{
  "query_image": "https://example.com/handbag.jpg",
  "matches": [
    {
      "id": 1,
      "sku": "HERMES-BIRKIN-35-TOGO-BLACK-GHW",
      "brand": "Hermès",
      "model_name": "Birkin 35",
      "pattern_name": "Togo",
      "colorway": "Black",
      "material": "Togo Leather",
      "hardware": "Gold",
      "size": "35cm",
      "condition": "Excellent",
      "price_usd": 15000.00,
      "image_url": "https://...",
      "similarity": 0.95,
      "source": "authenticated_catalog"
    }
  ],
  "count": 5
}
```

## Database Schema

```sql
CREATE TABLE handbags (
  id              bigserial PRIMARY KEY,
  sku             text UNIQUE NOT NULL,
  brand           text NOT NULL,
  model_name      text NOT NULL,
  pattern_name    text,
  colorway        text,
  material        text,
  hardware        text,
  size            text,
  condition       text,
  year            integer,
  price_usd       numeric(10, 2),
  price_jpy       numeric(10, 2),
  image_url       text NOT NULL,
  source          text,
  external_id     text,
  created_at      timestamp DEFAULT CURRENT_TIMESTAMP,
  updated_at      timestamp DEFAULT CURRENT_TIMESTAMP,
  embedding       vector(768),  -- CLIP embeddings
  is_active       boolean DEFAULT true,
  verified        boolean DEFAULT false
);
```

## Model Verification

Only handbags with verifiable model numbers are processed. Known patterns by brand:

- **Hermès**: Birkin, Kelly, Constance, Evelyne, Garden Party, Picotin, Bolide, Lindy
- **Chanel**: Classic Flap, Boy, 19, 22, Gabrielle, Diana, Coco Handle, 2.55
- **Louis Vuitton**: Neverfull, Speedy, Alma, Keepall, Pochette, Metis, Capucines
- **Dior**: Lady Dior, Saddle, Book Tote, 30 Montaigne
- **Gucci**: Marmont, Dionysus, Jackie, Bamboo, Soho, Ophidia
- **Prada**: Galleria, Cahier, Sidonie, Cleo, Re-edition
- And more...

## Performance

- **Embedding Generation**: ~50-100ms per image (CPU), ~10-20ms (GPU)
- **Vector Search**: <10ms for 100K vectors with IVFFlat index
- **Batch Processing**: ~1000 images/hour (CPU)

## Integration with Next.js

Update camera page to call matching API:

```typescript
const analyzeWithVector = async (imageBlob: Blob) => {
  const formData = new FormData()
  formData.append('file', imageBlob)
  formData.append('top_k', '5')
  formData.append('min_similarity', '0.7')

  const response = await fetch('http://localhost:8000/handbags/match/upload', {
    method: 'POST',
    body: formData
  })

  const data = await response.json()
  return data.matches
}
```

## Deployment

### Using PM2

```bash
pm2 start api/server.py --name handbag-matcher --interpreter python3
pm2 save
```

### Using systemd

```bash
sudo cp deployment/handbag-matcher.service /etc/systemd/system/
sudo systemctl enable handbag-matcher
sudo systemctl start handbag-matcher
```

## Maintenance

**Re-embed catalog after updates:**
```bash
python3 scripts/embed_handbags.py
```

**Rebuild vector index:**
```sql
REINDEX INDEX idx_handbags_embedding;
```

**Check embedding coverage:**
```sql
SELECT
  COUNT(*) as total,
  COUNT(embedding) as embedded,
  (COUNT(embedding)::float / COUNT(*)::float * 100) as coverage_pct
FROM handbags WHERE is_active = true;
```