`REPLICATE_MODELS_QUICK_REFERENCE.md`

304 lines
# Replicate Models for Dramatic Video Animation - Quick Reference

## Problem Identified
- **SVD (Stable Video Diffusion)**: Produces very subtle, minimal motion - NOT suitable for dramatic effects
- **Luma models**: Confusion about video output - some versions output single frames instead of video

## Solutions: Dramatic Motion Models

### 1. BEST FOR DRAMATIC SCENE ANIMATION (Starry Night swirling stars, etc.)

**Model ID**: `kwaivgi/kling-v2.5-turbo-pro`

**Why**: Complex camera moves, high-speed action, fluid motion, excellent prompt adherence

**Usage**:
```javascript
{
  model: "kwaivgi/kling-v2.5-turbo-pro",
  input: {
    prompt: "The stars in van Gogh's Starry Night swirl dramatically in spiraling patterns, the cypress tree sways, clouds move energetically across the sky",
    start_image: "https://url-to-image.jpg",
    duration: 10,  // seconds
    aspect_ratio: "16:9"
  }
}
```

**Output**: MP4 video URL with dramatic motion
**Cost**: $0.28 per second ($2.80 for 10s video)
**Runs**: 1.6M (most popular)

---

### 2. BEST FOR FACE ANIMATION - SINGING/TALKING

**Model ID**: `cjwbw/sadtalker`

**Why**: Purpose-built for audio-driven facial animation with realistic lip-sync, head movement, eye blinking

**Usage**:
```javascript
{
  model: "cjwbw/sadtalker",
  input: {
    source_image: "https://portrait.jpg",
    driven_audio: "https://song-or-speech.wav",
    use_enhancer: true,        // GFPGAN face enhancement
    use_eyeblink: true,         // Add eye blinking
    pose_style: 15,             // 0-45, controls head movement
    expression_scale: 1.0       // Expression intensity
  }
}
```

**Output**: Animated talking/singing face video
**Best For**: Museum portraits coming to life, historical figures speaking

---

### 3. BEST FOR CINEMATIC QUALITY + AUDIO

**Model ID**: `google/veo-3.1`

**Why**: State-of-the-art quality, realistic physics, generates audio automatically, supports reference images

**Usage**:
```javascript
{
  model: "google/veo-3.1",
  input: {
    image: "https://scene.jpg",
    prompt: "Orchestra playing with dramatic music and movement",
    duration: 8,                // 4, 6, or 8 seconds
    resolution: "1080p",        // 720p or 1080p
    generate_audio: true,       // Auto-generate synchronized audio
    aspect_ratio: "16:9"
  }
}
```

**Output**: High-fidelity video with synchronized audio
**Runs**: 272K

---

### 4. BEST FOR DRAMATIC HUMAN MOTION + CINEMATIC CAMERA

**Model ID**: `minimax/hailuo-2.3`

**Why**: Optimized for realistic human motion, cinematic VFX, handles complex physical movements (gymnastics, parkour)

**Usage**:
```javascript
{
  model: "minimax/hailuo-2.3",
  input: {
    prompt: "Renaissance marketplace with people walking, merchants selling, camera panning across busy scene",
    first_frame_image: "https://painting.jpg",
    duration: 10,               // 6 or 10 seconds
    resolution: "768p"          // 768p or 1080p (1080p limited to 6s)
  }
}
```

**Output**: Cinematic video with dramatic camera movements
**Runs**: 38K

---

### 5. BEST FOR PROFESSIONAL LIPSYNC

**Model ID**: `sync/lipsync-2-pro`

**Why**: Studio-grade quality with advanced emotional nuance, best for professional installations

**Usage**:
```javascript
{
  model: "sync/lipsync-2-pro",
  input: {
    video_or_image: "https://portrait.jpg",
    audio: "https://narration.mp3"
  }
}
```

**Output**: Professional-grade lipsynced video
**Alternative**: `sync/lipsync-2` (faster, cheaper version)

---

### 6. BEST FOR VIDEO-DRIVEN FACE ANIMATION

**Model ID**: `fofr/live-portrait`

**Why**: Transfer facial expressions/movements from any video to your portrait

**Usage**:
```javascript
{
  model: "fofr/live-portrait",
  input: {
    face_image: "https://portrait.jpg",
    driving_video: "https://singing-performance.mp4",  // Source of motion
    live_portrait_eye_retargeting: true,
    live_portrait_lip_retargeting: true
  }
}
```

**Output**: Portrait animated with motion from driving video
**Limitation**: Cannot be used commercially (licensing restriction)

---

### 7. BEST FOR AUDIO SYNC + VIDEO

**Model ID**: `wan-video/wan-2.5-i2v`

**Why**: One-pass audio/video sync, lip-sync features, custom audio integration

**Usage**:
```javascript
{
  model: "wan-video/wan-2.5-i2v",
  input: {
    image: "https://face.jpg",
    prompt: "Person singing with emotion",
    audio: "https://song.mp3",  // 3-30 seconds, ≤15MB
    duration: 10,
    resolution: "1080p"
  }
}
```

**Output**: Video with synchronized lip movements and audio
**Runs**: 153.5K

---

### 8. LUMA MODELS (Fixed)

**Correct Model ID for Video Output**: `luma/ray`

**Other Luma Models**:
- `luma/ray-2-720p` - 720p video output (5s and 9s)
- `luma/ray-2-540p` - 540p video output
- `luma/ray-flash-2-720p` - Faster 720p generation
- `luma/ray-flash-2-540p` - Faster 540p generation

**Usage**:
```javascript
{
  model: "luma/ray-2-720p",
  input: {
    prompt: "Dramatic motion in scene",
    image: "https://image.jpg"  // Optional for image-to-video
  }
}
```

**Output**: MP4 video URL (NOT single frames)
**Duration**: 5 or 9 seconds

---

## Model Comparison Matrix

| Use Case | Best Model | Runner-up | Why |
|----------|-----------|-----------|-----|
| **Dramatic scene motion** | `kwaivgi/kling-v2.5-turbo-pro` | `minimax/hailuo-2.3` | Complex camera moves, fluid motion, strong prompt adherence |
| **Face singing/talking** | `cjwbw/sadtalker` | `wan-video/wan-2.5-i2v` | Purpose-built for audio-driven faces, natural lip-sync |
| **Cinematic quality** | `google/veo-3.1` | `tencent/hunyuan-video` | State-of-the-art, realistic physics, native audio |
| **Professional lipsync** | `sync/lipsync-2-pro` | `sync/lipsync-2` | Studio-grade quality, emotional nuance |
| **Cost-effective** | `wan-video/wan-2.5-i2v-fast` | `bytedance/seedance-1-pro-fast` | Fast variants, good quality/cost balance |

---

## Key Insights

### What Makes Motion "Dramatic"?

1. **Prompt Engineering**: Use action verbs
   - Good: "swirling, dancing, spiraling, energetic motion"
   - Bad: "moving slightly"

2. **Model Selection**: Choose motion-focused models
   - SVD = subtle drift (avoid)
   - Kling/Hailuo = dramatic action (use)

3. **Duration**: Longer allows more dramatic development
   - 5s = limited motion
   - 10s = full dramatic arc

### Best Practices

1. **For museum portraits**: Use `sadtalker` with audio file
2. **For paintings coming alive**: Use `kling-v2.5-turbo-pro` with motion prompt
3. **For professional quality**: Use `veo-3.1` or `lipsync-2-pro`
4. **For quick tests**: Use `-fast` or `-turbo` variants
5. **Always specify motion**: Don't rely on model defaults

### Common Pitfalls

- Using SVD for dramatic effects (too subtle)
- Not specifying motion in prompts
- Trying to use text-only models for image-to-video
- Ignoring duration vs resolution tradeoffs
- Forgetting audio file size/length limits

---

## Migration from SVD

### Old Code (Subtle Motion):
```javascript
{
  model: "stability-ai/stable-video-diffusion",
  input: {
    image: "https://starry-night.jpg"
  }
}
// Result: Barely visible motion
```

### New Code (Dramatic Motion):
```javascript
{
  model: "kwaivgi/kling-v2.5-turbo-pro",
  input: {
    prompt: "Stars swirling dramatically in spiraling patterns, cypress tree swaying, clouds moving energetically",
    start_image: "https://starry-night.jpg",
    duration: 10,
    aspect_ratio: "16:9"
  }
}
// Result: Dramatic, visible, engaging motion
```

---

## Additional Resources

- Full research report: `DRAMATIC_VIDEO_MODELS_RESEARCH.json`
- Replicate Image-to-Video Collection: https://replicate.com/collections/image-to-video
- Replicate Lipsync Collection: https://replicate.com/collections/lipsync

---

## Quick Decision Tree

```
Need to animate a face?
├─ Yes → Using audio file?
│  ├─ Yes → sadtalker or wan-2.5-i2v
│  └─ No → live-portrait (with driving video)
│
└─ No → Animating a scene?
   ├─ Need dramatic motion? → kling-v2.5-turbo-pro
   ├─ Need cinematic quality? → veo-3.1
   ├─ Need human motion? → hailuo-2.3
   └─ Need it fast/cheap? → wan-2.5-i2v-fast
```