← back to Goodquestion

AI_PROVIDER_FALLBACK.md

293 lines

# AI Provider Fallback System

## Overview

The SWOT Analysis Platform now includes an intelligent AI provider fallback system that automatically switches between Anthropic's Claude and Google's Gemini based on token usage, rate limits, and API availability.

## Architecture

### Components

1. **AI Provider Manager** (`src/lib/ai-provider.ts`)
   - Unified interface for multiple AI providers
   - Automatic provider selection based on usage metrics
   - Token usage tracking with rolling window
   - Error handling and fallback logic

2. **Updated Workflows** (`src/lib/ai-workflows.ts`)
   - SWOT Analysis workflow
   - Historical Context workflow
   - Startup Costs workflow
   - All now use the unified AI provider

## Features

### Automatic Provider Selection

The system automatically selects the best provider based on:

1. **Rate Limit Detection**
   - Tracks API errors (429, low credit balance)
   - Implements 1-minute cooldown after rate limit errors
   - Automatically falls back to Gemini during cooldown

2. **Token Usage Monitoring**
   - Tracks output tokens in a rolling 1-minute window
   - Anthropic limit: 4,000 output tokens per minute
   - Switches to Gemini when usage exceeds 80% of limit (3,200 tokens)

3. **Error-Based Fallback**
   - Catches rate limit errors (HTTP 429)
   - Catches low credit balance errors (HTTP 400)
   - Automatically retries with Gemini on failure

### Provider Priority

1. **Primary**: Anthropic Claude Sonnet 4.5
   - Superior quality for complex business analysis
   - Used when available and within rate limits

2. **Fallback**: Google Gemini 1.5 Pro
   - High quality alternative
   - Generous rate limits
   - Activated automatically when needed

## Configuration

### Environment Variables

Add to `.env.local`:

```bash
# Anthropic API Configuration
ANTHROPIC_API_KEY=sk-ant-api03-...

# Google Gemini API Configuration
GOOGLE_API_KEY=your-gemini-api-key-here
```

### Get API Keys

- **Anthropic Claude**: https://console.anthropic.com/settings/keys
- **Google Gemini**: https://aistudio.google.com/app/apikey

## Usage

The fallback system works automatically. No code changes needed in your application:

```typescript
// Old approach (removed)
const message = await anthropic.messages.create({...})

// New approach (automatic fallback)
const response = await aiProvider.generateCompletion(prompt, {
  maxTokens: 5000,
  temperature: 1.0,
})

console.log(`Generated using ${response.provider}`) // 'anthropic' or 'gemini'
```

## Monitoring

### Console Logs

The system logs provider selection and usage:

```
[AI Provider] anthropic used 1234 tokens. Total in last minute: 2500
[SWOT] Generated using anthropic (1234 tokens)
[Historical] Generated using gemini (890 tokens)
[Startup Costs] Generated using anthropic (1567 tokens)
```

### Rate Limit Detection

```
[AI Provider] Anthropic is rate limited, using Gemini
[AI Provider] Approaching Anthropic token limit, using Gemini
[AI Provider] Anthropic failed (rate_limit_error), falling back to Gemini
```

### Statistics API

Access provider statistics programmatically:

```typescript
import { aiProvider } from '@/lib/ai-provider'

const stats = aiProvider.getStats()
// {
//   anthropic: {
//     provider: 'anthropic',
//     requestCount: 15,
//     totalTokens: 12500,
//     lastError: 'rate_limit_error',
//     lastErrorTime: Date
//   },
//   gemini: {
//     provider: 'gemini',
//     requestCount: 3,
//     totalTokens: 2800
//   }
// }

// Reset statistics
aiProvider.resetStats()
```

## Token Limits

### Anthropic Claude

- **Output Tokens**: 4,000 per minute
- **Input Tokens**: Separate limit (higher)
- **Fallback Threshold**: 3,200 tokens/min (80%)

### Google Gemini

- **Requests**: 15 per minute (free tier)
- **Tokens**: Much higher limits
- **Input/Output**: Combined into single limit

## Behavior Under Load

### Scenario 1: Normal Operation
- All requests use Anthropic Claude
- High quality responses
- ~1,200-1,500 output tokens per analysis

### Scenario 2: High Traffic
- First 2-3 requests use Anthropic (~4,000 tokens total)
- System detects approaching limit (3,200 threshold)
- Remaining requests use Gemini
- After 1 minute, Anthropic becomes available again

### Scenario 3: Rate Limited
- Anthropic returns 429 error
- Request automatically retries with Gemini
- 1-minute cooldown period activated
- All subsequent requests use Gemini
- After cooldown, Anthropic becomes available

### Scenario 4: Low Credits
- Anthropic returns "credit balance too low" error
- Request automatically retries with Gemini
- System remembers error for 1 minute
- Falls back to Gemini for all requests during cooldown

## Error Handling

The system handles errors gracefully:

1. **Anthropic Fails → Gemini Fallback**
   - Rate limit (429)
   - Low credits (400)
   - Connection timeout

2. **Both Fail → Error Returned**
   - If forced to use Gemini and it fails
   - Original error is thrown to caller

3. **Partial Results**
   - Route uses `Promise.allSettled()`
   - Returns partial data even if some workflows fail
   - Includes warnings array in response

## Testing

### Test Fallback Manually

Force Gemini usage:

```typescript
const response = await aiProvider.generateCompletion(prompt, {
  maxTokens: 5000,
  forceProvider: 'gemini'
})
```

### Test Rate Limit Behavior

1. Make multiple rapid requests to exhaust Anthropic quota
2. Watch logs for automatic Gemini fallback
3. Wait 60 seconds
4. Observe Anthropic becoming available again

### Verify Token Tracking

```typescript
import { aiProvider } from '@/lib/ai-provider'

// Make some requests
await generateAnalysis(...)

// Check stats
const stats = aiProvider.getStats()
console.log('Anthropic requests:', stats.anthropic.requestCount)
console.log('Anthropic tokens:', stats.anthropic.totalTokens)
```

## Benefits

1. **High Availability**: System continues working even when one provider fails
2. **Cost Optimization**: Uses free/cheaper Gemini when rate limits approached
3. **Quality First**: Prioritizes Claude for best results when available
4. **Transparent**: Automatic switching requires no code changes
5. **Observable**: Detailed logging for debugging and monitoring
6. **Resilient**: Graceful degradation with partial results

## Future Enhancements

Potential improvements:

1. **MCP Integration**: Store usage metrics in MCP for persistent monitoring
2. **Multiple Fallbacks**: Add OpenAI GPT-4 as tertiary option
3. **Load Balancing**: Distribute requests across providers proactively
4. **Quality Comparison**: A/B test provider outputs for quality metrics
5. **Cost Tracking**: Monitor and optimize based on API costs
6. **User Preference**: Allow users to select preferred provider
7. **Regional Routing**: Use different providers based on user location

## Troubleshooting

### Issue: Always using Gemini

**Cause**: Anthropic API key not configured or invalid

**Solution**:
- Check `.env.local` has valid `ANTHROPIC_API_KEY`
- Restart server after adding key
- Check console for authentication errors

### Issue: Gemini requests failing

**Cause**: Google API key not configured or quota exceeded

**Solution**:
- Add `GOOGLE_API_KEY` to `.env.local`
- Get free key from https://aistudio.google.com/app/apikey
- Check Google Cloud Console for quota limits

### Issue: Both providers failing

**Cause**: Network issues or both APIs down

**Solution**:
- Check internet connectivity
- Verify API status pages
- Check for firewall/proxy issues
- Review server logs for specific errors

## Summary

The AI Provider Fallback System provides enterprise-grade reliability through:

- Automatic provider selection based on real-time metrics
- Intelligent fallback on rate limits or errors
- Token usage tracking with rolling windows
- Graceful degradation with partial results
- Transparent operation requiring no application changes

This ensures the SWOT Analysis Platform remains available and responsive even under heavy load or API constraints.