`lib/yiddish-dictionary/README.md`

241 lines
# Yiddish Dictionary Integration for Dear Bubbe

This module provides a comprehensive Yiddish dictionary system for Dear Bubbe, allowing her to use authentic Yiddish terms with proper cultural context and safety filtering.

## Features

- **Multi-source dictionary import** - Combine multiple Yiddish dictionaries into one unified database
- **Bubbe-safe filtering** - Automatic detection and filtering of potentially offensive terms
- **Cultural context preservation** - Maintains usage examples and cultural notes
- **Usage tracking** - Learns which terms Bubbe uses most frequently
- **Contextual suggestions** - Provides appropriate terms based on conversation context
- **Hebrew script & transliteration** - Supports both יידיש and Latin transliteration

## Setup Instructions

### 1. Download Dictionary Sources

Download Yiddish dictionaries from these GitHub repositories:

- [Solomon Birnbaum Dictionary](https://github.com/SolomonBirnbaum/yiddish-dict)
- [Yiddish Book Center Dictionary](https://github.com/yiddishbookcenter/dictionaries)
- [Niborski Modern Yiddish](https://github.com/niborski/yiddish)
- [Weinreich Yiddish-English](https://github.com/weinreich/yiddish-english)

Save them in a `dictionaries/` subdirectory:
```
lib/yiddish-dictionary/
├── dictionaries/
│   ├── solomon_yiddish.json
│   ├── yiddish_book_center.tsv
│   ├── niborski_modern.csv
│   └── weinreich_dictionary.json
```

### 2. Build the Database

```bash
cd lib/yiddish-dictionary
python3 build_yiddish_db.py
```

This will create `yiddish_dictionary.sqlite` with:
- All dictionary entries merged and deduplicated
- Automatic filtering of problematic terms
- Pre-populated Bubbe favorites
- Usage tracking tables

### 3. Install Dependencies

```bash
npm install better-sqlite3
npm install --save-dev @types/better-sqlite3
```

### 4. Use in Your Code

```typescript
import { getDictionary } from '@/lib/yiddish-dictionary/dictionary-api';

// Get dictionary instance
const dict = getDictionary();

// Search for terms
const results = dict.searchEnglish('grandmother');
const yiddishResults = dict.searchYiddish('bubbe');

// Get random expressions
const insult = dict.getRandomInsult();
const expression = dict.getRandomExpression(5, 8); // intensity 5-8

// Get contextual suggestions
const complaints = dict.getSuggestions('complaint', 5);

// Track usage
dict.recordUsage('meshugana', 'scolding');

// Get statistics
const stats = dict.getStats();
console.log(`Dictionary has ${stats.approvedEntries} approved terms`);
```

## Database Schema

### Main Tables

- **entries** - Dictionary entries with translations
- **phrases** - Multi-word expressions and idioms  
- **sources** - Dictionary sources and licenses
- **bubbe_favorites** - Curated expressions with intensity ratings
- **banned_terms** - Terms to avoid (supplements banned-terms.json)
- **usage_stats** - Tracks term usage for learning

### Key Fields

- `bubbe_approved`: 1=safe, 0=needs review, -1=banned
- `intensity`: 1-10 scale for Bubbe's harshness
- `context`: greeting, insult, complaint, advice, etc.
- `cultural_note`: Important usage context

## API Methods

### Search Functions
- `searchEnglish(query, limit)` - Find by English translation
- `searchYiddish(query, limit)` - Find by Yiddish/transliteration
- `getPhrasesByContext(context, limit)` - Get contextual phrases

### Random Selection
- `getRandomInsult()` - Random scolding term
- `getRandomEndearment()` - Random term of endearment (used sarcastically)
- `getRandomExpression(min, max)` - Random by intensity level

### Contextual Suggestions
- `getSuggestions(context, limit)` - Get terms for specific contexts
- `getBubbeFavorites(context, minIntensity)` - Get Bubbe's preferred expressions

### Management
- `isBanned(term, language)` - Check if term is banned
- `addBannedTerm(...)` - Add new banned term
- `recordUsage(term, context)` - Track term usage
- `getMostUsedTerms(limit)` - Get frequently used terms

## Integration with Dear Bubbe

### In Chat Route (`/api/chat/route.ts`)

```typescript
import { getDictionary } from '@/lib/yiddish-dictionary/dictionary-api';

// Enhance Bubbe's response with authentic Yiddish
function enhanceWithYiddish(response: string, intensity: number): string {
  const dict = getDictionary();
  
  // Get a contextual expression
  const expression = dict.getRandomExpression(intensity - 2, intensity + 2);
  if (expression) {
    response = `${expression.english}! ${response}`;
    dict.recordUsage(expression.english, 'response');
  }
  
  // Add an insult if appropriate
  if (intensity > 5) {
    const insult = dict.getRandomInsult();
    if (insult) {
      response = response.replace('you', `you ${insult.english}`);
      dict.recordUsage(insult.english, 'insult');
    }
  }
  
  return response;
}
```

### Content Filtering

```typescript
// Check for banned terms before processing
function filterContent(text: string): string {
  const dict = getDictionary();
  
  const words = text.split(/\s+/);
  const filtered = words.map(word => {
    if (dict.isBanned(word, 'en')) {
      return '[removed]';
    }
    return word;
  });
  
  return filtered.join(' ');
}
```

## Maintenance

### Adding New Sources

Edit `INPUT_SOURCES` in `build_yiddish_db.py`:

```python
INPUT_SOURCES.append({
    "path": "dictionaries/new_source.json",
    "format": "json",
    "name": "New Dictionary Source",
    "url": "https://github.com/...",
    "license": "MIT",
    "field_map": {
        "word": "yiddish",
        "translation": "english",
        # Map source fields to database columns
    }
})
```

### Reviewing Terms

```sql
-- Find terms needing review
SELECT * FROM entries WHERE bubbe_approved = 0;

-- Approve a term
UPDATE entries SET bubbe_approved = 1 WHERE id = ?;

-- Ban a term
UPDATE entries SET bubbe_approved = -1 WHERE id = ?;
```

### Backup

```bash
# Backup database
cp yiddish_dictionary.sqlite yiddish_dictionary.backup.sqlite

# Export to JSON
sqlite3 yiddish_dictionary.sqlite ".mode json" ".output backup.json" \
  "SELECT * FROM entries WHERE bubbe_approved = 1" ".quit"
```

## Security Considerations

1. **Automatic Filtering** - Terms matching problematic patterns are flagged for review
2. **Manual Review** - Admin can review and approve/ban terms via dashboard
3. **Context Tracking** - Usage patterns help identify misuse
4. **Replacement Suggestions** - Banned terms include safe alternatives

## Performance

- SQLite database with indexes for fast lookups
- In-memory caching via singleton pattern
- Typical query time: <5ms for searches
- Database size: ~5-20MB depending on sources

## License

The dictionary integration code is part of Dear Bubbe (proprietary).
Individual dictionary sources maintain their original licenses (MIT, GPL, CC-BY-SA, etc.)

## Support

For issues or questions:
- Check the admin dashboard at http://45.61.58.125:5013
- Review flagged terms in the database
- Monitor usage statistics for patterns