`README.md`

339 lines
# CyPresAwards

**A comprehensive database of US non-profit organizations with cy pres statements for legal settlement distribution.**

## Overview

CyPresAwards is a specialized web application designed for legal professionals to identify suitable non-profit organizations for cy pres awards in class action settlements. The platform automatically scrapes, categorizes, and indexes non-profits that have published cy pres statements on their websites.

### What is Cy Pres?

Cy pres (pronounced "see pray") is a legal doctrine used in class action settlements when leftover or unclaimed settlement funds cannot be distributed to class members. Instead, these funds are awarded to non-profit organizations whose missions align with the nature of the legal case.

## Features

- **Automated Web Scraping**: Discovers cy pres pages across non-profit websites
- **Intelligent Categorization**: Uses NLP to automatically categorize organizations by topic
- **Legal Topic Matching**: Maps organizations to relevant legal practice areas
- **Advanced Search**: Filter by organization type, legal topics, location, and keywords
- **Comprehensive Database**: Stores mission statements, logos, cy pres statements, and contact info
- **Clean UI**: Professional interface designed for legal professionals

## Tech Stack

### Backend
- **Node.js** with Express.js
- **PostgreSQL** database with full-text search
- **Cheerio** & **Puppeteer** for web scraping
- **Natural** (NLP library) for text classification
- **Robots-parser** for ethical scraping

### Frontend
- Vanilla JavaScript (no framework dependencies)
- Modern CSS with responsive design
- REST API integration

## Project Structure

```
cypresawards/
├── backend/
│   ├── src/
│   │   ├── api/
│   │   │   └── routes.js          # API endpoints
│   │   ├── models/
│   │   │   ├── database.js        # PostgreSQL connection
│   │   │   └── Organization.js    # Data models & queries
│   │   ├── scraper/
│   │   │   ├── cypresFinder.js    # Web scraping logic
│   │   │   └── main.js            # Scraper orchestration
│   │   ├── utils/
│   │   └── server.js              # Express server
│   ├── package.json
│   └── .env.example
├── frontend/
│   ├── src/
│   │   ├── index.html             # Main HTML
│   │   ├── styles.css             # Styles
│   │   └── app.js                 # Frontend logic
│   └── package.json
├── database/
│   └── schema.sql                 # Database schema
├── docs/
│   ├── ARCHITECTURE.md            # System architecture
│   └── SCRAPING.md                # Scraping strategy
└── README.md
```

## Installation

### Prerequisites
- Node.js 18+ and npm
- PostgreSQL 14+
- Git

### 1. Clone the Repository

```bash
git clone <repository-url>
cd cypresawards
```

### 2. Setup Database

```bash
# Create PostgreSQL database
createdb cypresawards

# Run schema
psql -d cypresawards -f database/schema.sql
```

### 3. Configure Backend

```bash
cd backend
npm install

# Copy and configure environment variables
cp .env.example .env
# Edit .env with your database credentials
```

### 4. Configure Frontend

```bash
cd ../frontend
npm install
```

## Usage

### Start the Backend API

```bash
cd backend
npm run dev
```

The API will be available at `http://localhost:3000`

### Start the Frontend

```bash
cd frontend
npm start
```

The web interface will open at `http://localhost:8080`

### Run the Scraper

#### Test Mode (2 sample URLs)
```bash
cd backend
npm run scrape:test
```

#### Production Mode
```bash
# First, add URLs to scrape in backend/src/scraper/main.js
# Then run:
npm run scrape
```

## API Endpoints

### Organizations

- `GET /api/organizations` - Search and filter organizations
  - Query params: `search`, `legalTopics`, `categories`, `state`, `limit`, `offset`
- `GET /api/organizations/:id` - Get single organization

### Reference Data

- `GET /api/categories` - Get all organization categories
- `GET /api/legal-topics` - Get all legal topics
- `GET /api/stats` - Get database statistics

### System

- `GET /health` - Health check

## Scraper Details

The scraper employs multiple strategies to find cy pres pages:

1. **Pattern Matching**: Checks common URL patterns (`/cy-pres`, `/cypres`, etc.)
2. **Site Search**: Crawls homepage for cy pres-related links
3. **Content Validation**: Verifies pages contain genuine cy pres content
4. **Robots.txt Compliance**: Respects robots.txt directives
5. **Rate Limiting**: Configurable delays between requests

### Extracted Data

- Organization name and logo
- Mission statement
- Cy pres statement (full text)
- Website and cy pres page URLs
- Location information
- Automatic categorization (legal topics & org types)

## Search & Filtering

Users can filter organizations by:

- **Free-text search**: Name and mission statement (full-text search)
- **Legal topics**: Consumer Protection, Civil Rights, Environmental, etc.
- **Categories**: Legal Aid, Education, Health Services, etc.
- **Location**: Filter by US state
- **Combined filters**: All filters work together

## Database Schema

Key tables:
- `organizations` - Core organization data
- `cypres_statements` - Full cy pres statement text
- `categories` - Organization type taxonomy
- `legal_topics` - Legal practice area taxonomy
- `organization_categories` - Many-to-many relationships
- `organization_legal_topics` - Many-to-many relationships
- `scraping_logs` - Audit trail

See `database/schema.sql` for full details.

## Configuration

### Backend Environment Variables

```env
# Database
DB_HOST=localhost
DB_PORT=5432
DB_NAME=cypresawards
DB_USER=postgres
DB_PASSWORD=your_password

# Server
PORT=3000
NODE_ENV=development

# Scraper
SCRAPER_DELAY_MS=2000
SCRAPER_MAX_CONCURRENT=5
SCRAPER_USER_AGENT=CyPresAwards Bot/1.0
```

### Frontend Configuration

Update `API_BASE_URL` in `frontend/src/app.js` to point to your backend:

```javascript
const API_BASE_URL = 'http://localhost:3000/api';
```

## Development

### Adding New Legal Topics

Edit `database/schema.sql` and add to the `legal_topics` INSERT statement:

```sql
INSERT INTO legal_topics (name, description, keywords) VALUES
('Your Topic', 'Description', ARRAY['keyword1', 'keyword2']);
```

### Adding New Categories

Similarly, edit the `categories` INSERT statement.

### Customizing Scraper Behavior

Edit `backend/src/scraper/cypresFinder.js`:

- `getCyPresUrlPatterns()` - Add more URL patterns
- `containsCyPresKeywords()` - Adjust keyword matching
- `extractOrgDetails()` - Change extraction logic

## Deployment

### Production Checklist

- [ ] Set `NODE_ENV=production` in backend .env
- [ ] Configure production database
- [ ] Set up SSL/TLS certificates
- [ ] Configure CORS for production domain
- [ ] Set up logging and monitoring
- [ ] Schedule periodic scraping (cron job)
- [ ] Set up database backups

### Recommended Hosting

- **Backend**: Heroku, AWS, DigitalOcean
- **Database**: Heroku Postgres, AWS RDS, DigitalOcean Managed Postgres
- **Frontend**: Netlify, Vercel, AWS S3 + CloudFront

## Legal & Ethical Considerations

### Scraping Ethics

- Respects `robots.txt`
- Rate-limited requests (2 second default delay)
- Identifies bot with clear user agent
- Only scrapes public cy pres pages
- Logs all activity for transparency

### Data Usage

This tool is intended for:
- Legal professionals researching cy pres recipients
- Non-profits understanding cy pres opportunities
- Academic research on cy pres distributions

**Not intended for:**
- Bulk email harvesting
- Commercial solicitation
- Data resale

## Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

## Future Enhancements

- [ ] Add IRS 990 data integration
- [ ] Email notification system for new cy pres pages
- [ ] Export results to CSV/PDF
- [ ] Admin dashboard for manual data curation
- [ ] API authentication and rate limiting
- [ ] Elasticsearch integration for better search
- [ ] Multi-language support
- [ ] Mobile app

## License

MIT License - See LICENSE file for details

## Support

For issues, questions, or suggestions:
- Open a GitHub issue
- Contact: [your-email]

## Acknowledgments

Built with open-source tools and libraries. Special thanks to:
- Anthropic Claude for development assistance
- The non-profit sector for cy pres transparency
- The legal community for supporting charitable distributions

---

**Disclaimer**: This database is provided for informational purposes only. Always verify organization details and cy pres statements directly before making legal decisions.