← back to Cypress Awards

docs/SCRAPING.md

655 lines

# Web Scraping Strategy - CyPresAwards

## Overview

The CyPresAwards scraper is designed to ethically and efficiently discover, extract, and categorize cy pres statements from non-profit organization websites across the United States.

## Scraping Objectives

1. **Discover** cy pres pages on non-profit websites
2. **Extract** organization details (name, mission, logo)
3. **Extract** cy pres statement content
4. **Categorize** organizations by topic and legal relevance
5. **Store** all data in structured database format
6. **Log** all activity for transparency and debugging

## Ethical Scraping Principles

### 1. Robots.txt Compliance

The scraper checks `robots.txt` before accessing any URL:

```javascript
async checkRobotsTxt(url) {
  const robotsUrl = `${domain}/robots.txt`;
  const robots = robotsParser(robotsUrl, robotsTxtContent);
  return robots.isAllowed(url, this.userAgent);
}
```

**Behavior**: If robots.txt disallows access, the scraper skips the URL and logs it as blocked.

### 2. Rate Limiting

Default: 2000ms (2 seconds) between requests to the same domain

```javascript
await this.sleep(this.delay);
```

**Configurable via**: `SCRAPER_DELAY_MS` environment variable

**Rationale**: Prevents server overload and respects website resources

### 3. User-Agent Identification

Clear identification as a bot:

```
CyPresAwards Bot/1.0 (Legal Research Tool; +https://cypresawards.com/about)
```

**Benefits**:
- Transparency about scraping purpose
- Allows site owners to contact us
- Enables server-side bot detection

### 4. Scope Limitation

Only scrapes:
- Publicly accessible pages
- Cy pres-specific content
- No login/authentication required
- No personal data harvesting

Does NOT scrape:
- Private areas
- Email addresses (beyond public contact)
- Internal documents
- Gated content

## Cy Pres Page Discovery

### Strategy 1: Pattern-Based URL Guessing

Most common patterns for cy pres pages:

```javascript
const patterns = [
  `${domain}/cy-pres`,
  `${domain}/cy-pres/`,
  `${domain}/cypres`,
  `${domain}/cypres/`,
  `${domain}/cy-pres-awards`,
  `${domain}/legal/cy-pres`,
  `${domain}/awards/cy-pres`,
  `${domain}/support/cy-pres`,
  `${domain}/donate/cy-pres`,
  `${domain}/about/cy-pres`,
];
```

**Process**:
1. Try each pattern sequentially
2. Check if HTTP 200 response
3. Validate content contains cy pres keywords
4. If found, proceed to extraction
5. If not found, move to Strategy 2

**Success Rate**: ~60% of organizations with cy pres pages

### Strategy 2: Site Link Crawling

If pattern matching fails, crawl the homepage:

```javascript
async searchSiteForCyPres(baseUrl) {
  // 1. Fetch homepage
  const homepage = await axios.get(baseUrl);

  // 2. Find all links containing cy pres keywords
  $('a[href]').each((i, elem) => {
    const href = $(elem).attr('href');
    const text = $(elem).text().toLowerCase();

    if (href.includes('cy-pres') || text.includes('cy pres')) {
      links.push(normalizeUrl(href, baseUrl));
    }
  });

  // 3. Try each found link
  for (const link of links) {
    // Validate and extract
  }
}
```

**Success Rate**: ~30% of remaining sites

**Limitations**:
- Only checks homepage (doesn't deep crawl)
- Requires keyword in link text or URL
- Some sites may use JavaScript navigation

### Strategy 3: JavaScript-Rendered Sites

For sites that don't work with Axios/Cheerio:

```javascript
async scrapeDynamicSite(url) {
  const browser = await puppeteer.launch({ headless: 'new' });
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: 'networkidle2' });
  const html = await page.content();
  await browser.close();
  return html;
}
```

**When Used**: Fallback if static scraping fails

**Trade-offs**:
- Slower (launches Chrome)
- Higher resource usage
- More reliable for SPAs

## Content Validation

Before considering a page as valid cy pres content:

```javascript
containsCyPresKeywords(text) {
  const keywords = [
    'cy pres',
    'cy-pres',
    'cypres',
    'class action settlement',
    'unclaimed settlement funds',
    'residual funds',
    'settlement distribution',
    'class action award'
  ];

  let matchCount = 0;
  for (const keyword of keywords) {
    if (text.includes(keyword)) matchCount++;
  }

  return matchCount >= 2; // At least 2 matches required
}
```

**Rationale**: Prevents false positives from pages that merely mention cy pres

## Data Extraction

### 1. Organization Name

**Sources (in order of preference)**:
1. `<meta property="og:site_name">`
2. `<meta name="application-name">`
3. `<title>` tag
4. First `<h1>` element
5. `.site-title` or `#site-title` classes

**Fallback**: Domain name

### 2. Logo

**Sources**:
1. `<img alt="logo">` (case-insensitive)
2. `<img class="logo">`
3. `<img id="logo">`
4. `.logo img` selector
5. `#logo img` selector
6. First image in `<header>`

**Handling**:
- Converts relative URLs to absolute
- Validates URL format
- No logo = placeholder with first letter

### 3. Mission Statement

**Sources**:
1. `<meta name="description">`
2. `.mission` or `#mission` elements
3. `[class*="mission"]` elements
4. `.about` or `#about` elements

**Validation**:
- Must be at least 50 characters
- Limited to 1000 characters
- Cleaned of extra whitespace

### 4. Cy Pres Statement

**Extraction Process**:

```javascript
extractCyPresStatement(html) {
  // 1. Remove noise
  $('script, style, nav, footer, header').remove();

  // 2. Find main content area
  const mainSelectors = [
    'main',
    '[role="main"]',
    '.main-content',
    'article',
    '.content'
  ];

  // 3. Extract text
  let text = $(mainSelector).text().trim();

  // 4. Clean whitespace
  text = text.replace(/\s+/g, ' ');

  // 5. Limit length
  return text.substring(0, 5000);
}
```

**Also Stored**:
- Page title
- Original HTML (for reference)
- Extraction timestamp

## Automatic Categorization

### NLP-Based Classification

Uses Naive Bayes classifier trained on keyword sets:

```javascript
const trainingData = [
  {
    text: 'legal aid services lawyer attorney court',
    categories: ['Legal Aid']
  },
  {
    text: 'education school student learning university',
    categories: ['Education & Research']
  },
  // ... more training examples
];
```

**Process**:
1. Combine mission statement + cy pres text
2. Run through Bayes classifier
3. Take top 3 categories with confidence > 0.1
4. Link to organization in database

**Categories** (15 total):
- Legal Aid
- Advocacy & Policy
- Education & Research
- Community Development
- Health Services
- Environmental Conservation
- Arts & Culture
- Social Services
- Technology & Innovation
- Youth & Children
- Veterans Services
- Animal Welfare
- International Development
- Disability Rights
- Senior Services

### Legal Topic Identification

Keyword-based matching:

```javascript
const topicKeywords = {
  'Consumer Protection': [
    'consumer', 'fraud', 'advertising', 'ftc', 'unfair practices'
  ],
  'Civil Rights': [
    'civil rights', 'discrimination', 'equality', 'voting', 'ada'
  ],
  // ... 12 legal topics total
};

// Requires 2+ keyword matches
if (matchCount >= 2) topics.push(topicName);
```

**Legal Topics** (12 total):
- Consumer Protection
- Civil Rights
- Environmental Law
- Labor & Employment
- Privacy & Data Security
- Healthcare
- Education
- Housing
- Immigration
- Criminal Justice
- Technology & Internet
- Financial Services

## Batch Processing

### Concurrent Request Management

```javascript
async scrapeList(urls, maxConcurrent = 5) {
  for (let i = 0; i < urls.length; i += maxConcurrent) {
    const batch = urls.slice(i, i + maxConcurrent);
    const promises = batch.map(url => scrapeNonProfit(url));
    await Promise.allSettled(promises);
  }
}
```

**Configuration**:
- Default: 5 concurrent requests
- Configurable via `SCRAPER_MAX_CONCURRENT`

**Rationale**:
- Balances speed with politeness
- Prevents connection exhaustion
- Handles different domain speeds

### Error Handling

```javascript
try {
  const org = await scrapeNonProfit(url);
  await logScraping(url, 'success', 'Successfully scraped');
} catch (error) {
  console.error(`Error scraping ${url}:`, error.message);
  await logScraping(url, 'failed', error.message);
  // Continue to next URL (don't stop batch)
}
```

**Logged Statuses**:
- `success` - Found and extracted cy pres page
- `no_cypres_page` - No cy pres page found
- `blocked` - Blocked by robots.txt
- `failed` - Error occurred

## Database Storage

### Upsert Strategy

```javascript
INSERT INTO organizations (...)
VALUES (...)
ON CONFLICT (website_url)
DO UPDATE SET
  name = EXCLUDED.name,
  mission_statement = EXCLUDED.mission_statement,
  updated_at = CURRENT_TIMESTAMP
```

**Benefits**:
- Prevents duplicates
- Updates existing records
- Tracks last verification time

### Relationship Linking

```javascript
// Link categories
await Organization.addCategories(orgId, ['Legal Aid', 'Advocacy']);

// Link legal topics
await Organization.addLegalTopics(orgId, ['Consumer Protection']);
```

**Stored with**:
- Confidence scores (for ML-based classification)
- Relevance scores (for topic matching)

## Logging & Monitoring

### Scraping Logs Table

```sql
CREATE TABLE scraping_logs (
  id SERIAL PRIMARY KEY,
  url VARCHAR(1000) NOT NULL,
  status VARCHAR(50),
  error_message TEXT,
  scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

**Useful Queries**:

```sql
-- Success rate
SELECT
  status,
  COUNT(*) as count,
  ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) as percentage
FROM scraping_logs
GROUP BY status;

-- Recent failures
SELECT url, error_message, scraped_at
FROM scraping_logs
WHERE status = 'failed'
ORDER BY scraped_at DESC
LIMIT 20;
```

### Console Output

```
Scraping: https://example.org
✓ Successfully scraped: Example Org
  Categories: Legal Aid, Advocacy & Policy
  Legal Topics: Consumer Protection, Civil Rights

Progress: 50/100
```

## Running the Scraper

### Test Mode

```bash
npm run scrape:test
```

- Scrapes 2 sample URLs
- Sequential (not concurrent)
- Detailed console output

### Production Mode

```bash
npm run scrape
```

**Input Sources**:
1. **CSV File**: Load URLs from `data/nonprofits.csv`
2. **Database**: Query existing organizations for re-scraping
3. **API**: Fetch from external non-profit directory API
4. **Manual**: Edit `urls` array in `scraper/main.js`

**Recommended Schedule**:
- Initial scrape: Process all URLs
- Weekly updates: Re-scrape existing orgs
- Monthly audits: Verify data accuracy

## Performance

### Benchmarks

- **Static site** (Cheerio): ~1-2 seconds per site
- **Dynamic site** (Puppeteer): ~5-10 seconds per site
- **Rate limiting**: +2 seconds per request
- **Throughput**: ~100-200 orgs/hour (depending on complexity)

### Optimization Tips

1. **Increase concurrency** (carefully):
   ```javascript
   scraper.scrapeList(urls, 10); // More concurrent
   ```

2. **Reduce delay** for known-fast sites:
   ```javascript
   SCRAPER_DELAY_MS=1000 # 1 second
   ```

3. **Use Cheerio over Puppeteer** when possible

4. **Cache robots.txt** per domain (avoid repeated checks)

## Common Issues & Solutions

### Issue: Cy Pres Page Not Found

**Possible Causes**:
- Organization doesn't have cy pres page
- Page uses non-standard URL
- Page requires JavaScript

**Solutions**:
1. Manually check organization website
2. Add custom URL pattern
3. Use Puppeteer fallback
4. Add to manual review queue

### Issue: Incorrect Categorization

**Causes**:
- Limited training data
- Ambiguous mission statement
- Multiple organizational focuses

**Solutions**:
1. Add manual category override
2. Retrain classifier with more examples
3. Implement manual review workflow

### Issue: Blocked by Website

**Causes**:
- robots.txt disallows
- IP rate limiting
- Bot detection

**Solutions**:
1. Respect robots.txt (don't override)
2. Increase delay between requests
3. Rotate user agents (carefully)
4. Contact organization for permission

## Future Enhancements

### Planned Improvements

1. **Smart URL Discovery**
   - Machine learning to predict cy pres URLs
   - Sitemap.xml parsing
   - Google search API integration

2. **Advanced NLP**
   - Fine-tuned BERT model for categorization
   - Named entity recognition for key info
   - Sentiment analysis for mission alignment

3. **Distributed Scraping**
   - Message queue (RabbitMQ/Bull)
   - Worker pool architecture
   - Horizontal scaling

4. **Change Detection**
   - Content diffing
   - Email alerts for updates
   - Version history

5. **Data Enrichment**
   - IRS 990 data integration
   - GuideStar API integration
   - Social media links
   - Impact metrics

6. **Quality Assurance**
   - Manual review workflow
   - Confidence scoring
   - Data validation rules
   - Duplicate detection

## Compliance & Legal

### GDPR Considerations

- Only scrapes public information
- No personal data collection
- Respects data minimization principle
- Provides data removal on request

### CCPA Considerations

- Clear disclosure of data collection
- Opt-out mechanism available
- No sale of scraped data

### Terms of Service

Organizations can:
- Opt out via robots.txt
- Request data removal
- Update their information
- Report inaccuracies

Contact: [your-email]

## Testing the Scraper

### Unit Tests

```javascript
describe('CyPresFinder', () => {
  test('finds cy pres page by pattern', async () => {
    const finder = new CyPresFinder();
    const result = await finder.findCyPresPage('https://example.org');
    expect(result).toHaveProperty('url');
  });

  test('validates cy pres content', () => {
    const text = 'We accept cy pres awards from class action settlements';
    expect(finder.containsCyPresKeywords(text)).toBe(true);
  });
});
```

### Integration Tests

```javascript
test('full scraping pipeline', async () => {
  const scraper = new CyPresScraper();
  const org = await scraper.scrapeNonProfit('https://bettzedek.org');

  expect(org).toHaveProperty('name');
  expect(org).toHaveProperty('cypres_page_url');
  expect(org.cypres_page_url).toContain('cy-pres');
});
```

## Monitoring Dashboard (Future)

Proposed metrics:
- Total organizations scraped
- Success rate by day/week
- Average scraping time
- Error types distribution
- Top categories/topics
- Data freshness

---

**Last Updated**: 2024-10-13
**Version**: 1.0