← back to Cypress Awards

docs/ARCHITECTURE.md

512 lines

# CyPresAwards - System Architecture

## Overview

CyPresAwards is a three-tier web application consisting of:
1. **Frontend** - Single-page application (SPA)
2. **Backend API** - RESTful Node.js/Express server
3. **Database** - PostgreSQL with full-text search

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────────┐
│                         FRONTEND                            │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  HTML/CSS/JavaScript                                  │  │
│  │  - Search Interface                                   │  │
│  │  - Filter Controls                                    │  │
│  │  - Organization Cards                                 │  │
│  │  - Pagination                                         │  │
│  └──────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                           │
                           │ HTTP/REST API
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                     BACKEND API SERVER                      │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  Express.js Routes                                    │  │
│  │  - /api/organizations                                 │  │
│  │  - /api/categories                                    │  │
│  │  - /api/legal-topics                                  │  │
│  │  - /api/stats                                         │  │
│  └──────────────────────────────────────────────────────┘  │
│                           │                                 │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  Business Logic Layer                                 │  │
│  │  - Organization Model                                 │  │
│  │  - Query Builder                                      │  │
│  │  - Data Validation                                    │  │
│  └──────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                           │
                           │ SQL Queries
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                    POSTGRESQL DATABASE                      │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  Tables:                                              │  │
│  │  - organizations                                      │  │
│  │  - cypres_statements                                  │  │
│  │  - categories                                         │  │
│  │  - legal_topics                                       │  │
│  │  - organization_categories (junction)                 │  │
│  │  - organization_legal_topics (junction)               │  │
│  │  - scraping_logs                                      │  │
│  └──────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                           ▲
                           │
┌─────────────────────────────────────────────────────────────┐
│                    WEB SCRAPER (Batch)                      │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  CyPresScraper                                        │  │
│  │  ├─ CyPresFinder (page discovery)                     │  │
│  │  ├─ Content Extractor                                 │  │
│  │  ├─ NLP Classifier (categorization)                   │  │
│  │  └─ Database Writer                                   │  │
│  └──────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
```

## Component Details

### 1. Frontend Layer

**Technology**: Vanilla JavaScript, HTML5, CSS3

**Key Files**:
- `index.html` - Main page structure
- `styles.css` - Responsive styling
- `app.js` - Application logic and API communication

**Responsibilities**:
- User interface rendering
- Search and filter form handling
- API request management
- State management (client-side)
- Pagination controls
- Error handling and loading states

**Design Patterns**:
- Single Page Application (SPA)
- State management with JavaScript object
- Event-driven architecture
- Async/await for API calls
- Responsive grid layout

### 2. Backend API Layer

**Technology**: Node.js 18+, Express.js 4.x

**Key Files**:
- `server.js` - Express server setup and middleware
- `api/routes.js` - API endpoint definitions
- `models/Organization.js` - Data access layer
- `models/database.js` - PostgreSQL connection pool

**Middleware Stack**:
1. `helmet` - Security headers
2. `cors` - Cross-origin resource sharing
3. `compression` - Response compression
4. `express.json` - JSON body parsing
5. Custom logging middleware

**API Endpoints**:

```
GET /api/organizations
  Query params:
    - search: string (full-text search)
    - legalTopics: comma-separated topic names
    - categories: comma-separated category names
    - state: two-letter state code
    - limit: integer (default 50, max 100)
    - offset: integer (default 0)

  Response: {
    success: boolean,
    data: Organization[],
    meta: { total, limit, offset, count }
  }

GET /api/organizations/:id
  Response: {
    success: boolean,
    data: Organization
  }

GET /api/categories
  Response: {
    success: boolean,
    data: Category[]
  }

GET /api/legal-topics
  Response: {
    success: boolean,
    data: LegalTopic[]
  }

GET /api/stats
  Response: {
    success: boolean,
    data: {
      totalOrganizations,
      totalCategories,
      totalLegalTopics
    }
  }
```

**Error Handling**:
- 400 Bad Request - Invalid parameters
- 404 Not Found - Resource not found
- 500 Internal Server Error - Server error

### 3. Database Layer

**Technology**: PostgreSQL 14+

**Key Features**:
- Full-text search with GIN indexes
- Many-to-many relationships
- Automatic timestamps
- Cascading deletes
- UNIQUE constraints for data integrity

**Schema Design**:

```sql
organizations (1) ──< (M) organization_categories (M) >── (1) categories
organizations (1) ──< (M) organization_legal_topics (M) >── (1) legal_topics
organizations (1) ──< (1) cypres_statements
```

**Indexes**:
- B-tree indexes on foreign keys and frequently queried columns
- GIN indexes for full-text search on `name`, `mission_statement`, `statement_text`
- Composite indexes for common filter combinations

**Query Optimization**:
- Connection pooling (max 20 connections)
- Prepared statements via parameterized queries
- Array aggregation for related data
- Efficient JOIN strategies

### 4. Web Scraper

**Technology**: Cheerio (HTML parsing), Puppeteer (dynamic sites), Natural (NLP)

**Key Files**:
- `scraper/main.js` - Orchestration and batch processing
- `scraper/cypresFinder.js` - Page discovery and extraction

**Scraping Pipeline**:

```
1. URL Input
   ↓
2. Robots.txt Check
   ↓
3. Pattern-Based Discovery
   - Try common cy pres URL patterns
   ↓
4. Site Search Fallback
   - Crawl homepage for cy pres links
   ↓
5. Content Validation
   - Verify cy pres keywords present
   ↓
6. Data Extraction
   - Org name, logo, mission
   - Cy pres statement text
   ↓
7. NLP Classification
   - Categorize by content
   - Identify legal topics
   ↓
8. Database Storage
   - Insert/update organization
   - Link categories and topics
   - Save cy pres statement
   ↓
9. Logging
   - Record success/failure
```

**Scraping Strategies**:

1. **Static HTML Sites**: Cheerio (fast, lightweight)
2. **JavaScript-Heavy Sites**: Puppeteer (headless Chrome)
3. **Rate Limiting**: Configurable delay between requests
4. **Batch Processing**: Process N URLs concurrently (default 5)
5. **Error Recovery**: Continue on failure, log errors

**NLP Classification**:

Uses Naive Bayes classifier trained on:
- Legal topic keywords (e.g., "consumer fraud" → Consumer Protection)
- Organization category keywords (e.g., "legal aid" → Legal Aid)
- Mission statement analysis

**Ethical Considerations**:
- Respects robots.txt
- Identifies as bot in User-Agent
- Rate-limited (2 second default delay)
- Logs all activity
- Only scrapes public pages

## Data Flow

### Search Request Flow

```
1. User enters search query
   ↓
2. Frontend validates input
   ↓
3. Frontend sends GET /api/organizations
   ↓
4. Backend receives request
   ↓
5. Express routes to handler
   ↓
6. Organization.search() builds SQL query
   ↓
7. PostgreSQL executes full-text search
   ↓
8. Results joined with categories/topics
   ↓
9. Backend formats response
   ↓
10. Frontend receives JSON
    ↓
11. Frontend renders organization cards
```

### Scraping Data Flow

```
1. Scraper reads URL list
   ↓
2. For each URL (batch of 5):
   ├─ Check robots.txt
   ├─ Find cy pres page
   ├─ Extract content
   ├─ Classify content
   └─ Save to database
   ↓
3. Log results
```

## Security

### Backend Security

**Headers (via Helmet)**:
- Content-Security-Policy
- X-Frame-Options
- X-Content-Type-Options
- Strict-Transport-Security

**Database Security**:
- Parameterized queries (SQL injection prevention)
- Connection pooling with timeout
- Environment variable credentials

**Input Validation**:
- Query parameter sanitization
- Type checking
- Length limits

### Frontend Security

- No inline JavaScript (CSP-friendly)
- User input sanitization before display
- HTTPS enforcement (production)
- CORS configuration

## Performance Optimization

### Frontend
- Lazy loading for organization cards
- Pagination (limit results)
- Debounced search input
- Minimal dependencies (no heavy frameworks)

### Backend
- Connection pooling
- Response compression (gzip)
- Efficient SQL queries with indexes
- Query result caching potential (future)

### Database
- Full-text search indexes (GIN)
- Foreign key indexes
- ANALYZE and VACUUM for query planning
- Partial indexes where applicable

### Scraper
- Concurrent batch processing
- Reuses HTTP connections
- Cheerio for static sites (faster than Puppeteer)
- Puppeteer only for dynamic sites

## Scalability Considerations

### Horizontal Scaling

**Frontend**: Easily scalable (static files)
- CDN distribution
- Multiple web servers

**Backend**: Stateless API (scalable)
- Load balancer distribution
- Multiple API server instances
- Shared PostgreSQL connection

**Database**: Vertical scaling primary, horizontal possible
- Read replicas for queries
- Write to primary instance
- Connection pooling per instance

### Vertical Scaling

**Database**:
- Increase RAM for query caching
- Increase CPU for query processing
- SSD storage for faster I/O

**Backend**:
- Increase Node.js worker processes
- Increase connection pool size

### Performance Targets

- API response time: < 500ms (p95)
- Page load time: < 2 seconds
- Scraper throughput: 100+ orgs/hour
- Database queries: < 200ms (p95)

## Monitoring & Observability

**Recommended Tools**:
- **Logging**: Winston (structured logs)
- **APM**: New Relic, Datadog
- **Database**: PostgreSQL pg_stat_statements
- **Uptime**: Pingdom, UptimeRobot
- **Errors**: Sentry

**Key Metrics**:
- API request latency
- Error rate
- Database query performance
- Scraper success rate
- Active organizations count

## Deployment Architecture

### Development
```
Localhost:8080 (Frontend) → Localhost:3000 (Backend) → Localhost:5432 (PostgreSQL)
```

### Production (Example)
```
CloudFront/Netlify (Frontend)
    ↓
Application Load Balancer
    ↓
EC2/ECS Instances (Backend API)
    ↓
RDS PostgreSQL (Primary + Read Replica)
```

**Infrastructure as Code**: Consider Terraform or AWS CloudFormation

## Testing Strategy

### Unit Tests
- Organization model methods
- Data validation functions
- Utility functions

### Integration Tests
- API endpoint tests
- Database query tests
- Scraper extraction tests

### End-to-End Tests
- User search flows
- Filter combinations
- Pagination

**Framework**: Jest (Node.js), Playwright (E2E)

## Future Architecture Enhancements

1. **Caching Layer**: Redis for API response caching
2. **Message Queue**: Bull/RabbitMQ for scraping jobs
3. **Search Engine**: Elasticsearch for advanced search
4. **CDN**: CloudFront for static assets
5. **API Gateway**: Kong or AWS API Gateway
6. **Microservices**: Separate scraper into its own service
7. **GraphQL**: Consider GraphQL API alongside REST

## Dependencies

### Backend Core
- express: ^4.18.2
- pg: ^8.11.3 (PostgreSQL client)
- dotenv: ^16.3.1
- cors: ^2.8.5
- helmet: ^7.1.0

### Backend Scraping
- axios: ^1.6.0 (HTTP client)
- cheerio: ^1.0.0-rc.12 (HTML parsing)
- puppeteer: ^21.5.0 (browser automation)
- robots-parser: ^3.0.1
- natural: ^6.10.0 (NLP)

### Frontend
- None (vanilla JavaScript)

## Maintenance

### Regular Tasks
- Weekly scraper runs for new/updated content
- Monthly database backups
- Quarterly dependency updates
- Annual data accuracy audits

### Database Maintenance
```sql
-- Reindex for performance
REINDEX DATABASE cypresawards;

-- Update statistics
ANALYZE;

-- Clean up dead rows
VACUUM;
```

## Disaster Recovery

**Backup Strategy**:
- Daily automated PostgreSQL backups
- Point-in-time recovery capability
- Backup retention: 30 days

**Recovery Procedures**:
1. Restore from most recent backup
2. Replay transaction logs (if available)
3. Re-run scraper for latest data
4. Verify data integrity

**RTO (Recovery Time Objective)**: < 4 hours
**RPO (Recovery Point Objective)**: < 24 hours