`docs/LINK_VALIDATION.md`

289 lines
# Link Validation System

## Overview

The Link Validation System automatically checks all external links in the application to ensure they remain functional. It monitors property Airbnb URLs, property images, and activity website/maps/review links.

## Features

- **Automated Daily Checks**: Script can be run via cron to check all links daily
- **HTTP Status Tracking**: Records HTTP status codes for each link
- **Error Reporting**: Captures detailed error messages for broken links
- **Performance Monitoring**: Tracks response time for each link check
- **Manual Re-check**: API endpoints allow on-demand link validation
- **Admin Interface Ready**: Results stored in database for display in admin dashboard

## Database Schema

The `link_checks` table stores validation results:

```sql
- id: Primary key
- url: The URL being checked
- source_type: Type of source (property, activity_website, property_image, etc.)
- source_id: ID of the source record
- http_status: HTTP status code (200, 404, etc.)
- is_valid: Boolean indicating if link is accessible
- error_message: Error description if check failed
- response_time_ms: Response time in milliseconds
- last_checked_at: Timestamp of last check
- created_at: Record creation timestamp
- updated_at: Record update timestamp
```

## API Endpoints

### GET /api/link-validation/stats
Get overall link validation statistics.

**Response:**
```json
{
  "success": true,
  "data": {
    "total": 150,
    "valid": 145,
    "invalid": 5,
    "avgResponseTime": 342.5
  }
}
```

### GET /api/link-validation/broken
Get all broken links with pagination.

**Query Parameters:**
- `limit` (default: 100): Number of results to return
- `offset` (default: 0): Number of results to skip

**Response:**
```json
{
  "success": true,
  "data": [
    {
      "id": 1,
      "url": "https://example.com/broken",
      "source_type": "property",
      "source_id": 1,
      "http_status": 404,
      "is_valid": false,
      "error_message": "HTTP 404: Not Found",
      "response_time_ms": 234,
      "last_checked_at": "2025-01-14T12:00:00Z"
    }
  ],
  "count": 5
}
```

### GET /api/link-validation/all
Get all link check records with pagination.

**Query Parameters:**
- `limit` (default: 100)
- `offset` (default: 0)

### POST /api/link-validation/check-all
Manually trigger a full validation check of all links in the system.

**Response:**
```json
{
  "success": true,
  "message": "Link validation completed",
  "data": {
    "total": 150,
    "valid": 145,
    "invalid": 5,
    "results": [...]
  }
}
```

### POST /api/link-validation/check-url
Check a single URL on demand.

**Request Body:**
```json
{
  "url": "https://example.com/page"
}
```

**Response:**
```json
{
  "success": true,
  "data": {
    "url": "https://example.com/page",
    "isValid": true,
    "httpStatus": 200,
    "responseTimeMs": 234
  }
}
```

## Automated Script

### Running Manually

Check all links manually:

```bash
ts-node scripts/check-links.ts
```

### Setting Up Daily Automation

Add to crontab for daily execution at 2 AM:

```bash
crontab -e
```

Add this line:

```
0 2 * * * cd /root/Projects/jill-website && ts-node scripts/check-links.ts >> /var/log/link-checks.log 2>&1
```

Or using Node directly (after build):

```
0 2 * * * cd /root/Projects/jill-website && node dist/scripts/check-links.js >> /var/log/link-checks.log 2>&1
```

### Script Output

The script provides detailed console output:

```
🔍 Starting link validation...

📦 Checking property links...
  - Property: Casita del Sol (ID: 1)
    ✅ Airbnb URL valid (200)
  - Property: Ocean View Villa (ID: 2)
    ❌ Airbnb URL failed: Connection timeout

🎯 Checking activity links...
  - Activity: Surfing Lessons (ID: 1)
    ✅ Website valid (200)
    ✅ Maps link valid (200)

============================================================
📊 LINK VALIDATION SUMMARY
============================================================
Total links checked: 150
Valid links: 145 (96.7%)
Invalid links: 5 (3.3%)
Duration: 45.2s

Breakdown by source:
  property:
    Total: 10
    Valid: 10
    Invalid: 0
  property_image:
    Total: 120
    Valid: 115
    Invalid: 5
  activity_website:
    Total: 20
    Valid: 20
    Invalid: 0
============================================================
```

## Link Types Checked

### Properties
- **Airbnb URLs** (`source_type: 'property'`)
- **Image URLs** (`source_type: 'property_image'`)

### Activities
- **Website URLs** (`source_type: 'activity_website'`)
- **Google Maps Links** (`source_type: 'activity_maps'`)
- **Review Links** (`source_type: 'activity_review'`)

## Error Handling

The system captures various types of errors:

- **Connection timeout**: Link took too long to respond
- **Domain not found**: DNS lookup failed
- **Connection refused**: Server refused connection
- **HTTP 4xx/5xx**: Server returned error status
- **Network error**: General network issues

## Performance Considerations

- Links are checked sequentially to avoid overwhelming servers
- Default timeout: 10 seconds per link
- Maximum redirects: 5
- Uses HEAD requests (lighter than GET)
- Includes user agent to avoid bot blocking

## Future Enhancements

Potential improvements for future iterations:

1. **Email Notifications**: Send alerts when broken links are detected
2. **Admin Dashboard**: Visual interface showing link health
3. **Link History**: Track link status changes over time
4. **Bulk Operations**: Pause/resume link checking
5. **Priority Checking**: Check critical links more frequently
6. **Retry Logic**: Automatic retries for transient failures
7. **Integration Tests**: Verify link checking functionality

## Migration

To set up the database table, run:

```bash
npm run migrate
```

This will execute `src/migrations/005_create_link_checks_table.sql`.

## Testing

The link validation service can be tested independently:

```typescript
import { linkValidationService } from './services/linkValidationService';

// Check single URL
const result = await linkValidationService.checkLink('https://example.com');
console.log(result);
// { url: '...', isValid: true, httpStatus: 200, responseTimeMs: 234 }

// Check multiple URLs
const results = await linkValidationService.checkLinks([
  'https://example1.com',
  'https://example2.com'
]);
```

## Troubleshooting

### Script Not Running
- Check cron job is configured: `crontab -l`
- Verify TypeScript is installed: `npm install -g ts-node`
- Check logs: `tail -f /var/log/link-checks.log`

### Database Connection Errors
- Ensure `.env` has correct database credentials
- Verify database is running: `sudo systemctl status postgresql`
- Run migrations: `npm run migrate`

### Timeouts on All Links
- Check network connectivity
- Verify firewall allows outbound HTTPS requests
- Increase timeout in `linkValidationService.ts` if needed

## Support

For issues or questions about the link validation system, contact the development team or check the main project documentation.