`README.md`

73 lines
# Paul Conrad Master Archive

A **metadata-only** index of Paul Conrad (1924–2010) editorial cartoons and *where each one can legally be viewed*.
Ticket: TK-12199. See `REPORT.md` for current numbers, coverage, and every provenance caveat.

## Image rule (hard; revised by Steve 2026-09-24, TK-12199)
Conrad cartoon images are **never downloaded, proxied, cached, or stored** — anywhere (server, `data/`, cache, git).
`cartoon_sources.local_image` has a `CHECK (local_image IS NULL)` constraint and the crawler refuses image URLs,
image content-types and image magic bytes before reading a body (`crawlers/base.py`).
On the PRIVATE admin viewer only, a cartoon whose holding institution publishes a public image is shown by
**hotlinking the institution's own URL** (`src/conrad/web/imagehost.py`): exact-host allowlist
`tile.loc.gov`, `cdn.loc.gov` (Library of Congress), `5008.sydneyplus.com` (History Colorado), plus the web-image
hosts `ca-times.brightspotcdn.com`, `library.syracuse.edu`, `www.original-political-cartoon.com`, `pophistorydig.com`,
`www.truthdig.com` (added 2026-09-25 only after the publishing page and the image path were robots-allowed for our UA
and a HEAD returned 200 image/*; `image.invaluable.com` is excluded — its robots.txt answers 403); anything else →
`display_image: null`. A `Content-Security-Policy: img-src 'self' <those hosts>` header makes the browser refuse every
other image host. Every other record stays a text card with "View at &lt;repository&gt;" link-outs.
`tests/test_image_allowlist.py` enforces the allowlist (incl. a negative test proving the detector goes red when the
allowlist is bypassed) and scans the repo for image bytes. Perceptual-hash dedupe is skipped; dedupe is metadata-only.

## Record granularity
| granularity | meaning |
|---|---|
| `item` | one identified cartoon (LOC item, Wichita item, DPLA object, secondary citation) |
| `folder` | a folder of several cartoons (Syracuse folders, LOC proof set) |
| `box_range` | one Huntington CON number; date = the box's span (interpolated, `date_is_estimate=1`) |

## Setup
```sh
python3 -m venv .venv            # or: uv venv .venv
.venv/bin/pip install -e ".[dev]"   # or on a server: pip install -r requirements.txt  (pinned, Python >= 3.10)
.venv/bin/python scripts/init_db.py
```

## Run
```sh
.venv/bin/python scripts/crawl_all.py [--fresh] [--no-cache]   # seed + all crawlers, dedupe, exports, REPORT.md
.venv/bin/python scripts/crawl_source.py loc                     # one source
.venv/bin/python scripts/dedupe.py
.venv/bin/python scripts/export_csv.py                           # data/exports/*.csv|json + data/sources.json
.venv/bin/python scripts/report.py --gaps                        # REPORT.md + research gaps
.venv/bin/python -m pytest -q
```
Crawling is polite: robots.txt honoured (Crawl-delay wins over `REQUEST_DELAY=2.0`), `MAX_CONCURRENCY=3`, `TIMEOUT=30`,
tenacity exponential backoff, descriptive User-Agent, response cache in `data/cache/http/`, checkpoints in
`data/cache/checkpoints/`. No WAF/CAPTCHA/paywall/CDL bypass: such sources are recorded as blocked.
Seed data is read (never written) from `~/Projects/paul-conrad-cartoons/research/` (TK-12179).

## Viewer
```sh
.venv/bin/python -m conrad.web.app          # binds CONRAD_HOST:CONRAD_PORT (default 127.0.0.1:8787)
pm2 start .venv/bin/python --name paul-conrad-archive --interpreter none -- -m conrad.web.app
```
Locally on Mac2, port 8787 is already held by another service (DW-Local-AI), so the pm2 process runs with
`CONRAD_PORT=8788`.

| env | purpose |
|---|---|
| `CONRAD_HOST` / `CONRAD_PORT` | bind address (default `127.0.0.1` / `8787`) |
| `CONRAD_BASIC_USER` / `CONRAD_BASIC_PASS` | enable HTTP Basic Auth on every route (pages, `/api/*`, `/static/*`); `/robots.txt` stays readable |
| `CONRAD_REQUIRE_AUTH=1` | fail closed: the app refuses to start unless both credentials are set |
| `CONRAD_DB` | SQLite path (default `data/conrad.db`) |

Every response carries `X-Robots-Tag: noindex, nofollow`; pages carry `<meta name="robots" content="noindex,nofollow">`;
`/robots.txt` is `Disallow: /`. Search, year range + clickable timeline, president/person/topic/publication/repository/
collection filters, record-level selector, sort `<select>` (Newest, Oldest, Title A→Z, Repository, Year) and a density
slider (both persisted in `localStorage`). Cards show the record's created date + time.

## Layout
`src/conrad/{config,db,models,normalize,dedupe,rights,exports}.py`, `src/conrad/crawlers/*.py` (one module per source),
`src/conrad/web/` (FastAPI + static UI), `scripts/`, `tests/`, `data/{conrad.db,sources.json,exports/,cache/}`,
`docs/huntington-request-DRAFT.md` (unsent).