Paul Conrad Archive

repo: ~/Projects/paul-conrad-archive · 36 commits · 0 in last 24h, 36 in last 7d · ▄█

Search the build

36 commits indexed

  1. 167743e 2026-09-25 Robots re-audit + web-image hotlink allowlist (TK-12199)
  2. 1c16566 2026-09-25 Google-style robots matcher (wildcards, $, longest-match, product-token groups) + tests; web_images crawler: Syracuse exhibit, dealer, Invaluable, Pop History Dig, Truthdig (32 records w/ image URLs); probes: Stanford Daily API, Truthdig wp-json
  3. 80ae388 2026-09-25 auto-data-snapshot: 2026-09-25T11:23:40 (1 data files) — data/conrad.db
  4. 7735e27 2026-09-25 IA FTS: Editor & Publisher syndicate-ad pages, IA access-restricted check before recording page images, FTS 400 retry; probes: LBJ/JFK eMuseum 403 (no UA swap), Billy Ireland DC 504, Chronicling America 23 hits/0 cartoons; DPLA creator query
  5. 92c289d 2026-09-25 docs: 5 unsent draft research-access letters (CDNC, Colorado CHNC, Portal to Texas History, Utah Digital Newspapers, LA Times permissions)
  6. f2625ec 2026-09-25 IA full-text classifier: strict credit-line + prose-reprint rules, stated-title extraction, award/roster/music negatives
  7. 851fc79 2026-09-25 auto-data-snapshot: 2026-09-25T10:52:57 (1 data files) — data/conrad.db
  8. 25daa26 2026-09-25 LA Times: sitemap discovery + 2010 obituary gallery parser (26 cartoons w/ image URLs) + quote-verified letters citation; DPLA NPG Time covers; base.py gzip truncation fix; IA full-text + newspaper-probe crawlers; tests
  9. 020434a 2026-09-25 auto-data-snapshot: 2026-09-25T00:34:03 (1 data files) — data/conrad.db
  10. 95d0fc0 2026-09-24 auto-data-snapshot: 2026-09-24T23:32:41 (1 data files) — data/sources.json
  11. 56bb0e9 2026-09-24 Rights note: reflect hotlinked institutional images (no copy stored) instead of 'no image is reproduced' (68 rows)
  12. fb7c43e 2026-09-24 Reserve card image height before lazy load so filtered grids don't collapse
  13. 63f07b5 2026-09-24 Show institution-published cartoon images by hotlink only (allowlist + CSP), All cartoons view + Has image filter; replace blanket no-img test with allowlist tests
  14. ddb5417 2026-09-24 Draft (unsent) data-request letters to Syracuse, Wichita State, Ohio State Billy Ireland
  15. 0f34d5c 2026-09-24 auto-data-snapshot: 2026-09-24T18:58:12 (1 data files) — data/conrad.db
  16. ee3f521 2026-09-24 auto-data-snapshot: 2026-09-24T18:27:35 (1 data files) — data/conrad.db
  17. e1a4fb4 2026-09-24 NCR citation: use live natcath.org original (200, no robots) instead of Wayback copy — Wayback-only-for-retired-origins policy now holds 29/29 (Cody c3)
  18. f12cde5 2026-09-24 Fix PEP701 nested-quote f-strings (broke Kamatera py3.10); add py3.10 py_compile test over all sources (proven red on the bug)
  19. e4fc551 2026-09-24 Copyright rail layer 2: refuse image/PDF responses by Content-Type or magic bytes before body read; negative test proven red without guard
  20. 5ed3bca 2026-09-24 Cycle 3 data refresh: dedupe/exports/REPORT after live 10-sample re-verification (10/10) + non-PBS re-check
  21. e088856 2026-09-24 Cycle 3: quote-verified secondary-citation crawler (PBS gallery, LA Times obit, LOC bulletin, Wikipedia, NCR, Canyon News) + Commons/Wikidata/Pulitzer/Nixon/NYT/WaPo probes; Http refuses image URLs; REPORT 'How to get more'; tests
  22. f0b0302 2026-09-24 Daily Iowan: text-layer probe (sidecars 404, dir 403, search=Google CSE robots-disallowed, IDL 403) -> no permitted text layer, 0 records; full 1945-1950 issue list export (1,845 issues) for manual review; REPORT/sources updated; tests
  23. 5d9802d 2026-09-24 Viewer search: q now matches repository, collection_name, publication (denormalised cartoon_index.search_text, migration + auto-rebuild); tests
  24. b6988b4 2026-09-24 LOC deep enumeration run: +1 group record (2010634752), 2 lots reclassified folder, unprocessed single prints stay items; exports + REPORT regenerated with acquisition_method counts
  25. 0fbcd4e 2026-09-24 auto-data-snapshot: 2026-09-24T17:26:27 (1 data files) — data/conrad.db
  26. 84113e0 2026-09-24 LOC deep enumeration: robots audit per path, more format endpoints, notes-based Conrad attribution, group-component check, bounded checkpointed neighbour-LCCN probing; tests
  27. e9c5e1c 2026-09-24 Dedupe: generic/[bracketed] title matches need identifier or finer-than-year date agreement; different LOC pks never auto-link; negative test
  28. 88b34d2 2026-09-24 Add acquisition_method provenance field: idempotent migration + backfill, API/search flags, exports, detail-page reader-bypass badge, REPORT counts, tests
  29. 9144a7e 2026-09-24 Ignore SQLite WAL/SHM sidecar files
  30. 0441366 2026-09-24 Search index, cross-year duplicate review, README, Huntington request draft, py3.10 requirements, data + REPORT
  31. 00dd3c6 2026-09-24 Dedupe, exports, REPORT, FastAPI viewer (link-out only, basic auth, noindex), tests incl. negative no-image test
  32. a9bba5a 2026-09-24 Live crawlers: LOC item JSON, OAC, Syracuse/OSU/Wichita probes, paywalled registrations, Daily Iowan inventory, IA books, DPLA/SI discovery
  33. bc3f93c 2026-09-24 auto-data-snapshot: 2026-09-24T16:24:12 (3 data files) — data/conrad.db-shm data/conrad.db-wal pyproject.toml
  34. 8ff2c42 2026-09-24 Live Huntington EAD crawler: 8628 CON slots verified, box inventory, charset fix
  35. 380110b 2026-09-24 Core schema, models, normalizers, polite HTTP base and seed importer
  36. e3065ee 2026-09-24 initial scaffold

Authors

Agents used

  • none detected

Skills used

  • /claude8
  • /code8
  • /conrad8
  • /sources2
  • /image1
  • /thumbnail1
  • /roster1
  • /music1
  • /exports1
  • /search1

Creative ideas + design notes

Commits with substantial prose (≥120 chars) — the rationale behind each move.

167743e · 2026-09-25 · Robots re-audit + web-image hotlink allowlist (TK-12199) 
- Re-checked all 17,795 cartoon_sources record/image/thumbnail URLs against fresh robots.txt (26 hosts) with RobotRules.
  latimes photogallery is ALLOWED for our UA: 'Disallow: /*photogallery' is only in the Googlebot-News group, not '*'.
  0 source rows deleted. Tests pin the real latimes robots.txt (fixture) and prove Http.get refuses the gallery
  before any I/O if the '*' group ever disallows it.
- imagehost: allowlist ca-times.brightspotcdn.com, library.syracuse.edu, www.original-political-cartoon.com,
  pophistorydig.com, www.truthdig.com (49/49 image URLs verified 200 image/* by HEAD, no bytes kept). Page URLs on
  those hosts still return null. image.invaluable.com excluded (robots.txt 403 -> treated disallowed). CSP follows.
- Re-ran dedupe, exports, report --gaps; pytest 153 passed. Viewer: 67 records with display_image.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SW5KHgwrh2Rr9TKHM6ndqM
1c16566 · 2026-09-25 · Google-style robots matcher (wildcards, $, longest-match, product-token groups) + tests; web_images crawler: Syracuse exhibit, dealer, Invaluable, Pop History Dig, Truthdig (32 records w/ image URLs); probes: Stanford Daily API, Truthdig wp-json 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SW5KHgwrh2Rr9TKHM6ndqM
7735e27 · 2026-09-25 · IA FTS: Editor & Publisher syndicate-ad pages, IA access-restricted check before recording page images, FTS 400 retry; probes: LBJ/JFK eMuseum 403 (no UA swap), Billy Ireland DC 504, Chronicling America 23 hits/0 cartoons; DPLA creator query 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SW5KHgwrh2Rr9TKHM6ndqM
92c289d · 2026-09-25 · docs: 5 unsent draft research-access letters (CDNC, Colorado CHNC, Portal to Texas History, Utah Digital Newspapers, LA Times permissions) 
Draft only, never sent. Google News Archive skipped (no contact channel); Billy Ireland already covered by osu-request-DRAFT.md. TK-12199

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SW5KHgwrh2Rr9TKHM6ndqM
f2625ec · 2026-09-25 · IA full-text classifier: strict credit-line + prose-reprint rules, stated-title extraction, award/roster/music negatives 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SW5KHgwrh2Rr9TKHM6ndqM
25daa26 · 2026-09-25 · LA Times: sitemap discovery + 2010 obituary gallery parser (26 cartoons w/ image URLs) + quote-verified letters citation; DPLA NPG Time covers; base.py gzip truncation fix; IA full-text + newspaper-probe crawlers; tests 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SW5KHgwrh2Rr9TKHM6ndqM
fb7c43e · 2026-09-24 · Reserve card image height before lazy load so filtered grids don't collapse 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5NGeZrf7GXNQrfMRCTnRc
63f07b5 · 2026-09-24 · Show institution-published cartoon images by hotlink only (allowlist + CSP), All cartoons view + Has image filter; replace blanket no-img test with allowlist tests 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5NGeZrf7GXNQrfMRCTnRc

File tree

92 files tracked. Click any to browse the source at HEAD.

Other build journals

← Patterndesignlab  ·  all 4 projects  ·  Paul Conrad Cartoons →

Export

commits.csv · feed.atom · project.json · commits.json

rendered in 1ms