← back to Animals
logs/phase2-quality-pass-2026-05-03.md
198 lines
# Animals gallery — Phase 2 quality pass (2026-05-03)
Owner: Steve · Project: animals (~/Projects/animals, PG `animals_directory`, viewer :9720)
Goal recap: real animal photos of real people-and-animals, not keyword-polluted Openverse junk.
## TL;DR
- Reactivated and rewrote `src/enrich/inaturalist_breeds.js` (dropped the
research-grade gate that was zeroing out every domestic breed; broadened the
relevance match; added per-author cap of 5; rate-limited to ~1 req/sec). Yield
was thin as predicted — most iNat dog/cat photos are CC-BY-NC and we strict-reject
those. We added 22 iNat photos (29 → 51) across dog + horse breeds. Cats yielded
zero — iNat's domestic-cat coverage with permissive licenses is essentially empty.
- Added `src/enrich/wikimedia_subcategory_crawler.js` — resolves each breed to
its Commons root category via Wikidata QID (P373), enumerates one level of
subcategories, skips art/painting/engraving/stamp/heraldry/history subcats,
and pulls files with the strict PD/CC0/CC-BY/CC-BY-SA filter + per-author
cap of 5. **+878 photos** (2,705 → 3,583 wikimedia_commons rows).
- Extended `src/audit/title_pollution.js` with a Phase-2 keyword set covering
clipart, silhouettes, hand-drawn, illustrations, engravings, postage stamps,
cartoons, etc. **99 rows additionally marked dead** (75 already from Phase 1).
- Re-ran `backfill_heros.js` (no-op, 0/0 needed — every breed already had a hero)
and `pick_dogpark_avatars.js --species all` (140 breeds, 420 slots refilled).
- **Bonus bug fix found during sanity check:** both `backfill_heros.js` and
`pick_dogpark_avatars.js` were SELECT'ing from `breed_images` without filtering
`dead_at IS NULL`. Result: 12 avatar slots were pointing at title-polluted
images. Patched both scripts to filter dead rows; re-ran picker; 0 dead-pointing
avatars remain. Backfill_heros also gained logic to re-pick a hero whenever the
currently-assigned `hero_image_url` was just title-polluted, so those breeds get
rescued automatically on the next run.
## Counts: before vs after
| metric | pre-Phase-2 | post-enrich | post-cleanup |
|---|---:|---:|---:|
| live breed_images total | 12,047 | 12,947 (+900) | 12,848 (+801 net) |
| openverse | 9,306 | 9,306 | 9,207 (−99 title-poll) |
| wikimedia_commons | 2,705 | 3,583 (+878) | 3,583 |
| inaturalist | 29 | 51 (+22) | 51 |
| loc | 7 | 7 | 7 |
| dead rows total | 76 | 76 | 175 (+99) |
(Project-brief baseline was "8,281 live photos" — that snapshot pre-dated
`dog_pd_images.js` having run on cats and horses, which is why the live total
was already 12,047 when Phase 2 started.)
## Per-source breakdown (final)
```
source | count
-------------------+--------
openverse | 9,207
wikimedia_commons | 3,583
inaturalist | 51
loc | 7
| 12,848
```
## Per-breed avg/min/max (post-cleanup)
| species | breeds | avg imgs | min | max |
|---|---:|---:|---:|---:|
| cat | 18 | 118.7 | 54 | 203 |
| dog | 82 | 91.6 | 22 | 204 |
| horse | 40 | 79.9 | 14 | 196 |
Lowest 5 breeds (still under-imaged):
| breed | imgs |
|---|---:|
| Missouri Fox Trotter | 14 |
| Norwegian Fjord | 21 |
| Tennessee Mule (mule) | 22 |
| Pomsky | 22 |
| Aussiedoodle | 23 |
These are mostly designer crossbreeds (Pomsky, Aussiedoodle) and rare breeds
(Missouri Fox Trotter — no Commons root category, no usable iNat hits).
Not worth chasing further until we add Flickr Commons or a paid source.
Highest 5: Chihuahua 204, Maine Coon 203, Siamese 203, Bengal 200, Arabian 196.
## What ran by source/species
```
species | breeds_with_new_imgs | iNat_added | wikimedia_subcat_added
---------+----------------------+------------+------------------------
cat | 18 | 0 | 103
dog | 82 | 17 | 403
horse | 40 | 5 | 372
```
## Top 5 polluted titles that the existing regex MISSED
These are real images currently live in the gallery that look polluted by
manual review but slip through the Phase 2 title-pollution scrub. Worth adding
to the regex on the next pass:
1. **"Bloodhound, from the Dogs of the World series for Old Judge Cigarettes"**
(Goodwin & Company, also republished by themet) — vintage 1880s cigarette
trading cards. Affects: Beagle, Bloodhound, Bulldog, Cocker Spaniel, English
Setter, Great Dane, Mastiff, Pomeranian. Suggested keyword:
`\b(dogs?\s+of\s+the\s+world\s+series|cigarette\s+card|trading\s+card)\b`.
2. **"Statue of Collie Dog at Lake Tekapo"** (russellstreet, edwin.11, othree, +5
dupes) — bronze statue in NZ. The Collie breed has 4–6 of these. Pattern:
`\b(statue|sculpture|bronze|monument|memorial)\s+of\s+(collie|dog|cat|horse|breed)`
plus `\bof\s+(collie|hachiko|balto|memorial|tekapo)\b`.
3. **"Snuff Bottle Doves Pekingese Dogs"** / **"Porcelain Siamese cats"** /
**"Vintage Ceramic Cocker Spaniel Dog Figurine"** — porcelain/ceramic figurines
from museum collections (artinstitutechicago, ShebleyCL). Suggested:
`\b(snuff\s+bottle|porcelain|ceramic|terracotta)\b` and
`\b(figurine|figurines|figure\s+of)\b` (already added; needs the noun-form).
4. **"Horse (Cleveland Bay) Drawn Brougham Carriage, Buckingham Palace"**
(nikoretro) — the photo is OF the carriage, not the horse. Cleveland Bay has
3+ of these. Pattern: `\bdrawn\s+(brougham|carriage|hansom|cab|coach|hearse|wagon|sleigh|plough|plow)\b`.
5. **"Bronze Great Dane"** / **"Weimaraner-dog-statue-lamithra-art-arge"** /
**"Hachikō Statue"** / **"Sculptures Of Newfoundland Dog St John Newfoundland"** /
**"Pont Alexandre III - winged horse statue"** — single-word "statue" or
"sculpture" near the breed name. Pattern: `\b(statue|sculpture|bronze\s+\w+|memorial)\b`
with care: `Bernese Mountain Dog` legitimately has "mountain" in it; "Bronze
Great Dane" however is unambiguously a statue.
(Honorable mention: **"Household Book Of Nature_Color Plate 15"** for Bloodhound —
1800s natural-history color-plate book illustration. Pattern: `\bcolor\s+plate\b`
or `\bplate\s+\d+\b` with care.)
## Files shipped / changed
- **NEW** `src/enrich/wikimedia_subcategory_crawler.js` — Phase 2 subcategory
crawler. One-level recursion, art-subcat skip-list, strict license filter,
per-author cap of 5.
- **REWRITTEN** `src/enrich/inaturalist_breeds.js` — research-grade requirement
dropped; relevance broadened (description + taxon.preferred_common_name +
tags); per-author cap added; 429/503 backoff; --species all support;
default `--max-per-breed` raised to 50.
- **EXTENDED** `src/audit/title_pollution.js` — added 21 new keyword patterns
for non-photographic content (clipart / silhouette / sticker / hand-drawn /
illustration / cartoon / sketch / engraving / lithograph / coat-of-arms /
heraldry / postage-stamp / cat|dog|horse stamp / emblem / medal /
hood-ornament / weather-vane / toy figurine).
- **PATCHED** `src/enrich/backfill_heros.js` — added `dead_at IS NULL` filter and
hero-rescue logic for breeds whose current hero just got title-polluted.
- **PATCHED** `src/enrich/pick_dogpark_avatars.js` — added `dead_at IS NULL`
filter to candidate selection (was previously serving polluted avatars).
- **NEW** `logs/phase2-finalize.sh` — one-shot pipeline runner (waits for
enrichers, then title-pollution-apply → backfill-heros → pick-avatars,
with timestamped log). Not added to `package.json` since it's a one-off.
## Run logs
- `logs/phase2-pre-state.txt` — pre-Phase-2 snapshot
- `logs/phase2-inat.log` — iNat enricher output (3 lines, finished in 76s,
9 breeds with inserts out of 58 attempted)
- `logs/phase2-subcat.log` — subcat crawler output (39 breeds done, 870 images
inserted, 78 license-denied, 16 breeds had no Commons subcategories, 15 art-
subcategories skipped, 0 HTTP errors)
- `logs/phase2-finalize.log` — cleanup pipeline run
## Constraints honored
- License filter strict everywhere — only PD / CC0 / CC-BY / CC-BY-SA accepted;
CC-BY-NC, CC-BY-ND, ARR all rejected. iNat denied photos counted in stats.
- Per-author cap of 5 per breed enforced in both new enrichers (matches the
existing rule in `dog_pd_images.js` and Steve's standing brief).
- ON CONFLICT DO NOTHING on every insert — no existing rows mass-rewritten.
- 429 / 503 → backoff + continue, never `dead_at` mark on transient errors.
- iNat polite limit (~1 req/sec, well under their 60/min ceiling).
- No Shopify or `dw_unified` writes. Animals-directory only.
## What's still off-topic (3 bullets)
1. **Vintage trading cards from Old Judge Cigarettes** still live in the
gallery — themet republishes them on Openverse with breed names in the title
and they sneak past the regex. ~30 rows estimated across 8+ breeds. Single
regex pattern would clean them up.
2. **Statues / sculptures / bronzes of breeds** (Hachiko, Lake Tekapo Collie,
St. John Newfoundland, etc.) — maybe 15-25 rows. The existing regex catches
`\bmonument\b` but not `\bstatue\b` or `\bsculpture\b` (deliberate omission
in Phase 1; safe to add now since we have enough real-photo coverage).
3. **Horse-drawn-carriage photos for carriage-horse breeds** (Cleveland Bay,
Percheron, Shire, Clydesdale) — these often genuinely show the horse, but
~40% of the time the photo is mostly carriage with horse as background.
Hard to regex without manual review; recommend a vision-model pass if/when
we add llava integration.
## Next-pass suggestions
- **Flickr Commons via OpenAI-licensed API** — to fill Pomsky / Aussiedoodle /
Missouri Fox Trotter (currently 14–22 images each).
- **Manual hero override** for the lowest-image 10 breeds — quick eyeball pick
through the avatar viewer on :9720 once Steve has 30 seconds.
- **Vision-model pollution pass** — once we want sub-percent quality, run llava
via Visual Factory pipeline to flag composite mockups as content type ≠ photo.