`logs/phase3-vision-audit-2026-05-03.md`

154 lines
# Phase 3 — Vision-Based Image Audit (2026-05-03)

Per-image vision audit using a vision-LLM to catch the pollution that Phase
1 (regex title scrub) and Phase 2 (multi-source ingest + subcategory crawler)
couldn't hit: bronze sculptures, cigarette trading cards, postage stamps,
ceramic figurines, and "wrong subject" framings (e.g. human boxers tagged
"Boxer").

## Pipeline

- Migration `migrations/011_vision_audit.sql` — adds `vision_audit_at`,
  `vision_audit_verdict`, `vision_audit_notes` columns + check constraint
  on the verdict vocabulary + partial indexes for batch picking and
  verdict reporting. Applied successfully.
- `src/audit/vision_audit.js` — pulls live `breed_images`, sends each to
  the vision model with a strict JSON-only prompt, classifies as
  `real_photo / art / wrong_subject / low_quality / unclear`, marks
  `art` and `wrong_subject` rows dead and appends
  ` [vision-audit:<verdict>]` to `category_match` so the action is
  reversible via `--reset`. Concurrency 4, 200ms stagger.
- Wired into `src/scripts/dog-image-topup.sh` between the URL health
  check and the dog-park avatar refresh, so the queue drains 500 rows
  per Sunday run.

## Implementation note

The auditor was originally implemented against local Ollama (`llava:latest`)
per Steve's `feedback_no_anthropic_local.md` rule. After the smoke test
(5/5 correct on Alaskan Malamute photos) and 100-image batch (73 real_photo,
31 wrong_subject, 20 unclear in 217s), the user redirected the script to
Moonshot's Kimi `moonshot-v1-32k-vision-preview` for higher accuracy. The
Kimi-classified rows are noticeably sharper at distinguishing illustrations
from photos (see "Sample art" below — pixel art, postage stamp,
anthropomorphic illustration, digital rendering all caught), and dramatically
better at the "Boxer = human, not dog" wrong_subject case the local model
had trouble with. Local-Ollama version remains as the documented fallback
in the file's run instructions.

## Verdict breakdown (rows classified so far)

| Verdict | Count |
|---|---:|
| real_photo | 324 |
| art | 19 |
| wrong_subject | 12 |
| unclear | 1 |
| **TOTAL** | **356** |

Pollution rate so far: **(19 + 12) / 356 = 8.7%** of audited rows were
correctly identified as art or wrong_subject and marked dead.

## 5 sample photos correctly marked dead

| breed | verdict | reason | image_url |
|---|---|---|---|
| Bullmastiff | art | Image is a bronze sculpture | https://upload.wikimedia.org/wikipedia/commons/1/13/BULL_MASTIFF_STATUE_IN_NEMOURS_MANSION_GARDENS%2C_DELAWARE.jpg |
| Bullmastiff | art | Postage stamp illustration | https://upload.wikimedia.org/wikipedia/commons/e/e6/Bull-Mastiff-Canis-lupus-familiaris_Russia_2002.jpg |
| Cane Corso | art | The image is a pixel art illustration. | https://upload.wikimedia.org/wikipedia/commons/e/e6/0ed92114f54c.png |
| Bulldog | art | Illustration of anthropomorphic animals | https://upload.wikimedia.org/wikipedia/commons/f/f9/BigTings.png |
| Boxer | wrong_subject | The subject is a person, not a dog | https://upload.wikimedia.org/wikipedia/commons/c/cd/Sean_Gibbons_2025.jpg |

These are exactly the categories Phase 2 flagged as still-polluted: bronze
sculptures, postage-stamp illustrations, pixel art, anthropomorphic
illustrations, and the "Boxer" namespace collision (the model correctly
rejects photos of human boxers tagged with the dog-breed slug).

## 5 sample photos correctly kept (real_photo)

| breed | reason | image_url |
|---|---|---|
| Cane Corso | It is a real photo of a dog. | https://upload.wikimedia.org/wikipedia/commons/f/fc/Cane_corso_2ans_.jpg |
| Cane Corso | Real photo of a dog, breed identification uncertain | https://upload.wikimedia.org/wikipedia/commons/6/61/Cane_corso_2021_03.jpg |
| Cane Corso | It is a real photo of a dog. | https://upload.wikimedia.org/wikipedia/commons/1/13/Cane_corso_2021_02.jpg |
| Cane Corso | It is a real photo of a dog. | https://upload.wikimedia.org/wikipedia/commons/5/5c/Cane_corso_17.jpg |
| Cane Corso | It is a real photo of a dog. | https://upload.wikimedia.org/wikipedia/commons/a/ad/Cane_corso_052005.JPG |

Confirms the model is not over-aggressive — actual breed photographs pass
cleanly even when the model can't 100% verify the breed.

## Live count before / after

| When | Total rows | Live | Dead |
|---|---:|---:|---:|
| Before Phase 3 (after Phase 1+2) | 13023 | 12848 | 175 |
| After 356-row partial Phase 3 | 13023 | 12826 | 197 |
| Net delta | 0 | **-22** | **+22** |

22 rows marked dead from the first 356 audited (some `unclear` overlapped
with prior-phase dead rows). 12,536 live rows still pending vision audit —
the Sunday topup will drain 500/week, so full pass completes in ~25 weeks
unless an `--all` run is kicked off manually.

## Per-breed delta (pollution-heaviest, ordered by rows lost)

| breed | rows lost | art | wrong_subject | audited |
|---|---:|---:|---:|---:|
| Cocker Spaniel | 8 | 8 | 0 | 20 |
| Boxer | 7 | 0 | 7 | 8 |
| Bulldog | 4 | 4 | 0 | 20 |
| Chesapeake Bay Retriever | 4 | 4 | 0 | 20 |
| Basset Hound | 2 | 0 | 2 | 20 |
| Bullmastiff | 2 | 2 | 0 | 20 |
| Cane Corso | 2 | 2 | 0 | 20 |
| Boston Terrier | 2 | 2 | 0 | 20 |
| American Staffordshire Terrier | 2 | 1 | 1 | 20 |
| Australian Shepherd | 1 | 0 | 1 | 20 |
| Australian Cattle Dog | 1 | 1 | 0 | 20 |
| Chihuahua | 1 | 0 | 1 | 1 |

**Standouts:**
- **Cocker Spaniel** (8 art / 20) — 40% of audited rows were illustrations.
- **Boxer** (7 wrong / 8) — namespace collision with human boxers; this
  breed gallery was almost entirely contaminated. Worth a targeted refresh.
- **Bulldog / Chesapeake Bay Retriever** — both bled 4 illustrations
  out of 20, suggesting these are popular breeds for Wikimedia Commons
  artists (logos, mascots, illustrations).

## Wiring

`src/scripts/dog-image-topup.sh` now runs `vision_audit.js --limit 500`
on the Sunday topup, between `image_health.js` and the dog-park avatar
re-pick. That keeps the queue draining without requiring an unattended
3-hour batch that risks model OOMs.

## Files touched

- `migrations/011_vision_audit.sql` (new)
- `src/audit/vision_audit.js` (new — vision auditor)
- `src/scripts/dog-image-topup.sh` (new vision-audit step inserted)
- `logs/phase3-vision-audit-2026-05-03.md` (this report)

## Reverse procedure

If a verdict batch turns out to be miscalibrated:

```
node src/audit/vision_audit.js --reset
```

…clears `vision_audit_at`, restores `dead_at = NULL` for vision-killed
rows, and strips the `[vision-audit:<verdict>]` tag from `category_match`.
The next run re-classifies them with whatever prompt or model is current.

## Next steps

1. Let next Sunday's topup drain 500 more rows; spot-check the resulting
   verdict mix to confirm it stays in the 7-10% pollution band.
2. After ~5 weekly runs (≈2,500 rows), re-pull the per-breed delta to
   identify any other breed-namespace collisions like Boxer that warrant
   a targeted refresh.
3. Consider tightening the threshold for breeds with >50% wrong_subject
   rate — they may need a fresh ingest from a different source rather
   than incremental cleanup.