`README.md`

72 lines
# RENTV Ad Engine

A searchable directory of **every advertiser/sponsor** and **every guest** across
RENTV's Vimeo video library (`vimeo.com/user9059207`), harvested from Vimeo's free
auto-caption transcripts. Local-only, **$0** — no paid APIs.

Born from TK-10343. DTD verdict (5/5): standalone project, reads
`~/Projects/rentv/data/vimeo-library.json` **READ-ONLY** as the single source of the
video list (no drift). Designed to mount into `rentv.agentabrams.com/advertisers`
later with Steve's approval.

## Pipeline

| Node | Script | What it does | Cost |
|------|--------|--------------|------|
| A — Harvest | `scripts/harvest-transcripts.py` | Fetches each video's auto-caption VTT via the Vimeo viewer-JWT; writes `data/transcripts/<id>.txt` + `data/manifest.json` | $0 |
| B+C — Extract | `scripts/extract.py` | Local Ollama (qwen2.5, JSON mode) + regex backstop → advertisers (name/website/phone/category/pitch/episodes) and guests (name/title/company/location/topics/quotes) | $0 |
| D — Ad-card OCR | *(Phase 2)* | ffmpeg intro/outro frames → tesseract/qwen2.5vl for on-screen logos + mailing addresses not spoken aloud | $0 |
| E — Viewer | `server.js` + `public/index.html` | Zero-dep Node server; searchable/sortable advertiser + guest directory, Basic-Auth gated | $0 |

## Run

```bash
python3 scripts/harvest-transcripts.py   # refresh transcripts (re-run when new episodes drop)
python3 scripts/extract.py               # rebuild advertisers.json + guests.json (incremental cache)
node server.js                           # serve at http://localhost:9789  (admin / DW2024!)
```

Env: `OLLAMA_MODEL` (default `qwen2.5:latest`), `OLLAMA_URL`, `PORT` (9789), `BASIC_AUTH`.

## Coverage note

Vimeo auto-captions the 22 long "CRE Talk" interview episodes richly (55–75k chars each).
Conference panels and short clips often have thin/no auto-captions — those yield little,
by design of Vimeo's captioner, not a bug here. `data/manifest.json` records per-video
`caption_chars` + `has_captions`; the Coverage tab surfaces exactly what was transcribed.

## Data files (git-tracked, regenerable)

- `data/manifest.json` — per-video transcript coverage
- `data/advertisers.json` — the deduped advertiser registry (the engine's payload)
- `data/guests.json` — per-episode guest records
- `data/extractions/<id>.json` — per-video raw extraction cache (incremental)
- `data/transcripts/<id>.txt` — plain-text transcripts (`.vtt` raw is gitignored)

## SoundCloud podcast source (soundcloud.com/rentvnews)

RENTV's Vimeo library is the "CRE Talk" **interview** show. The SoundCloud channel is a
SEPARATE weekly-news podcast — **"RENTV's Weekly CRE News & Views with Steve Bloom"** (3
public episodes: Ep. 9/10/11, Aug–Oct 2020). It's harvested as a first-class source and
merged into the SAME unified `advertisers.json`, so podcast sponsors flow straight into the
engagement-email drafter.

Because SoundCloud has **no auto-captions** (Vimeo did), we transcribe locally with
`faster-whisper` (CTranslate2, int8, CPU) — still **$0**.

| Node | Script | What it does | Cost |
|------|--------|--------------|------|
| A — Harvest | `scripts/harvest-soundcloud.py --download` | `yt-dlp` → audio (mp3) + `info.json`; builds `data/soundcloud/manifest.json` (title, date, artwork, view/like counts, show-note **chapters**) | $0 |
| B — Transcribe | `scripts/transcribe-soundcloud.py` | faster-whisper → `data/transcripts/<id>.txt` (path `extract.py` already reads) + `data/soundcloud/segments/<id>.json` (timestamped); upserts each episode into `data/manifest.json` | $0 |
| C — Extract | `scripts/extract.py` *(shared, unmodified)* | picks up the podcast transcripts → unified advertisers + guests | $0 |
| D — Ad clips | `scripts/clip-ads.py` | locates each advertiser's spoken mention in the timestamped segments → `ffmpeg`-cuts the **actual spoken-ad audio** → `data/soundcloud/adclips/*.mp3`; enriches `advertisers.json` with per-episode `source` + `adclips` + `has_podcast` | $0 |
| E — Viewer | `server.js` + Podcasts tab | episode cards with full-episode audio, chapter-seek, spoken-ad clip players, transcript expander, and a one-click "Draft engagement email" | $0 |

Run it all: **`bash scripts/build-soundcloud.sh`** then `pm2 restart rentv-ad-engine`.

New endpoints: `/api/soundcloud`, `/api/adclips`, `/api/transcript?id=<id>`, `/media/<path>`
(audio with HTTP Range seeking, path-guarded to `data/soundcloud/`).

SoundCloud data files: `data/soundcloud/manifest.json`, `.../segments/<id>.json`,
`.../adclips.json` + `.../adclips/*.mp3`, `.../audio/<id>.mp3` (audio gitignored).