`references/retrieval-playbook.md`

76 lines
# Retrieval playbook — getting a tip's real content off TikTok

TikTok is hostile to reading: video pages are JS-rendered (HTML = the "Make Your
Day" shell), there is no public transcript API, and short `/t/<code>` links are
redirects. The answer is the same discipline as `gurisingh-scout`: **fan out
across the no-JS primitives, take the first that yields the caption + claim, and
never conclude "no new tips" from one failure.**

## Step order (stop as soon as you have the tip's CLAIM)

1. **A single dropped link → `tt-oembed.js`.**
   ```bash
   node scripts/tt-oembed.js "https://www.tiktok.com/t/ZP8GY4Rgo/"
   ```
   Follows the redirect to the canonical `@handle/video/<id>`, returns caption +
   hashtags + author + `claude_relevant`. `$0`, no auth. For most tips the
   caption already states the claim ("Never ask Claude code to do this!").

2. **A "what's new" sweep → `tt-list.js`.**
   ```bash
   node scripts/tt-list.js tiktok_architect 12
   ```
   Newest N videos, each flagged `claude_relevant`. Scout ONLY the flagged ones
   — the feed is mixed (motorcycles, hobbies). `$0`. The "attempting
   impersonation" WARNING on stderr is normal, not a failure.

3. **Caption too vague → `tt-transcript.sh`** (optional, `$0` local).
   ```bash
   bash scripts/tt-transcript.sh "https://www.tiktok.com/@tiktok_architect/video/<id>"
   ```
   Tries yt-dlp subtitles first (usually empty on TikTok), then LOCAL whisper on
   the downloaded audio. If no local whisper exists it prints the audio path and
   says so — it never fabricates a transcript. Reach for this only when the
   claim can't be pinned from the caption + on-screen premise.

4. **Verify / second-source the claim → Exa or WebSearch.** Search the tip's
   phrasing to find reposts, write-ups, or discussion (Reddit/X/blogs) — and,
   critically, to check whether the claim is TRUE for current Claude Code. Exa
   is metered; show/log the cost. Also consult the `claude-code-guide` subagent
   and the `claude-docs` / `claude-api` skills — those are `$0` and
   authoritative for "is this actually how Claude Code works."

5. **Tip names an open-source tool/repo → `gh-repo-stats.js`.**
   ```bash
   node scripts/gh-repo-stats.js owner/repo
   ```
   Some tips ("get Claude Code for free", "install this MCP") point at a repo.
   Enrich it (stars / license / freshness / is-claude-skill) exactly like the
   tool-scout, and hand the tool half to `gurisingh-scout`'s adoption path.

## Turning a video into a candidate tip

For each `claude_relevant` video, capture:

- **claim** — the one-sentence assertion ("never ask Claude Code to write your
  whole app in one prompt"), normalized to an imperative rule.
- **kind** — anti-pattern ("never do X") | best-practice ("always do X") |
  workflow | tool/repo | prompt-pack | setting/config.
- **specifics** — the *why* and the *instead-do-this*, from caption/on-screen/
  transcript. A tip with no reasoning is weaker (see rubric Leverage).
- **references a repo?** — if yes, resolve + enrich it with `gh-repo-stats.js`.

## Failure modes to expect (and how to read them)

- **`curl`/WebFetch on the video page returns "Make Your Day"** → expected; the
  page is JS-rendered. Use oEmbed/yt-dlp, never the raw HTML.
- **`tt-list.js` exits 3 (0 videos)** → TikTok throttled or blocked, NOT "no new
  tips". Retry once, then fall back to Exa/WebSearch for recent reposts.
- **yt-dlp "attempting impersonation" WARNING** → informational. If a *download*
  (audio) actually fails, install a curl_cffi impersonate target
  (`yt-dlp#impersonation`) or accept caption-only for that video.
- **No subtitles on a video** → normal for TikTok. Use audio+local-whisper, or
  just work from the caption + on-screen premise.
- **The tip can't be verified either way** → do NOT codify it. Mark it WATCH and
  say so; an unverifiable behavioral claim is not a rule.