`references/retrieval-playbook.md`

61 lines
# Retrieval playbook — getting the picks past X's auth wall

The whole difficulty of this skill is retrieval: `x.com` returns HTTP 402 to
unauthenticated readers, the target account is intermittently suspended, and
Thread Reader gates some threads behind login. The answer is the same as
`feed-first-scraper`: **never depend on one source; fan out and take the first
that yields the post text + repo link.**

## Step order (stop as soon as you have text + repo)

1. **Discover with Exa.** Run `mcp__exa__web_search_exa` with 2–3 query
   variants, e.g.:
   - `heygurisingh open sourced Claude Code skill "it's called"`
   - `Guri Singh gurisaroy Claude Code tool GitHub`
   - `heygurisingh someone built agent skill terminal`
   Exa returns Thread Reader unrolls **and** LinkedIn (`gurisaroy`) mirrors that
   already contain the full caption + repo link. This alone usually answers the
   whole scan. (Metered — show/log the cost per your always-show-costs rule.)

2. **Confirm/expand with `WebSearch`** (free). Same intent, different index —
   good for catching tweet URLs/IDs and the tool name when Exa misses.

3. **Pull the full body from a mirror** if the search highlight was truncated:
   `mcp__exa__web_fetch_exa` on the LinkedIn `gurisaroy_...` URL or the
   `threadreaderapp.com/thread/<id>.html` URL. Prefer the LinkedIn mirror — it
   rarely login-walls and usually has the `lnkd.in` repo shortlink.

4. **Hydrate a specific tweet** only if you have an id/URL and want the verbatim
   X text: `node scripts/hydrate-tweet.js <id-or-url>`. Exit `3` = tombstone
   (suspended/deleted) → the account is likely suspended; go back to mirrors,
   don't report "nothing new".

5. **Resolve `lnkd.in` / `t.co` shortlinks** to the real GitHub URL by fetching
   the shortlink (WebFetch/Exa follows the redirect) or by reading the repo name
   out of the post body and searching GitHub for it.

## Turning a post into a candidate

For each post that names an open-source tool:

- Extract: **tool name**, **GitHub repo URL**, **one-line what-it-does**,
  **category** (claude-skill | claude-plugin | CLI/TUI | MCP server | agent
  framework | prompt-pack).
- Enrich the repo: `node scripts/gh-repo-stats.js <repo-url-or-owner/repo>`.
  This returns stars, license, `days_since_push`, archived flag, and
  `is_claude_skill` (detects `SKILL.md` / `.claude/` / plugin manifest) plus
  quick `flags` (unlicensed / stale / archived / low_traction). Set
  `GITHUB_TOKEN` in env to avoid the 60/hr unauthenticated rate limit.

## Failure modes to expect (and how to read them)

- **All tweets tombstone** → account suspended, not "no content". Use mirrors.
- **Exa `SOURCE_NOT_AVAILABLE` on an x.com URL** → expected; never fetch x.com
  directly, go through a mirror.
- **Thread Reader login wall on one thread** → try the LinkedIn mirror or the
  Thread Reader *user* page instead of that specific thread.
- **`lnkd.in` won't resolve** → grep the repo name from the post and hit
  `api.github.com/search/repositories?q=<name>` (or just `gh-repo-stats.js
  owner/repo` once you know the slug).
- **GitHub 403** → unauthenticated rate limit; set `GITHUB_TOKEN` and retry.