`skills/last30/references/sources.md`

126 lines
# /last30 — source matrix & design notes

Detailed reference for the `/last30` skill. Loaded by the agent only when it needs
the full source config, the engine/web-search split rationale, or scoring detail.
Keep procedural instructions in `SKILL.md`; keep depth here.

## The two-layer design

`/last30` splits sources into two layers by what they cost and how they're reached:

| Layer | Sources | Reached by | Why |
|---|---|---|---|
| **Engine** (`scripts/last30.py`) | Hacker News, arXiv, GitHub, Polymarket | Free public JSON/Atom APIs, no key | Clean, deterministic, parallel, structured engagement numbers |
| **Web-search** (agent) | Reddit, X/Twitter, YouTube, TikTok, Instagram, general news | Host tool (Exa MCP → WebSearch) | These are IP-blocked (Reddit) or key-gated / login-walled (X/TikTok/Instagram); the host already indexes what's public |

The original `mvanhorn/last30days-skill` reaches the second layer with a paid
**ScrapeCreators** key and a vendored X client. This reimplementation deliberately
avoids new paid keys — it uses the web-search tool the host **already has** (on
Steve's stack, the Exa MCP + built-in WebSearch), honoring the standing
"generation/enrichment runs local/free, paid infra is opt-in" rule.

## Engine sources (free, no key)

| Source | Endpoint | Engagement signal | Notes |
|---|---|---|---|
| **Hacker News** | `hn.algolia.com/api/v1/search` | points | `numericFilters=created_at_i>` enforces the window server-side |
| **arXiv** | `export.arxiv.org/api/query` (Atom) | none (recency-led) | sorted by `submittedDate` desc; window filtered client-side |
| **GitHub** | `api.github.com/search/repositories` | stars | `pushed:>=<date>` qualifier; unauth = 10 req/min (fine for one run). If the `gh` CLI is authed, higher limits are available but not required. |
| **Polymarket** | `gamma-api.polymarket.com/public-search` | $ volume + live odds | odds are money-backed probability, not opinion |

All four confirmed working from a clean IP (`scripts/last30.py doctor`).

### Reddit (engine source, opt-in, best-effort)
Reddit's unauthenticated JSON (`www.reddit.com/search.json`, `/r/<sub>/top.json`)
returns **403 Blocked** or **429** on many networks — it's an IP/policy block, not
a User-Agent issue (a desktop-browser UA does not fix it). So Reddit is:
- **kept in the engine** (`--sources reddit,...`) for networks where it works;
- **off by default** (not in `DEFAULT_SOURCES`) so a default run is all-green;
- a **WARN, never a failure** in `doctor`;
- **covered by the web-search layer** in `SKILL.md` Step 3 **only when that layer
  is Exa** (see the caveat below).

If you need Reddit's real upvote/comment structure and the engine is blocked, the
web-search supplement scoped to `reddit.com` is the fallback — **but only via
Exa**. Anthropic's built-in **WebSearch** tool is *also* blocked from `reddit.com`
(returns `400 — domain not accessible to our user agent`), so on a WebSearch-only
host Reddit is dark on **both** paths (engine 403 + WebSearch 400). In that case,
say so in the brief rather than implying Reddit was covered. (Verified 2026-07-13.)

### TikTok & Instagram (web-search layer, best-effort)
These are the completion of the original `/last30days` source set (X, Reddit,
YouTube, TikTok, Instagram, arXiv, HN, Polymarket). Neither has a free no-key API,
so — like X and YouTube — they live in the **web-search layer** (`SKILL.md` Step 3),
never in the engine. Coverage is deliberately **best-effort**:

- **TikTok** — `allowed_domains: ["tiktok.com"]`. Public video pages are indexed
  but engagement (views/likes) shows up only when the search snippet includes it.
  Capture it when visible; never fabricate a count.
- **Instagram** — `allowed_domains: ["instagram.com"]`. Login-walled, so the index
  is thin; treat any relevant hit as a bonus, not a required section.

A stub engine fetcher for either would only ever return an `error` marker (no free
endpoint), so it is intentionally **not** wired into `FETCHERS`/`DEFAULT_SOURCES`.
For real per-post engagement numbers, the optional **ScrapeCreators** key below
(10k free calls) is the paid upgrade path — off by design.

## Optional keyed sources (documented, not required)

Not wired into this engine — enable only if the user explicitly wants them and
accepts the cost. Add as a new fetcher following the fault-isolated pattern
(return `{"source","items"}` or `{"source","error"}`; never raise past the boundary):

| Platform | Key / access | Cost |
|---|---|---|
| X / Twitter | `XAI_API_KEY` (xAI) or browser cookies (`AUTH_TOKEN`+`CT0`) | key = PAYG; cookies = free but account-risk |
| YouTube transcripts | `yt-dlp` CLI on PATH | free |
| TikTok / Instagram | ScrapeCreators key | 10k free calls, then PAYG |
| Bluesky | app password from bsky.app | free |

Cookie-scraping X carries account-suspension risk and is off by design.

## Scoring model (`score_items`)

Each item gets a normalized `score` in **0–100**:

1. **Per-source engagement normalization** — divide by that source's max, so
   50k Reddit upvotes and 5k HN points are comparable within the run.
2. **Recency weight** — linear decay from `1.0` (now) to `~0.15` at the window edge.
3. **Blend** — engagement-led sources: `score = 100·(0.7·eng_norm + 0.3·recency)`.
   Discovery sources with no engagement signal (arXiv, thin Polymarket): recency-only.

`top_across()` flattens every source and returns the global top-N by `score` —
that's the "Top across all sources" block the brief leads with.

## Relevance gate (`apply_relevance`)

Runs **before** scoring to kill keyword collisions (the classic case: a query for
"Kimi K2" pulling Polymarket's F1 driver "Kimi Antonelli"). It tokenizes the topic
(lowercasing, dropping stopwords and separators like "vs"), then keeps an item only
if it matches at least `ceil(len(tokens) * threshold)` distinct topic tokens
(minimum 1). Defaults to `--min-relevance 0.4`:

- 3-token topic ("kimi k2 deepseek") → needs ≥2 hits → the F1 "Kimi" item (1 hit) drops.
- 2-token topic ("nvidia earnings") → needs 1 hit → lenient, keeps either term.
- 1-token topic ("openclaw") → keeps anything containing it.

Every kept item carries a `relevance` fraction; each source reports `filtered_out`
so drops are visible (JSON field + a markdown footer note), never silent. Disable
with `--no-filter` or `--min-relevance 0`; tighten toward `0.6` for noisy topics.

## Extending

- **New free source** → add a `fetch_<name>()` returning the standard shape, add it
  to `FETCHERS`, and (if it should run by default) to `DEFAULT_SOURCES`.
- **Pin a fixed source set** → pass `--sources` explicitly on every call.
- **Faster runs** → `--no-comments` (skips Reddit comment enrichment) and a lower
  `--limit`.

## Lineage / credit

Idea and command shape from **[@mvanhorn](https://github.com/mvanhorn/last30days-skill)**'s
`/last30days` (MIT), surfaced by [@Saboo_Shubham_](https://x.com/saboo_shubham_) — whose
own pro tip was "use this open-source skill … to make it your own." This is that:
an independent, no-key, $0-local reimplementation tuned to a host that already has
web search. MIT-licensed in kind.