`references/evaluation-rubric.md`

79 lines
# Evaluation rubric — should Steve actually adopt this?

Same six-dimension family rubric as `gurisingh-scout`, with Axel-specific
modifiers layered on. Score each candidate 0–3 per dimension, 18 max.

## Dimensions

1. **Stack fit (0–3)** — does it serve something Steve actually runs? High:
   Claude Code skills/subagents/hooks, MCP servers, catalog/scraper/pricing
   tooling, cost/observability, marketing/social automation, DW commerce,
   directory builders, video pipelines. Low: generic boilerplate, crypto,
   anything with no path into his workflow.
2. **Novelty vs. existing kit (0–3)** — 3 = fills a real gap; 0 = an
   equal-or-better skill/agent already exists. Run the dedupe check first.
3. **Leverage (0–3)** — measurable hours back or revenue unlocked, not
   "feels productive". Axel's income projections do NOT count as evidence.
4. **Install/run cost (0–3, higher = cheaper)** — 3 = one paste, local, no
   new paid API; 0 = heavy infra, new metered API, or a rewrite. Estimate
   from the repo README, never from the tweet's "2 minutes" framing.
5. **Trust & license (0–3)** — stars/traction (ACTUAL, from
   `gh-repo-stats.js`), license present, recently maintained, no security
   smell. Unlicensed + unmaintained caps at 1.
6. **Safety/gating exposure (0–3, higher = safer)** — 3 = read-only/local,
   no secrets; 0 = wants prod creds, sends outbound, or auto-executes. Low
   scores route to pending-approval, they don't auto-kill.

**Verdict bands:** ≥14 → **ADOPT** (draft the install memo). 9–13 → **WATCH**
(log, revisit on traction). ≤8 → **SKIP** (one-line why).

## Axel modifiers (applied after base scoring)

- **Trust discount:** dimension 5 caps at **2/3** when the only evidence is
  Axel's own post. The cap lifts only when an independent source (repo
  metrics, a second educator, hands-on test) confirms the core claim. Reason:
  the account's engagement-farming style and crypto-shill history mean its
  claims systematically overshoot.
- **Claim-delta flag:** if a verifiable claim (stars, "free", capability)
  misses reality by >2x, note it in the output row AND subtract 1 from
  dimension 5. Repeated deltas across sweeps should be recorded in
  `references/sources.md` under Hype calibration.
- **Income claims are weightless:** "$X/year side hustle" framing
  contributes nothing to Leverage. Score leverage from what the tool does
  for Steve's actual businesses.

## Dedupe procedure (before scoring novelty)

```bash
ls ~/.claude/skills/ | grep -i "<keyword>"
grep -ril "<capability keyword>" ~/.claude/skills/*/SKILL.md 2>/dev/null
```

Also sanity-check the agent cabinet (`~/Projects/agent-cabinet/cabinet.yaml`)
and CNCP. Known equivalents: video creation → `hyperframes` family; research
→ `last30`, `deep-research`, `web-extract`, `knowledge-layer`; cost →
`cost-tracker`; model evaluation → `ollama-model-eval`; browser automation →
`dev-browser`, `browserbase`, `crawl4ai`. If an equivalent exists, output the
**diff** ("video-use edits raw footage off a transcript; hyperframes composes
from scratch — no overlap on ingest"), not a redundant recommend.

## Gating — what the scout may and may NOT do

Discovery, retrieval, enrichment, scoring, and the shortlist run freely
(reversible, read-only). Everything that changes Steve's machine or spends is
gated — draft to `~/.claude/yolo-queue/pending-approval/` with an
APPROVE/REVISE/BLOCK block; never auto-run:

- Installing a skill/plugin (`git clone`, `/plugin install`, paste into `~/.claude`).
- `npx`/running unfamiliar code that executes or reads `~/.claude`.
- Anything needing a new paid API key or that spends.
- Anything the tool itself does that is customer-facing/destructive/outbound.

## Output shape (per run)

Ranked table: `Tool | Repo (claimed vs actual stars, license, last-push) |
What it does | Bucket | Dedupe (have/gap) | Score | Verdict | 1-line reason`.
Then per ADOPT: a pending-approval memo — what it is, why it fits, exact
install steps, cost, risks, APPROVE/REVISE/BLOCK. Log genuine wins (an
adopted pick), never the scan itself.