`docs/community_signal.md`

106 lines
# Community-signal ingest — Reddit, Nextdoor, and the free path

The animals directory has structured public sources (state vet boards, AAHA,
OSM, Petfinder). Community sources fill in everything they miss: which vet is
actually loved, who's the best groomer in a 5-mile radius, who's a scammy
breeder. This doc covers the free ingest path.

## What's wired

| Source     | File                            | Status      | Cost  |
| ---------- | ------------------------------- | ----------- | ----- |
| Reddit     | `src/ingest/reddit.js`          | LIVE        | $0    |
| Nextdoor   | `src/ingest/nextdoor.js`        | STUB        | (paid) |
| Patch.com  | (planned `src/ingest/patch.js`) | not started | $0    |

## Reddit pipeline

```
fetch r/<sub>/new.json (anonymous, ~60 req/min)
   → regex prefilter (drops 80% of noise without an LLM call)
   → gemma3:4b triage  ("is this about a real pet-care business?")
   → qwen3:14b extract (JSON: {business_name, service_type, city, state, sentiment})
   → INSERT raw_records (idempotent on hash)
```

Both LLMs are local Ollama. Zero API cost.

### Subreddits

Default set (`src/ingest/reddit.js`): pet-care subs (`AskVet`, `dogs`, `cats`,
`Pets`, `Petloss`, `puppy101`, `CatAdvice`) + top-50 US metro city subs
(`LosAngeles`, `nyc`, `chicago`, etc).

Override:

```sh
SUBREDDITS=LosAngeles,nyc,chicago LIMIT=25 npm run ingest:reddit
DRY=1 npm run ingest:reddit:dry        # see what would land without writing
```

### What gets stored

`raw_records` rows with `entity_type='reddit_post'` and `raw_json` containing:

```json
{
  "post":     { /* full reddit post */ },
  "mentions": [
    { "business_name": "...", "service_type": "vet_clinic",
      "city": "Los Angeles", "state": "CA",
      "sentiment": "positive", "context": "..." }
  ],
  "sub": "LosAngeles"
}
```

A downstream enrichment pass (next iteration) walks `raw_records`, fuzzy-matches
`business_name + city/state` against `businesses`, and either:

- Adds a `community_mention` row to a new `business_mentions` table (TODO), or
- Promotes a brand-new business if no match exists with confidence ≥ 0.7.

## Weekly schedule

Loaded as a launchd agent on Mac Studio 2:

- Plist: `~/Library/LaunchAgents/com.steve.animals-reddit-weekly.plist`
- Schedule: every Sunday at 07:00 local
- Logs: `~/Projects/animals/logs/reddit-weekly.{log,err}`

Manage:

```sh
launchctl bootstrap   gui/$UID ~/Library/LaunchAgents/com.steve.animals-reddit-weekly.plist
launchctl bootout     gui/$UID/com.steve.animals-reddit-weekly
launchctl kickstart   gui/$UID/com.steve.animals-reddit-weekly      # run now
launchctl print       gui/$UID/com.steve.animals-reddit-weekly
```

## Nextdoor — why it's a stub

Nextdoor has no public API and aggressively blocks anonymous scrapers
(Cloudflare bot detection + login wall). Three free-ish surfaces exist:

1. **Public business pages** (`/pages/<slug>/`) — useful for enriching a
   known business; rate-limits anonymous fetches to a few per minute.
2. **Public sitemap** (`nextdoor.com/sitemap.xml`) — best free starting
   point for discovery; politely crawl with 5s delays.
3. **Logged-in Browserbase session** — required for actual neighborhood
   feed posts and "recommendations" threads. **Not free** — Browserbase
   has paid tiers and Nextdoor itself requires an account. Per Steve's
   "must be free" constraint, not implemented.

When ready to enable the Browserbase route:

```sh
NEXTDOOR_EMAIL=...  NEXTDOOR_PASS=...  USE_BROWSERBASE=1 npm run ingest:nextdoor
```

## Free Nextdoor substitute

**Patch.com** is the closest free analog — city-by-city local-news + community
business mentions, no login wall. A `src/ingest/patch.js` module is the next
brick. Coverage is good in 1,200+ US cities (better than Nextdoor in many
secondary markets).