`docs/CRE_PR_INTELLIGENCE_PLAN.md`

120 lines
# CRE PR Intelligence — Architecture & Integration Plan

_Branch: `feature/rentv-cre-pr-ca-az`_ · _Owner: RENTV admin_ · _Status: in build_

This document is the required "how the existing architecture works and how this feature
fits into it" artifact. It is written **after** a full audit of the existing `rentv`
application and **before** the feature changes anything outside its own namespace.

---

## 1. Existing architecture (audited, not assumed)

| Concern | What actually exists in `rentv` |
|---|---|
| **Runtime / framework** | Plain **Node (CommonJS) + Express 4**. Single `server.js` (~740 lines). No build step for the server; `build.mjs` is a *static-site* generator for `public/`. |
| **Dependencies** | `package.json` lists **only `express`**. No ORM, no DB client, no queue, no test runner, no TypeScript. |
| **Data layer** | **JSON flat-files** in `data/` (`news.json`, `deals.json`, `posts.json`, `subscribers.jsonl`, `data/consulting/*.json`). Access helpers: `readJSON`, `asArr`, `readPosts/writePosts`, `cRead/cWrite`. |
| **Auth** | **Two-tier HTTP Basic Auth.** `ROLE_CREDS = {admin:[...], user:[...]}` → `CRED_ROLE` map. A global middleware resolves `req.role`; `adminOnly(req,res,next)` guards every internal API + shell. `OPEN=1` bypasses auth for local preview (sets admin). Env overrides: `BASIC_AUTH_ADMIN`, `BASIC_AUTH_EXTRA`. `/api/health` sits **above** the gate. |
| **Page serving** | `sendPage(res, absFile)` reads an HTML file and injects a toggle + customer footer. Internal shells are matched by the `INTERNAL_PAGE` regex (no footer) and additionally protected by the `INTERNAL_STATIC` regex + a defense-in-depth static guard. Gated shells that must never be statically served (e.g. `desk-admin.html`) live **outside** `public/` (in `admin/`). |
| **API convention** | `app.get('/api/<x>', adminOnly, handler)` returning JSON, with `Cache-Control` headers. Writes validate + clamp input (`clean(s,n)`), return `{ok:true,...}`. |
| **Existing contact data** | Newsletter `subscribers.jsonl`; `/api/audience` CRM (newsletter + sublease brokers); `/desk` Broker/Owner Intelligence proxying the loopback **usrealestate** CA registry (382k brokers / 34k firms / sublease); `/consulting` intake + CRM buckets. |
| **Outreach letters** | **None.** A full grep found no outreach letter, press-list template, email template, or campaign code. (Only newsletter "unsubscribe" text exists.) → We create the fallback letter as an **editable, non-sending** base template. |
| **Email integration** | **None wired into this app.** (A global George/Gmail MCP exists at the machine level but is not part of `rentv`.) → We add an abstract email-provider interface with a Gmail adapter behind env/OAuth. |
| **Jobs / cron** | `scripts/pull-*.mjs` run by external cron to refresh the JSON caches. No in-app queue. |
| **Deploy** | `.deploy.conf` → pm2 `rentv` on **:9704** at Kamatera, **double gate** (nginx htpasswd in front + app CREDS). Deploy via `~/Projects/_shared/scripts/deploy.sh`. |
| **Tests** | None. |

### Available infrastructure on the dev machine
- **PostgreSQL 14** is running locally (socket `/tmp/.s.PGSQL.5432`, existing DBs `dw_unified`, `postgres`).

---

## 2. Design decisions (and why)

The task permits sensible defaults **only when a subsystem truly doesn't exist**, and requires
*extending* the existing architecture rather than replacing it. The decisions below optimize for
**minimum divergence** from the existing plain-JS/Express app while giving the intelligence
system the relational spine it genuinely needs.

1. **Persistence → PostgreSQL via raw `pg` + plain-SQL migrations (NOT Prisma).**
   A research/CRM system (organizations, people, per-field source evidence, relationships,
   dedup, scoring, campaigns, outreach threads, tasks, suppression, audit, resumable runs) is
   inherently relational and far exceeds what JSON flat-files can do safely at 250+ orgs / 750+
   contacts. Postgres is already running locally and is the house standard across Steve's stack,
   which is accessed with **raw `pg` + SQL everywhere** — so raw `pg` (not Prisma) is the
   *faithful* extension and adds no heavy codegen toolchain to a plain-Express app.
   - New database **`rentv_pr`** (isolated). We **never** touch `dw_unified` or any existing prod table.
   - Configured by `PR_DATABASE_URL` (fallback: local `/tmp` socket, db `rentv_pr`).
   - **Migrations only** — a tiny idempotent runner applies `src/pr/migrations/NNN_*.sql` in order and records them in `pr_migrations`. Nothing is dropped/renamed/truncated.
   - **Graceful degradation:** the entire PR module is lazily initialized and fully isolated. If Postgres is unreachable, `/api/pr/*` returns a clear `503 {ok:false, db:'unavailable'}` and **the rest of `rentv` is completely unaffected**.

2. **Language → stay plain JS (CommonJS) + Express. No Next.js, no TypeScript.**
   The existing server is CJS `require`. All new server code lives under `src/pr/**` as CJS and
   mounts through a single `require('./src/pr')(app, deps)` call — a one-line, low-risk diff to
   `server.js`. Admin UI = server-rendered static HTML shells (same pattern as `/desk`, `/admin`)
   that fetch JSON from `/api/pr/*`.

3. **Queue → database-backed (`pr_jobs` table + worker loop).**
   Idempotent, resumable, checkpointed jobs with retry + exponential backoff, pausable from the
   admin, filterable by state/metro/category. Run with `npm run pr:worker`. No external queue dep.

4. **Tests → Node's built-in `node:test` + `node:assert` for unit + integration.**
   Zero new dependency (keeps the "only express" footprint), matches the constraint to use the
   existing framework or a minimal default. Browser tests are scaffolded as Playwright specs
   under `test/pr/browser/` and require `npx playwright install` (documented, dev-only).

5. **Routes are namespaced and admin-gated.**
   - Shells: `/admin/pr-intelligence`, `/admin/pr-intelligence/{research,organizations,people,review,campaigns,letters,inbox,tasks,sources,runs,settings}` — all `adminOnly`, all excluded from the customer footer, all added to `INTERNAL_STATIC`.
   - APIs: `/api/pr/*` — all `adminOnly`.

6. **Geographic rollout is enforced in code.**
   California-first. Arizona discovery jobs are **hard-gated**: the dispatcher refuses to enqueue
   AZ discovery unless `pr_settings.arizona_unlocked = true`, which can only flip after the
   California quality gate passes (or an explicit admin override, which is audit-logged). A test
   asserts an AZ broad run is blocked before the gate.

7. **Honesty & compliance are structural, not advisory.**
   - `verification_status` / `email_verification_status` / `linkedin_status` are first-class enums; an inferred email can be **stored** but never sent and is visually flagged.
   - Every important fact is backed by a `pr_sources` row (URL + excerpt + retrieved_at + usage note). No field is marked "verified" without corroboration.
   - `pr_suppression` + `pr_audit_log` + CAN-SPAM sender identity/unsubscribe fields are part of the schema and the outreach path.
   - LinkedIn: we store only **publicly indexed** URLs/snippets from an authorized search API; no logged-in scraping, cookie reuse, or connection automation.

---

## 3. How the pieces fit (module map)

```
server.js                      ── one added line: require('./src/pr')(app, {adminOnly, sendPage, PUB, __dirname})
src/pr/
  index.js                     ── mount(app, deps): registers /api/pr/* + /admin/pr-intelligence/* ; lazy DB init
  db.js                        ── pg Pool, query(), tx(), runMigrations(), health()
  migrations/001_init.sql      ── all tables + enums + indexes
  migrations/002_seed_config.sql ── metros, query matrix, settings, base letter + vertical blocks
  lib/{normalize,scoring,status,dedupe,geo,taxonomy}.js
  services/{organizations,people,sources,relationships,campaigns,letters,
            outreach,replies,tasks,suppression,audit,runs,importexport}.js
  adapters/{index,csv,manual,website,edgar,search,registry,rss,json,linkedin}.js
  jobs/{index + 15 job modules}.js
  worker.js                    ── DB-backed queue runner (npm run pr:worker)
  seed/ca-seed.js              ── real, source-cited CA CRE orgs (npm run pr:seed:ca)
public/admin/pr-intelligence/  ── 12 HTML shells + pr.js + pr.css
docs/CRE_PR_*.md               ── this plan + data-sources, research-ops, email-setup, compliance, admin-guide
test/pr/*.test.js              ── node:test unit + integration ; test/pr/browser/*.spec.js Playwright
```

Data flow: **adapters** (discovery) → **research runs / jobs** (resumable) → **services** (normalize,
dedupe, score, persist with evidence) → **review queue** (human accept/edit/merge) →
**letters/campaigns** (grounded drafts) → **outreach + inbox** (provider draft, explicit send,
reply sync) → **suppression + audit** (compliance). Every step writes `pr_audit_log`.

---

## 4. Migration & safety rules honored
- Additive migrations only; no existing table is dropped, renamed, or truncated.
- New objects live in a new database (`rentv_pr`); zero coupling to `dw_unified`.
- Secrets via env only (`.env.example` documents placeholders; nothing committed).
- The PR module never sends email, never auto-sends a research-generated message, and never
  begins broad Arizona research before the California quality gate.
```