← back to Rentv
auto-save: 2026-07-30T10:47:06 (1 files) — docs/
fecb50bbc84e3c899c10835f3ee11cfa531ca748 · 2026-07-30 10:47:08 -0700 · Steve Abrams
Files touched
A docs/CRE_PR_INTELLIGENCE_PLAN.md
Diff
commit fecb50bbc84e3c899c10835f3ee11cfa531ca748
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Thu Jul 30 10:47:08 2026 -0700
auto-save: 2026-07-30T10:47:06 (1 files) — docs/
---
docs/CRE_PR_INTELLIGENCE_PLAN.md | 119 +++++++++++++++++++++++++++++++++++++++
1 file changed, 119 insertions(+)
diff --git a/docs/CRE_PR_INTELLIGENCE_PLAN.md b/docs/CRE_PR_INTELLIGENCE_PLAN.md
new file mode 100644
index 00000000..f72e7cfa
--- /dev/null
+++ b/docs/CRE_PR_INTELLIGENCE_PLAN.md
@@ -0,0 +1,119 @@
+# CRE PR Intelligence — Architecture & Integration Plan
+
+_Branch: `feature/rentv-cre-pr-ca-az`_ · _Owner: RENTV admin_ · _Status: in build_
+
+This document is the required "how the existing architecture works and how this feature
+fits into it" artifact. It is written **after** a full audit of the existing `rentv`
+application and **before** the feature changes anything outside its own namespace.
+
+---
+
+## 1. Existing architecture (audited, not assumed)
+
+| Concern | What actually exists in `rentv` |
+|---|---|
+| **Runtime / framework** | Plain **Node (CommonJS) + Express 4**. Single `server.js` (~740 lines). No build step for the server; `build.mjs` is a *static-site* generator for `public/`. |
+| **Dependencies** | `package.json` lists **only `express`**. No ORM, no DB client, no queue, no test runner, no TypeScript. |
+| **Data layer** | **JSON flat-files** in `data/` (`news.json`, `deals.json`, `posts.json`, `subscribers.jsonl`, `data/consulting/*.json`). Access helpers: `readJSON`, `asArr`, `readPosts/writePosts`, `cRead/cWrite`. |
+| **Auth** | **Two-tier HTTP Basic Auth.** `ROLE_CREDS = {admin:[...], user:[...]}` → `CRED_ROLE` map. A global middleware resolves `req.role`; `adminOnly(req,res,next)` guards every internal API + shell. `OPEN=1` bypasses auth for local preview (sets admin). Env overrides: `BASIC_AUTH_ADMIN`, `BASIC_AUTH_EXTRA`. `/api/health` sits **above** the gate. |
+| **Page serving** | `sendPage(res, absFile)` reads an HTML file and injects a toggle + customer footer. Internal shells are matched by the `INTERNAL_PAGE` regex (no footer) and additionally protected by the `INTERNAL_STATIC` regex + a defense-in-depth static guard. Gated shells that must never be statically served (e.g. `desk-admin.html`) live **outside** `public/` (in `admin/`). |
+| **API convention** | `app.get('/api/<x>', adminOnly, handler)` returning JSON, with `Cache-Control` headers. Writes validate + clamp input (`clean(s,n)`), return `{ok:true,...}`. |
+| **Existing contact data** | Newsletter `subscribers.jsonl`; `/api/audience` CRM (newsletter + sublease brokers); `/desk` Broker/Owner Intelligence proxying the loopback **usrealestate** CA registry (382k brokers / 34k firms / sublease); `/consulting` intake + CRM buckets. |
+| **Outreach letters** | **None.** A full grep found no outreach letter, press-list template, email template, or campaign code. (Only newsletter "unsubscribe" text exists.) → We create the fallback letter as an **editable, non-sending** base template. |
+| **Email integration** | **None wired into this app.** (A global George/Gmail MCP exists at the machine level but is not part of `rentv`.) → We add an abstract email-provider interface with a Gmail adapter behind env/OAuth. |
+| **Jobs / cron** | `scripts/pull-*.mjs` run by external cron to refresh the JSON caches. No in-app queue. |
+| **Deploy** | `.deploy.conf` → pm2 `rentv` on **:9704** at Kamatera, **double gate** (nginx htpasswd in front + app CREDS). Deploy via `~/Projects/_shared/scripts/deploy.sh`. |
+| **Tests** | None. |
+
+### Available infrastructure on the dev machine
+- **PostgreSQL 14** is running locally (socket `/tmp/.s.PGSQL.5432`, existing DBs `dw_unified`, `postgres`).
+
+---
+
+## 2. Design decisions (and why)
+
+The task permits sensible defaults **only when a subsystem truly doesn't exist**, and requires
+*extending* the existing architecture rather than replacing it. The decisions below optimize for
+**minimum divergence** from the existing plain-JS/Express app while giving the intelligence
+system the relational spine it genuinely needs.
+
+1. **Persistence → PostgreSQL via raw `pg` + plain-SQL migrations (NOT Prisma).**
+ A research/CRM system (organizations, people, per-field source evidence, relationships,
+ dedup, scoring, campaigns, outreach threads, tasks, suppression, audit, resumable runs) is
+ inherently relational and far exceeds what JSON flat-files can do safely at 250+ orgs / 750+
+ contacts. Postgres is already running locally and is the house standard across Steve's stack,
+ which is accessed with **raw `pg` + SQL everywhere** — so raw `pg` (not Prisma) is the
+ *faithful* extension and adds no heavy codegen toolchain to a plain-Express app.
+ - New database **`rentv_pr`** (isolated). We **never** touch `dw_unified` or any existing prod table.
+ - Configured by `PR_DATABASE_URL` (fallback: local `/tmp` socket, db `rentv_pr`).
+ - **Migrations only** — a tiny idempotent runner applies `src/pr/migrations/NNN_*.sql` in order and records them in `pr_migrations`. Nothing is dropped/renamed/truncated.
+ - **Graceful degradation:** the entire PR module is lazily initialized and fully isolated. If Postgres is unreachable, `/api/pr/*` returns a clear `503 {ok:false, db:'unavailable'}` and **the rest of `rentv` is completely unaffected**.
+
+2. **Language → stay plain JS (CommonJS) + Express. No Next.js, no TypeScript.**
+ The existing server is CJS `require`. All new server code lives under `src/pr/**` as CJS and
+ mounts through a single `require('./src/pr')(app, deps)` call — a one-line, low-risk diff to
+ `server.js`. Admin UI = server-rendered static HTML shells (same pattern as `/desk`, `/admin`)
+ that fetch JSON from `/api/pr/*`.
+
+3. **Queue → database-backed (`pr_jobs` table + worker loop).**
+ Idempotent, resumable, checkpointed jobs with retry + exponential backoff, pausable from the
+ admin, filterable by state/metro/category. Run with `npm run pr:worker`. No external queue dep.
+
+4. **Tests → Node's built-in `node:test` + `node:assert` for unit + integration.**
+ Zero new dependency (keeps the "only express" footprint), matches the constraint to use the
+ existing framework or a minimal default. Browser tests are scaffolded as Playwright specs
+ under `test/pr/browser/` and require `npx playwright install` (documented, dev-only).
+
+5. **Routes are namespaced and admin-gated.**
+ - Shells: `/admin/pr-intelligence`, `/admin/pr-intelligence/{research,organizations,people,review,campaigns,letters,inbox,tasks,sources,runs,settings}` — all `adminOnly`, all excluded from the customer footer, all added to `INTERNAL_STATIC`.
+ - APIs: `/api/pr/*` — all `adminOnly`.
+
+6. **Geographic rollout is enforced in code.**
+ California-first. Arizona discovery jobs are **hard-gated**: the dispatcher refuses to enqueue
+ AZ discovery unless `pr_settings.arizona_unlocked = true`, which can only flip after the
+ California quality gate passes (or an explicit admin override, which is audit-logged). A test
+ asserts an AZ broad run is blocked before the gate.
+
+7. **Honesty & compliance are structural, not advisory.**
+ - `verification_status` / `email_verification_status` / `linkedin_status` are first-class enums; an inferred email can be **stored** but never sent and is visually flagged.
+ - Every important fact is backed by a `pr_sources` row (URL + excerpt + retrieved_at + usage note). No field is marked "verified" without corroboration.
+ - `pr_suppression` + `pr_audit_log` + CAN-SPAM sender identity/unsubscribe fields are part of the schema and the outreach path.
+ - LinkedIn: we store only **publicly indexed** URLs/snippets from an authorized search API; no logged-in scraping, cookie reuse, or connection automation.
+
+---
+
+## 3. How the pieces fit (module map)
+
+```
+server.js ── one added line: require('./src/pr')(app, {adminOnly, sendPage, PUB, __dirname})
+src/pr/
+ index.js ── mount(app, deps): registers /api/pr/* + /admin/pr-intelligence/* ; lazy DB init
+ db.js ── pg Pool, query(), tx(), runMigrations(), health()
+ migrations/001_init.sql ── all tables + enums + indexes
+ migrations/002_seed_config.sql ── metros, query matrix, settings, base letter + vertical blocks
+ lib/{normalize,scoring,status,dedupe,geo,taxonomy}.js
+ services/{organizations,people,sources,relationships,campaigns,letters,
+ outreach,replies,tasks,suppression,audit,runs,importexport}.js
+ adapters/{index,csv,manual,website,edgar,search,registry,rss,json,linkedin}.js
+ jobs/{index + 15 job modules}.js
+ worker.js ── DB-backed queue runner (npm run pr:worker)
+ seed/ca-seed.js ── real, source-cited CA CRE orgs (npm run pr:seed:ca)
+public/admin/pr-intelligence/ ── 12 HTML shells + pr.js + pr.css
+docs/CRE_PR_*.md ── this plan + data-sources, research-ops, email-setup, compliance, admin-guide
+test/pr/*.test.js ── node:test unit + integration ; test/pr/browser/*.spec.js Playwright
+```
+
+Data flow: **adapters** (discovery) → **research runs / jobs** (resumable) → **services** (normalize,
+dedupe, score, persist with evidence) → **review queue** (human accept/edit/merge) →
+**letters/campaigns** (grounded drafts) → **outreach + inbox** (provider draft, explicit send,
+reply sync) → **suppression + audit** (compliance). Every step writes `pr_audit_log`.
+
+---
+
+## 4. Migration & safety rules honored
+- Additive migrations only; no existing table is dropped, renamed, or truncated.
+- New objects live in a new database (`rentv_pr`); zero coupling to `dw_unified`.
+- Secrets via env only (`.env.example` documents placeholders; nothing committed).
+- The PR module never sends email, never auto-sends a research-generated message, and never
+ begins broad Arizona research before the California quality gate.
+```
← 07c5e2f9 Front page: move Deal Wire video row to the top (above Marke
·
back to Rentv
·
pr-intelligence: plan doc, isolated pg layer (rentv_pr), sch 459558da →