`docs/loopnet-and-rent-rolls.md`

57 lines
# LoopNet + Rent Rolls — what's live, what's gated, and why

Steve asked: **"Pull all rent rolls PDF data. Add LoopNet for all listings."** Both have real
feasibility / legal walls, so here is the honest breakdown and what was actually built.

## LoopNet (CoStar) — for all listings

| Tier | Status | What it gives | Cost |
|---|---|---|---|
| **Deep-links on every card** | ✅ LIVE | `LoopNet ↗` (exact site-search for the address) + `(comps)` (city/type browse) on all 1,407 listings | $0 |
| **Per-property live comps** | ✅ LIVE (pre-existing) | `🔄 Live comps` button → Browserbase cloud browser → LoopNet city/type search | ~$0.03/call |
| **Structured listing pull (price/cap/units/NOI/broker)** | 🔒 GATED | A proper credentialed feed for every listing | needs CoStar/LoopNet access |

**Why not just scrape LoopNet for all listings?** Two hard reasons:
1. **Legal.** LoopNet is CoStar Group. CoStar **actively litigates scrapers** — most relevant here, CoStar
   sued **Crexi** (whose data this tool already uses) over harvesting. Mass-scraping LoopNet is a real
   legal-risk action, not a technical detail. It is therefore **gated** — not something to launch autonomously.
2. **Technical.** LoopNet sits behind CoStar anti-bot (Cloudflare/HUMAN-class). Prior discovery found **0 JSON
   endpoints**. A fleet-wide scrape would need Browserbase + a CAPTCHA solver per page — fragile and, per #1, exposed.

**The honest "LoopNet for all listings" we shipped** is the deep-link cross-reference: every card now opens the
exact property on LoopNet (or comparable for-sale comps) in one click — full coverage, $0, zero exposure. The
existing $0.03 Browserbase puller stays the opt-in "pull live data now" action.

**To light up tier 3** (structured data for every listing), provide ONE of:
- a **CoStar/LoopNet enterprise API key** → set `LOOPNET_API_KEY` (+ `LOOPNET_BASE_URL`) in `.env`; or
- an **authenticated CoStar/LoopNet session** Steve is licensed to use.

`scripts/sources/loopnet.js` is the inert adapter waiting for that credential (`status()` returns
`{available:false, needs:'…'}` until set) — identical pattern to `title-records.js`.

## Rent rolls — parse what you have, don't scrape what you can't

**There is no public corpus of rent rolls to "pull."** A rent roll is the confidential unit-by-unit lease
schedule for a property. It lives behind a signed **Confidentiality Agreement (CA)** on the listing broker's
deal room (Crexi/Buildout/RCM OM data rooms) or comes from the borrower. Mass-harvesting them would mean
**circumventing those CAs** — off the table.

**What we built instead — a rent-roll PDF *ingester* (`scripts/rent-roll-ingest.js`):**
- Drop a rent-roll PDF you legitimately have into `data/rent-rolls/*.pdf`.
- It parses (via `pdftotext`, $0, local) the unit / tenant / rent / status table and computes:
  **Gross Scheduled Rent** (monthly + annual), **in-place rent**, **vacancy %**, **avg rent/unit**, and an
  **NOI proxy** (in-place × (1 − 40% expense ratio) — a transparent placeholder until the OM's actual T-12).
- Writes a structured sidecar `data/rent-rolls/<name>.json`.

This upgrades the property rent figure from a **Census-median proxy** to **document-grade** for any deal where
Steve has the rent roll — exactly the input for DSCR / debt-sizing. It reads files; it never fetches them.

**Run:** `node scripts/rent-roll-ingest.js` (all PDFs in the folder) or `node scripts/rent-roll-ingest.js one.pdf`.

## The one decision for Steve

To get **real** LoopNet listing data (not just deep-links) across the catalog, the gated path is a
**CoStar/LoopNet credential**. That's the same shape as the title-records / appraisal-panel / FNMA-CPM logins
already scaffolded — drop the credential and the adapter goes live with no caller changes. Until then, the
free deep-links give full LoopNet coverage on every card today.