← back to Commercialrealestate
LoopNet for all listings (free deep-links on every card + gated CoStar adapter) + local rent-roll PDF ingester + feasibility doc
9e99f28ae9911af5c67d12e54cee7fe81b609fe0 · 2026-06-28 17:22:07 -0700 · Steve
Files touched
A data/rent-rolls/.gitignoreA data/rent-rolls/.gitkeepA docs/loopnet-and-rent-rolls.mdM public/index.htmlA scripts/rent-roll-ingest.jsA scripts/sources/loopnet.js
Diff
commit 9e99f28ae9911af5c67d12e54cee7fe81b609fe0
Author: Steve <steve@designerwallcoverings.com>
Date: Sun Jun 28 17:22:07 2026 -0700
LoopNet for all listings (free deep-links on every card + gated CoStar adapter) + local rent-roll PDF ingester + feasibility doc
---
data/rent-rolls/.gitignore | 4 ++
data/rent-rolls/.gitkeep | 0
docs/loopnet-and-rent-rolls.md | 56 ++++++++++++++++++++
public/index.html | 13 ++++-
scripts/rent-roll-ingest.js | 117 +++++++++++++++++++++++++++++++++++++++++
scripts/sources/loopnet.js | 93 ++++++++++++++++++++++++++++++++
6 files changed, 282 insertions(+), 1 deletion(-)
diff --git a/data/rent-rolls/.gitignore b/data/rent-rolls/.gitignore
new file mode 100644
index 0000000..d1d142b
--- /dev/null
+++ b/data/rent-rolls/.gitignore
@@ -0,0 +1,4 @@
+# rent-roll PDFs are confidential (CA-gated). Drop them here; they are NOT committed.
+*.pdf
+*.json
+!.gitkeep
diff --git a/data/rent-rolls/.gitkeep b/data/rent-rolls/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/docs/loopnet-and-rent-rolls.md b/docs/loopnet-and-rent-rolls.md
new file mode 100644
index 0000000..220d5f3
--- /dev/null
+++ b/docs/loopnet-and-rent-rolls.md
@@ -0,0 +1,56 @@
+# LoopNet + Rent Rolls — what's live, what's gated, and why
+
+Steve asked: **"Pull all rent rolls PDF data. Add LoopNet for all listings."** Both have real
+feasibility / legal walls, so here is the honest breakdown and what was actually built.
+
+## LoopNet (CoStar) — for all listings
+
+| Tier | Status | What it gives | Cost |
+|---|---|---|---|
+| **Deep-links on every card** | ✅ LIVE | `LoopNet ↗` (exact site-search for the address) + `(comps)` (city/type browse) on all 1,407 listings | $0 |
+| **Per-property live comps** | ✅ LIVE (pre-existing) | `🔄 Live comps` button → Browserbase cloud browser → LoopNet city/type search | ~$0.03/call |
+| **Structured listing pull (price/cap/units/NOI/broker)** | 🔒 GATED | A proper credentialed feed for every listing | needs CoStar/LoopNet access |
+
+**Why not just scrape LoopNet for all listings?** Two hard reasons:
+1. **Legal.** LoopNet is CoStar Group. CoStar **actively litigates scrapers** — most relevant here, CoStar
+ sued **Crexi** (whose data this tool already uses) over harvesting. Mass-scraping LoopNet is a real
+ legal-risk action, not a technical detail. It is therefore **gated** — not something to launch autonomously.
+2. **Technical.** LoopNet sits behind CoStar anti-bot (Cloudflare/HUMAN-class). Prior discovery found **0 JSON
+ endpoints**. A fleet-wide scrape would need Browserbase + a CAPTCHA solver per page — fragile and, per #1, exposed.
+
+**The honest "LoopNet for all listings" we shipped** is the deep-link cross-reference: every card now opens the
+exact property on LoopNet (or comparable for-sale comps) in one click — full coverage, $0, zero exposure. The
+existing $0.03 Browserbase puller stays the opt-in "pull live data now" action.
+
+**To light up tier 3** (structured data for every listing), provide ONE of:
+- a **CoStar/LoopNet enterprise API key** → set `LOOPNET_API_KEY` (+ `LOOPNET_BASE_URL`) in `.env`; or
+- an **authenticated CoStar/LoopNet session** Steve is licensed to use.
+
+`scripts/sources/loopnet.js` is the inert adapter waiting for that credential (`status()` returns
+`{available:false, needs:'…'}` until set) — identical pattern to `title-records.js`.
+
+## Rent rolls — parse what you have, don't scrape what you can't
+
+**There is no public corpus of rent rolls to "pull."** A rent roll is the confidential unit-by-unit lease
+schedule for a property. It lives behind a signed **Confidentiality Agreement (CA)** on the listing broker's
+deal room (Crexi/Buildout/RCM OM data rooms) or comes from the borrower. Mass-harvesting them would mean
+**circumventing those CAs** — off the table.
+
+**What we built instead — a rent-roll PDF *ingester* (`scripts/rent-roll-ingest.js`):**
+- Drop a rent-roll PDF you legitimately have into `data/rent-rolls/*.pdf`.
+- It parses (via `pdftotext`, $0, local) the unit / tenant / rent / status table and computes:
+ **Gross Scheduled Rent** (monthly + annual), **in-place rent**, **vacancy %**, **avg rent/unit**, and an
+ **NOI proxy** (in-place × (1 − 40% expense ratio) — a transparent placeholder until the OM's actual T-12).
+- Writes a structured sidecar `data/rent-rolls/<name>.json`.
+
+This upgrades the property rent figure from a **Census-median proxy** to **document-grade** for any deal where
+Steve has the rent roll — exactly the input for DSCR / debt-sizing. It reads files; it never fetches them.
+
+**Run:** `node scripts/rent-roll-ingest.js` (all PDFs in the folder) or `node scripts/rent-roll-ingest.js one.pdf`.
+
+## The one decision for Steve
+
+To get **real** LoopNet listing data (not just deep-links) across the catalog, the gated path is a
+**CoStar/LoopNet credential**. That's the same shape as the title-records / appraisal-panel / FNMA-CPM logins
+already scaffolded — drop the credential and the adapter goes live with no caller changes. Until then, the
+free deep-links give full LoopNet coverage on every card today.
diff --git a/public/index.html b/public/index.html
index 676a3b4..fe6d609 100644
--- a/public/index.html
+++ b/public/index.html
@@ -239,6 +239,17 @@ function rentLine(p){
}
// For ranking, treat implausible (>18%) pro-forma yields as low-confidence so bad-units outliers don't top the list.
function rentRoiScore(p){ const r=rentEstimate(p); if(!r||!r.grossYieldPro) return -1e9; return r.grossYieldPro>18 ? -1 : r.grossYieldPro; }
+// LoopNet cross-reference for EVERY listing — free deep-links ($0, no scrape). Mirrors scripts/sources/loopnet.js.
+const LN_TYPE={'Multifamily':'multifamily','Office':'office','Retail':'retail','Industrial':'industrial','Mixed-use':'mixed-use','Mixed Use':'mixed-use','Land':'land'};
+function loopnetLink(p){
+ const sl=s=>String(s||'').toLowerCase().trim().replace(/[^a-z0-9]+/g,'-').replace(/^-|-$/g,'');
+ const city=p.city?`${sl(p.city)}-ca`:'los-angeles-ca';
+ const ptype=LN_TYPE[p.type]||'';
+ // exact: site-search resolves the specific LoopNet listing for this address if it exists.
+ const exact=`https://www.google.com/search?q=${encodeURIComponent(`site:loopnet.com "${[p.address,p.city].filter(Boolean).join(' ')}"`)}`;
+ const browse=ptype?`https://www.loopnet.com/search/${ptype}/${city}/for-sale/`:`https://www.loopnet.com/search/commercial-real-estate/${city}/for-sale/`;
+ return `<a href="${exact}" target="_blank" rel="noopener noreferrer" title="Find this exact property on LoopNet (CoStar) — free search, opens in a new tab">LoopNet ↗</a> <a href="${browse}" target="_blank" rel="noopener noreferrer" class="small" title="Browse comparable ${p.type||'commercial'} for-sale on LoopNet in ${p.city||'LA'}" style="opacity:.7">(comps)</a>`;
+}
// Rich filter state
const F = { types:new Set(), firms:new Set(), cityQuery:'', priceMin:null, priceMax:null,
capMin:0, unitsMin:null, yearMin:null, status:'all', afford:false };
@@ -297,7 +308,7 @@ function card(p){
${p.upside_note?`<div class="small">💡 ${p.upside_note}</div>`:''}
${demoLine(p)}
${rentLine(p)}
- <div class="src"><a href="${p.source}" target="_blank" rel="noopener noreferrer">View listing ↗</a> · <button class="lcbtn" data-id="${p.id}" title="Pull fresh comps from LoopNet via a cloud browser (~$0.03)">🔄 Live comps</button> · <button class="histbtn" data-id="${p.id}" data-address="${(p.address||'').replace(/"/g,'"')}" data-city="${(p.city||'').replace(/"/g,'"')}" data-zip="${p.zip||''}" title="LA County assessor + permit + film history (free)">📜 Property history</button></div>
+ <div class="src"><a href="${p.source}" target="_blank" rel="noopener noreferrer">View listing ↗</a> · ${loopnetLink(p)} · <button class="lcbtn" data-id="${p.id}" title="Pull fresh comps from LoopNet via a cloud browser (~$0.03)">🔄 Live comps</button> · <button class="histbtn" data-id="${p.id}" data-address="${(p.address||'').replace(/"/g,'"')}" data-city="${(p.city||'').replace(/"/g,'"')}" data-zip="${p.zip||''}" title="LA County assessor + permit + film history (free)">📜 Property history</button></div>
<div class="lcout" id="lc-${p.id}"></div>
<div class="histout" id="hist-${p.id}"></div>
</div>`;
diff --git a/scripts/rent-roll-ingest.js b/scripts/rent-roll-ingest.js
new file mode 100644
index 0000000..7290bd9
--- /dev/null
+++ b/scripts/rent-roll-ingest.js
@@ -0,0 +1,117 @@
+// rent-roll-ingest.js — turn a rent-roll PDF into structured underwriting numbers. LOCAL, $0.
+//
+// WHAT THIS IS: a loan originator gets rent rolls (the unit-by-unit lease schedule) as PDFs on the deals
+// they work — from the OM, the listing broker's deal room, or the borrower. This tool parses ONE such PDF
+// into a clean table (unit / tenant / rent / lease term / status) and computes Gross Scheduled Rent, vacancy,
+// average rent/unit, and an NOI proxy — the inputs for DSCR / debt-sizing.
+//
+// WHAT THIS IS NOT: a scraper. Rent rolls are confidential, almost always behind a signed CONFIDENTIALITY
+// AGREEMENT (CA) on the broker's deal room. There is no public corpus to "pull all rent rolls" from, and
+// circumventing a CA to harvest them is off the table. So this operates ONLY on PDFs Steve legitimately has
+// and drops into the folder. It reads files; it does not fetch them.
+//
+// USAGE:
+// 1. Drop rent-roll PDFs into data/rent-rolls/*.pdf
+// 2. node scripts/rent-roll-ingest.js # process all new PDFs
+// node scripts/rent-roll-ingest.js path/to/one.pdf # process one
+// 3. Each PDF gets a sidecar data/rent-rolls/<name>.json with units[] + underwriting{}.
+//
+// DEP: pdftotext (poppler) — already installed. Zero npm deps.
+'use strict';
+const fs = require('fs');
+const path = require('path');
+const { execFileSync } = require('child_process');
+
+const ROOT = path.join(__dirname, '..');
+const DIR = path.join(ROOT, 'data', 'rent-rolls');
+
+const money = s => {
+ if (s == null) return null;
+ const n = parseFloat(String(s).replace(/[^0-9.]/g, ''));
+ return isFinite(n) && n > 0 ? n : null;
+};
+
+// Pull the text out of a PDF with layout preserved (keeps columns roughly aligned).
+function pdfText(file) {
+ try { return execFileSync('pdftotext', ['-layout', file, '-'], { encoding: 'utf8', maxBuffer: 32 * 1024 * 1024 }); }
+ catch (e) { return null; }
+}
+
+// Heuristic row parser. Rent rolls vary wildly, so we look for lines that carry a plausible rent figure
+// and pull the unit id (first token) + the largest dollar figure on the line as the rent. Conservative:
+// a line must have a $ amount in a sane monthly-rent band to count as a unit.
+function parseRows(text) {
+ const units = [];
+ const lines = text.split('\n');
+ const RENT_LO = 200, RENT_HI = 60000; // sane monthly residential/small-commercial rent band
+ for (const raw of lines) {
+ const line = raw.replace(/\s+$/, '');
+ if (!line.trim()) continue;
+ if (/\b(total|subtotal|grand total|gross|scheduled|summary|average|vacancy)\b/i.test(line)) continue; // skip aggregate lines
+ const dollars = (line.match(/\$?\s*[\d,]+(?:\.\d{2})?/g) || [])
+ .map(money).filter(n => n && n >= RENT_LO && n <= RENT_HI);
+ if (!dollars.length) continue;
+ // unit id: a label word + identifier (Unit 101 / Apt A-12 / Suite 200 / #4) if present, else the leading token.
+ const labeled = line.trim().match(/^(?:unit|apt|apartment|suite|ste|room|rm|#)\s*#?\s*([A-Za-z0-9\-]+)/i);
+ const firstTok = labeled ? labeled[1] : ((line.trim().match(/^[A-Za-z0-9#\-]+/) || [])[0] || '');
+ if (!firstTok) continue;
+ const rent = Math.max(...dollars); // largest in-band figure on the line ≈ the rent
+ const vacant = /\bvacant\b|\bvac\b/i.test(line);
+ const beds = (line.match(/\b(\d)\s*(?:bd|br|bed)\b/i) || [])[1];
+ units.push({ unit: firstTok, rent: vacant ? 0 : rent, marketRent: rent, status: vacant ? 'vacant' : 'occupied', beds: beds ? +beds : null });
+ }
+ return units;
+}
+
+function underwrite(units) {
+ const occ = units.filter(u => u.status === 'occupied');
+ const vac = units.filter(u => u.status === 'vacant');
+ const gsrMonthly = units.reduce((s, u) => s + (u.marketRent || 0), 0); // gross SCHEDULED (market) rent
+ const actualMonthly = occ.reduce((s, u) => s + (u.rent || 0), 0); // in-place collected rent
+ const annualGSR = gsrMonthly * 12, annualActual = actualMonthly * 12;
+ const vacancyRate = units.length ? +(vac.length / units.length * 100).toFixed(1) : null;
+ // NOI proxy: small-multifamily operating expenses run ~35-45% of EGI; use 40% as a transparent placeholder.
+ const EXP_RATIO = 0.40;
+ const noiProxy = +(annualActual * (1 - EXP_RATIO)).toFixed(0);
+ return {
+ unitCount: units.length, occupied: occ.length, vacant: vac.length, vacancyRate,
+ grossScheduledRentMonthly: +gsrMonthly.toFixed(0), grossScheduledRentAnnual: +annualGSR.toFixed(0),
+ inPlaceRentMonthly: +actualMonthly.toFixed(0), inPlaceRentAnnual: +annualActual.toFixed(0),
+ avgRentPerUnit: occ.length ? +(actualMonthly / occ.length).toFixed(0) : null,
+ noiProxyAnnual: noiProxy, noiBasis: `in-place rent × (1 − ${EXP_RATIO} expense ratio); replace with the OM's actual T-12 when available`,
+ confidence: 'parsed from rent-roll PDF — verify unit count + totals against the document before underwriting',
+ };
+}
+
+function processOne(file) {
+ const text = pdfText(file);
+ if (!text) return { file: path.basename(file), error: 'pdftotext failed (not a readable PDF?)' };
+ const units = parseRows(text);
+ const uw = underwrite(units);
+ const out = {
+ source: path.basename(file), parsedAt: new Date().toISOString(),
+ units, underwriting: uw,
+ note: 'Confidential rent-roll document provided by Steve. Parsed locally ($0). Not scraped, not redistributed.',
+ };
+ const sidecar = file.replace(/\.pdf$/i, '.json');
+ fs.writeFileSync(sidecar, JSON.stringify(out, null, 2));
+ return { file: path.basename(file), units: units.length, ...uw, sidecar: path.basename(sidecar) };
+}
+
+function main() {
+ const arg = process.argv[2];
+ if (arg) { console.log(JSON.stringify(processOne(path.resolve(arg)), null, 2)); return; }
+ if (!fs.existsSync(DIR)) { fs.mkdirSync(DIR, { recursive: true }); }
+ const pdfs = fs.readdirSync(DIR).filter(f => /\.pdf$/i.test(f));
+ if (!pdfs.length) {
+ console.log(`No rent-roll PDFs found. Drop them into ${DIR}/ and re-run.`);
+ console.log('This tool parses rent rolls you legitimately have (deal OMs, broker deal rooms, borrower docs).');
+ console.log('It does NOT scrape — rent rolls are CA-gated confidential documents with no public source.');
+ return;
+ }
+ const results = pdfs.map(f => processOne(path.join(DIR, f)));
+ console.log(JSON.stringify({ processed: results.length, results }, null, 2));
+}
+
+if (require.main === module) main();
+module.exports = { processOne, parseRows, underwrite };
diff --git a/scripts/sources/loopnet.js b/scripts/sources/loopnet.js
new file mode 100644
index 0000000..fc0360a
--- /dev/null
+++ b/scripts/sources/loopnet.js
@@ -0,0 +1,93 @@
+// sources/loopnet.js — LoopNet (CoStar) cross-reference for EVERY listing.
+//
+// TWO honest tiers, no fabrication:
+//
+// 1. deepLink({address,city,zip,type}) — FREE, $0, zero legal exposure.
+// Pure URL construction. Returns links that OPEN LoopNet for this property:
+// • exact — a `site:loopnet.com "<address>"` Google deep-link that surfaces the
+// specific LoopNet listing if one exists (most reliable for a single address).
+// • browse — a LoopNet city + property-type for-sale search page.
+// This is how we put "LoopNet on all listings" instantly: a click-through, not a scrape.
+//
+// 2. getListing({address,city,zip}) — CREDENTIALED, GATED, returns {available:false,...} until
+// Steve provides a CoStar/LoopNet authenticated session or enterprise API access. Structured
+// LoopNet listing data (price, cap rate, units, NOI, broker) is behind CoStar's login + anti-bot,
+// and CoStar actively litigates scrapers (CoStar v. Crexi). So we DO NOT mass-scrape it here.
+// The existing per-property Browserbase puller (/api/comps -> live-comps-bb.js, ~$0.03/call) stays
+// the opt-in "pull live data" path; this adapter is the slot for a proper credentialed feed.
+//
+// PLUG-IN CONTRACT (mirrors title-records.js so it composes with the rest of the source layer):
+// status() -> { available:boolean, needs?:string, provider?:string }
+// deepLink(prop) -> { exact:string, browse:string } // always available, $0
+// getListing(prop) -> { available:false, needs:string } // until credentialed
+// -> { available:true, matched:boolean, listing }// when credentialed
+'use strict';
+
+const CREDENTIAL_ENV = 'LOOPNET_API_KEY'; // CoStar/LoopNet enterprise key or session token
+const BASE_URL_ENV = 'LOOPNET_BASE_URL'; // provider API base when applicable
+
+function readCredential() {
+ const fromEnv = process.env[CREDENTIAL_ENV];
+ if (fromEnv) return { key: fromEnv, baseUrl: process.env[BASE_URL_ENV] || null };
+ try {
+ const fs = require('fs'), path = require('path');
+ const env = fs.readFileSync(path.join(__dirname, '..', '..', '.env'), 'utf8');
+ const m = env.match(new RegExp('^' + CREDENTIAL_ENV + '=(.*)$', 'm'));
+ if (m) return {
+ key: m[1].replace(/['"]/g, '').trim(),
+ baseUrl: (env.match(new RegExp('^' + BASE_URL_ENV + '=(.*)$', 'm')) || [])[1]?.replace(/['"]/g, '').trim() || null,
+ };
+ } catch (_) {}
+ return null;
+}
+
+function status() {
+ const c = readCredential();
+ if (!c) return {
+ available: false,
+ provider: 'loopnet/costar',
+ needs: 'A CoStar/LoopNet authenticated session or enterprise API key. Set LOOPNET_API_KEY (+ optional ' +
+ 'LOOPNET_BASE_URL) in .env. NOTE: free deep-links work today with no credential; the credential ' +
+ 'only unlocks structured listing pulls (price/cap/units/NOI). Mass-scraping LoopNet is NOT done here — ' +
+ 'CoStar litigates scrapers (CoStar v. Crexi).',
+ };
+ return { available: true, provider: 'loopnet/costar' };
+}
+
+// our asset TYPE -> LoopNet property-type slug (best-effort; falls back to a generic search).
+const TYPE_SLUG = {
+ 'Multifamily': 'multifamily', 'multifamily': 'multifamily',
+ 'Office': 'office', 'office': 'office',
+ 'Retail': 'retail', 'retail': 'retail',
+ 'Industrial': 'industrial', 'industrial': 'industrial',
+ 'Mixed-use': 'mixed-use', 'mixed-use': 'mixed-use', 'Mixed Use': 'mixed-use',
+ 'Land': 'land', 'land': 'land',
+};
+
+const slug = s => String(s || '').toLowerCase().trim().replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '');
+
+// FREE, $0, no fetch — just builds the URLs. Always available.
+function deepLink({ address, city, zip, type, state } = {}) {
+ const st = (state || 'ca').toLowerCase();
+ const citySlug = city ? `${slug(city)}-${st}` : `los-angeles-${st}`;
+ const ptype = TYPE_SLUG[type] || '';
+ // exact: Google site-search reliably resolves the specific LoopNet listing for an address.
+ const q = encodeURIComponent(`site:loopnet.com "${[address, city].filter(Boolean).join(' ')}"`.trim());
+ const exact = `https://www.google.com/search?q=${q}`;
+ // browse: LoopNet city (+ type) for-sale search page.
+ const browse = ptype
+ ? `https://www.loopnet.com/search/${ptype}/${citySlug}/for-sale/`
+ : `https://www.loopnet.com/search/commercial-real-estate/${citySlug}/for-sale/`;
+ return { exact, browse };
+}
+
+// CREDENTIALED structured pull — inert until a credential lands. Fabricates NOTHING.
+async function getListing(prop) {
+ const s = status();
+ if (!s.available) return { available: false, needs: s.needs };
+ // When LOOPNET_API_KEY is configured, implement the real provider call here.
+ // Until then we never reach this branch.
+ return { available: false, needs: s.needs };
+}
+
+module.exports = { status, deepLink, getListing, TYPE_SLUG };
← 3054e11 CRCP: collapse-all-until-activated panels + chip density sli
·
back to Commercialrealestate
·
Property map: all 10,875 geocoded properties on Leaflet, col f4cd532 →