← back to Lawyer Directory Builder
Loop housekeeping: migrations, ingest scripts, compliance docs, build-agent automation
03e8b75238415a1be6ddf3afe0dab3d51eb477e0 · 2026-05-03 23:19:24 -0700 · SteveStudio2
Migrations 003-018: site_audits, app_users (+profile), leads, upgrade_orders,
prof_loc_unique, site_mockups (+extras), data_orders, contact_crawled_at,
user_tiers, ratings, community + replies/views, message_user_fk fix, site_intel.
Ingest:
- calbar_full_scrape.ts (full attorney scrape, bar# 29960→360000, conc=5)
- here_discover.ts (HERE Maps Discover API enricher for phone/website,
free-tier 30k/mo, 8,244 LA firms target)
Enrich:
- discover_websites.ts (web-search firm-website discovery)
- geocode_nominatim.ts (lat/lng for organizations)
- render_mockups.ts (Playwright mockup-template renderer)
- site_audit.ts (HEAD/GET timing + headers + marketing_score)
Scripts:
- build-agent/* (launchd watchdog/progress/nightly/weekly cadence)
- take-shots.mjs, click_through.ts (Playwright walkers)
- create_admin.ts (admin user provisioner)
- probe_calbar.ts (smoke test)
- setup_stripe_*.ts (lawyer-pro price + webhook + tier-webhook)
- run_overnight.ts (overnight pipeline orchestrator)
Docs:
- COMPLIANCE.md (Cal §6155 / Rule 7.x / 5.4 / CCPA standing rules)
- legal/attorney-engagement-letter.md (canonical engagement-letter draft)
- legal/MEETING_FOLLOWUP_2026-05-03.md (codex meeting verification)
- legal/PLATFORM_DECISION_2026-05-03.md (separate-repo vs monorepo decision)
- docs/COMMUNITY_PLATFORM_SPEC.md (PawCircles-style community sub-brand spec)
- CHANGES.md / CHANGES2.md (codex review applied-fix log)
Frontend:
- public/dashboard.html (admin dashboard, will be migrated to portal.ts later)
Build/walker artifacts:
- _clickthrough.mjs (Playwright site walker w/ Steve avatar PiP, video out)
- _shoot_mobile.mjs (375x812 mobile probe for overflow detection)
.gitignore: exclude public/screenshots, public/mockups, public/qa-shots,
public/mockup-templates (6+ GB of generated artifacts), *.webm, *.mp4.
.env.example: placeholder env template (no real keys).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Files touched
A .env.exampleM .gitignoreA CHANGES.mdA CHANGES2.mdA COMPLIANCE.mdA _clickthrough.mjsA _shoot_mobile.mjsA docs/COMMUNITY_PLATFORM_SPEC.mdA legal/MEETING_FOLLOWUP_2026-05-03.mdA legal/PLATFORM_DECISION_2026-05-03.mdA legal/attorney-engagement-letter.mdA migrations/003_site_audits.sqlA migrations/004_app_users.sqlA migrations/005_leads.sqlA migrations/006_app_users_profile.sqlA migrations/007_upgrade_orders.sqlA migrations/008_prof_loc_unique.sqlA migrations/009_site_mockups.sqlA migrations/010_data_orders.sqlA migrations/011_contact_crawled_at.sqlA migrations/012_site_mockups_extras.sqlA migrations/013_user_tiers.sqlA migrations/014_ratings.sqlA migrations/015_community.sqlA migrations/016_replies_and_views.sqlA migrations/017_fix_message_user_fk.sqlA migrations/018_site_intel.sqlM package-lock.jsonM package.jsonA public/dashboard.htmlA scripts/build-agent/nightly-email.shA scripts/build-agent/progress.shA scripts/build-agent/watchdog.shA scripts/build-agent/weekly-refresh.shA scripts/take-shots.mjsA src/enrich/discover_websites.tsM src/enrich/firm_website_contacts.tsA src/enrich/geocode_nominatim.tsA src/enrich/render_mockups.tsA src/enrich/site_audit.tsA src/ingest/calbar_full_scrape.tsA src/ingest/here_discover.tsA src/scripts/click_through.tsA src/scripts/create_admin.tsA src/scripts/probe_calbar.tsM src/scripts/run_overnight.tsA src/scripts/setup_stripe_lawyer_pro_price.tsA src/scripts/setup_stripe_tier_webhook.tsA src/scripts/setup_stripe_webhook.ts
Diff
commit 03e8b75238415a1be6ddf3afe0dab3d51eb477e0
Author: SteveStudio2 <stevestudio2@SteveStacStudio.lan>
Date: Sun May 3 23:19:24 2026 -0700
Loop housekeeping: migrations, ingest scripts, compliance docs, build-agent automation
Migrations 003-018: site_audits, app_users (+profile), leads, upgrade_orders,
prof_loc_unique, site_mockups (+extras), data_orders, contact_crawled_at,
user_tiers, ratings, community + replies/views, message_user_fk fix, site_intel.
Ingest:
- calbar_full_scrape.ts (full attorney scrape, bar# 29960→360000, conc=5)
- here_discover.ts (HERE Maps Discover API enricher for phone/website,
free-tier 30k/mo, 8,244 LA firms target)
Enrich:
- discover_websites.ts (web-search firm-website discovery)
- geocode_nominatim.ts (lat/lng for organizations)
- render_mockups.ts (Playwright mockup-template renderer)
- site_audit.ts (HEAD/GET timing + headers + marketing_score)
Scripts:
- build-agent/* (launchd watchdog/progress/nightly/weekly cadence)
- take-shots.mjs, click_through.ts (Playwright walkers)
- create_admin.ts (admin user provisioner)
- probe_calbar.ts (smoke test)
- setup_stripe_*.ts (lawyer-pro price + webhook + tier-webhook)
- run_overnight.ts (overnight pipeline orchestrator)
Docs:
- COMPLIANCE.md (Cal §6155 / Rule 7.x / 5.4 / CCPA standing rules)
- legal/attorney-engagement-letter.md (canonical engagement-letter draft)
- legal/MEETING_FOLLOWUP_2026-05-03.md (codex meeting verification)
- legal/PLATFORM_DECISION_2026-05-03.md (separate-repo vs monorepo decision)
- docs/COMMUNITY_PLATFORM_SPEC.md (PawCircles-style community sub-brand spec)
- CHANGES.md / CHANGES2.md (codex review applied-fix log)
Frontend:
- public/dashboard.html (admin dashboard, will be migrated to portal.ts later)
Build/walker artifacts:
- _clickthrough.mjs (Playwright site walker w/ Steve avatar PiP, video out)
- _shoot_mobile.mjs (375x812 mobile probe for overflow detection)
.gitignore: exclude public/screenshots, public/mockups, public/qa-shots,
public/mockup-templates (6+ GB of generated artifacts), *.webm, *.mp4.
.env.example: placeholder env template (no real keys).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
.env.example | 14 +
.gitignore | 6 +
CHANGES.md | 30 ++
CHANGES2.md | 9 +
COMPLIANCE.md | 108 +++++
_clickthrough.mjs | 185 ++++++++
_shoot_mobile.mjs | 60 +++
docs/COMMUNITY_PLATFORM_SPEC.md | 184 ++++++++
legal/MEETING_FOLLOWUP_2026-05-03.md | 80 ++++
legal/PLATFORM_DECISION_2026-05-03.md | 84 ++++
legal/attorney-engagement-letter.md | 50 +++
migrations/003_site_audits.sql | 45 ++
migrations/004_app_users.sql | 39 ++
migrations/005_leads.sql | 33 ++
migrations/006_app_users_profile.sql | 11 +
migrations/007_upgrade_orders.sql | 39 ++
migrations/008_prof_loc_unique.sql | 23 +
migrations/009_site_mockups.sql | 14 +
migrations/010_data_orders.sql | 51 +++
migrations/011_contact_crawled_at.sql | 7 +
migrations/012_site_mockups_extras.sql | 5 +
migrations/013_user_tiers.sql | 24 +
migrations/014_ratings.sql | 45 ++
migrations/015_community.sql | 60 +++
migrations/016_replies_and_views.sql | 25 ++
migrations/017_fix_message_user_fk.sql | 4 +
migrations/018_site_intel.sql | 12 +
package-lock.json | 647 ++++++++++++++++++++++++++-
package.json | 12 +-
public/dashboard.html | 553 +++++++++++++++++++++++
scripts/build-agent/nightly-email.sh | 68 +++
scripts/build-agent/progress.sh | 53 +++
scripts/build-agent/watchdog.sh | 75 ++++
scripts/build-agent/weekly-refresh.sh | 24 +
scripts/take-shots.mjs | 43 ++
src/enrich/discover_websites.ts | 265 +++++++++++
src/enrich/firm_website_contacts.ts | 34 +-
src/enrich/geocode_nominatim.ts | 181 ++++++++
src/enrich/render_mockups.ts | 517 +++++++++++++++++++++
src/enrich/site_audit.ts | 306 +++++++++++++
src/ingest/calbar_full_scrape.ts | 342 ++++++++++++++
src/ingest/here_discover.ts | 308 +++++++++++++
src/scripts/click_through.ts | 131 ++++++
src/scripts/create_admin.ts | 36 ++
src/scripts/probe_calbar.ts | 72 +++
src/scripts/run_overnight.ts | 41 +-
src/scripts/setup_stripe_lawyer_pro_price.ts | 75 ++++
src/scripts/setup_stripe_tier_webhook.ts | 57 +++
src/scripts/setup_stripe_webhook.ts | 52 +++
49 files changed, 5109 insertions(+), 30 deletions(-)
diff --git a/.env.example b/.env.example
new file mode 100644
index 0000000..5a338fb
--- /dev/null
+++ b/.env.example
@@ -0,0 +1,14 @@
+DATABASE_URL=postgres://USER:PASSWORD@HOST:5432/DATABASE
+PORT=9701
+USER_AGENT=LawyerDirectoryBuilder/0.1 (research; contact: CONTACT_EMAIL)
+RAW_HTML_DIR=./data/raw_html
+STATE_BAR_RPS=0.5
+GOOGLE_PLACES_API_KEY=your-google-places-api-key
+HERE_API_KEY=your-here-api-key
+HERE_DELAY_MS=250
+ADMIN_USERNAME=admin@example.com
+ADMIN_PASSWORD=change-me
+CREATE_ADMIN_PASSWORD=change-me
+STRIPE_SECRET_KEY=sk_test_your-secret-key
+STRIPE_PUBLISHABLE_KEY=pk_test_your-publishable-key
+STRIPE_WEBHOOK_SECRET=whsec_your-webhook-secret
diff --git a/.gitignore b/.gitignore
index f9960cf..b0aa65d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,3 +7,9 @@ logs/
.DS_Store
dist/
exports/
+public/screenshots/
+public/mockups/
+public/qa-shots/
+public/mockup-templates/
+*.webm
+*.mp4
diff --git a/CHANGES.md b/CHANGES.md
new file mode 100644
index 0000000..26bf632
--- /dev/null
+++ b/CHANGES.md
@@ -0,0 +1,30 @@
+## Applied
+- .env.example:1 - added placeholder-only environment template - documents required keys without copying real secrets from `.env`.
+- src/server/portal.ts:352 - added malformed/missing body guard on signup - avoids a 500 when `req.body` is absent.
+- src/server/portal.ts:391 - added malformed/missing body guard on login - avoids a 500 when `req.body` is absent.
+- src/server/portal.ts:534 - added malformed/missing body guard on admin plan updates - avoids a 500 when `req.body` is absent.
+- src/server/leads.ts:186 - added malformed/missing body guard on public lead submission - avoids a 500 when `req.body` is absent.
+- src/server/leads.ts:337 - added malformed/missing body guards on admin lead updates - avoids a 500 when `req.body` is absent.
+- src/server/directory.ts:468 - added malformed/missing body guard on directory inquiries - avoids a 500 when `req.body` is absent.
+- src/server/upgrade.ts:149 - added malformed/missing body guard on checkout creation - avoids a 500 when `req.body` is absent.
+- src/server/upgrade.ts:314 - added malformed/missing body guard on order status updates - avoids a 500 when `req.body` is absent.
+- src/enrich/site_audit.ts:268 - wrapped Playwright browser lifecycle in `try/finally` - ensures the browser closes on worker failures.
+- src/scripts/click_through.ts:30 - wrapped Playwright browser lifecycle in `try/finally` - ensures smoke-test browser cleanup.
+- src/scripts/probe_calbar.ts:14 - wrapped Playwright browser lifecycle in `try/finally` - ensures probe browser cleanup.
+- src/scripts/create_admin.ts:18 - replaced CLI password argument with `CREATE_ADMIN_PASSWORD`/`ADMIN_PASSWORD` env lookup - avoids exposing the password in process args.
+
+## Deferred (needs Steve)
+- P0 - .env:1, .env:9-14 - real secrets are in the project tree - rotation/removal requires Steve; `.env` was left untouched.
+- P0 - src/server/index.ts:19, src/server/upgrade.ts:242 - Stripe raw webhook parser is mounted after global JSON parser - not in safe list.
+- P0 - src/server/upgrade.ts:173 - checkout redirects trust forwarded host headers - needs a fixed `PUBLIC_BASE_URL` decision.
+- P0 - src/server/portal.ts:388 - login lacks rate limit/audit trail - auth policy change, not in safe list.
+- P0 - src/server/index.ts:34 - async Express handlers lack central error wrapping - broad routing change, not in safe list.
+- P0 - src/server/leads.ts:186, src/server/directory.ts:468 - public write endpoints lack spam controls - product/security policy decision.
+- P1 - package.json:10 - scripts reference missing files - deferred because restoring/removing ops commands was not in safe list.
+- P1 - package-lock.json:971 - Express runtime/types mismatch - package/version change is out of scope.
+- P1 - src/scripts/run_overnight.ts:88 - required child job failures do not fail orchestrator - behavior change outside safe list.
+- P1 - src/scripts/migrate.ts:28 - migrations not transactional - migration/DDL area is explicitly deferred.
+- P1 - src/enrich/discover_websites.ts:127 - CLI `LIMIT` validation/parameterization - not in safe list.
+- P1 - src/enrich/firm_website_contacts.ts:127 - duplicate race needs unique indexes/`ON CONFLICT` - schema change is explicitly deferred.
+- P1 - src/lib/match_firms.ts:72 - repeated audit subqueries may slow down - query refactor outside safe list.
+- P1 - package-lock.json:1544 - existing typecheck failures - not changed; tests/typecheck were not run by request.
diff --git a/CHANGES2.md b/CHANGES2.md
new file mode 100644
index 0000000..d93deb6
--- /dev/null
+++ b/CHANGES2.md
@@ -0,0 +1,9 @@
+## Round 2 Applied
+- src/server/data_market.ts:321 — added malformed/missing body guard before reading `b.kind` and other body fields — fixes CODEX_REREVIEW.md "Missed call-sites" finding for `POST /data/checkout`, where absent/non-object `req.body` could 500.
+
+## Still Deferred
+- Previously deferred P0/P1 items in CHANGES.md remain for Steve because they require policy, ops, package, schema, migration, or broader routing decisions outside this safe round-2 scope.
+- Real secret rotation/removal from `.env` remains deferred; no secrets were changed.
+- Stripe webhook parser ordering and forwarded-host redirect trust remain deferred; both need broader server/config decisions.
+- Auth rate limiting/audit trail and public endpoint spam controls remain deferred; both need product/security policy.
+- Package script/runtime/typecheck issues, orchestrator failure behavior, transactional migrations, CLI parameterization, duplicate-race database constraints, and query performance refactors remain deferred.
diff --git a/COMPLIANCE.md b/COMPLIANCE.md
new file mode 100644
index 0000000..88520f2
--- /dev/null
+++ b/COMPLIANCE.md
@@ -0,0 +1,108 @@
+# Counsel & Bar — California Legal Compliance Notes
+
+**Status:** Pre-launch internal notes. Not legal advice. Do not rely on this in lieu of consulting a CA-licensed business attorney before public marketing or paid acquisition.
+
+**Last reviewed:** 2026-05-02 (Steve + Claude session)
+
+---
+
+## 1. Cal. Bus. & Prof. Code § 6155 — Lawyer Referral Service certification
+
+### What it says
+> "An individual, partnership, corporation, association, or any other entity shall not operate for the direct or indirect purpose, in whole or in part, of referring potential clients to attorneys… unless [it] holds a certificate from the State Bar."
+
+### Why it matters here
+The current `/find-a-lawyer` flow:
+1. Consumer submits a case description, practice area, ZIP, urgency, budget
+2. Server runs `match_firms.ts` (zip-centroid + haversine + bounding box)
+3. Lead is routed to 3-5 matched firms in `leads.matched_firm_ids`
+4. The page promises "we put it in front of three to five matched counsels within twenty-four hours"
+
+That is **textbook §6155 referral activity**. The fee structure ($499 EZ Upgrade, $29/mo Lawyer Pro, data marketplace) doesn't change that — §6155 applies to **direct or indirect** referral, regardless of how the operator monetizes.
+
+### What §6155 + Rule 3.830-3.836 actually require
+- **LRS certification** by the State Bar of California
+- **Tariff filed** disclosing referral fees
+- **Annual operations report** to the State Bar
+- **Geographic and practice-area panels** with non-discriminatory attorney admission criteria
+- **Mandatory client-protection trust account**
+- **Specific advertising disclosures** (must include cert number, must say "California Lawyer Referral Service")
+- **No fee-splitting** between LRS and panel attorneys above a regulated cap
+- Annual fees + setup costs (varies; budget several thousand dollars year 1)
+
+### Three paths forward
+
+**Path A — Get certified.** Apply for LRS certification. Roughly 6 months end-to-end. Required if the matching feature is the core value prop.
+
+**Path B — Restructure to a directory.** Drop the active matching/routing. Become a passive **directory + claim/advertising** product:
+- Lawyers list (free or paid)
+- Consumers browse, filter, click direct contact
+- No "we route you" promise
+- No matched_firm_ids in leads — leads, if collected, go only to the user (consumer), not pushed to firms
+
+This pattern is exempt from §6155 (Rule 3.830 cmt. 1: "directories are not LRSes"). Examples: Avvo (directory), Justia, Martindale.
+
+**Path C — Hybrid.** Run as a directory, but offer attorney-tier subscribers a **shared inbox** of consumer-submitted leads they self-serve from. Avoid words like "match," "refer," "route." Position consumer submissions as "client-initiated direct contact requests, broadcast to firms whose self-listed practice areas match." This is grayer; needs the attorney's review.
+
+### Recommendation
+Until LRS certification is in hand, **change the lead flow** before any public marketing push:
+- `/find-a-lawyer` form copy: drop "we put it in front of three to five matched counsels"
+- Backend: stop populating `matched_firm_ids`, stop emailing leads to firms
+- New flow: lead → admin queue → manual outreach (Steve), or → consumer self-serve directory results page
+- Migration plan tracked in §3 below
+
+---
+
+## 2. Cal. Rules of Prof. Conduct 7.1 / 7.2 / 7.3 / 7.5 — attorney advertising
+
+These rules apply to **attorneys** advertising their services. They reach Counsel & Bar **indirectly** because the platform displays attorney content and may be deemed to publish on attorneys' behalf.
+
+### Concrete risks
+
+| Issue | Where it shows up | Fix |
+|---|---|---|
+| **Misleading firm names** (Rule 7.5) | `/showcase` mockups invent firm taglines like "$184M recovered" — if displayed publicly with the real firm's name, that's a 7.5/7.1 false-statement-of-recovery violation | Either (a) the giant **MOCKUP** banner stays prominent and the page header makes 100% clear these are concept designs, OR (b) strip the placeholder stats from variants A/B/C and use only generic copy |
+| **Comparative claims** | "Best California attorney" / "highest rated" — must be substantiated | Don't make subjective superlative claims without source data + footnote |
+| **Schema.org structured data** | Adding `LegalService` / `Attorney` JSON-LD with placeholder ratings = false advertising for the firm | Don't inject schema until each firm's data is verified |
+| **Mockup screenshot usage** | If a screenshot of a mockup goes into a sales email to the firm — the firm receives it and it's clearly a mockup. Low risk. | Continue requiring "Verify before publication" disclaimer on every mockup tile |
+
+---
+
+## 3. Brand-name risk
+
+Adding the word **"Bar"** to a brand operating a referral service in California compounds §6155 risk by implying State Bar affiliation. The State Bar of California has historically pursued cease-and-desist against names creating that implication.
+
+**Decision:** before public launch, choose a brand that does NOT contain "Bar," "Esq," "California State," or any term implying government/regulatory affiliation. Top alternates from the 2026-05-02 search:
+- `counsel.law` (premium TLD, strongest brand)
+- `briefandcounsel.com`
+- `counselgrade.com`
+
+Avoid: `counselandbar.com`, `lawyerbar.com`, anything with "Bar" or "California Bar."
+
+---
+
+## 4. CCPA / CPRA — consumer data
+
+The lead form collects: name, email, phone, ZIP, case description, urgency, budget, practice area.
+
+- Add a **privacy policy** before public launch describing what's collected, how it's used, and CCPA opt-out rights (sale of personal info).
+- Consumer data must NEVER be sold via the data marketplace. Marketplace sells only **State Bar public-record attorney data** + publicly indexed firm contacts. Wall the two databases at the application layer (`leads` table is not a source for `data_orders`).
+- Compliance footer on `/data` already covers this. Verify it stays there.
+
+---
+
+## 5. Action items before public launch
+
+| # | Action | Owner | Status |
+|---|---|---|---|
+| 1 | Remove "we route to 3-5 matched counsels" copy from `/find-a-lawyer` | Steve | open |
+| 2 | Decide §6155 path (A: cert / B: directory / C: hybrid) | Steve + CA-business-attorney consult | open |
+| 3 | Pick non-"Bar" brand name (see §3 alternates) | Steve | open |
+| 4 | Confirm mockup tiles always show MOCKUP banner before any external use | Done in `mockup_templates.ts` | ✓ |
+| 5 | Privacy policy page | Steve | open |
+| 6 | Wall leads table from data_orders queries | Steve / Claude | ✓ (already separate routers) |
+| 7 | Have a CA business attorney review §6155 path before any paid acquisition | Steve | open |
+
+---
+
+**Bottom line:** the codebase is in good shape. The legal posture is not. None of these are tomorrow-emergencies, but they ARE before-public-marketing tasks. The §6155 question is the biggest one — it determines whether the matching feature stays as-is, gets replaced with a directory, or runs through a 6-month certification process.
diff --git a/_clickthrough.mjs b/_clickthrough.mjs
new file mode 100644
index 0000000..c4ed647
--- /dev/null
+++ b/_clickthrough.mjs
@@ -0,0 +1,185 @@
+import { chromium } from 'playwright';
+import fs from 'node:fs/promises';
+import { spawn } from 'node:child_process';
+
+const BASE = 'http://localhost:9701';
+const SHOTS = '/tmp/lawyer-clickthrough';
+await fs.mkdir(SHOTS, { recursive: true });
+
+const surfaces = [
+ { name: '01-landing', path: '/', caption: 'Landing — every CA-licensed attorney, indexed against the State Bar public roll.' },
+ { name: '02-find', path: '/find-a-lawyer', caption: 'Consumer configurator — three short sections; you choose who to call.' },
+ { name: '03-thanks', path: '/find-a-lawyer/thanks?zip=90210&area=Personal+Injury', caption: 'Inquiry logged — we don\'t contact firms on your behalf. §6155-safe receipt.' },
+ { name: '03b-results', path: '/find-a-lawyer/results?zip=90210&area=Personal+Injury', caption: 'Your firms — real California-licensed firms, sorted by proximity. You choose who to call.' },
+ { name: '04-signup', path: '/signup', caption: 'Attorney signup — claim your listing from the live CalBar registry.' },
+ { name: '05-login', path: '/login', caption: 'Sign in — minimal auth surface.' },
+ { name: '06-audit', path: '/audit.html', caption: '1,016 attorney sites independently audited and scored.' },
+ { name: '07-privacy', path: '/privacy.html', caption: 'Privacy — CCPA/CPRA compliant. Lead-form data never sold.' },
+ { name: '08-terms', path: '/terms.html', caption: 'Terms — not a law firm; not a lawyer referral service.' },
+ { name: '09-data', path: '/data', caption: 'Data marketplace — CSV exports of CA attorney directory, public-record only.' },
+];
+
+const OPERATOR = 'Steve Abrams';
+const PROJECT = 'Counsel & Bar';
+const URL_ = 'lawyers.agentabrams.com';
+
+function avatarCss() {
+ return `
+ #steve-overlay {
+ position: fixed; right: 22px; bottom: 22px; z-index: 99999;
+ width: 320px; padding: 14px 16px 14px 14px;
+ background: linear-gradient(180deg, rgba(19,19,22,0.97), rgba(10,10,12,0.95));
+ border: 1px solid #b89968; box-shadow: 0 18px 48px rgba(0,0,0,0.7), 0 0 0 1px rgba(184,153,104,0.18);
+ color: #f4f1ea; font-family: -apple-system, "Inter", system-ui, sans-serif;
+ backdrop-filter: blur(6px); -webkit-backdrop-filter: blur(6px);
+ display: flex; gap: 12px; align-items: flex-start;
+ transform: translateY(8px); opacity: 0;
+ animation: so-rise 700ms cubic-bezier(.16,.84,.3,1) forwards;
+ }
+ @keyframes so-rise { to { transform: none; opacity: 1; } }
+ #steve-overlay .so-avatar {
+ width: 46px; height: 46px; flex: 0 0 46px; border-radius: 50%;
+ background: #b89968;
+ display: flex; align-items: center; justify-content: center;
+ font-family: "Cormorant Garamond", Georgia, serif;
+ color: #0a0a0c; font-size: 18px; font-weight: 500; letter-spacing: 0.02em;
+ border: 1px solid #d4b683;
+ }
+ #steve-overlay .so-body { flex: 1; min-width: 0; }
+ #steve-overlay .so-name { font-size: 11px; letter-spacing: 0.18em; text-transform: uppercase; color: #d4b683; font-weight: 500; margin-bottom: 2px; }
+ #steve-overlay .so-meta { font-size: 9px; letter-spacing: 0.22em; text-transform: uppercase; color: #8b857a; margin-bottom: 10px; font-weight: 500; }
+ #steve-overlay .so-caption { font-size: 12px; line-height: 1.5; color: #d8d2c5; font-weight: 300; }
+ #steve-overlay .so-caption b { color: #f4f1ea; font-weight: 500; }
+ #steve-overlay .so-rec { position: absolute; top: 12px; right: 14px; font-size: 9px; letter-spacing: 0.22em; color: #8b857a; font-weight: 500; font-variant-numeric: tabular-nums; }
+ `;
+}
+
+function avatarHtml(idx, total, caption) {
+ return `
+ <div id="steve-overlay" role="presentation">
+ <div class="so-rec">REC ${String(idx).padStart(2, '0')}/${String(total).padStart(2, '0')}</div>
+ <div class="so-avatar">SA</div>
+ <div class="so-body">
+ <div class="so-name">${OPERATOR}</div>
+ <div class="so-meta">${PROJECT} · ${URL_}</div>
+ <div class="so-caption">${caption}</div>
+ </div>
+ </div>`;
+}
+
+const results = [];
+
+const browser = await chromium.launch();
+const ctx = await browser.newContext({
+ viewport: { width: 1440, height: 900 },
+ recordVideo: { dir: SHOTS, size: { width: 1440, height: 900 } },
+});
+
+// Single page reused across all surfaces so the video shows continuous click-through
+const page = await ctx.newPage();
+let surfaceIdx = 0;
+for (const s of surfaces) {
+ surfaceIdx++;
+ const consoleErrs = [];
+ const failedReq = [];
+ const pageErrs = [];
+ const onConsole = m => { if (m.type() === 'error') consoleErrs.push(m.text().slice(0, 200)); };
+ const onPageErr = e => pageErrs.push(String(e).slice(0, 300));
+ const onReqFail = req => failedReq.push(`${req.method()} ${req.url()} — ${req.failure()?.errorText || '?'}`);
+ page.on('console', onConsole);
+ page.on('pageerror', onPageErr);
+ page.on('requestfailed', onReqFail);
+
+ let status = '?', title = '?', err = null;
+ try {
+ const resp = await page.goto(BASE + s.path, { waitUntil: 'networkidle', timeout: 15000 });
+ status = resp ? resp.status() : '?';
+ // Inject the Steve overlay (refreshes per surface so the caption updates)
+ await page.addStyleTag({ content: avatarCss() }).catch(() => {});
+ await page.evaluate(({ html }) => {
+ const old = document.getElementById('steve-overlay');
+ if (old) old.remove();
+ const wrap = document.createElement('div');
+ wrap.innerHTML = html;
+ document.body.appendChild(wrap.firstElementChild);
+ }, { html: avatarHtml(surfaceIdx, surfaces.length, s.caption) });
+ await page.waitForTimeout(800);
+ await page.evaluate(() => window.scrollTo({ top: document.body.scrollHeight / 3, behavior: 'smooth' }));
+ await page.waitForTimeout(900);
+ await page.evaluate(() => window.scrollTo({ top: 0, behavior: 'smooth' }));
+ await page.waitForTimeout(700);
+ title = await page.title();
+ await page.screenshot({ path: `${SHOTS}/${s.name}.png` });
+ } catch (e) {
+ err = String(e).slice(0, 300);
+ }
+ page.off('console', onConsole);
+ page.off('pageerror', onPageErr);
+ page.off('requestfailed', onReqFail);
+
+ // Heuristic checks for public surfaces
+ let hasMonogram = false, hasVolStrip = false, hasFooterDisclaimer = false;
+ try {
+ hasMonogram = await page.evaluate(() => !!document.querySelector('svg circle[stroke*="mono"], svg circle, svg [d^="M9.6 9.5"]')) || await page.evaluate(() => !!document.querySelector('svg.monogram, .brand-mono svg, header svg'));
+ hasVolStrip = await page.evaluate(() => !!document.querySelector('.vol-strip, .public-vol, .signup-vol'));
+ hasFooterDisclaimer = await page.evaluate(() => /not a law firm|not a lawyer referral service/i.test(document.body.textContent || ''));
+ } catch {}
+
+ results.push({
+ name: s.name,
+ path: s.path,
+ status,
+ title,
+ err,
+ consoleErrs,
+ pageErrs,
+ failedReq: failedReq.filter(r => !/favicon\.ico/.test(r)),
+ hasMonogram,
+ hasVolStrip,
+ hasFooterDisclaimer,
+ });
+}
+
+await page.close();
+await ctx.close(); // flushes the video file
+await browser.close();
+
+// Find the rendered video and rename it for clarity
+const dirEntries = await fs.readdir(SHOTS);
+const webm = dirEntries.find(f => f.endsWith('.webm'));
+let videoPath = null;
+if (webm) {
+ const stamp = new Date().toISOString().replace(/[:.]/g, '-').slice(0, 19);
+ videoPath = `${SHOTS}/walkthrough-${stamp}.webm`;
+ await fs.rename(`${SHOTS}/${webm}`, videoPath);
+}
+
+await fs.writeFile(`${SHOTS}/report.json`, JSON.stringify(results, null, 2));
+
+// Compact text report
+const lines = [];
+lines.push('CLICK-THROUGH REPORT — ' + new Date().toISOString());
+lines.push('='.repeat(72));
+for (const r of results) {
+ lines.push(`\n[${r.status}] ${r.name.padEnd(14)} ${r.path}`);
+ lines.push(` title: ${r.title}`);
+ lines.push(` brand: monogram=${r.hasMonogram} volStrip=${r.hasVolStrip} disclaimer=${r.hasFooterDisclaimer}`);
+ if (r.err) lines.push(` ERR: ${r.err}`);
+ if (r.consoleErrs.length) lines.push(` console errors (${r.consoleErrs.length}):`);
+ r.consoleErrs.slice(0, 5).forEach(e => lines.push(` - ${e}`));
+ if (r.pageErrs.length) lines.push(` page errors (${r.pageErrs.length}):`);
+ r.pageErrs.slice(0, 3).forEach(e => lines.push(` - ${e}`));
+ if (r.failedReq.length) lines.push(` failed reqs (${r.failedReq.length}):`);
+ r.failedReq.slice(0, 5).forEach(e => lines.push(` - ${e}`));
+}
+if (videoPath) {
+ lines.push('');
+ lines.push('VIDEO ' + videoPath);
+}
+const txt = lines.join('\n');
+await fs.writeFile(`${SHOTS}/report.txt`, txt);
+console.log(txt);
+
+if (videoPath) {
+ spawn('open', [videoPath], { detached: true, stdio: 'ignore' }).unref();
+}
diff --git a/_shoot_mobile.mjs b/_shoot_mobile.mjs
new file mode 100644
index 0000000..7c92040
--- /dev/null
+++ b/_shoot_mobile.mjs
@@ -0,0 +1,60 @@
+import { chromium } from 'playwright';
+import fs from 'node:fs/promises';
+
+const BASE = 'http://localhost:9701';
+const OUT = '/tmp/lawyer-mobile';
+await fs.mkdir(OUT, { recursive: true });
+
+const surfaces = [
+ { name: '01-landing', path: '/' },
+ { name: '02-find', path: '/find-a-lawyer' },
+ { name: '03-thanks', path: '/find-a-lawyer/thanks?zip=90210&area=Personal+Injury' },
+ { name: '03b-results', path: '/find-a-lawyer/results?zip=90210&area=Personal+Injury' },
+ { name: '04-signup', path: '/signup' },
+ { name: '05-login', path: '/login' },
+ { name: '06-audit', path: '/audit.html' },
+ { name: '07-privacy', path: '/privacy.html' },
+ { name: '08-terms', path: '/terms.html' },
+ { name: '09-data', path: '/data' },
+ { name: '10-404', path: '/this-page-does-not-exist' },
+];
+
+const browser = await chromium.launch();
+const ctx = await browser.newContext({
+ viewport: { width: 375, height: 812 },
+ deviceScaleFactor: 3,
+ userAgent: 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 Mobile',
+});
+const page = await ctx.newPage();
+const findings = [];
+
+for (const s of surfaces) {
+ try {
+ const resp = await page.goto(BASE + s.path, { waitUntil: 'networkidle', timeout: 12000 });
+ await page.waitForTimeout(500);
+ await page.screenshot({ path: `${OUT}/${s.name}-fold.png` });
+ await page.screenshot({ path: `${OUT}/${s.name}-full.png`, fullPage: true });
+ // Probe for obvious overflow / clipping
+ const probe = await page.evaluate(() => {
+ const docW = document.documentElement.scrollWidth;
+ const winW = window.innerWidth;
+ const overflowsRight = docW > winW + 4;
+ // Find any element wider than viewport
+ const wide = [];
+ document.querySelectorAll('header, main, section, .config-section, .firm-card, .stats, .firms').forEach(el => {
+ const r = el.getBoundingClientRect();
+ if (r.right > winW + 4) wide.push(`${el.tagName.toLowerCase()}.${el.className.slice(0, 30)} right=${Math.round(r.right)}`);
+ });
+ return { docW, winW, overflowsRight, wide: wide.slice(0, 3) };
+ });
+ findings.push({ name: s.name, status: resp ? resp.status() : '?', ...probe });
+ } catch (e) {
+ findings.push({ name: s.name, err: String(e).slice(0, 120) });
+ }
+}
+await browser.close();
+console.log('=== MOBILE 375×812 ===');
+findings.forEach(f => {
+ const flag = f.overflowsRight ? '⚠️' : '✓';
+ console.log(`${flag} [${f.status || 'err'}] ${f.name.padEnd(14)} docW=${f.docW || '?'} wide=${(f.wide || []).join(' / ') || '—'}`);
+});
diff --git a/docs/COMMUNITY_PLATFORM_SPEC.md b/docs/COMMUNITY_PLATFORM_SPEC.md
new file mode 100644
index 0000000..0291f38
--- /dev/null
+++ b/docs/COMMUNITY_PLATFORM_SPEC.md
@@ -0,0 +1,184 @@
+# Lawyer Directory — Community + Ratings Platform Spec
+**Status:** DRAFT — pending Steve's confirms on price/Reddit/scope, then claude-codex debate, then ralph scaffold.
+
+## Vision
+Turn the lawyer directory from a sales tool into a two-sided marketplace:
+- **Clients** find + research + rate lawyers
+- **Lawyers** claim + curate their profiles, respond to clients, network with peers
+- Steve (admin) oversees + monetizes via subscriptions + the existing $499 EZ Upgrade site-rebuild offer
+
+## Four user tiers (lawyer variant)
+
+| Tier | Identity | Auth | Capabilities | Price |
+|------|----------|------|--------------|-------|
+| **Guest** | anonymous | none | browse firms, read public bios, see ratings | free |
+| **Client** | individual | Google OAuth (free signup) | rate lawyers, post in client-side community, DM other clients, toggle comments on their own reviews | free |
+| **Lawyer** | claimed-firm-member | Google OAuth + claim-via-bar# | edit own profile content, respond to client reviews, DM clients, DM peers, paid tier required | **$29/mo** (default; needs Steve confirm) |
+| **Admin** | Steve only | hardcoded | see every firm + every pitch URL + every dashboard, manage takedowns, override ratings, send broadcast pitches | n/a |
+
+**Auth strategy:** Google OAuth via existing `google-oauth-integration` skill — single provider, simplest UX, no password reset hell. Lawyer tier additionally requires bar-number verification (we already have 88,511 bar #s in `professionals` table; lawyers paste their bar# at signup, system matches against `professionals.bar_number` + sends verification email to address on file at CalBar).
+
+## Schema additions
+
+```sql
+-- 013_user_tiers.sql
+ALTER TABLE app_users
+ ADD COLUMN tier TEXT NOT NULL DEFAULT 'client' CHECK (tier IN ('guest','client','lawyer','admin')),
+ ADD COLUMN google_sub TEXT UNIQUE, -- Google OAuth subject ID
+ ADD COLUMN claimed_professional_id BIGINT REFERENCES professionals(id),
+ ADD COLUMN bar_number TEXT, -- self-reported, verified
+ ADD COLUMN bar_verified_at TIMESTAMPTZ,
+ ADD COLUMN paid_until TIMESTAMPTZ, -- subscription expiry
+ ADD COLUMN stripe_customer_id TEXT,
+ ADD COLUMN stripe_subscription_id TEXT,
+ ADD COLUMN comments_enabled BOOLEAN NOT NULL DEFAULT true; -- per-user toggle
+
+-- 014_ratings.sql
+CREATE TABLE ratings (
+ id BIGSERIAL PRIMARY KEY,
+ professional_id BIGINT REFERENCES professionals(id) ON DELETE CASCADE,
+ organization_id BIGINT REFERENCES organizations(id) ON DELETE CASCADE,
+ reviewer_user_id BIGINT REFERENCES app_users(id) ON DELETE SET NULL,
+ source TEXT NOT NULL CHECK (source IN ('user','avvo','google','yelp','reddit','manual')),
+ source_url TEXT,
+ stars NUMERIC(2,1) NOT NULL CHECK (stars BETWEEN 0 AND 5),
+ service_score NUMERIC(2,1), -- responsiveness, communication
+ price_score NUMERIC(2,1), -- value
+ quality_score NUMERIC(2,1), -- outcome
+ comment TEXT,
+ posted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ CONSTRAINT rating_target_check CHECK (professional_id IS NOT NULL OR organization_id IS NOT NULL)
+);
+CREATE INDEX idx_ratings_pro ON ratings(professional_id);
+CREATE INDEX idx_ratings_org ON ratings(organization_id);
+
+-- 015_community.sql
+CREATE TABLE threads (
+ id BIGSERIAL PRIMARY KEY,
+ channel TEXT NOT NULL CHECK (channel IN ('client_public','lawyer_public','dm_client_lawyer','dm_client_client','dm_lawyer_lawyer')),
+ participant_user_ids BIGINT[] NOT NULL, -- for DMs; for public channels = empty
+ topic TEXT,
+ organization_id BIGINT REFERENCES organizations(id), -- threads scoped to a firm
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ last_message_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+CREATE INDEX idx_threads_channel ON threads(channel, last_message_at DESC);
+CREATE INDEX idx_threads_participants ON threads USING GIN (participant_user_ids);
+
+CREATE TABLE messages (
+ id BIGSERIAL PRIMARY KEY,
+ thread_id BIGINT NOT NULL REFERENCES threads(id) ON DELETE CASCADE,
+ user_id BIGINT NOT NULL REFERENCES app_users(id) ON DELETE SET NULL,
+ body TEXT NOT NULL,
+ posted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ edited_at TIMESTAMPTZ,
+ hidden_at TIMESTAMPTZ -- soft-delete for moderation
+);
+CREATE INDEX idx_messages_thread ON messages(thread_id, posted_at);
+```
+
+## Public-rating ingestion (no Reddit, per Steve confirm pending)
+
+Sources to scrape ONCE per firm, store with `ratings.source` set:
+- **Avvo** — has lawyer-specific ratings 1–10, public profile pages
+- **Google Business** — star + review count via Places API ($)
+- **Yelp Fusion API** — star + review count, free tier (5k req/day)
+- (skipped: Reddit — too high-risk per default)
+
+For each firm with a website, the ingester:
+1. Searches Google for `"<firm name>" site:avvo.com` → first profile result
+2. Scrapes the rating + review count
+3. Inserts into `ratings` with `source='avvo'`, `reviewer_user_id=NULL`
+4. Same for Yelp (via API), Google (via Places API if budget allows)
+
+Aggregate ranking formula (initial):
+```
+overall_score =
+ 0.5 * weighted_avg(stars across all sources, weight=review_count)
+ + 0.2 * service_score (avg of user-submitted)
+ + 0.2 * quality_score (avg of user-submitted)
+ + 0.1 * price_score (avg of user-submitted)
+```
+Cold-start: firms with 0 user reviews use only the public-source weighted avg.
+
+## Pages to build
+
+### Guest-visible
+- `/firms/:id` — firm public page (existing data + ratings widget + "claim this firm" CTA for unclaimed)
+- `/lawyers/:id` — individual attorney page (bio + ratings + practice areas)
+- `/community` — read-only feed of public threads (paywall preview at message #5)
+
+### Client-tier
+- `/community` — full read + post in client-public channel
+- `/community/dm` — initiate DM with another client OR lawyer
+- `/profile` — manage account, toggle `comments_enabled`, see all reviews they've posted
+- `/firms/:id/rate` — leave a star rating + dimension scores + comment
+
+### Lawyer-tier (paid)
+- `/dashboard/lawyer` — claimed firm overview, recent reviews, unread DMs
+- `/dashboard/lawyer/edit` — edit own bio, practice areas, photo, contact
+- `/dashboard/lawyer/respond` — reply to client reviews (one reply per review)
+- `/community` — post in lawyer-public + DM peers + DM clients
+- `/billing` — Stripe portal
+
+### Admin (Steve only)
+- `/admin/firms` — every firm + audit score + has-mockup + has-claim flag
+- `/admin/pitches` — every per-firm pitch URL `/p/:id` with status (sent / opened / replied)
+- `/admin/users` — every user, force-tier-change, refund button
+- `/admin/broadcast` — bulk-pitch composer (template + recipient filter from dashboard)
+- `/admin/moderation` — flagged reviews/messages queue
+
+## Pitch persona
+
+**From:** Steve Abrams `<hello@agentabrams.com>`
+**Signature:**
+> Steve Abrams
+> Entrepreneur · Designer · Small-business owner
+> agentabrams.com
+
+Pitch email body (template, per firm) — already drafted in `/p/:firm_id` page; outbound version uses same screenshot+mockup layout in HTML email.
+
+## Build phases (for ralph)
+
+**Phase 1 — Auth + Tiers (week 1)**
+- Migration 013, 014, 015
+- Google OAuth integration (use existing skill)
+- Tier middleware + route guards
+- Admin user seeded via env
+
+**Phase 2 — Ratings (week 2)**
+- Public-source rating ingester (Avvo/Yelp/Google)
+- Star-rating UI on firm + attorney pages
+- Comment toggle on user profile
+
+**Phase 3 — Community (week 3)**
+- Threads + messages schema + REST routes
+- Public client + lawyer feeds
+- DM channels (client-lawyer, peer-peer)
+
+**Phase 4 — Stripe paywall (week 4)**
+- Subscription product creation
+- Webhook handlers (payment_succeeded, subscription_canceled)
+- Tier upgrade flow
+
+**Phase 5 — Lawyer-claim flow (week 4–5)**
+- Bar-number verification email (PurelyMail send via hello@agentabrams.com)
+- Claim approval (auto if email matches CalBar; manual queue otherwise)
+
+**Phase 6 — Admin pages (ongoing)**
+- Stats + moderation + broadcast pitch composer
+
+## Open questions (need Steve)
+
+1. ⏳ **Lawyer subscription price** — $29/mo default, confirm or change
+2. ⏳ **Reddit scraping** — default = SKIP, use Avvo+Yelp+Google only. Confirm.
+3. ⏳ **Vertical scope** — default = build inside `lawyer-directory-builder` (no platform extraction). Confirm.
+4. ⏳ **Google OAuth client** — use existing `dw-sso-integration` skill or fresh OAuth app for `lawyers.agentabrams.com`?
+5. ⏳ **Public domain for client signups** — `lawyers.agentabrams.com` is already configured per CNCP. Use that, or new subdomain `directory.agentabrams.com`?
+
+## What happens NOW (no confirms needed)
+
+- Mac1 keeps generating mockups in background (independent of this spec)
+- This spec waits for Steve's confirms
+- Then `/claude-codex` is invoked to debate the architecture (specifically: tier middleware shape, message-thread schema choice, public-rating attribution legality)
+- Then `/ralph` is invoked with the phase-by-phase task list above to scaffold autonomously
diff --git a/legal/MEETING_FOLLOWUP_2026-05-03.md b/legal/MEETING_FOLLOWUP_2026-05-03.md
new file mode 100644
index 0000000..8899322
--- /dev/null
+++ b/legal/MEETING_FOLLOWUP_2026-05-03.md
@@ -0,0 +1,80 @@
+# Claude-Codex Evening Meeting — Followup
+**2026-05-03 17:00 PDT · Direct verification by parallel-act agent**
+
+The meeting report contained material errors. Verified findings below.
+
+---
+
+## ❌ Report claims that were WRONG
+
+### 1. "PM2 inaccessible — fifth consecutive missed status check"
+**FALSE.** PM2 is online. `pm2 ping` returns `pong`. Full fleet of **62 processes** visible. Direct read worked first try.
+
+```
+pm2 --version → 6.0.14
+pm2 ping → pong
+pm2 jlist → 62 processes
+```
+
+The meeting's premise — "we have no observability" — is incorrect. Observability was always available; the meeting agent never tried.
+
+### 2. "george-gmail OAuth unrotated, no Sunday confirmation"
+**FALSE / MISLEADING.** george-gmail is online and healthy:
+
+```
+status=online · restarts=4 (over the service's lifetime)
+profile API call → steve@designerwallcoverings.com · 36,660 messages
+```
+
+No evidence the OAuth needs rotation. The "5 days overdue" framing has no supporting data. If rotation is *policy* (not response to incident), set a calendar reminder; don't surface as a P0 every meeting.
+
+### 3. "No commits landed today" → "complete standstill"
+**MISLEADING.** This session alone shipped 7 iterations on `lawyer-directory-builder` (landing, /find-a-lawyer, /thanks, /signup, static surfaces, hero proof strip, /audit editorial). Working tree has 63 modified files awaiting a single batched commit. Different from "standstill."
+
+---
+
+## 🚨 Real issues the meeting MISSED
+
+### A. Three zombie PM2 processes (2,000+ restarts each)
+| Service | Restarts | cwd |
+|---|---:|---|
+| `gemini-classifier` | **2,312** | `kamatera-mirror/DW-Agents/scripts` |
+| `gemini-tag-engine` | **2,309** | `kamatera-mirror/.../gemini-tag-engine` |
+| `gemini-catalog-tagger` | **189** | `kamatera-mirror/.../shopify/scripts` |
+
+All status `stopped` — PM2 gave up after thousands of crash cycles. They run from `kamatera-mirror/`, which is the **read-only Kamatera mirror**, not the live source-of-truth. Likely cause: someone enabled them locally on Mac2 thinking they were the prod path. **Recommended: `pm2 delete` all three to stop polluting the registry.**
+
+### B. Three projects with NO git remote (data-loss risk)
+| Project | Uncommitted | Last commit | Remote |
+|---|---:|---|---|
+| `professional-directory` (doctors) | **49 files** | 34h ago | **NONE** |
+| `animals` | **3,153 files** ⚠️ | 2 days ago | **NONE** |
+| `visual-factory` | 1 file | 2 days ago | **NONE** |
+
+**This is the actual P0 the meeting should have surfaced.** If Mac2's disk dies, this work is unrecoverable. Most of the `animals` 3,153 is `logs/debug-loop-*` noise (gitignorable), but `agents/animal-agent/server.js` is real new code.
+
+### C. `am-recrawl` is not a project, it's a debug artifact
+`/Users/stevestudio2/Projects/codex-review-2026-04-30/am-recrawl/` contains **only** `poke.log`, `poke.md`, `recent-errors.log` — a one-time analysis output from the 2026-04-30 codex review. PM2 entry should be deleted, not "diagnosed."
+
+---
+
+## Actions taken in this turn (autonomous, low-risk)
+
+1. ✅ This followup written
+2. ✅ `legal/PLATFORM_DECISION_2026-05-03.md` — lawyers + doctors architectural decision (committed to working tree)
+3. ✅ `~/Projects/animals/.gitignore` — added `logs/debug-loop-*` pattern to drop ~3000 untracked files from the noise pile
+
+## Actions deferred (need explicit Steve auth — state changes)
+
+1. ⏸ `pm2 delete am-recrawl gemini-classifier gemini-tag-engine gemini-catalog-tagger` — registry cleanup
+2. ⏸ `gh repo create` for `professional-directory`, `animals`, `visual-factory` + initial push — backup gap
+3. ⏸ `git add -A && git commit` of in-flight working trees — Steve's review-gate
+
+## Suggested Monday opener (revised)
+
+1. **Confirm pm2 delete on the 4 zombie/stale entries** (am-recrawl + 3 gemini-*) — 30 sec
+2. **Create GitHub repos for the 3 unbacked projects** + initial push — 5 min
+3. **Decide on animals .gitignore + commit batch** — 5 min
+4. **Resume iter loop** — back to the lawyers graphic loop where iter 8 is queued
+
+PM2 access was never the blocker. The meeting agent likely never tried.
diff --git a/legal/PLATFORM_DECISION_2026-05-03.md b/legal/PLATFORM_DECISION_2026-05-03.md
new file mode 100644
index 0000000..647244f
--- /dev/null
+++ b/legal/PLATFORM_DECISION_2026-05-03.md
@@ -0,0 +1,84 @@
+# Platform Decision — Lawyers + Doctors (and the rest)
+**2026-05-03 · Long-overdue from `apply-templates.js` (~58h stale)**
+
+## TL;DR
+**Keep the projects as separate git repos. Extract a shared "vertical-directory" core into `~/Projects/vertical-directory-core` as published modules. Don't merge them into one monorepo — the compliance copy diverges too sharply by vertical.**
+
+---
+
+## What the projects share TODAY (verified 2026-05-03)
+
+| Capability | lawyer-directory-builder | professional-directory (doctors) | animals | restaurant-prof-dir (LA County Eats) |
+|---|---|---|---|---|
+| Stack | Express + tsx + PG | Express + tsx + PG | Express + tsx + PG | Express + tsx + PG |
+| Schema pattern | `professionals`/`organizations`/`practice_areas` | same | analog (`businesses`/`categories`) | analog (`restaurants`/`permits`) |
+| Mockup engine | yes (4-variant per firm) | yes | yes (3-variant per record) | planned |
+| Audit pipeline | yes (`/audit.html`) | yes (`/audit.html`) | no | planned |
+| Lead form | yes (`/find-a-lawyer`) | yes | yes | planned |
+| Brand/copy | "Counsel & Bar" §6155 strict | "—" no equivalent compliance load | community-positive ("PawCircles") | civic-record positive |
+
+## What's specific TO each (non-shareable)
+
+- **Lawyers**: Cal B&P §6155, Rule 5.4/7.x, "not a referral service" copy, State Bar disclaimers on every page. **Operator must remain a non-attorney.** This is binding regulatory context — cannot leak into other verticals.
+- **Doctors**: would need parallel HIPAA-aware language if any patient PII enters; current state doesn't, but Steve has flagged this as a regulatory landmine to avoid.
+- **Animals**: community/UGC tone, no per-record liability concerns
+- **Restaurants**: civic public-records vibe, no advertising-rule load
+
+The compliance copy is **not template-able**. A shared "directory.html" template with pluggable copy would invite cross-vertical bleed (e.g., a Rule-5.4 disclaimer ending up on the doctors site, or a HIPAA risk creeping onto lawyers).
+
+## Decision: SEPARATE repos · SHARED core · VERTICAL copy
+
+```
+~/Projects/vertical-directory-core/ # NEW, published as @vd-core/* npm scope
+ ├─ schema/ # canonical PG schema (professionals/orgs/practice_areas)
+ ├─ ingest/ # base scrapers, autocomplete, dedup
+ ├─ mockup-engine/ # 4-variant mockup pipeline (vendor-agnostic)
+ ├─ audit-pipeline/ # site-audit (lighthouse + signals)
+ ├─ ui-tokens/ # noir/champagne palette, Cormorant+Inter, monogram SVG factory
+ └─ ui-components/ # configurator, Volume strip, brand-mono, public-trust block
+
+~/Projects/lawyer-directory-builder/ # CURRENT, repo on GitHub
+ imports @vd-core/* · adds Cal §6155 copy + State Bar audits + EZ Upgrade product
+
+~/Projects/professional-directory/ # CURRENT, NEEDS REMOTE
+ imports @vd-core/* · adds doctor-specific intake + medical-board verification
+
+~/Projects/animals/ # CURRENT, NEEDS REMOTE
+ imports @vd-core/{ui-tokens,mockup-engine} · skips audit-pipeline · adds PawCircles community
+
+~/Projects/lacountyeats/ # PLANNED (registered 2026-05-02)
+ imports @vd-core/* · adds civic-permit ingest
+```
+
+## Why not a monorepo
+
+1. **Compliance copy must NOT be DRY.** A monorepo invites accidental sharing of regulatory disclaimers across verticals. Lawyer Rule 7.x ≠ doctor HIPAA ≠ animal community policy. Each vertical's `legal/` and visible disclaimers stay 100% in-vertical.
+2. **Independent deploy cadence.** Lawyers ships a hero proof strip on Sunday; restaurants ships next Tuesday. Branch protection / CI lockup of a monorepo would block the loop iteration cadence Steve runs.
+3. **Independent secrets.** Each vertical may end up with its own Stripe account, GA4 property, Sendgrid sender. A leaked `.env` in one vertical shouldn't blast-radius to the others.
+4. **Independent compliance reviews.** When Steve hires the CA business attorney for §6155 sign-off (per `legal/attorney-engagement-letter.md`), the review scope is *just* lawyer-directory-builder — not 4 verticals at once.
+
+## Why a published core (not just copy/paste)
+
+- Bug fix in mockup engine flows to all verticals on `pnpm update @vd-core/mockup-engine`
+- UI token changes (palette, type) propagate uniformly — the brand audit Steve has been running this session would have one canonical source instead of 4 drifted forks
+- Schema migrations versioned — `@vd-core/schema` v0.4 → v0.5 forces every vertical to migrate or pin
+
+## Migration plan (proposed, not committed)
+
+| Phase | Work | Time |
+|---|---|---|
+| 0 | This decision doc reviewed/signed by Steve | 1 day |
+| 1 | Create `~/Projects/vertical-directory-core/` repo + initial extract from lawyer-directory-builder (cleanest pattern) | 1 week |
+| 2 | Migrate professional-directory (doctors) to consume `@vd-core/*` — proves portability | 1 week |
+| 3 | Add GitHub remotes to professional-directory + animals + visual-factory (pre-existing P0 from MEETING_FOLLOWUP) | 30 min |
+| 4 | Migrate animals subset, then plan lacountyeats build on `@vd-core` from day 1 | 2 weeks |
+
+## What `apply-templates.js` (Steve's stale 58h-old script) was probably trying
+
+Best guess from filename: copy a tested HTML template from one vertical to others. **That's the wrong layer to share.** Templates with embedded copy are the part that MUST diverge. Share *components* (Volume strip, monogram, configurator chrome), not *templates* (full pages with disclaimer text).
+
+**Recommend: archive `apply-templates.js`. Replace with the `@vd-core/ui-components` migration in Phase 1.**
+
+---
+
+**Decision required from Steve:** approve the shape above, or counter. If approve, Phase 1 starts on next loop iteration after the lawyer landing surfaces converge.
diff --git a/legal/attorney-engagement-letter.md b/legal/attorney-engagement-letter.md
new file mode 100644
index 0000000..c927acd
--- /dev/null
+++ b/legal/attorney-engagement-letter.md
@@ -0,0 +1,50 @@
+# Outreach email to a California business attorney
+
+**Use:** Send this to 3–5 CA-licensed business attorneys who advertise flat-fee work for tech / SaaS founders. Ask for written responses; collect bids; pick the most reasonable.
+
+**Suggested search terms:** "California business attorney flat fee SaaS", "California attorney advertising compliance flat fee", "Avvo / Martindale: business attorney + Los Angeles + flat-fee consultation".
+
+---
+
+**To:** *(attorney name)*
+**Subject:** Flat-fee opinion request — California directory platform compliance (4 questions)
+
+---
+
+I'm a non-lawyer operator of a California-licensed-attorney directory and software platform. The site is live at lawyers.agentabrams.com (working brand: Counsel & Bar). I'm writing for a written opinion on four specific compliance questions before I begin any paid customer acquisition. I'm prepared to pay a flat fee for written answers — please quote.
+
+**About the operation:**
+- I display California-licensed attorneys, sourced from the State Bar's public licensing database, organized by city and practice area.
+- Consumers can fill out an inquiry form and view a list of licensed firms in their area; **the consumer chooses who to contact**. I do not auto-route, broker, or share consumer details with firms.
+- I sell two products to attorneys: a one-time $499 website rebuild ("EZ Upgrade") and a $29/month subscription ("Lawyer Pro") for engagement features (review responses, profile editing, lead alerts surfaced inside the platform). Pricing is flat — never per case, per lead, or contingent on outcome.
+- I sell a separate data product (CSV exports of attorney directory information sourced from State Bar public records) to business buyers under a use restriction prohibiting unsolicited bulk SMS, robocalls, or unlawful purposes.
+- I am not licensed to practice law in any jurisdiction.
+
+**The four questions** (please answer each in writing, brief is fine):
+
+1. **§6155 LRS scope.** Given that I do not auto-route consumer details, do not charge attorneys for placement of any specific consumer, and present a directory the consumer browses themselves — does my operation fall outside California Business & Professions Code §6155's definition of a "lawyer referral service"? If you see exposure, what specifically would I need to change?
+
+2. **Rule 5.4 / fee-splitting.** Do the flat $499 (one-time website rebuild) and $29/month (engagement-tools subscription) fees create any fee-splitting concern under Cal. Rules of Prof. Conduct 5.4, given that neither price is contingent on a case outcome or a specific consumer being placed?
+
+3. **Rule 7.x lawyer-advertising compliance.** I display attorney profiles populated from State Bar public records, plus content the attorney supplies after claiming a listing. Do I (the platform) have any independent advertising-rule obligation under 7.1 / 7.2 / 7.3 / 7.5 — separate from the attorney's own obligation for content they post — that I should address with disclaimers, gating, or content review?
+
+4. **Data marketplace use restrictions.** The CSV export carries a contractual restriction (no bulk SMS, no robocalls, no unlawful purposes). If a buyer violates that restriction and the violated party complains to the State Bar or AG, what's my exposure as the seller? What contractual or operational safeguards should I add (acceptance click-through, audit trail, indemnification language) before going public with this product?
+
+**Format I'd like:**
+- Brief written answer to each of the four questions
+- A short list of any additional risks you spot in the operation as described
+- An optional 30-minute call to discuss your responses
+
+**Logistics:**
+- I'd like a flat fee for this scope (please quote separately if a follow-up call is included)
+- Engagement is single-scope; I am not retaining you ongoing
+- I'm happy to sign a standard engagement letter and pay 50% on engagement, 50% on delivery of the written opinion
+- Turnaround: ideally 14 days from engagement
+
+I have a written internal compliance memo I can share under our engagement that walks through my current positions, the policies I've published (privacy, terms), and the technical controls I've put in place (no auto-routing, fail-closed Stripe webhook verification, admin-gated marketing mockups). Happy to send once you respond.
+
+Thank you for your time.
+
+**Steve Abrams**
+*Operator, Counsel & Bar (lawyers.agentabrams.com)*
+steve@designerwallcoverings.com
diff --git a/migrations/003_site_audits.sql b/migrations/003_site_audits.sql
new file mode 100644
index 0000000..17b2317
--- /dev/null
+++ b/migrations/003_site_audits.sql
@@ -0,0 +1,45 @@
+-- Site audits — captures a snapshot of each firm's marketing presence:
+-- color palette, content signals, weight, and a 0-100 "marketing health" score.
+--
+-- This is NOT a real traffic ranking (those require paid APIs). It's a
+-- composite health score from publicly observable on-page signals only.
+-- "Better marketing" is a defensible inference; "more traffic" would be a lie.
+
+CREATE TABLE IF NOT EXISTS site_audits (
+ id BIGSERIAL PRIMARY KEY,
+ organization_id BIGINT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ audited_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ url TEXT NOT NULL,
+ final_url TEXT,
+ status_code INTEGER,
+
+ -- Visual
+ screenshot_path TEXT,
+ primary_color TEXT,
+ palette JSONB, -- [{hex,r,g,b,fraction},...] top-N
+
+ -- On-page signals (booleans)
+ has_https BOOLEAN,
+ has_favicon BOOLEAN,
+ has_og_image BOOLEAN,
+ has_meta_viewport BOOLEAN,
+ has_analytics BOOLEAN, -- GA4 / GTM / Plausible / Fathom
+ has_h1 BOOLEAN,
+ has_schema_org BOOLEAN,
+ has_phone_visible BOOLEAN,
+
+ -- Page facts
+ page_size_bytes INTEGER,
+ load_time_ms INTEGER,
+ font_count INTEGER,
+ image_count INTEGER,
+ link_count INTEGER,
+
+ -- Score & suggestions
+ marketing_score INTEGER, -- 0..100
+ suggestions TEXT[]
+);
+
+CREATE INDEX IF NOT EXISTS idx_site_audits_org ON site_audits(organization_id);
+CREATE INDEX IF NOT EXISTS idx_site_audits_score ON site_audits(marketing_score DESC NULLS LAST);
+CREATE INDEX IF NOT EXISTS idx_site_audits_when ON site_audits(audited_at DESC);
diff --git a/migrations/004_app_users.sql b/migrations/004_app_users.sql
new file mode 100644
index 0000000..c1ff4b9
--- /dev/null
+++ b/migrations/004_app_users.sql
@@ -0,0 +1,39 @@
+-- App users + sessions for the admin/user portals.
+-- Separate from `professionals` (which is the bar-licensed-attorney directory)
+-- and `organizations` (firms). An app_user MAY claim a professional/firm.
+
+CREATE TABLE IF NOT EXISTS app_users (
+ id BIGSERIAL PRIMARY KEY,
+ email TEXT UNIQUE NOT NULL,
+ password_hash TEXT NOT NULL,
+ full_name TEXT,
+ role TEXT NOT NULL DEFAULT 'user' CHECK (role IN ('user', 'admin')),
+ plan TEXT NOT NULL DEFAULT 'free' CHECK (plan IN ('free', 'premium', 'pro')),
+ status TEXT NOT NULL DEFAULT 'active' CHECK (status IN ('active', 'suspended', 'deleted')),
+ organization_id BIGINT REFERENCES organizations(id) ON DELETE SET NULL,
+ professional_id BIGINT REFERENCES professionals(id) ON DELETE SET NULL,
+ bar_number TEXT,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ last_login_at TIMESTAMPTZ
+);
+
+CREATE INDEX IF NOT EXISTS idx_app_users_email ON app_users(LOWER(email));
+CREATE INDEX IF NOT EXISTS idx_app_users_role ON app_users(role);
+CREATE INDEX IF NOT EXISTS idx_app_users_status ON app_users(status);
+
+CREATE TABLE IF NOT EXISTS app_sessions (
+ id TEXT PRIMARY KEY,
+ user_id BIGINT NOT NULL REFERENCES app_users(id) ON DELETE CASCADE,
+ expires_at TIMESTAMPTZ NOT NULL,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ ip TEXT,
+ user_agent TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_app_sessions_user ON app_sessions(user_id);
+CREATE INDEX IF NOT EXISTS idx_app_sessions_expires ON app_sessions(expires_at);
+
+CREATE OR REPLACE TRIGGER trg_app_users_updated
+ BEFORE UPDATE ON app_users
+ FOR EACH ROW EXECUTE FUNCTION set_updated_at();
diff --git a/migrations/005_leads.sql b/migrations/005_leads.sql
new file mode 100644
index 0000000..1d45bb4
--- /dev/null
+++ b/migrations/005_leads.sql
@@ -0,0 +1,33 @@
+-- Inbound consumer leads — the LendingTree-for-lawyers funnel.
+-- Submitted via the public /find-a-lawyer form, reviewed/routed by admins.
+
+CREATE TABLE IF NOT EXISTS leads (
+ id BIGSERIAL PRIMARY KEY,
+ full_name TEXT NOT NULL,
+ email TEXT NOT NULL,
+ phone TEXT,
+ practice_area TEXT NOT NULL, -- 'personal_injury' | 'family' | 'criminal' | 'immigration' | 'business' | 'estate' | 'other'
+ zip TEXT,
+ city TEXT,
+ case_description TEXT,
+ budget TEXT, -- 'under_500' | '500_2k' | '2k_10k' | '10k_plus' | 'unsure'
+ urgency TEXT, -- 'immediate' | 'within_week' | 'within_month' | 'just_researching'
+ consent_to_contact BOOLEAN NOT NULL DEFAULT true,
+ status TEXT NOT NULL DEFAULT 'new' CHECK (status IN ('new', 'matched', 'contacted', 'won', 'lost', 'spam')),
+ matched_firm_ids BIGINT[],
+ admin_notes TEXT,
+ ip TEXT,
+ user_agent TEXT,
+ source TEXT, -- 'public_form' | 'partner' | 'organic' | 'paid'
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+CREATE INDEX IF NOT EXISTS idx_leads_status ON leads(status);
+CREATE INDEX IF NOT EXISTS idx_leads_practice_area ON leads(practice_area);
+CREATE INDEX IF NOT EXISTS idx_leads_created ON leads(created_at DESC);
+CREATE INDEX IF NOT EXISTS idx_leads_zip ON leads(zip);
+
+CREATE OR REPLACE TRIGGER trg_leads_updated
+ BEFORE UPDATE ON leads
+ FOR EACH ROW EXECUTE FUNCTION set_updated_at();
diff --git a/migrations/006_app_users_profile.sql b/migrations/006_app_users_profile.sql
new file mode 100644
index 0000000..71fac8b
--- /dev/null
+++ b/migrations/006_app_users_profile.sql
@@ -0,0 +1,11 @@
+-- Add the 5 signup-form fields directly on app_users.
+-- (We could derive firm_name/website from organizations via organization_id,
+-- but storing the user-entered values keeps the auth flow simple and lets the
+-- user override the directory data if it's stale.)
+
+ALTER TABLE app_users
+ ADD COLUMN IF NOT EXISTS firm_name TEXT,
+ ADD COLUMN IF NOT EXISTS website TEXT,
+ ADD COLUMN IF NOT EXISTS phone TEXT;
+
+CREATE INDEX IF NOT EXISTS idx_app_users_firm_name ON app_users(LOWER(firm_name));
diff --git a/migrations/007_upgrade_orders.sql b/migrations/007_upgrade_orders.sql
new file mode 100644
index 0000000..ba94259
--- /dev/null
+++ b/migrations/007_upgrade_orders.sql
@@ -0,0 +1,39 @@
+-- EZ Upgrade order intent. Captures "yes I want to pay $499" before Stripe is
+-- wired so admin can hand-process payment links until automation lands.
+
+CREATE TABLE IF NOT EXISTS upgrade_orders (
+ id BIGSERIAL PRIMARY KEY,
+ app_user_id BIGINT REFERENCES app_users(id) ON DELETE SET NULL,
+ professional_id BIGINT REFERENCES professionals(id) ON DELETE SET NULL,
+ organization_id BIGINT REFERENCES organizations(id) ON DELETE SET NULL,
+
+ full_name TEXT NOT NULL,
+ email TEXT NOT NULL,
+ phone TEXT,
+ bar_number TEXT,
+ firm_name TEXT,
+ website TEXT,
+
+ plan TEXT NOT NULL DEFAULT 'ez_upgrade_499',
+ amount_cents INTEGER NOT NULL DEFAULT 49900,
+
+ status TEXT NOT NULL DEFAULT 'pending_payment'
+ CHECK (status IN ('pending_payment','paid','in_production','delivered','refunded','cancelled')),
+ payment_link TEXT,
+ paid_at TIMESTAMPTZ,
+ delivered_at TIMESTAMPTZ,
+ notes TEXT,
+
+ ip TEXT,
+ user_agent TEXT,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+CREATE INDEX IF NOT EXISTS idx_upgrade_orders_status ON upgrade_orders(status);
+CREATE INDEX IF NOT EXISTS idx_upgrade_orders_email ON upgrade_orders(LOWER(email));
+CREATE INDEX IF NOT EXISTS idx_upgrade_orders_created ON upgrade_orders(created_at DESC);
+
+CREATE OR REPLACE TRIGGER trg_upgrade_orders_updated
+ BEFORE UPDATE ON upgrade_orders
+ FOR EACH ROW EXECUTE FUNCTION set_updated_at();
diff --git a/migrations/008_prof_loc_unique.sql b/migrations/008_prof_loc_unique.sql
new file mode 100644
index 0000000..d6562a5
--- /dev/null
+++ b/migrations/008_prof_loc_unique.sql
@@ -0,0 +1,23 @@
+-- professional_locations had no unique constraint, so ON CONFLICT DO NOTHING
+-- in calbar_full_scrape.ts was a no-op — every resume of the scrape created
+-- duplicate primary firm links. Dedupe existing rows then add the partial
+-- unique index the importer can target.
+--
+-- PG 14-compatible: uses a partial index instead of NULLS NOT DISTINCT.
+-- The importer only inserts when organization_id is non-null, so the partial
+-- predicate matches the upsert site exactly.
+
+-- 1. Collapse duplicates: keep the lowest-id row per (professional_id, organization_id).
+DELETE FROM professional_locations a
+USING professional_locations b
+WHERE a.id > b.id
+ AND a.professional_id = b.professional_id
+ AND a.organization_id IS NOT NULL
+ AND b.organization_id IS NOT NULL
+ AND a.organization_id = b.organization_id;
+
+-- 2. Partial unique index — supports ON CONFLICT (professional_id, organization_id)
+-- WHERE organization_id IS NOT NULL.
+CREATE UNIQUE INDEX IF NOT EXISTS prof_loc_unique_pro_org
+ ON professional_locations (professional_id, organization_id)
+ WHERE organization_id IS NOT NULL;
diff --git a/migrations/009_site_mockups.sql b/migrations/009_site_mockups.sql
new file mode 100644
index 0000000..e9c13c4
--- /dev/null
+++ b/migrations/009_site_mockups.sql
@@ -0,0 +1,14 @@
+-- One row per (firm, variant) — stores the path of the rendered mockup PNG
+-- and the variant label/template id. Variants A/B/C are deterministic templates.
+
+CREATE TABLE IF NOT EXISTS site_mockups (
+ id BIGSERIAL PRIMARY KEY,
+ organization_id BIGINT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ variant TEXT NOT NULL, -- 'a' | 'b' | 'c'
+ template_label TEXT NOT NULL, -- 'Editorial Noir', 'Clean Modern Light', 'Bold Confident'
+ screenshot_path TEXT NOT NULL, -- /mockups/firm-{id}-{variant}.png
+ generated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ UNIQUE (organization_id, variant)
+);
+
+CREATE INDEX IF NOT EXISTS idx_site_mockups_org ON site_mockups (organization_id);
diff --git a/migrations/010_data_orders.sql b/migrations/010_data_orders.sql
new file mode 100644
index 0000000..e1894a1
--- /dev/null
+++ b/migrations/010_data_orders.sql
@@ -0,0 +1,51 @@
+-- Data marketplace: customers buy lists of California-licensed attorneys filtered
+-- by city. Two purchase kinds:
+-- 'one_time' — pay once, single CSV download
+-- 'subscription'— recurring monthly fee, unlimited downloads while active
+--
+-- Distinct from upgrade_orders (lawyer-facing site rebuild). Different kind of
+-- buyer, different SKU, different reporting.
+
+CREATE TABLE IF NOT EXISTS data_orders (
+ id BIGSERIAL PRIMARY KEY,
+ kind TEXT NOT NULL CHECK (kind IN ('one_time','subscription')),
+
+ -- buyer
+ email TEXT NOT NULL,
+ full_name TEXT,
+ company TEXT,
+ phone TEXT,
+
+ -- product spec
+ cities TEXT[] NOT NULL DEFAULT '{}'::TEXT[],
+ attorney_count INT, -- snapshot at order time
+ amount_cents INT NOT NULL,
+
+ -- status
+ status TEXT NOT NULL DEFAULT 'pending_payment'
+ CHECK (status IN ('pending_payment','paid','refunded','cancelled','sub_cancelled')),
+
+ -- stripe
+ stripe_session_id TEXT,
+ stripe_customer_id TEXT,
+ stripe_subscription_id TEXT,
+ payment_link TEXT,
+
+ -- delivery
+ download_token TEXT NOT NULL UNIQUE, -- random 32-byte hex
+ downloads_count INT NOT NULL DEFAULT 0,
+ last_downloaded_at TIMESTAMPTZ,
+
+ -- timestamps
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ paid_at TIMESTAMPTZ,
+ cancelled_at TIMESTAMPTZ,
+
+ -- audit
+ ip TEXT,
+ user_agent TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_data_orders_email ON data_orders (LOWER(email));
+CREATE INDEX IF NOT EXISTS idx_data_orders_status ON data_orders (status);
+CREATE INDEX IF NOT EXISTS idx_data_orders_created ON data_orders (created_at DESC);
diff --git a/migrations/011_contact_crawled_at.sql b/migrations/011_contact_crawled_at.sql
new file mode 100644
index 0000000..788f86b
--- /dev/null
+++ b/migrations/011_contact_crawled_at.sql
@@ -0,0 +1,7 @@
+-- Track per-firm contact-page crawl attempts so reruns can skip already-attempted firms.
+ALTER TABLE organizations
+ ADD COLUMN IF NOT EXISTS contact_crawled_at TIMESTAMPTZ;
+
+CREATE INDEX IF NOT EXISTS idx_orgs_contact_crawled_at
+ ON organizations (contact_crawled_at)
+ WHERE type = 'law_firm' AND website ~ '^https?://';
diff --git a/migrations/012_site_mockups_extras.sql b/migrations/012_site_mockups_extras.sql
new file mode 100644
index 0000000..d2747d5
--- /dev/null
+++ b/migrations/012_site_mockups_extras.sql
@@ -0,0 +1,5 @@
+-- Bring site_mockups in line with what render_mockups.ts persists.
+ALTER TABLE site_mockups
+ ADD COLUMN IF NOT EXISTS html_path TEXT,
+ ADD COLUMN IF NOT EXISTS llm_model TEXT,
+ ADD COLUMN IF NOT EXISTS gen_ms INTEGER;
diff --git a/migrations/013_user_tiers.sql b/migrations/013_user_tiers.sql
new file mode 100644
index 0000000..7a3d7ed
--- /dev/null
+++ b/migrations/013_user_tiers.sql
@@ -0,0 +1,24 @@
+-- Four-tier auth: guest / client / lawyer / admin
+-- Adds Google OAuth fields, bar-number claim flow, Stripe subscription fields,
+-- and per-user comment-toggle.
+
+ALTER TABLE app_users
+ ADD COLUMN IF NOT EXISTS tier TEXT NOT NULL DEFAULT 'client'
+ CHECK (tier IN ('guest','client','lawyer','admin')),
+ ADD COLUMN IF NOT EXISTS google_sub TEXT,
+ ADD COLUMN IF NOT EXISTS google_email TEXT,
+ ADD COLUMN IF NOT EXISTS google_picture TEXT,
+ ADD COLUMN IF NOT EXISTS claimed_professional_id BIGINT REFERENCES professionals(id) ON DELETE SET NULL,
+ ADD COLUMN IF NOT EXISTS bar_number TEXT,
+ ADD COLUMN IF NOT EXISTS bar_verified_at TIMESTAMPTZ,
+ ADD COLUMN IF NOT EXISTS paid_until TIMESTAMPTZ,
+ ADD COLUMN IF NOT EXISTS stripe_customer_id TEXT,
+ ADD COLUMN IF NOT EXISTS stripe_subscription_id TEXT,
+ ADD COLUMN IF NOT EXISTS comments_enabled BOOLEAN NOT NULL DEFAULT true;
+
+-- Unique only when set (allow multiple rows with NULL google_sub)
+CREATE UNIQUE INDEX IF NOT EXISTS uq_app_users_google_sub
+ ON app_users (google_sub) WHERE google_sub IS NOT NULL;
+
+CREATE INDEX IF NOT EXISTS idx_app_users_tier ON app_users (tier);
+CREATE INDEX IF NOT EXISTS idx_app_users_claimed_pro ON app_users (claimed_professional_id);
diff --git a/migrations/014_ratings.sql b/migrations/014_ratings.sql
new file mode 100644
index 0000000..e8fff7f
--- /dev/null
+++ b/migrations/014_ratings.sql
@@ -0,0 +1,45 @@
+-- Star ratings + dimension scores (service / price / quality) for firms and attorneys.
+-- Sources: user-submitted, Avvo, Yelp, Google, Reddit (future), manual.
+
+CREATE TABLE IF NOT EXISTS ratings (
+ id BIGSERIAL PRIMARY KEY,
+ professional_id BIGINT REFERENCES professionals(id) ON DELETE CASCADE,
+ organization_id BIGINT REFERENCES organizations(id) ON DELETE CASCADE,
+ reviewer_user_id BIGINT REFERENCES app_users(id) ON DELETE SET NULL,
+ source TEXT NOT NULL CHECK (source IN ('user','avvo','google','yelp','reddit','manual')),
+ source_url TEXT,
+ external_review_id TEXT, -- e.g. Yelp review id, for dedup
+ stars NUMERIC(2,1) NOT NULL CHECK (stars BETWEEN 0 AND 5),
+ service_score NUMERIC(2,1),
+ price_score NUMERIC(2,1),
+ quality_score NUMERIC(2,1),
+ comment TEXT,
+ reviewer_name TEXT, -- for external sources where we only have a display name
+ posted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ ingested_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ hidden_at TIMESTAMPTZ, -- moderation soft-delete
+ hidden_reason TEXT,
+ CONSTRAINT rating_target_check
+ CHECK (professional_id IS NOT NULL OR organization_id IS NOT NULL)
+);
+CREATE INDEX IF NOT EXISTS idx_ratings_pro ON ratings (professional_id) WHERE professional_id IS NOT NULL;
+CREATE INDEX IF NOT EXISTS idx_ratings_org ON ratings (organization_id) WHERE organization_id IS NOT NULL;
+CREATE INDEX IF NOT EXISTS idx_ratings_source ON ratings (source);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_ratings_external
+ ON ratings (source, external_review_id) WHERE external_review_id IS NOT NULL;
+
+-- Aggregate cache (rebuilt from triggers / cron — keeps firm/attorney pages fast)
+CREATE TABLE IF NOT EXISTS rating_aggregates (
+ target_kind TEXT NOT NULL CHECK (target_kind IN ('professional','organization')),
+ target_id BIGINT NOT NULL,
+ total_reviews INT NOT NULL DEFAULT 0,
+ user_review_count INT NOT NULL DEFAULT 0,
+ avg_stars NUMERIC(3,2),
+ avg_service NUMERIC(3,2),
+ avg_price NUMERIC(3,2),
+ avg_quality NUMERIC(3,2),
+ overall_score NUMERIC(4,2), -- weighted formula from spec
+ by_source JSONB, -- {avvo:{count,avg}, yelp:{...}, google:{...}, user:{...}}
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (target_kind, target_id)
+);
diff --git a/migrations/015_community.sql b/migrations/015_community.sql
new file mode 100644
index 0000000..c46dfe0
--- /dev/null
+++ b/migrations/015_community.sql
@@ -0,0 +1,60 @@
+-- Community + DM threads. Channels: client_public / lawyer_public /
+-- dm_client_lawyer / dm_client_client / dm_lawyer_lawyer.
+
+CREATE TABLE IF NOT EXISTS threads (
+ id BIGSERIAL PRIMARY KEY,
+ channel TEXT NOT NULL CHECK (channel IN (
+ 'client_public','lawyer_public',
+ 'dm_client_lawyer','dm_client_client','dm_lawyer_lawyer'
+ )),
+ topic TEXT,
+ organization_id BIGINT REFERENCES organizations(id) ON DELETE SET NULL,
+ professional_id BIGINT REFERENCES professionals(id) ON DELETE SET NULL,
+ participant_user_ids BIGINT[] NOT NULL DEFAULT '{}',
+ created_by BIGINT REFERENCES app_users(id) ON DELETE SET NULL,
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ last_message_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ message_count INT NOT NULL DEFAULT 0,
+ closed_at TIMESTAMPTZ
+);
+CREATE INDEX IF NOT EXISTS idx_threads_channel_recent ON threads (channel, last_message_at DESC);
+CREATE INDEX IF NOT EXISTS idx_threads_participants ON threads USING GIN (participant_user_ids);
+CREATE INDEX IF NOT EXISTS idx_threads_org ON threads (organization_id) WHERE organization_id IS NOT NULL;
+
+CREATE TABLE IF NOT EXISTS messages (
+ id BIGSERIAL PRIMARY KEY,
+ thread_id BIGINT NOT NULL REFERENCES threads(id) ON DELETE CASCADE,
+ user_id BIGINT NOT NULL REFERENCES app_users(id) ON DELETE SET NULL,
+ body TEXT NOT NULL,
+ posted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ edited_at TIMESTAMPTZ,
+ hidden_at TIMESTAMPTZ,
+ hidden_reason TEXT
+);
+CREATE INDEX IF NOT EXISTS idx_messages_thread ON messages (thread_id, posted_at);
+CREATE INDEX IF NOT EXISTS idx_messages_user ON messages (user_id, posted_at DESC);
+
+-- Reads/seen tracker so we can show unread counts in UI.
+CREATE TABLE IF NOT EXISTS thread_reads (
+ thread_id BIGINT NOT NULL REFERENCES threads(id) ON DELETE CASCADE,
+ user_id BIGINT NOT NULL REFERENCES app_users(id) ON DELETE CASCADE,
+ last_read_message_id BIGINT REFERENCES messages(id) ON DELETE SET NULL,
+ last_read_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ PRIMARY KEY (thread_id, user_id)
+);
+
+-- Bump thread metadata on each message insert
+CREATE OR REPLACE FUNCTION bump_thread_on_message() RETURNS TRIGGER AS $$
+BEGIN
+ UPDATE threads
+ SET last_message_at = NEW.posted_at,
+ message_count = message_count + 1
+ WHERE id = NEW.thread_id;
+ RETURN NEW;
+END;
+$$ LANGUAGE plpgsql;
+
+DROP TRIGGER IF EXISTS trg_bump_thread_on_message ON messages;
+CREATE TRIGGER trg_bump_thread_on_message
+ AFTER INSERT ON messages
+ FOR EACH ROW EXECUTE FUNCTION bump_thread_on_message();
diff --git a/migrations/016_replies_and_views.sql b/migrations/016_replies_and_views.sql
new file mode 100644
index 0000000..6607515
--- /dev/null
+++ b/migrations/016_replies_and_views.sql
@@ -0,0 +1,25 @@
+-- Lawyer can post a single reply per rating (their answer/correction).
+CREATE TABLE IF NOT EXISTS rating_replies (
+ id BIGSERIAL PRIMARY KEY,
+ rating_id BIGINT NOT NULL REFERENCES ratings(id) ON DELETE CASCADE,
+ responder_user_id BIGINT NOT NULL REFERENCES app_users(id) ON DELETE CASCADE,
+ body TEXT NOT NULL,
+ posted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ edited_at TIMESTAMPTZ,
+ hidden_at TIMESTAMPTZ,
+ UNIQUE (rating_id, responder_user_id) -- one reply per (rating, user)
+);
+CREATE INDEX IF NOT EXISTS idx_rating_replies_rating ON rating_replies (rating_id);
+
+-- Pitch-page open tracking — every visit to /p/:firm_id
+CREATE TABLE IF NOT EXISTS pitch_views (
+ id BIGSERIAL PRIMARY KEY,
+ organization_id BIGINT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
+ visitor_ip TEXT,
+ visitor_ua TEXT,
+ referrer TEXT,
+ utm_source TEXT, -- ?utm_source= for tracking which campaign drove it
+ viewed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+CREATE INDEX IF NOT EXISTS idx_pitch_views_org ON pitch_views (organization_id, viewed_at DESC);
+CREATE INDEX IF NOT EXISTS idx_pitch_views_recent ON pitch_views (viewed_at DESC);
diff --git a/migrations/017_fix_message_user_fk.sql b/migrations/017_fix_message_user_fk.sql
new file mode 100644
index 0000000..d3bdf93
--- /dev/null
+++ b/migrations/017_fix_message_user_fk.sql
@@ -0,0 +1,4 @@
+-- migration 015 declared messages.user_id as NOT NULL with ON DELETE SET NULL,
+-- which contradicts and prevents app_users hard-delete. Make it nullable so the
+-- SET NULL behavior actually fires when a user is removed.
+ALTER TABLE messages ALTER COLUMN user_id DROP NOT NULL;
diff --git a/migrations/018_site_intel.sql b/migrations/018_site_intel.sql
new file mode 100644
index 0000000..c3474c3
--- /dev/null
+++ b/migrations/018_site_intel.sql
@@ -0,0 +1,12 @@
+-- Cache real-site content extracted at mockup-gen time, so re-renders skip the fetch.
+ALTER TABLE organizations
+ ADD COLUMN IF NOT EXISTS site_intel_title TEXT,
+ ADD COLUMN IF NOT EXISTS site_intel_h1 TEXT,
+ ADD COLUMN IF NOT EXISTS site_intel_meta_desc TEXT,
+ ADD COLUMN IF NOT EXISTS site_intel_body TEXT,
+ ADD COLUMN IF NOT EXISTS site_intel_practice TEXT[],
+ ADD COLUMN IF NOT EXISTS site_intel_attorneys TEXT[],
+ ADD COLUMN IF NOT EXISTS site_intel_at TIMESTAMPTZ;
+
+CREATE INDEX IF NOT EXISTS idx_org_site_intel_pending ON organizations (id)
+ WHERE type='law_firm' AND website IS NOT NULL AND site_intel_at IS NULL;
diff --git a/package-lock.json b/package-lock.json
index c3040a2..1e4d8d6 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -9,12 +9,17 @@
"version": "0.1.0",
"license": "UNLICENSED",
"dependencies": {
+ "bcryptjs": "^3.0.3",
"cheerio": "^1.0.0",
+ "cookie": "^1.1.1",
"csv-stringify": "^6.5.2",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"pg": "^8.13.1",
+ "playwright": "^1.59.1",
"robots-parser": "^3.0.1",
+ "sharp": "^0.34.5",
+ "stripe": "^22.1.0",
"undici": "^7.2.0"
},
"devDependencies": {
@@ -25,6 +30,16 @@
"typescript": "^5.7.3"
}
},
+ "node_modules/@emnapi/runtime": {
+ "version": "1.10.0",
+ "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz",
+ "integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==",
+ "license": "MIT",
+ "optional": true,
+ "dependencies": {
+ "tslib": "^2.4.0"
+ }
+ },
"node_modules/@esbuild/aix-ppc64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.27.7.tgz",
@@ -467,6 +482,471 @@
"node": ">=18"
}
},
+ "node_modules/@img/colour": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/@img/colour/-/colour-1.1.0.tgz",
+ "integrity": "sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/@img/sharp-darwin-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.34.5.tgz",
+ "integrity": "sha512-imtQ3WMJXbMY4fxb/Ndp6HBTNVtWCUI0WdobyheGf5+ad6xX8VIDO8u2xE4qc/fr08CKG/7dDseFtn6M6g/r3w==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-darwin-arm64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-darwin-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.34.5.tgz",
+ "integrity": "sha512-YNEFAF/4KQ/PeW0N+r+aVVsoIY0/qxxikF2SWdp+NRkmMB7y9LBZAVqQ4yhGCm/H3H270OSykqmQMKLBhBJDEw==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-darwin-x64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-libvips-darwin-arm64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.2.4.tgz",
+ "integrity": "sha512-zqjjo7RatFfFoP0MkQ51jfuFZBnVE2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-darwin-x64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.2.4.tgz",
+ "integrity": "sha512-1IOd5xfVhlGwX+zXv2N93k0yMONvUlANylbJw1eTah8K/Jtpi15KC+WSiaX/nBmbm2HxRM1gZ0nSdjSsrZbGKg==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-arm": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.2.4.tgz",
+ "integrity": "sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==",
+ "cpu": [
+ "arm"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-arm64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.2.4.tgz",
+ "integrity": "sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-ppc64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-ppc64/-/sharp-libvips-linux-ppc64-1.2.4.tgz",
+ "integrity": "sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==",
+ "cpu": [
+ "ppc64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-riscv64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-riscv64/-/sharp-libvips-linux-riscv64-1.2.4.tgz",
+ "integrity": "sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==",
+ "cpu": [
+ "riscv64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-s390x": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.2.4.tgz",
+ "integrity": "sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==",
+ "cpu": [
+ "s390x"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linux-x64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.2.4.tgz",
+ "integrity": "sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linuxmusl-arm64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.2.4.tgz",
+ "integrity": "sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-libvips-linuxmusl-x64": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.2.4.tgz",
+ "integrity": "sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-linux-arm": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm/-/sharp-linux-arm-0.34.5.tgz",
+ "integrity": "sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==",
+ "cpu": [
+ "arm"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-arm": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.34.5.tgz",
+ "integrity": "sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-arm64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-ppc64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-ppc64/-/sharp-linux-ppc64-0.34.5.tgz",
+ "integrity": "sha512-7zznwNaqW6YtsfrGGDA6BRkISKAAE1Jo0QdpNYXNMHu2+0dTrPflTLNkpc8l7MUP5M16ZJcUvysVWWrMefZquA==",
+ "cpu": [
+ "ppc64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-ppc64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-riscv64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-riscv64/-/sharp-linux-riscv64-0.34.5.tgz",
+ "integrity": "sha512-51gJuLPTKa7piYPaVs8GmByo7/U7/7TZOq+cnXJIHZKavIRHAP77e3N2HEl3dgiqdD/w0yUfiJnII77PuDDFdw==",
+ "cpu": [
+ "riscv64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-riscv64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-s390x": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.34.5.tgz",
+ "integrity": "sha512-nQtCk0PdKfho3eC5MrbQoigJ2gd1CgddUMkabUj+rBevs8tZ2cULOx46E7oyX+04WGfABgIwmMC0VqieTiR4jg==",
+ "cpu": [
+ "s390x"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-s390x": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linux-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linux-x64/-/sharp-linux-x64-0.34.5.tgz",
+ "integrity": "sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linux-x64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linuxmusl-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.34.5.tgz",
+ "integrity": "sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linuxmusl-arm64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-linuxmusl-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.34.5.tgz",
+ "integrity": "sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0",
+ "optional": true,
+ "os": [
+ "linux"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-libvips-linuxmusl-x64": "1.2.4"
+ }
+ },
+ "node_modules/@img/sharp-wasm32": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-wasm32/-/sharp-wasm32-0.34.5.tgz",
+ "integrity": "sha512-OdWTEiVkY2PHwqkbBI8frFxQQFekHaSSkUIJkwzclWZe64O1X4UlUjqqqLaPbUpMOQk6FBu/HtlGXNblIs0huw==",
+ "cpu": [
+ "wasm32"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later AND MIT",
+ "optional": true,
+ "dependencies": {
+ "@emnapi/runtime": "^1.7.0"
+ },
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-win32-arm64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-win32-arm64/-/sharp-win32-arm64-0.34.5.tgz",
+ "integrity": "sha512-WQ3AgWCWYSb2yt+IG8mnC6Jdk9Whs7O0gxphblsLvdhSpSTtmu69ZG1Gkb6NuvxsNACwiPV6cNSZNzt0KPsw7g==",
+ "cpu": [
+ "arm64"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-win32-ia32": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.34.5.tgz",
+ "integrity": "sha512-FV9m/7NmeCmSHDD5j4+4pNI8Cp3aW+JvLoXcTUo0IqyjSfAZJ8dIUmijx1qaJsIiU+Hosw6xM5KijAWRJCSgNg==",
+ "cpu": [
+ "ia32"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
+ "node_modules/@img/sharp-win32-x64": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/@img/sharp-win32-x64/-/sharp-win32-x64-0.34.5.tgz",
+ "integrity": "sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw==",
+ "cpu": [
+ "x64"
+ ],
+ "license": "Apache-2.0 AND LGPL-3.0-or-later",
+ "optional": true,
+ "os": [
+ "win32"
+ ],
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ }
+ },
"node_modules/@types/body-parser": {
"version": "1.19.6",
"resolved": "https://registry.npmjs.org/@types/body-parser/-/body-parser-1.19.6.tgz",
@@ -524,7 +1004,7 @@
"version": "22.19.17",
"resolved": "https://registry.npmjs.org/@types/node/-/node-22.19.17.tgz",
"integrity": "sha512-wGdMcf+vPYM6jikpS/qhg6WiqSV/OhG+jeeHT/KlVqxYfD40iYJf9/AE1uQxVWFvU7MipKRkRv8NSHiCGgPr8Q==",
- "dev": true,
+ "devOptional": true,
"license": "MIT",
"dependencies": {
"undici-types": "~6.21.0"
@@ -596,6 +1076,15 @@
"integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==",
"license": "MIT"
},
+ "node_modules/bcryptjs": {
+ "version": "3.0.3",
+ "resolved": "https://registry.npmjs.org/bcryptjs/-/bcryptjs-3.0.3.tgz",
+ "integrity": "sha512-GlF5wPWnSa/X5LKM1o0wz0suXIINz1iHRLvTS+sLyi7XPbe5ycmYI3DlZqVGZZtDgl4DmasFg7gOB3JYbphV5g==",
+ "license": "BSD-3-Clause",
+ "bin": {
+ "bcrypt": "bin/bcrypt"
+ }
+ },
"node_modules/body-parser": {
"version": "1.20.5",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.5.tgz",
@@ -755,12 +1244,16 @@
}
},
"node_modules/cookie": {
- "version": "0.7.2",
- "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz",
- "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==",
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/cookie/-/cookie-1.1.1.tgz",
+ "integrity": "sha512-ei8Aos7ja0weRpFzJnEA9UHJ/7XQmqglbRwnf2ATjcB9Wq874VKH9kfjjirM6UhU2/E5fFYadylyhFldcqSidQ==",
"license": "MIT",
"engines": {
- "node": ">= 0.6"
+ "node": ">=18"
+ },
+ "funding": {
+ "type": "opencollective",
+ "url": "https://opencollective.com/express"
}
},
"node_modules/cookie-signature": {
@@ -831,6 +1324,15 @@
"npm": "1.2.8000 || >= 1.4.16"
}
},
+ "node_modules/detect-libc": {
+ "version": "2.1.2",
+ "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
+ "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
+ "license": "Apache-2.0",
+ "engines": {
+ "node": ">=8"
+ }
+ },
"node_modules/dom-serializer": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/dom-serializer/-/dom-serializer-2.0.0.tgz",
@@ -1085,6 +1587,15 @@
"url": "https://opencollective.com/express"
}
},
+ "node_modules/express/node_modules/cookie": {
+ "version": "0.7.2",
+ "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz",
+ "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==",
+ "license": "MIT",
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
"node_modules/finalhandler": {
"version": "1.3.2",
"resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.2.tgz",
@@ -1582,6 +2093,50 @@
"split2": "^4.1.0"
}
},
+ "node_modules/playwright": {
+ "version": "1.59.1",
+ "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.59.1.tgz",
+ "integrity": "sha512-C8oWjPR3F81yljW9o5OxcWzfh6avkVwDD2VYdwIGqTkl+OGFISgypqzfu7dOe4QNLL2aqcWBmI3PMtLIK233lw==",
+ "license": "Apache-2.0",
+ "dependencies": {
+ "playwright-core": "1.59.1"
+ },
+ "bin": {
+ "playwright": "cli.js"
+ },
+ "engines": {
+ "node": ">=18"
+ },
+ "optionalDependencies": {
+ "fsevents": "2.3.2"
+ }
+ },
+ "node_modules/playwright-core": {
+ "version": "1.59.1",
+ "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.59.1.tgz",
+ "integrity": "sha512-HBV/RJg81z5BiiZ9yPzIiClYV/QMsDCKUyogwH9p3MCP6IYjUFu/MActgYAvK0oWyV9NlwM3GLBjADyWgydVyg==",
+ "license": "Apache-2.0",
+ "bin": {
+ "playwright-core": "cli.js"
+ },
+ "engines": {
+ "node": ">=18"
+ }
+ },
+ "node_modules/playwright/node_modules/fsevents": {
+ "version": "2.3.2",
+ "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz",
+ "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
+ "hasInstallScript": true,
+ "license": "MIT",
+ "optional": true,
+ "os": [
+ "darwin"
+ ],
+ "engines": {
+ "node": "^8.16.0 || ^10.6.0 || >=11.0.0"
+ }
+ },
"node_modules/postgres-array": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
@@ -1730,6 +2285,18 @@
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",
"license": "MIT"
},
+ "node_modules/semver": {
+ "version": "7.7.4",
+ "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz",
+ "integrity": "sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA==",
+ "license": "ISC",
+ "bin": {
+ "semver": "bin/semver.js"
+ },
+ "engines": {
+ "node": ">=10"
+ }
+ },
"node_modules/send": {
"version": "0.19.2",
"resolved": "https://registry.npmjs.org/send/-/send-0.19.2.tgz",
@@ -1781,6 +2348,50 @@
"integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==",
"license": "ISC"
},
+ "node_modules/sharp": {
+ "version": "0.34.5",
+ "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.34.5.tgz",
+ "integrity": "sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg==",
+ "hasInstallScript": true,
+ "license": "Apache-2.0",
+ "dependencies": {
+ "@img/colour": "^1.0.0",
+ "detect-libc": "^2.1.2",
+ "semver": "^7.7.3"
+ },
+ "engines": {
+ "node": "^18.17.0 || ^20.3.0 || >=21.0.0"
+ },
+ "funding": {
+ "url": "https://opencollective.com/libvips"
+ },
+ "optionalDependencies": {
+ "@img/sharp-darwin-arm64": "0.34.5",
+ "@img/sharp-darwin-x64": "0.34.5",
+ "@img/sharp-libvips-darwin-arm64": "1.2.4",
+ "@img/sharp-libvips-darwin-x64": "1.2.4",
+ "@img/sharp-libvips-linux-arm": "1.2.4",
+ "@img/sharp-libvips-linux-arm64": "1.2.4",
+ "@img/sharp-libvips-linux-ppc64": "1.2.4",
+ "@img/sharp-libvips-linux-riscv64": "1.2.4",
+ "@img/sharp-libvips-linux-s390x": "1.2.4",
+ "@img/sharp-libvips-linux-x64": "1.2.4",
+ "@img/sharp-libvips-linuxmusl-arm64": "1.2.4",
+ "@img/sharp-libvips-linuxmusl-x64": "1.2.4",
+ "@img/sharp-linux-arm": "0.34.5",
+ "@img/sharp-linux-arm64": "0.34.5",
+ "@img/sharp-linux-ppc64": "0.34.5",
+ "@img/sharp-linux-riscv64": "0.34.5",
+ "@img/sharp-linux-s390x": "0.34.5",
+ "@img/sharp-linux-x64": "0.34.5",
+ "@img/sharp-linuxmusl-arm64": "0.34.5",
+ "@img/sharp-linuxmusl-x64": "0.34.5",
+ "@img/sharp-wasm32": "0.34.5",
+ "@img/sharp-win32-arm64": "0.34.5",
+ "@img/sharp-win32-ia32": "0.34.5",
+ "@img/sharp-win32-x64": "0.34.5"
+ }
+ },
"node_modules/side-channel": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz",
@@ -1871,6 +2482,23 @@
"node": ">= 0.8"
}
},
+ "node_modules/stripe": {
+ "version": "22.1.0",
+ "resolved": "https://registry.npmjs.org/stripe/-/stripe-22.1.0.tgz",
+ "integrity": "sha512-w/xHyJGxXWnLPbNHG13sz/fae0MrFGC80Oz7YbICQymbfpqfEcsoG+6yG+9BWb81PWc4rrkeSO4wmTcmefmbLw==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=18"
+ },
+ "peerDependencies": {
+ "@types/node": ">=18"
+ },
+ "peerDependenciesMeta": {
+ "@types/node": {
+ "optional": true
+ }
+ }
+ },
"node_modules/toidentifier": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",
@@ -1880,6 +2508,13 @@
"node": ">=0.6"
}
},
+ "node_modules/tslib": {
+ "version": "2.8.1",
+ "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
+ "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
+ "license": "0BSD",
+ "optional": true
+ },
"node_modules/tsx": {
"version": "4.21.0",
"resolved": "https://registry.npmjs.org/tsx/-/tsx-4.21.0.tgz",
@@ -1940,7 +2575,7 @@
"version": "6.21.0",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz",
"integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==",
- "dev": true,
+ "devOptional": true,
"license": "MIT"
},
"node_modules/unpipe": {
diff --git a/package.json b/package.json
index 51516d7..21e4ec5 100644
--- a/package.json
+++ b/package.json
@@ -13,15 +13,25 @@
"export:csv": "tsx src/scripts/export_csv.ts",
"server": "tsx src/server/index.ts",
"dev": "tsx watch src/server/index.ts",
- "stats": "tsx src/scripts/stats.ts"
+ "stats": "tsx src/scripts/stats.ts",
+ "overnight": "tsx src/scripts/run_overnight.ts",
+ "geocode": "tsx src/enrich/geocode_nominatim.ts",
+ "discover": "tsx src/enrich/discover_websites.ts",
+ "enrich:contacts": "tsx src/enrich/firm_website_contacts.ts",
+ "create-admin": "tsx src/scripts/create_admin.ts"
},
"dependencies": {
+ "bcryptjs": "^3.0.3",
"cheerio": "^1.0.0",
+ "cookie": "^1.1.1",
"csv-stringify": "^6.5.2",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"pg": "^8.13.1",
+ "playwright": "^1.59.1",
"robots-parser": "^3.0.1",
+ "sharp": "^0.34.5",
+ "stripe": "^22.1.0",
"undici": "^7.2.0"
},
"devDependencies": {
diff --git a/public/dashboard.html b/public/dashboard.html
new file mode 100644
index 0000000..bc4dd59
--- /dev/null
+++ b/public/dashboard.html
@@ -0,0 +1,553 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width,initial-scale=1">
+<title>Lawyer Directory · dashboard</title>
+<link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css" crossorigin="">
+<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js" crossorigin=""></script>
+<style>
+ :root {
+ --bg:#0a0a0c; --panel:#131316; --panel-2:#1a1a1f; --border:#2a2724;
+ --fg:#f4f1ea; --fg-soft:#d8d2c5; --muted:#8b857a;
+ --accent:#b89968; --accent-glow:#d4b683; --accent-deep:#8a7044;
+ --good:#10b981; --warn:#facc15; --bad:#ef4444; --info:#60a5fa;
+ }
+ *,*::before,*::after { box-sizing:border-box; }
+ body { margin:0; font:14px/1.5 -apple-system,BlinkMacSystemFont,"Segoe UI",system-ui,sans-serif; background:var(--bg); color:var(--fg); -webkit-font-smoothing:antialiased; }
+ a { color:var(--accent); text-decoration:none; }
+ a:hover { color:var(--accent-glow); }
+ button { font:inherit; cursor:pointer; }
+
+ header { padding:18px 28px; background:var(--panel); border-bottom:1px solid var(--border); display:flex; align-items:baseline; gap:24px; flex-wrap:wrap; }
+ h1 { margin:0; font-size:18px; font-weight:500; letter-spacing:.02em; }
+ h1 .accent { color:var(--accent); }
+ .stats { display:flex; gap:18px; flex-wrap:wrap; color:var(--muted); font-size:12px; }
+ .stats span { cursor:pointer; transition:color .15s; }
+ .stats span:hover { color:var(--fg); }
+ .stats b { color:var(--fg); font-weight:600; font-size:13px; }
+
+ .controls { padding:14px 28px; background:var(--panel); border-bottom:1px solid var(--border); display:flex; align-items:center; gap:14px; flex-wrap:wrap; }
+ .seg { display:inline-flex; background:var(--bg); border:1px solid var(--border); border-radius:8px; overflow:hidden; }
+ .seg button { background:transparent; color:var(--muted); border:0; padding:7px 14px; }
+ .seg button.active { background:var(--accent); color:var(--bg); font-weight:600; }
+ select, input[type=text] { background:var(--bg); color:var(--fg); border:1px solid var(--border); border-radius:6px; padding:7px 10px; font:inherit; }
+ .slider-group { display:flex; align-items:center; gap:8px; color:var(--muted); font-size:12px; }
+ .slider-group input[type=range] { accent-color:var(--accent); width:140px; }
+ .slider-group .val { color:var(--fg); font-variant-numeric:tabular-nums; min-width:24px; text-align:right; }
+ .pager { margin-left:auto; color:var(--muted); display:flex; align-items:center; gap:8px; }
+ .pager button { background:var(--bg); border:1px solid var(--border); color:var(--fg); border-radius:6px; padding:6px 10px; }
+ .pager button:disabled { opacity:.4; cursor:not-allowed; }
+ .pager .info { font-variant-numeric:tabular-nums; min-width:80px; text-align:center; }
+
+ /* GRID */
+ #results.grid { display:grid; gap:14px; padding:18px 28px; grid-template-columns:repeat(var(--cols,5),minmax(0,1fr)); }
+ .card { background:var(--panel); border:1px solid var(--border); border-radius:10px; padding:14px; display:flex; flex-direction:column; gap:6px; min-height:88px; }
+ .card .name { font-weight:600; font-size:14px; color:var(--fg); display:flex; justify-content:space-between; gap:8px; align-items:flex-start; }
+ .card .meta { color:var(--muted); font-size:12px; line-height:1.5; }
+ .card .meta a { color:var(--accent); }
+ .badge { display:inline-block; padding:1px 6px; border-radius:4px; background:#1c2237; color:var(--accent); font-size:10.5px; margin-right:4px; vertical-align:middle; }
+ .badge.size { background:#1c2237; color:var(--fg-soft); }
+ .badge.has { background:#173b2a; color:var(--good); }
+ .badge.needs { background:#3b3015; color:var(--warn); }
+ .badge.audit { color:var(--bg); font-weight:700; padding:2px 7px; }
+ .swatch { display:inline-block; width:12px; height:12px; border-radius:2px; vertical-align:middle; margin-right:4px; border:1px solid var(--border); }
+ .actions { display:flex; gap:10px; margin-top:6px; flex-wrap:wrap; font-size:12px; }
+ .actions button { background:none; border:0; color:var(--accent); padding:0; }
+ .actions button.muted { color:var(--muted); }
+ .actions button:hover { color:var(--accent-glow); }
+ .pitch-btn { background:var(--accent) !important; color:var(--bg) !important; padding:4px 10px !important; border-radius:4px !important; font-weight:600 !important; }
+ .pitch-btn:hover { background:var(--accent-glow) !important; }
+
+ /* LIST */
+ #results.list { padding:0 28px 28px; overflow-x:auto; }
+ #results.list table { width:100%; border-collapse:collapse; font-size:13px; }
+ #results.list th { text-align:left; padding:10px 12px; background:var(--panel); border-bottom:1px solid var(--border); position:sticky; top:0; font-weight:600; color:var(--muted); text-transform:uppercase; font-size:11px; letter-spacing:.5px; cursor:pointer; user-select:none; white-space:nowrap; }
+ #results.list th:hover { color:var(--fg); }
+ #results.list th .sort-arrow { color:var(--accent); margin-left:4px; }
+ #results.list td { padding:8px 12px; border-bottom:1px solid var(--border); white-space:nowrap; max-width:280px; overflow:hidden; text-overflow:ellipsis; }
+ #results.list tr:hover td { background:var(--panel); }
+
+ /* MAP */
+ #results.map { padding:0; height:calc(100vh - 180px); }
+ #results.map #leaflet { width:100%; height:100%; background:var(--panel-2); }
+ .leaflet-container { background:#0a0a0c !important; }
+ .leaflet-popup-content-wrapper { background:var(--panel) !important; color:var(--fg) !important; border:1px solid var(--border); }
+ .leaflet-popup-tip { background:var(--panel) !important; border:1px solid var(--border); }
+ .leaflet-popup-content { font:13px/1.45 system-ui; color:var(--fg); }
+ .leaflet-popup-content .pname { font-weight:600; font-size:14px; margin-bottom:4px; }
+
+ .empty { padding:40px 28px; color:var(--muted); text-align:center; }
+</style>
+</head>
+<body>
+
+<header>
+ <h1>Lawyer Directory <span class="accent">·</span> dashboard</h1>
+ <div class="stats" id="stats-bar">
+ <span data-filter="firms"> <b id="s-firms">…</b> firms</span>
+ <span data-filter="attorneys"> <b id="s-attorneys">…</b> attorneys</span>
+ <span data-filter="with_site"> <b id="s-with_site">…</b> with sites</span>
+ <span data-filter="with_phone"> <b id="s-with_phone">…</b> with phone</span>
+ <span data-filter="phones"> <b id="s-phones">…</b> phones</span>
+ <span data-filter="emails"> <b id="s-emails">…</b> emails</span>
+ <span data-filter="audits"> <b id="s-audits">…</b> audited</span>
+ <span data-filter="cities"> <b id="s-cities">…</b> cities</span>
+ <span style="margin-left:auto"><a href="/audit.html">/audit.html</a> · <a href="/directory">/directory</a> · <a href="/" >/marketing</a></span>
+ </div>
+</header>
+
+<div class="controls">
+ <div class="seg" id="entity-toggle">
+ <button data-entity="firms" class="active">Firms</button>
+ <button data-entity="attorneys">Attorneys</button>
+ </div>
+
+ <select id="prospect-select" title="Lead-quality filter (firms only)">
+ <option value="">Any prospect</option>
+ <option value="has_site">Has website</option>
+ <option value="needs_site">Needs website</option>
+ <option value="has_phone">Has phone</option>
+ <option value="audited">Audited</option>
+ <option value="high_score">High audit score (≥70)</option>
+ <option value="low_score">Low audit score (<50)</option>
+ </select>
+
+ <select id="size-select" title="Firm size band">
+ <option value="">Any size</option>
+ <option value="solo">Solo</option>
+ <option value="small">Small (2–10)</option>
+ <option value="mid">Mid (11–50)</option>
+ <option value="large">Large (51–200)</option>
+ <option value="biglaw">BigLaw (200+)</option>
+ </select>
+
+ <select id="sort-select" title="Sort">
+ <option value="score">Sort: lead score</option>
+ <option value="attorneys">Sort: # attorneys</option>
+ <option value="recent">Sort: recently crawled</option>
+ <option value="name">Sort: name (A–Z)</option>
+ <option value="zip">Sort: ZIP</option>
+ </select>
+
+ <div class="seg" id="view-toggle">
+ <button data-view="grid" class="active">Grid</button>
+ <button data-view="list">List</button>
+ <button data-view="map">Map</button>
+ </div>
+
+ <div class="slider-group" id="grid-controls">
+ <span>Per row</span>
+ <input type="range" min="3" max="10" value="5" id="grid-cols">
+ <span class="val" id="grid-cols-val">5</span>
+ </div>
+
+ <input type="text" id="filter-q" placeholder="Filter by name…" style="width:180px">
+
+ <div class="pager">
+ <button id="prev">← Prev</button>
+ <span class="info" id="pageinfo">0–0</span>
+ <button id="next">Next →</button>
+ </div>
+</div>
+
+<div id="results" class="grid"></div>
+
+<script>
+(() => {
+ const $ = (s) => document.querySelector(s);
+ const results = $('#results');
+ const gridColsEl = $('#grid-cols');
+ const gridColsVal = $('#grid-cols-val');
+ const gridControls = $('#grid-controls');
+ const filterQ = $('#filter-q');
+ const pageInfo = $('#pageinfo');
+ const prevBtn = $('#prev');
+ const nextBtn = $('#next');
+
+ const PAGE_BY_VIEW = { grid: 100, list: 250, map: 1000 };
+ const SAVE = (k, v) => localStorage.setItem('ldb.' + k, String(v));
+ const LOAD = (k, d) => { const v = localStorage.getItem('ldb.' + k); return v == null ? d : v; };
+
+ const state = {
+ entity: LOAD('entity', 'firms'),
+ view: LOAD('view', 'grid'),
+ cols: Number(LOAD('cols', 5)),
+ prospect: LOAD('prospect', ''),
+ size: LOAD('size', ''),
+ sort: LOAD('sort', 'score'),
+ listSortCol: LOAD('listSortCol', ''),
+ listSortDir: LOAD('listSortDir', 'asc'),
+ offset: 0,
+ q: '',
+ rows: [],
+ };
+
+ function pageSize() { return PAGE_BY_VIEW[state.view] || 100; }
+
+ function escapeHtml(s) {
+ return String(s == null ? '' : s).replace(/[&<>"']/g, c => ({'&':'&','<':'<','>':'>','"':'"',"'":'''}[c]));
+ }
+
+ async function loadStats() {
+ const r = await fetch('/api/dashboard/stats').then(r => r.json());
+ for (const k of ['firms','attorneys','with_site','with_phone','phones','emails','audits','cities']) {
+ const el = document.getElementById('s-' + k);
+ if (el && r[k] != null) el.textContent = Number(r[k]).toLocaleString();
+ }
+ }
+
+ function buildQS() {
+ const p = new URLSearchParams({ limit: pageSize(), offset: state.offset });
+ if (state.q) p.set('q', state.q);
+ if (state.entity === 'firms') {
+ if (state.prospect) p.set('prospect', state.prospect);
+ if (state.size) p.set('firm_size_band', state.size);
+ if (state.sort) p.set('sort', state.sort);
+ } else {
+ if (state.sort === 'name' || state.sort === 'admission' || state.sort === 'bar') p.set('sort', state.sort);
+ }
+ return p;
+ }
+
+ async function load() {
+ const url = (state.entity === 'firms' ? '/api/firms' : '/api/attorneys') + '?' + buildQS();
+ try {
+ const r = await fetch(url).then(r => r.json());
+ state.rows = r.rows || [];
+ pageInfo.textContent = state.rows.length
+ ? (state.offset + 1) + '–' + (state.offset + state.rows.length)
+ : '0–0';
+ prevBtn.disabled = state.offset === 0;
+ nextBtn.disabled = state.rows.length < pageSize();
+ render();
+ } catch (e) {
+ results.innerHTML = '<div class="empty">Error loading data: ' + escapeHtml(e.message) + '</div>';
+ }
+ }
+
+ function applyView() {
+ results.className = state.view;
+ document.documentElement.style.setProperty('--cols', String(state.cols));
+ gridColsVal.textContent = state.cols;
+ gridControls.style.display = state.view === 'grid' ? '' : 'none';
+ document.querySelectorAll('#view-toggle button').forEach(b => b.classList.toggle('active', b.dataset.view === state.view));
+ document.querySelectorAll('#entity-toggle button').forEach(b => b.classList.toggle('active', b.dataset.entity === state.entity));
+ const isFirms = state.entity === 'firms';
+ document.getElementById('prospect-select').style.display = isFirms ? '' : 'none';
+ document.getElementById('size-select').style.display = isFirms ? '' : 'none';
+ }
+
+ // ─── GRID ─────────────────────────────────────────────────────────────────
+ function renderGridFirms() {
+ if (!state.rows.length) return '<div class="empty">No firms match.</div>';
+ return state.rows.map(r => {
+ const sc = r.audit_score == null ? null : Number(r.audit_score);
+ const scColor = sc == null ? '#3a3a44' : sc >= 70 ? '#10b981' : sc >= 50 ? '#facc15' : '#ef4444';
+ const scoreBadge = sc == null
+ ? '<span class="badge" style="background:#1c2237;color:#737373">no audit</span>'
+ : '<span class="badge audit" style="background:' + scColor + '">' + sc + '</span>';
+ const swatch = r.audit_color ? '<span class="swatch" style="background:' + escapeHtml(r.audit_color) + '"></span>' : '';
+ const siteBadge = r.website
+ ? '<span class="badge has">site</span>'
+ : '<span class="badge needs">★ needs site</span>';
+ const sizeBadge = r.firm_size_band ? '<span class="badge size">' + escapeHtml(r.firm_size_band) + '</span>' : '';
+ const lawyerBadge = r.lawyer_count > 0 ? '<span class="badge size">' + r.lawyer_count + ' atty</span>' : '';
+ const phoneBadge = (r.phone_count > 0 || r.phone) ? '<span class="badge" style="color:#10b981">📞 ' + (r.phone_count || 1) + '</span>' : '';
+ const emailBadge = r.email_count > 0 ? '<span class="badge" style="color:#60a5fa">✉ ' + r.email_count + '</span>' : '';
+ const phoneLink = r.phone ? '<a href="tel:' + escapeHtml(r.phone.replace(/[^+0-9]/g,'')) + '">' + escapeHtml(r.phone) + '</a>' : '';
+ const siteLink = r.website ? '<a href="' + escapeHtml(r.website) + '" target="_blank" rel="noopener">' + escapeHtml(r.website.replace(/^https?:\/\//,'').slice(0,40)) + '</a>' : '';
+ const mapLink = (r.lat && r.lng)
+ ? '<a href="https://www.google.com/maps/search/?api=1&query=' + r.lat + ',' + r.lng + '" target="_blank" rel="noopener">🗺 map</a>'
+ : '';
+ return '<div class="card">'
+ + '<div class="name"><span>' + escapeHtml(r.name) + '</span><span style="display:flex;gap:4px;align-items:center">' + swatch + scoreBadge + '</span></div>'
+ + '<div>' + sizeBadge + lawyerBadge + siteBadge + phoneBadge + emailBadge + '</div>'
+ + '<div class="meta">'
+ + (r.address ? escapeHtml(r.address) + '<br>' : '')
+ + escapeHtml([r.neighborhood || r.city, r.zip].filter(Boolean).join(', '))
+ + (phoneLink ? '<br>' + phoneLink : '')
+ + (siteLink ? '<br>' + siteLink : '')
+ + '</div>'
+ + '<div class="actions">'
+ + (mapLink ? mapLink + ' · ' : '')
+ + '<button class="pitch-btn" data-pitch="' + r.id + '">✉ pitch</button>'
+ + '<button class="muted" data-copy="' + r.id + '">📋 copy</button>'
+ + '</div>'
+ + '</div>';
+ }).join('');
+ }
+
+ function renderGridAttorneys() {
+ if (!state.rows.length) return '<div class="empty">No attorneys match.</div>';
+ return state.rows.map(r => {
+ const statusBadge = r.license_status === 'Active'
+ ? '<span class="badge has">Active</span>'
+ : '<span class="badge needs">' + escapeHtml(r.license_status || '?') + '</span>';
+ const barBadge = r.bar_number ? '<span class="badge size">Bar #' + escapeHtml(r.bar_number) + '</span>' : '';
+ return '<div class="card">'
+ + '<div class="name">' + escapeHtml(r.full_name) + '</div>'
+ + '<div>' + barBadge + statusBadge + '</div>'
+ + '<div class="meta">'
+ + (r.law_school ? escapeHtml(r.law_school) + '<br>' : '')
+ + (r.firm_name ? '<a href="#" data-firm="' + r.firm_id + '">' + escapeHtml(r.firm_name) + '</a><br>' : '')
+ + escapeHtml([r.city, r.zip].filter(Boolean).join(', '))
+ + (r.firm_phone ? '<br><a href="tel:' + escapeHtml(r.firm_phone.replace(/[^+0-9]/g,'')) + '">' + escapeHtml(r.firm_phone) + '</a>' : '')
+ + '</div>'
+ + '</div>';
+ }).join('');
+ }
+
+ // ─── LIST (sortable headers) ─────────────────────────────────────────────
+ const FIRM_COLS = [
+ { key: 'audit_score', label: 'Score', acc: r => r.audit_score == null ? -1 : Number(r.audit_score) },
+ { key: 'name', label: 'Name', acc: r => (r.name || '').toLowerCase() },
+ { key: 'firm_size_band', label: 'Size', acc: r => (r.firm_size_band || '') },
+ { key: 'lawyer_count', label: '# Atty',acc: r => Number(r.lawyer_count || r.attorney_count || 0) },
+ { key: 'phone_count', label: '# Phone',acc: r => Number(r.phone_count || 0) },
+ { key: 'email_count', label: '# Email',acc: r => Number(r.email_count || 0) },
+ { key: 'phone', label: 'Phone', acc: r => r.phone || '' },
+ { key: 'address', label: 'Address',acc: r => r.address || '' },
+ { key: 'city', label: 'City', acc: r => (r.neighborhood || r.city || '') },
+ { key: 'zip', label: 'ZIP', acc: r => r.zip || '' },
+ { key: 'website', label: 'Website',acc: r => r.website || '' },
+ ];
+ const ATTORNEY_COLS = [
+ { key: 'full_name', label: 'Name', acc: r => (r.full_name || '').toLowerCase() },
+ { key: 'last_name', label: 'Last', acc: r => (r.last_name || '').toLowerCase() },
+ { key: 'bar_number', label: 'Bar #', acc: r => Number(r.bar_number) || 0 },
+ { key: 'license_status', label: 'Status', acc: r => r.license_status || '' },
+ { key: 'admission_date', label: 'Admitted',acc: r => r.admission_date || '' },
+ { key: 'law_school', label: 'School', acc: r => r.law_school || '' },
+ { key: 'firm_name', label: 'Firm', acc: r => r.firm_name || '' },
+ { key: 'city', label: 'City', acc: r => r.city || '' },
+ { key: 'zip', label: 'ZIP', acc: r => r.zip || '' },
+ { key: 'firm_phone', label: 'Phone', acc: r => r.firm_phone || '' },
+ ];
+
+ function renderList() {
+ const cols = state.entity === 'firms' ? FIRM_COLS : ATTORNEY_COLS;
+ if (!state.rows.length) return '<div class="empty">No matches.</div>';
+
+ let rows = state.rows.slice();
+ if (state.listSortCol) {
+ const col = cols.find(c => c.key === state.listSortCol);
+ if (col) {
+ const dir = state.listSortDir === 'desc' ? -1 : 1;
+ rows.sort((a, b) => {
+ const av = col.acc(a), bv = col.acc(b);
+ if (av < bv) return -1 * dir;
+ if (av > bv) return 1 * dir;
+ return 0;
+ });
+ }
+ }
+
+ const head = '<tr>' + cols.map(c => {
+ const arrow = state.listSortCol === c.key ? (state.listSortDir === 'desc' ? '▼' : '▲') : '';
+ return '<th data-col="' + c.key + '">' + escapeHtml(c.label) + ' <span class="sort-arrow">' + arrow + '</span></th>';
+ }).join('') + '</tr>';
+
+ let body;
+ if (state.entity === 'firms') {
+ body = rows.map(r => {
+ const sc = r.audit_score == null ? null : Number(r.audit_score);
+ const scColor = sc == null ? '#737373' : sc >= 70 ? '#10b981' : sc >= 50 ? '#facc15' : '#ef4444';
+ return '<tr>'
+ + '<td><span style="color:' + scColor + ';font-weight:600">' + (sc == null ? '—' : sc) + '</span></td>'
+ + '<td>' + escapeHtml(r.name) + '</td>'
+ + '<td>' + escapeHtml(r.firm_size_band || '') + '</td>'
+ + '<td>' + (r.lawyer_count || r.attorney_count || '') + '</td>'
+ + '<td>' + (r.phone_count || '') + '</td>'
+ + '<td>' + (r.email_count || '') + '</td>'
+ + '<td>' + escapeHtml(r.phone || '') + '</td>'
+ + '<td>' + escapeHtml(r.address || '') + '</td>'
+ + '<td>' + escapeHtml(r.neighborhood || r.city || '') + '</td>'
+ + '<td>' + escapeHtml(r.zip || '') + '</td>'
+ + '<td>' + (r.website ? '<a href="' + escapeHtml(r.website) + '" target="_blank" rel="noopener">' + escapeHtml(r.website.replace(/^https?:\/\//,'').slice(0,40)) + '</a>' : '') + '</td>'
+ + '</tr>';
+ }).join('');
+ } else {
+ body = rows.map(r => {
+ return '<tr>'
+ + '<td>' + escapeHtml(r.full_name) + '</td>'
+ + '<td>' + escapeHtml(r.last_name || '') + '</td>'
+ + '<td>' + escapeHtml(r.bar_number || '') + '</td>'
+ + '<td>' + escapeHtml(r.license_status || '') + '</td>'
+ + '<td>' + escapeHtml(r.admission_date ? String(r.admission_date).slice(0, 10) : '') + '</td>'
+ + '<td>' + escapeHtml(r.law_school || '') + '</td>'
+ + '<td>' + escapeHtml(r.firm_name || '') + '</td>'
+ + '<td>' + escapeHtml(r.city || '') + '</td>'
+ + '<td>' + escapeHtml(r.zip || '') + '</td>'
+ + '<td>' + escapeHtml(r.firm_phone || '') + '</td>'
+ + '</tr>';
+ }).join('');
+ }
+ return '<table>' + head + body + '</table>';
+ }
+
+ // ─── MAP ─────────────────────────────────────────────────────────────────
+ let map = null, mapLayer = null;
+ function renderMap() {
+ results.innerHTML = '<div id="leaflet"></div>';
+ const el = document.getElementById('leaflet');
+ if (!window.L) { el.innerHTML = '<div class="empty">Leaflet failed to load</div>'; return; }
+ map = L.map(el, { preferCanvas: true }).setView([34.05, -118.25], 10);
+ L.tileLayer('https://cartodb-basemaps-{s}.global.ssl.fastly.net/dark_all/{z}/{x}/{y}.png', {
+ attribution: '© OpenStreetMap, © CartoDB', maxZoom: 18,
+ }).addTo(map);
+ mapLayer = L.layerGroup().addTo(map);
+ addPins();
+ }
+ function addPins() {
+ if (!mapLayer) return;
+ mapLayer.clearLayers();
+ const bounds = [];
+ for (const r of state.rows) {
+ if (!r.lat || !r.lng) continue;
+ const sc = r.audit_score == null ? null : Number(r.audit_score);
+ const color = sc == null ? '#8b857a' : sc >= 70 ? '#10b981' : sc >= 50 ? '#facc15' : '#ef4444';
+ const dot = L.circleMarker([r.lat, r.lng], {
+ radius: 5, color, weight: 1, fillColor: color, fillOpacity: 0.85,
+ });
+ const safe = (s) => String(s == null ? '' : s).replace(/[&<>]/g, c => ({'&':'&','<':'<','>':'>'}[c]));
+ const popup = '<div class="pname">' + safe(r.name) + '</div>'
+ + (r.firm_size_band ? '<div>' + safe(r.firm_size_band) + '</div>' : '')
+ + (r.address ? '<div>' + safe(r.address) + '</div>' : '')
+ + (r.phone ? '<div>📞 ' + safe(r.phone) + '</div>' : '')
+ + (r.website ? '<div>🔗 <a href="' + safe(r.website) + '" target="_blank">' + safe(r.website.replace(/^https?:\/\//,'')) + '</a></div>' : '')
+ + '<div style="margin-top:6px"><a href="/p/' + r.id + '" target="_blank">→ pitch</a></div>';
+ dot.bindPopup(popup);
+ dot.addTo(mapLayer);
+ bounds.push([r.lat, r.lng]);
+ }
+ if (bounds.length) map.fitBounds(bounds, { padding: [40, 40] });
+ }
+
+ function render() {
+ if (state.view === 'map') {
+ if (map) { try { map.remove(); } catch(_){} map = null; mapLayer = null; }
+ renderMap();
+ } else {
+ if (map) { try { map.remove(); } catch(_){} map = null; mapLayer = null; }
+ results.innerHTML = state.view === 'grid'
+ ? (state.entity === 'firms' ? renderGridFirms() : renderGridAttorneys())
+ : renderList();
+ }
+ }
+
+ // ─── delegated handlers ──────────────────────────────────────────────────
+ results.addEventListener('click', async (e) => {
+ // sortable column header
+ const th = e.target.closest('th[data-col]');
+ if (th) {
+ const col = th.getAttribute('data-col');
+ if (state.listSortCol === col) {
+ state.listSortDir = state.listSortDir === 'asc' ? 'desc' : 'asc';
+ } else {
+ state.listSortCol = col; state.listSortDir = 'asc';
+ }
+ SAVE('listSortCol', state.listSortCol);
+ SAVE('listSortDir', state.listSortDir);
+ render();
+ return;
+ }
+ // pitch
+ const pBtn = e.target.closest('[data-pitch]');
+ if (pBtn) {
+ const id = pBtn.getAttribute('data-pitch');
+ window.open('/p/' + id, '_blank');
+ return;
+ }
+ // copy contact card
+ const cBtn = e.target.closest('[data-copy]');
+ if (cBtn) {
+ const id = cBtn.getAttribute('data-copy');
+ const r = state.rows.find(x => String(x.id) === id);
+ if (!r) return;
+ const text = [
+ r.name,
+ r.phone || '',
+ [r.address, r.city, r.zip].filter(Boolean).join(' · '),
+ r.website || ''
+ ].filter(Boolean).join('\n');
+ try {
+ await navigator.clipboard.writeText(text);
+ cBtn.textContent = '✓ copied';
+ setTimeout(() => { cBtn.textContent = '📋 copy'; }, 1200);
+ } catch (_) {}
+ }
+ });
+
+ // ─── stats bar click → quick filter ─────────────────────────────────────
+ document.getElementById('stats-bar').addEventListener('click', (e) => {
+ const span = e.target.closest('span[data-filter]');
+ if (!span) return;
+ const f = span.getAttribute('data-filter');
+ if (f === 'attorneys') { state.entity = 'attorneys'; }
+ else if (f === 'firms') { state.entity = 'firms'; state.prospect = ''; }
+ else if (f === 'with_site') { state.entity = 'firms'; state.prospect = 'has_site'; }
+ else if (f === 'with_phone') { state.entity = 'firms'; state.prospect = 'has_phone'; }
+ else if (f === 'audits') { state.entity = 'firms'; state.prospect = 'audited'; }
+ SAVE('entity', state.entity); SAVE('prospect', state.prospect);
+ document.getElementById('prospect-select').value = state.prospect;
+ state.offset = 0; applyView(); load();
+ });
+
+ // ─── controls wiring ─────────────────────────────────────────────────────
+ document.querySelectorAll('#view-toggle button').forEach(b => {
+ b.addEventListener('click', () => {
+ const prev = state.view;
+ state.view = b.dataset.view;
+ SAVE('view', state.view);
+ applyView();
+ if (PAGE_BY_VIEW[prev] !== PAGE_BY_VIEW[state.view]) {
+ state.offset = 0; load();
+ } else {
+ render();
+ }
+ });
+ });
+ document.querySelectorAll('#entity-toggle button').forEach(b => {
+ b.addEventListener('click', () => {
+ state.entity = b.dataset.entity;
+ SAVE('entity', state.entity);
+ state.offset = 0; applyView(); load();
+ });
+ });
+ gridColsEl.addEventListener('input', () => {
+ state.cols = Number(gridColsEl.value);
+ SAVE('cols', state.cols);
+ document.documentElement.style.setProperty('--cols', String(state.cols));
+ gridColsVal.textContent = state.cols;
+ });
+ let qTimer;
+ filterQ.addEventListener('input', () => {
+ state.q = filterQ.value;
+ clearTimeout(qTimer);
+ qTimer = setTimeout(() => { state.offset = 0; load(); }, 250);
+ });
+ document.getElementById('prospect-select').addEventListener('change', (e) => {
+ state.prospect = e.target.value; SAVE('prospect', state.prospect); state.offset = 0; load();
+ });
+ document.getElementById('size-select').addEventListener('change', (e) => {
+ state.size = e.target.value; SAVE('size', state.size); state.offset = 0; load();
+ });
+ document.getElementById('sort-select').addEventListener('change', (e) => {
+ state.sort = e.target.value; SAVE('sort', state.sort); state.offset = 0; load();
+ });
+ prevBtn.addEventListener('click', () => { state.offset = Math.max(0, state.offset - pageSize()); load(); });
+ nextBtn.addEventListener('click', () => { state.offset += pageSize(); load(); });
+
+ // ─── init ───────────────────────────────────────────────────────────────
+ gridColsEl.value = state.cols;
+ document.getElementById('prospect-select').value = state.prospect;
+ document.getElementById('size-select').value = state.size;
+ document.getElementById('sort-select').value = state.sort;
+ applyView();
+ loadStats();
+ load();
+})();
+</script>
+</body>
+</html>
diff --git a/scripts/build-agent/nightly-email.sh b/scripts/build-agent/nightly-email.sh
new file mode 100755
index 0000000..2887db4
--- /dev/null
+++ b/scripts/build-agent/nightly-email.sh
@@ -0,0 +1,68 @@
+#!/bin/bash
+# Lawyer Build Agent — nightly digest (8 PM PT) → steve@designerwallcoverings.com via George.
+set -euo pipefail
+
+PROJ="/Users/stevestudio2/Projects/lawyer-directory-builder"
+LOG="$PROJ/logs/build-agent/email.log"
+SNAP="$PROJ/logs/build-agent/snapshot.json"
+mkdir -p "$(dirname "$LOG")"
+ts() { date -u +%Y-%m-%dT%H:%M:%SZ; }
+
+[[ -f "$SNAP" ]] || { echo "[$(ts)] no snapshot — skipping email" >> "$LOG"; exit 0; }
+
+PSQL="psql -d lawyer_professional_directory -tA"
+
+# Stats from snapshot + DB
+TOTAL=$($PSQL -c "SELECT COUNT(*) FROM site_mockups")
+UNIQ=$($PSQL -c "SELECT COUNT(*) FROM site_mockups WHERE variant IN ('x1','x2','x3')")
+COMPLETE=$($PSQL -c "SELECT COUNT(*) FROM (SELECT organization_id FROM site_mockups WHERE variant IN ('x1','x2','x3') GROUP BY organization_id HAVING COUNT(DISTINCT variant)=3) z")
+LAST24=$($PSQL -c "SELECT COUNT(*) FROM site_mockups WHERE variant IN ('x1','x2','x3') AND generated_at > NOW() - INTERVAL '24 hours'")
+LEADS_24=$($PSQL -c "SELECT COUNT(*) FROM leads WHERE source='pitch_page_reply' AND created_at > NOW() - INTERVAL '24 hours'")
+PITCH_VIEWS_24=$($PSQL -c "SELECT COUNT(*) FROM pitch_views WHERE viewed_at > NOW() - INTERVAL '24 hours'")
+RECENT_FIRMS=$($PSQL <<'SQL'
+SELECT string_agg('• ' || o.name || ' (#' || o.id || ')', E'\n')
+FROM (
+ SELECT DISTINCT ON (organization_id) organization_id, generated_at
+ FROM site_mockups
+ WHERE variant IN ('x1','x2','x3')
+ AND generated_at > NOW() - INTERVAL '24 hours'
+ ORDER BY organization_id, generated_at DESC
+ LIMIT 8
+) m
+JOIN organizations o ON o.id = m.organization_id
+SQL
+)
+
+# HTML body
+read -r -d '' BODY <<EOF
+<html><body style="font-family:-apple-system,system-ui,sans-serif;color:#1a1814;max-width:640px">
+<h2 style="font-family:Georgia,serif;font-weight:400;color:#a82a1f;border-bottom:1px solid #ddd;padding-bottom:8px">Lawyer Directory — overnight build digest</h2>
+<table cellpadding="6" cellspacing="0" style="border-collapse:collapse;width:100%;font-size:14px">
+ <tr><td style="color:#7a6e5a">Mockups in DB</td><td style="text-align:right"><b>$TOTAL</b></td></tr>
+ <tr><td style="color:#7a6e5a">Unique (x1/x2/x3)</td><td style="text-align:right"><b>$UNIQ</b></td></tr>
+ <tr><td style="color:#7a6e5a">Firms with all 3 unique variants</td><td style="text-align:right"><b>$COMPLETE</b></td></tr>
+ <tr><td style="color:#7a6e5a">Generated in last 24h</td><td style="text-align:right"><b>$LAST24</b></td></tr>
+ <tr style="border-top:1px solid #eee"><td style="color:#7a6e5a">Pitch-page views (24h)</td><td style="text-align:right"><b>$PITCH_VIEWS_24</b></td></tr>
+ <tr><td style="color:#7a6e5a">Pitch-page replies (24h)</td><td style="text-align:right"><b>$LEADS_24</b></td></tr>
+</table>
+<h3 style="margin-top:24px;font-family:Georgia,serif;font-weight:400">Recent firms</h3>
+<pre style="background:#f5f1e8;padding:12px;border-radius:4px;font-family:Menlo,Monaco,monospace;font-size:12px;white-space:pre-wrap">${RECENT_FIRMS:-(none)}</pre>
+<p style="font-size:12px;color:#7a6e5a;margin-top:24px;border-top:1px solid #eee;padding-top:12px">
+ Live gallery: <a href="http://192.168.1.53:9701/mockups">http://192.168.1.53:9701/mockups</a><br/>
+ Per-firm pitch: <a href="http://192.168.1.53:9701/p/5">http://192.168.1.53:9701/p/<firm_id></a><br/>
+ Watchdog log: $PROJ/logs/build-agent/watchdog.log
+</p>
+</body></html>
+EOF
+
+# Subject (RFC 2047 encoded)
+SUBJECT="=?UTF-8?B?$(printf '%s' "Lawyer Directory · $LAST24 mockups today · $COMPLETE firms complete" | base64)?="
+
+# Send via George
+RESP=$(curl -s -u "admin:DWSecure2024!" \
+ -X POST "http://localhost:9850/api/send?account=info" \
+ -H "Content-Type: application/json" \
+ -d "$(jq -n --arg t "steve@designerwallcoverings.com" --arg s "$SUBJECT" --arg b "$BODY" \
+ '{to: $t, subject: $s, body: $b, html: true}')" 2>&1 | tr -d '\n' | head -c 500)
+
+echo "[$(ts)] sent: total=$TOTAL uniq=$UNIQ complete=$COMPLETE 24h=$LAST24 → $RESP" >> "$LOG"
diff --git a/scripts/build-agent/progress.sh b/scripts/build-agent/progress.sh
new file mode 100755
index 0000000..18b2859
--- /dev/null
+++ b/scripts/build-agent/progress.sh
@@ -0,0 +1,53 @@
+#!/bin/bash
+# Lawyer Build Agent — hourly progress snapshot.
+# Records: variant counts, firms-with-3-unique, last-hour throughput.
+set -euo pipefail
+
+PROJ="/Users/stevestudio2/Projects/lawyer-directory-builder"
+LOG="$PROJ/logs/build-agent/progress.log"
+SNAP="$PROJ/logs/build-agent/snapshot.json"
+mkdir -p "$(dirname "$LOG")"
+ts() { date -u +%Y-%m-%dT%H:%M:%SZ; }
+
+# Snapshot of current build state. read returns 1 if no trailing newline, hence `|| true`.
+read -r MOCK_TOTAL MOCK_X1 MOCK_X2 MOCK_X3 MOCK_X1H FIRMS_3UNIQUE FIRMS_LA_LEFT < <(
+ psql -d lawyer_professional_directory -tA -F ' ' <<'SQL' | tr '\n' ' '
+SELECT
+ (SELECT COUNT(*) FROM site_mockups),
+ (SELECT COUNT(*) FROM site_mockups WHERE variant='x1'),
+ (SELECT COUNT(*) FROM site_mockups WHERE variant='x2'),
+ (SELECT COUNT(*) FROM site_mockups WHERE variant='x3'),
+ (SELECT COUNT(*) FROM site_mockups WHERE variant IN ('x1','x2','x3') AND generated_at > NOW() - INTERVAL '1 hour'),
+ (SELECT COUNT(*) FROM (
+ SELECT organization_id FROM site_mockups WHERE variant IN ('x1','x2','x3')
+ GROUP BY organization_id HAVING COUNT(DISTINCT variant) = 3
+ ) z),
+ (SELECT COUNT(*) FROM organizations
+ WHERE type='law_firm' AND website IS NOT NULL
+ AND LOWER(city) IN (
+ 'los angeles','beverly hills','santa monica','pasadena','long beach','glendale',
+ 'burbank','culver city','west hollywood','el segundo','torrance','hermosa beach',
+ 'manhattan beach','sherman oaks','encino','studio city','van nuys','marina del rey',
+ 'hollywood','north hollywood','inglewood','redondo beach','westwood','brentwood',
+ 'century city','pacific palisades','malibu','calabasas','woodland hills','tarzana'
+ )
+ AND id NOT IN (
+ SELECT organization_id FROM site_mockups WHERE variant IN ('x1','x2','x3')
+ GROUP BY organization_id HAVING COUNT(DISTINCT variant) = 3
+ )
+ )
+SQL
+) || true
+
+cat > "$SNAP" <<EOF
+{
+ "ts": "$(ts)",
+ "mockups_total": $MOCK_TOTAL,
+ "x1": $MOCK_X1, "x2": $MOCK_X2, "x3": $MOCK_X3,
+ "unique_last_hour": $MOCK_X1H,
+ "firms_with_3_unique": $FIRMS_3UNIQUE,
+ "la_county_remaining": $FIRMS_LA_LEFT
+}
+EOF
+
+echo "[$(ts)] mockups=$MOCK_TOTAL x1=$MOCK_X1 x2=$MOCK_X2 x3=$MOCK_X3 1hr=$MOCK_X1H complete=$FIRMS_3UNIQUE remaining=$FIRMS_LA_LEFT" >> "$LOG"
diff --git a/scripts/build-agent/watchdog.sh b/scripts/build-agent/watchdog.sh
new file mode 100755
index 0000000..f057310
--- /dev/null
+++ b/scripts/build-agent/watchdog.sh
@@ -0,0 +1,75 @@
+#!/bin/bash
+# Lawyer Build Agent — watchdog
+# Run every 5 min via launchd. Ensures Mac1 mockup-render worker is alive.
+# Mac2 worker is intentionally NOT auto-started (RAM constrained).
+set -euo pipefail
+
+PROJ="/Users/stevestudio2/Projects/lawyer-directory-builder"
+LOG="$PROJ/logs/build-agent/watchdog.log"
+mkdir -p "$(dirname "$LOG")"
+ts() { date -u +%Y-%m-%dT%H:%M:%SZ; }
+
+# ── Mac1 worker check ──────────────────────────────────────────────────────
+mac1_running() {
+ ssh -o ConnectTimeout=4 -o BatchMode=yes mac1ts \
+ 'pgrep -f "tsx.*src/enrich/render_mockups.ts" >/dev/null 2>&1' \
+ >/dev/null 2>&1
+}
+
+mac1_tunnel_alive() {
+ ssh -o ConnectTimeout=4 -o BatchMode=yes mac1ts \
+ 'lsof -nP -iTCP:15432 -sTCP:LISTEN >/dev/null 2>&1' \
+ >/dev/null 2>&1
+}
+
+ensure_mac1_tunnel() {
+ if mac1_tunnel_alive; then return 0; fi
+ echo "[$(ts)] mac1 tunnel down — reopening" | tee -a "$LOG"
+ ssh -o ConnectTimeout=4 mac1ts \
+ "ssh -o StrictHostKeyChecking=accept-new -fN -L 15432:/tmp/.s.PGSQL.5432 stevestudio2@100.65.187.120" \
+ >>"$LOG" 2>&1 || true
+ sleep 2
+}
+
+start_mac1_worker() {
+ echo "[$(ts)] starting mac1 worker" | tee -a "$LOG"
+ local logname="mockup-mac1-unique-$(date +%H%M%S).log"
+ ssh -o ConnectTimeout=4 mac1ts \
+ "cd ~/Projects/lawyer-directory-builder && \
+ nohup env PATH=\"\$HOME/local/bin:\$PATH\" \
+ DATABASE_URL='postgres://stevestudio2@127.0.0.1:15432/lawyer_professional_directory' \
+ OLLAMA_URL='http://localhost:11434' \
+ LLM_MOCKUP_MODEL=gemma3:12b \
+ LLM_TIMEOUT_MS=360000 \
+ SITE_INTEL_TIMEOUT_MS=12000 \
+ \"\$HOME/local/bin/node\" \"\$HOME/Projects/lawyer-directory-builder/node_modules/.bin/tsx\" \
+ src/enrich/render_mockups.ts --la-county --id-mod 2 --id-rem 1 \
+ > logs/$logname 2>&1 & disown" \
+ >>"$LOG" 2>&1
+}
+
+# ── ensure DB schema is up to date (migrations sometimes get reverted) ─────
+SCHEMA_OK=$(psql -d lawyer_professional_directory -tAc \
+ "SELECT 1 FROM information_schema.tables WHERE table_name='site_mockups' LIMIT 1" 2>/dev/null || true)
+if [[ "$SCHEMA_OK" != "1" ]]; then
+ echo "[$(ts)] schema MISSING site_mockups — running migrations" | tee -a "$LOG"
+ ( cd "$PROJ" && npm run migrate >> "$LOG" 2>&1 ) || true
+ psql -d lawyer_professional_directory -c \
+ "UPDATE app_users SET tier='admin' WHERE role='admin' AND (tier IS NULL OR tier='client')" \
+ >> "$LOG" 2>&1 || true
+fi
+
+# ── main ───────────────────────────────────────────────────────────────────
+ensure_mac1_tunnel
+if mac1_running; then
+ echo "[$(ts)] mac1 worker OK" >> "$LOG"
+else
+ echo "[$(ts)] mac1 worker DOWN — restarting" | tee -a "$LOG"
+ start_mac1_worker
+fi
+
+# ── prune old logs (keep 30 most recent per host) ──────────────────────────
+ls -t "$PROJ/logs/"mockup-mac2-unique-*.log 2>/dev/null | tail -n +31 | xargs rm -f 2>/dev/null || true
+ssh -o ConnectTimeout=4 mac1ts \
+ 'ls -t ~/Projects/lawyer-directory-builder/logs/mockup-mac1-unique-*.log 2>/dev/null | tail -n +31 | xargs rm -f 2>/dev/null || true' \
+ >/dev/null 2>&1 || true
diff --git a/scripts/build-agent/weekly-refresh.sh b/scripts/build-agent/weekly-refresh.sh
new file mode 100755
index 0000000..376010c
--- /dev/null
+++ b/scripts/build-agent/weekly-refresh.sh
@@ -0,0 +1,24 @@
+#!/bin/bash
+# Lawyer Build Agent — weekly maintenance.
+# Sun 4 AM PT: clear stale site_intel (>30d) so Mac1 worker re-crawls them.
+set -euo pipefail
+
+PROJ="/Users/stevestudio2/Projects/lawyer-directory-builder"
+LOG="$PROJ/logs/build-agent/weekly.log"
+mkdir -p "$(dirname "$LOG")"
+ts() { date -u +%Y-%m-%dT%H:%M:%SZ; }
+
+CLEARED=$(psql -d lawyer_professional_directory -tA <<'SQL'
+WITH cleared AS (
+ UPDATE organizations
+ SET site_intel_at = NULL
+ WHERE type = 'law_firm'
+ AND website IS NOT NULL
+ AND site_intel_at < NOW() - INTERVAL '30 days'
+ RETURNING id
+)
+SELECT COUNT(*) FROM cleared
+SQL
+)
+
+echo "[$(ts)] cleared site_intel for $CLEARED firms — Mac1 worker will re-fetch" >> "$LOG"
diff --git a/scripts/take-shots.mjs b/scripts/take-shots.mjs
new file mode 100644
index 0000000..2827d65
--- /dev/null
+++ b/scripts/take-shots.mjs
@@ -0,0 +1,43 @@
+/**
+ * Capture full-page screenshots of every key URL in the lawyer directory.
+ * Output: public/qa-shots/*.png → served at http://localhost:9701/qa-shots/
+ */
+import { chromium } from 'playwright';
+import fs from 'node:fs';
+import path from 'node:path';
+
+const OUT = path.resolve('public/qa-shots');
+fs.mkdirSync(OUT, { recursive: true });
+const BASE = 'http://localhost:9701';
+
+const urls = [
+ ['dashboard', '/dashboard.html'],
+ ['mockups', '/mockups'],
+ ['mockups-x-only', '/mockups?variant=x'],
+ ['community', '/community'],
+ ['firm-bodie-law', '/firms/4'],
+ ['firm-large-sample', '/firms/26'],
+ ['attorney-amos-lopez', '/attorneys/16692'],
+ ['pitch-bodie', '/p/4'],
+ ['marketing-landing', '/'],
+ ['audit-viewer', '/audit.html'],
+];
+
+const browser = await chromium.launch();
+const ctx = await browser.newContext({ viewport: { width: 1440, height: 900 }, deviceScaleFactor: 1 });
+for (const [name, url] of urls) {
+ const page = await ctx.newPage();
+ try {
+ await page.goto(BASE + url, { waitUntil: 'load', timeout: 20000 });
+ await page.waitForTimeout(800);
+ await page.screenshot({ path: path.join(OUT, `${name}.png`), fullPage: true, type: 'png' });
+ console.log(`✓ ${name} ${BASE}${url}`);
+ } catch (e) {
+ console.error(`✗ ${name} ${e.message}`);
+ } finally {
+ await page.close();
+ }
+}
+await browser.close();
+console.log(`\nScreenshots → ${OUT}`);
+console.log(`View → ${BASE}/qa-shots/`);
diff --git a/src/enrich/discover_websites.ts b/src/enrich/discover_websites.ts
new file mode 100644
index 0000000..8eaaecb
--- /dev/null
+++ b/src/enrich/discover_websites.ts
@@ -0,0 +1,265 @@
+/**
+ * Website discovery via DNS + name verification.
+ *
+ * For every law firm without a `website`, generate candidate domains from the
+ * firm name (stripped of legal suffixes), DNS-resolve, then HTTP-GET the
+ * homepage and confirm the firm name actually appears in the title/body before
+ * accepting.
+ *
+ * No API keys, no third-party services — just public DNS + a single GET per
+ * candidate. Conservative: name must match before we attribute a website.
+ *
+ * Once a website is set, the existing src/enrich/firm_website_contacts.ts
+ * enricher (which only operates on firms WITH a website) can pull phone/email
+ * from the contact pages.
+ *
+ * CLI:
+ * npx tsx src/enrich/discover_websites.ts # full pass
+ * npx tsx src/enrich/discover_websites.ts --smoke # first 10 firms
+ * npx tsx src/enrich/discover_websites.ts --limit 100
+ */
+import 'dotenv/config';
+import { promises as dns } from 'node:dns';
+import { pool, query } from '../db/pool.ts';
+
+const SOURCE_NAME = 'DNS website discovery';
+const RATE_DELAY_MS = Number(process.env.WEBSITE_DISCOVERY_DELAY_MS || 50);
+const USER_AGENT = process.env.USER_AGENT
+ || 'LawyerDirectoryBuilder/0.1 (research; contact: steveabramsdesigns@gmail.com)';
+
+type Firm = { id: number; name: string };
+
+const argv = process.argv.slice(2);
+const SMOKE = argv.includes('--smoke');
+const LIMIT_IDX = argv.indexOf('--limit');
+const LIMIT = LIMIT_IDX >= 0 ? Number(argv[LIMIT_IDX + 1]) : (SMOKE ? 10 : 0);
+
+// Strip these suffix tokens before forming candidates.
+const SUFFIX_TOKENS = new Set([
+ 'llp', 'llc', 'pllc', 'pc', 'pa', 'lp', 'inc', 'ltd',
+ 'esq', 'esquire', 'chartered',
+ 'and', '&',
+ 'law', 'laws', 'legal', 'attorney', 'attorneys', 'lawyer', 'lawyers',
+ 'office', 'offices', 'group', 'firm', 'associates', 'partners', 'partnership',
+ 'a', 'the', 'of', 'at',
+]);
+
+// Skip these — too generic / would yield mass false positives.
+const NAME_BLOCKLIST = [
+ /^law office$/i, /^law offices$/i, /^attorney$/i, /^we the people$/i,
+ /^preferred$/i, /^the law (office|firm)$/i, /^attorneys at law$/i,
+];
+
+function tokens(name: string): string[] {
+ return name
+ .toLowerCase()
+ .replace(/&/g, ' and ')
+ .replace(/[^a-z0-9\s]/g, ' ')
+ .split(/\s+/)
+ .filter(Boolean);
+}
+
+function meaningfulTokens(name: string): string[] {
+ return tokens(name).filter(t => !SUFFIX_TOKENS.has(t) && t.length >= 2);
+}
+
+function candidateDomains(name: string): string[] {
+ const out = new Set<string>();
+ const meaningful = meaningfulTokens(name);
+ if (meaningful.length === 0) return [];
+
+ // 1. join all meaningful tokens
+ out.add(meaningful.join('') + '.com');
+
+ // 2. join meaningful + "law"
+ out.add(meaningful.join('') + 'law.com');
+
+ // 3. first 2 meaningful tokens
+ if (meaningful.length >= 2) {
+ out.add(meaningful.slice(0, 2).join('') + '.com');
+ out.add(meaningful.slice(0, 2).join('') + 'law.com');
+ }
+
+ // 4. surname + "law" — for 2+ token names, the LAST meaningful token is
+ // the more reliable surname signal (firm names rarely lead with a surname
+ // and "DavidLaw" type matches generic-firm domains too easily).
+ if (meaningful.length >= 2 && meaningful[meaningful.length - 1].length >= 5) {
+ out.add(meaningful[meaningful.length - 1] + 'law.com');
+ }
+
+ // 5. initials + "law" (3+ tokens only — avoids 2-letter false positives)
+ if (meaningful.length >= 3) {
+ const initials = meaningful.map(t => t[0]).join('');
+ if (initials.length >= 3) out.add(initials + 'law.com');
+ }
+
+ // Filter out anything < 6 chars total (e.g. "abc.com")
+ return [...out].filter(d => d.length >= 7);
+}
+
+async function ensureSource(): Promise<number> {
+ await query(`
+ INSERT INTO sources (source_name, source_type, base_url, terms_notes, allowed_method, rate_limit_rps)
+ VALUES ($1, 'directory', '(public DNS + per-firm websites)',
+ 'DNS resolution + single HTTP GET per candidate. Robots-respected per host.',
+ 'crawl', 5.0)
+ ON CONFLICT (source_name) DO NOTHING
+ `, [SOURCE_NAME]);
+ const r = await query<{ id: number }>(`SELECT id FROM sources WHERE source_name = $1`, [SOURCE_NAME]);
+ return r.rows[0].id;
+}
+
+async function startJob(sourceId: number, label: string): Promise<number> {
+ const r = await query<{ id: number }>(`
+ INSERT INTO scrape_jobs (source_id, job_label, status, started_at)
+ VALUES ($1, $2, 'running', NOW()) RETURNING id
+ `, [sourceId, label]);
+ return r.rows[0].id;
+}
+
+async function finishJob(jobId: number, fields: Record<string, unknown>) {
+ const sets: string[] = []; const params: unknown[] = []; let i = 1;
+ for (const [k, v] of Object.entries(fields)) { sets.push(`${k} = $${i++}`); params.push(v); }
+ sets.push(`finished_at = NOW()`); params.push(jobId);
+ await query(`UPDATE scrape_jobs SET ${sets.join(', ')} WHERE id = $${i}`, params);
+}
+
+async function loadFirms(): Promise<Firm[]> {
+ const lim = LIMIT > 0 ? `LIMIT ${LIMIT}` : '';
+ const r = await query<Firm>(`
+ SELECT id, name FROM organizations
+ WHERE type='law_firm' AND website IS NULL
+ ORDER BY id
+ ${lim}
+ `);
+ return r.rows.filter(f => !NAME_BLOCKLIST.some(rx => rx.test(f.name)));
+}
+
+async function dnsExists(domain: string): Promise<boolean> {
+ try {
+ const records = await dns.resolve4(domain);
+ return Array.isArray(records) && records.length > 0;
+ } catch {
+ try {
+ const recs = await dns.resolve6(domain);
+ return Array.isArray(recs) && recs.length > 0;
+ } catch { return false; }
+ }
+}
+
+async function fetchHomepage(domain: string): Promise<{ ok: boolean; html: string; finalUrl: string }> {
+ const { fetch } = await import('undici');
+ for (const proto of ['https', 'http']) {
+ try {
+ const url = `${proto}://${domain}/`;
+ const r = await fetch(url, {
+ method: 'GET',
+ headers: { 'User-Agent': USER_AGENT, Accept: 'text/html,application/xhtml+xml' },
+ signal: AbortSignal.timeout(8000),
+ redirect: 'follow',
+ });
+ if (r.ok && r.headers.get('content-type')?.includes('text/html')) {
+ const html = (await r.text()).slice(0, 20000);
+ return { ok: true, html, finalUrl: r.url };
+ }
+ } catch { /* try next proto */ }
+ }
+ return { ok: false, html: '', finalUrl: '' };
+}
+
+function nameMatchesPage(firmName: string, html: string): boolean {
+ const meaningful = meaningfulTokens(firmName);
+ if (meaningful.length === 0) return false;
+ const lower = html.toLowerCase();
+ // For multi-token names: at least 2 meaningful tokens must appear together
+ // (adjacent or as a joined string). Single-word matches against "law" pages
+ // produce too many false positives (any "Smith Law" firm matches any other
+ // "Smith"-themed law site).
+ if (meaningful.length >= 2) {
+ const firstTwo = meaningful.slice(0, 2);
+ if (lower.includes(firstTwo.join(' '))) return true;
+ if (lower.includes(firstTwo.join(''))) return true;
+ // Also allow last+first or any adjacent pair (firm name may be reversed on site)
+ for (let i = 0; i < meaningful.length - 1; i++) {
+ if (lower.includes(meaningful[i] + ' ' + meaningful[i + 1])) return true;
+ }
+ return false;
+ }
+ // Single-token firm name: require both the token AND the firm-name length
+ // to be substantial, plus a legal-services keyword nearby.
+ const only = meaningful[0];
+ if (only.length >= 6 && /\b(law|attorney|legal|esq|counsel)\b/.test(lower) && lower.includes(only)) return true;
+ return false;
+}
+
+async function discover(firm: Firm): Promise<{ website: string; matched: string } | null> {
+ const candidates = candidateDomains(firm.name);
+ for (const c of candidates) {
+ if (!(await dnsExists(c))) continue;
+ const r = await fetchHomepage(c);
+ if (!r.ok) continue;
+ if (nameMatchesPage(firm.name, r.html)) {
+ return { website: r.finalUrl || `https://${c}/`, matched: c };
+ }
+ }
+ return null;
+}
+
+const CONCURRENCY = Number(process.env.DISCOVERY_CONCURRENCY || 20);
+
+async function main() {
+ const sourceId = await ensureSource();
+ const label = SMOKE ? 'website-discovery:smoke' : (LIMIT > 0 ? `website-discovery:limit-${LIMIT}` : 'website-discovery:la-firms');
+ const jobId = await startJob(sourceId, label);
+
+ const firms = await loadFirms();
+ console.log(`[disc] scanning ${firms.length} firms (concurrency=${CONCURRENCY})…`);
+
+ let seen = 0, found = 0, errs = 0;
+
+ async function processOne(firm: Firm) {
+ seen++;
+ try {
+ const r = await discover(firm);
+ if (r) {
+ await query(
+ `UPDATE organizations SET website = $2, updated_at = NOW() WHERE id = $1 AND website IS NULL`,
+ [firm.id, r.website]);
+ found++;
+ console.log(`[disc] ✓ #${firm.id} ${firm.name.slice(0,50).padEnd(50)} → ${r.website}`);
+ }
+ } catch {
+ errs++;
+ }
+ if (seen % 200 === 0) console.log(`[disc] ${seen}/${firms.length} found=${found} errs=${errs}`);
+ }
+
+ try {
+ // Parallelism via N concurrent workers pulling from a shared queue.
+ let cursor = 0;
+ const workers = Array.from({ length: CONCURRENCY }, async () => {
+ while (true) {
+ const idx = cursor++;
+ if (idx >= firms.length) break;
+ await processOne(firms[idx]);
+ }
+ });
+ await Promise.all(workers);
+ } finally {
+ await finishJob(jobId, {
+ status: 'completed',
+ records_found: seen,
+ records_updated: found,
+ records_skipped: seen - found,
+ error_message: errs > 0 ? `${errs} per-firm errors` : null,
+ });
+ }
+ console.log(`[disc] done. seen=${seen} found=${found} errs=${errs}`);
+ await pool.end();
+}
+
+main().catch(async (err) => {
+ console.error('[disc] fatal:', err);
+ try { await pool.end(); } catch {}
+ process.exit(1);
+});
diff --git a/src/enrich/firm_website_contacts.ts b/src/enrich/firm_website_contacts.ts
index 54a2372..f24eab4 100644
--- a/src/enrich/firm_website_contacts.ts
+++ b/src/enrich/firm_website_contacts.ts
@@ -153,26 +153,38 @@ async function main() {
const sourceId = await ensureSource();
const jobId = await startJob(sourceId, 'enrich:firm-website-contacts');
+ const skipDoneDays = Number(process.env.SKIP_DONE_DAYS ?? '30');
+ const limit = Number(process.env.LIMIT ?? '0');
const r = await query<{ id: number; name: string; website: string }>(`
SELECT id, name, website FROM organizations
WHERE type='law_firm' AND website ~ '^https?://'
+ AND (contact_crawled_at IS NULL OR contact_crawled_at < NOW() - ($1 || ' days')::interval)
ORDER BY id
- `);
+ ${limit > 0 ? `LIMIT ${limit}` : ''}
+ `, [String(skipDoneDays)]);
const firms = r.rows;
- console.log(`[enrich] ${firms.length} firms with websites`);
+ console.log(`[enrich] ${firms.length} firms to crawl (skip-done-days=${skipDoneDays}${limit ? `, limit=${limit}` : ''})`);
+ const concurrency = Number(process.env.CONCURRENCY ?? '8');
let phoneTot = 0, emailTot = 0, processed = 0, errors = 0;
- for (const f of firms) {
- try {
- const { phones, emails } = await enrichOne(f, sourceId);
- phoneTot += phones; emailTot += emails;
- processed++;
- if (processed % 25 === 0) console.log(`[enrich] ${processed}/${firms.length}: phones+=${phoneTot} emails+=${emailTot}`);
- } catch (e) {
- errors++;
- console.error(`[enrich] err ${f.name}: ${(e as Error).message}`);
+ let cursor = 0;
+ async function worker() {
+ while (cursor < firms.length) {
+ const f = firms[cursor++];
+ try {
+ const { phones, emails } = await enrichOne(f, sourceId);
+ phoneTot += phones; emailTot += emails;
+ await query(`UPDATE organizations SET contact_crawled_at = NOW() WHERE id = $1`, [f.id]);
+ processed++;
+ if (processed % 25 === 0) console.log(`[enrich] ${processed}/${firms.length}: phones+=${phoneTot} emails+=${emailTot}`);
+ } catch (e) {
+ errors++;
+ console.error(`[enrich] err ${f.name}: ${(e as Error).message}`);
+ }
}
}
+ console.log(`[enrich] concurrency=${concurrency}`);
+ await Promise.all(Array.from({ length: concurrency }, () => worker()));
await finishJob(jobId, {
status: 'completed',
diff --git a/src/enrich/geocode_nominatim.ts b/src/enrich/geocode_nominatim.ts
new file mode 100644
index 0000000..b5f72ff
--- /dev/null
+++ b/src/enrich/geocode_nominatim.ts
@@ -0,0 +1,181 @@
+/**
+ * Nominatim geocoder — fills lat/lng for firms that have an address but no coordinates.
+ *
+ * Free, no key. OpenStreetMap's nominatim.openstreetmap.org per their usage policy:
+ * - 1 request/second max
+ * - Meaningful User-Agent required
+ * - Cache results (we already do — geocoded_at gates re-runs)
+ *
+ * https://operations.osmfoundation.org/policies/nominatim/
+ *
+ * CLI:
+ * npx tsx src/enrich/geocode_nominatim.ts # full pass
+ * npx tsx src/enrich/geocode_nominatim.ts --smoke # first 5
+ * npx tsx src/enrich/geocode_nominatim.ts --limit 100 # cap at N
+ */
+import 'dotenv/config';
+import { pool, query } from '../db/pool.ts';
+
+const SOURCE_NAME = 'OSM Nominatim Geocoder';
+const NOMINATIM_URL = 'https://nominatim.openstreetmap.org/search';
+const RATE_DELAY_MS = Number(process.env.NOMINATIM_DELAY_MS || 1100);
+const USER_AGENT = process.env.USER_AGENT
+ || 'LawyerDirectoryBuilder/0.1 (research; contact: steveabramsdesigns@gmail.com)';
+
+type Firm = {
+ id: number;
+ name: string;
+ address: string;
+ city: string | null;
+ state: string | null;
+ zip: string | null;
+};
+
+type NomHit = {
+ lat: string;
+ lon: string;
+ display_name: string;
+ importance?: number;
+};
+
+const argv = process.argv.slice(2);
+const SMOKE = argv.includes('--smoke');
+const LIMIT_IDX = argv.indexOf('--limit');
+const LIMIT = LIMIT_IDX >= 0 ? Number(argv[LIMIT_IDX + 1]) : (SMOKE ? 5 : 0);
+
+async function ensureSource(): Promise<number> {
+ await query(`
+ INSERT INTO sources (source_name, source_type, base_url, terms_notes, allowed_method, rate_limit_rps)
+ VALUES ($1, 'api', $2,
+ 'OSM Nominatim public instance. ODbL. Polite-use ≤1 rps with meaningful UA. Cache results.',
+ 'api', 1.0)
+ ON CONFLICT (source_name) DO NOTHING
+ `, [SOURCE_NAME, NOMINATIM_URL]);
+ const r = await query<{ id: number }>(`SELECT id FROM sources WHERE source_name = $1`, [SOURCE_NAME]);
+ return r.rows[0].id;
+}
+
+async function startJob(sourceId: number, label: string): Promise<number> {
+ const r = await query<{ id: number }>(`
+ INSERT INTO scrape_jobs (source_id, job_label, status, started_at)
+ VALUES ($1, $2, 'running', NOW()) RETURNING id
+ `, [sourceId, label]);
+ return r.rows[0].id;
+}
+
+async function finishJob(jobId: number, fields: Record<string, unknown>) {
+ const sets: string[] = []; const params: unknown[] = []; let i = 1;
+ for (const [k, v] of Object.entries(fields)) { sets.push(`${k} = $${i++}`); params.push(v); }
+ sets.push(`finished_at = NOW()`); params.push(jobId);
+ await query(`UPDATE scrape_jobs SET ${sets.join(', ')} WHERE id = $${i}`, params);
+}
+
+async function loadFirms(): Promise<Firm[]> {
+ const lim = LIMIT > 0 ? `LIMIT ${LIMIT}` : '';
+ const r = await query<Firm>(`
+ SELECT id, name, address, city, state, zip
+ FROM organizations
+ WHERE type = 'law_firm'
+ AND address IS NOT NULL
+ AND lat IS NULL
+ ORDER BY id
+ ${lim}
+ `);
+ return r.rows;
+}
+
+function buildQuery(firm: Firm): string {
+ // Nominatim does best with a single comma-separated string. Don't double-include
+ // city/state if they're already in the address blob.
+ const haystack = firm.address.toLowerCase();
+ const parts = [firm.address];
+ if (firm.city && !haystack.includes(firm.city.toLowerCase())) parts.push(firm.city);
+ if (firm.state && !haystack.includes(`, ${firm.state.toLowerCase()}`)) parts.push(firm.state);
+ if (firm.zip && !haystack.includes(firm.zip)) parts.push(firm.zip);
+ return parts.join(', ');
+}
+
+async function geocode(firm: Firm): Promise<NomHit | null> {
+ const { fetch } = await import('undici');
+ const params = new URLSearchParams({
+ q: buildQuery(firm),
+ format: 'json',
+ limit: '1',
+ countrycodes: 'us',
+ addressdetails: '0',
+ });
+ const r = await fetch(`${NOMINATIM_URL}?${params.toString()}`, {
+ headers: { 'User-Agent': USER_AGENT, Accept: 'application/json' },
+ signal: AbortSignal.timeout(20000),
+ });
+ if (!r.ok) {
+ if (r.status === 429) throw new Error('NOMINATIM_429');
+ return null;
+ }
+ const j = await r.json() as NomHit[];
+ return j && j.length > 0 ? j[0] : null;
+}
+
+async function applyHit(firmId: number, hit: NomHit): Promise<void> {
+ const lat = parseFloat(hit.lat);
+ const lng = parseFloat(hit.lon);
+ if (!isFinite(lat) || !isFinite(lng)) return;
+ // Guardrail: discard hits outside the LA County bounding box.
+ // ~33.7° to 34.85° N, -119.0° to -117.6° W (loose)
+ if (lat < 33.5 || lat > 35.0 || lng < -119.5 || lng > -117.5) return;
+ await query(
+ `UPDATE organizations SET lat = $2, lng = $3, geocoded_at = NOW(), updated_at = NOW() WHERE id = $1 AND lat IS NULL`,
+ [firmId, lat, lng]);
+}
+
+async function main() {
+ const sourceId = await ensureSource();
+ const label = SMOKE ? 'nominatim:smoke' : (LIMIT > 0 ? `nominatim:limit-${LIMIT}` : 'nominatim:la-firms');
+ const jobId = await startJob(sourceId, label);
+
+ const firms = await loadFirms();
+ console.log(`[nom] geocoding ${firms.length} firms…`);
+
+ let seen = 0, hit = 0, miss = 0, errs = 0, outside = 0;
+ try {
+ for (const firm of firms) {
+ seen++;
+ try {
+ const h = await geocode(firm);
+ if (!h) { miss++; }
+ else {
+ const lat = parseFloat(h.lat); const lng = parseFloat(h.lon);
+ if (lat < 33.5 || lat > 35.0 || lng < -119.5 || lng > -117.5) outside++;
+ else { await applyHit(firm.id, h); hit++; }
+ }
+ } catch (e) {
+ errs++;
+ const msg = (e as Error).message;
+ if (msg === 'NOMINATIM_429') {
+ console.error('[nom] 429 — backing off 60s');
+ await new Promise(r => setTimeout(r, 60000));
+ } else {
+ console.error(`[nom] firm=${firm.id} err: ${msg}`);
+ }
+ }
+ if (seen % 50 === 0) console.log(`[nom] ${seen}/${firms.length} hit=${hit} miss=${miss} outside=${outside} errs=${errs}`);
+ await new Promise(r => setTimeout(r, RATE_DELAY_MS));
+ }
+ } finally {
+ await finishJob(jobId, {
+ status: errs > firms.length / 2 ? 'failed' : 'completed',
+ records_found: seen,
+ records_updated: hit,
+ records_skipped: miss + outside,
+ error_message: errs > 0 ? `${errs} per-firm errors, ${outside} outside-LA discarded` : null,
+ });
+ }
+ console.log(`[nom] done. seen=${seen} hit=${hit} miss=${miss} outside=${outside} errs=${errs}`);
+ await pool.end();
+}
+
+main().catch(async (err) => {
+ console.error('[nom] fatal:', err);
+ try { await pool.end(); } catch {}
+ process.exit(1);
+});
diff --git a/src/enrich/render_mockups.ts b/src/enrich/render_mockups.ts
new file mode 100644
index 0000000..40ed981
--- /dev/null
+++ b/src/enrich/render_mockups.ts
@@ -0,0 +1,517 @@
+/**
+ * Render mockup screenshots for law firms.
+ *
+ * For each firm scope (default: Beverly Hills), generate 4 variants and
+ * persist into site_mockups:
+ * a — Editorial Noir (deterministic, mockup_templates.ts)
+ * b — Clean Modern Light (deterministic, mockup_templates.ts)
+ * c — Bold Confident (deterministic, mockup_templates.ts)
+ * x — LLM Bespoke (gemma3:12b custom HTML per firm)
+ *
+ * All HTML is saved to public/mockups/firm-{id}-{variant}.html and rendered
+ * to a PNG at public/mockups/firm-{id}-{variant}.png. Both files are served
+ * statically by the existing express.static('/').
+ *
+ * CLI:
+ * npx tsx src/enrich/render_mockups.ts --city "Beverly Hills"
+ * npx tsx src/enrich/render_mockups.ts --city "Beverly Hills" --skip-llm
+ * npx tsx src/enrich/render_mockups.ts --zips 90210,90211,90212,90213
+ * npx tsx src/enrich/render_mockups.ts --limit 10 --re-render
+ * npx tsx src/enrich/render_mockups.ts --la-county
+ */
+import 'dotenv/config';
+import { chromium, Browser } from 'playwright';
+import { load } from 'cheerio';
+import fs from 'node:fs';
+import path from 'node:path';
+import { pool, query } from '../db/pool.ts';
+import { fetchCompliant } from '../lib/compliance.ts';
+
+const MOCKUP_DIR = path.resolve(process.cwd(), 'public/mockups');
+const TEMPLATES_DIR = path.resolve(process.cwd(), 'public/mockup-templates');
+const OLLAMA_URL = process.env.OLLAMA_URL || 'http://localhost:11434';
+const LLM_MODEL = process.env.LLM_MOCKUP_MODEL || 'gemma3:12b';
+const LLM_TIMEOUT_MS = Number(process.env.LLM_TIMEOUT_MS || 120000);
+
+// ─── load top-N rotation pool of leading-firm templates ─────────────────────
+type Template = { id: number; name: string; website: string; score: number; primary_color: string; body_file: string };
+let TEMPLATES: Template[] = [];
+try {
+ const tpath = path.join(TEMPLATES_DIR, 'top12.json');
+ if (fs.existsSync(tpath)) {
+ TEMPLATES = JSON.parse(fs.readFileSync(tpath, 'utf8'));
+ }
+} catch { /* empty pool — fall back to old prompt */ }
+function pickTemplate(firmId: number): Template | null {
+ if (TEMPLATES.length === 0) return null;
+ return TEMPLATES[firmId % TEMPLATES.length];
+}
+function loadTemplateBody(t: Template): string {
+ try {
+ // Cap at 2.5KB so the prompt stays under model context budget
+ return fs.readFileSync(path.join(TEMPLATES_DIR, t.body_file), 'utf8').slice(0, 2500);
+ } catch { return ''; }
+}
+
+const argv = process.argv.slice(2);
+const has = (f: string) => argv.includes(f);
+const arg = (f: string): string | undefined => {
+ const i = argv.indexOf(f);
+ return i >= 0 ? argv[i + 1] : undefined;
+};
+const CITY = arg('--city') || null;
+const ZIPS = (arg('--zips') || '').split(',').map(s => s.trim()).filter(Boolean);
+const LIMIT = Number(arg('--limit') || 0);
+const ID_MOD = Number(arg('--id-mod') || 0);
+const ID_REM = Number(arg('--id-rem') || 0);
+const RE_RENDER = has('--re-render');
+const SKIP_LLM = has('--skip-llm');
+const LA_COUNTY = has('--la-county');
+
+// LA County cities (a workable subset — Beverly Hills, surrounding, downtown LA)
+const LA_COUNTY_CITIES = [
+ 'Los Angeles','Beverly Hills','Santa Monica','Pasadena','Long Beach','Glendale',
+ 'Burbank','Culver City','West Hollywood','El Segundo','Torrance','Hermosa Beach',
+ 'Manhattan Beach','Sherman Oaks','Encino','Studio City','Van Nuys','Marina Del Rey',
+ 'Hollywood','North Hollywood','Inglewood','Redondo Beach','Westwood','Brentwood',
+ 'Century City','Pacific Palisades','Malibu','Calabasas','Woodland Hills','Tarzana',
+];
+
+// ─── firm loader ────────────────────────────────────────────────────────────
+
+type Firm = {
+ id: number; name: string;
+ city: string | null; address: string | null;
+ zip: string | null; phone: string | null; website: string | null;
+ practice_areas: string[] | null;
+ site_intel_title: string | null;
+ site_intel_h1: string | null;
+ site_intel_meta_desc: string | null;
+ site_intel_body: string | null;
+ site_intel_practice: string[] | null;
+ site_intel_attorneys: string[] | null;
+ site_intel_at: Date | null;
+ audit_primary_color: string | null;
+ audit_palette: { hex: string; fraction: number }[] | null;
+};
+
+type SiteIntel = {
+ title: string | null;
+ h1: string | null;
+ metaDesc: string | null;
+ body: string | null; // first ~2KB of cleaned text
+ practice: string[];
+ attorneys: string[];
+};
+
+const SITE_INTEL_TIMEOUT_MS = Number(process.env.SITE_INTEL_TIMEOUT_MS || 12000);
+const SITE_INTEL_MAX_AGE_DAYS = Number(process.env.SITE_INTEL_MAX_AGE_DAYS || 30);
+
+// HSL hue from firm.id via golden-ratio scrambling — guarantees ~5000+ distinct hues
+// across the queue, with adjacent ids landing on visually distinct colors.
+function uniqueAccent(firmId: number): { hex: string; hsl: string } {
+ const phi = 0.6180339887;
+ const hue = Math.floor(((firmId * phi) % 1) * 360);
+ const sat = 65 + ((firmId * 13) % 20); // 65-84
+ const lit = 52 + ((firmId * 7) % 12); // 52-63
+ return { hex: hslToHex(hue, sat, lit), hsl: `hsl(${hue} ${sat}% ${lit}%)` };
+}
+function hslToHex(h: number, s: number, l: number): string {
+ s /= 100; l /= 100;
+ const k = (n: number) => (n + h / 30) % 12;
+ const a = s * Math.min(l, 1 - l);
+ const f = (n: number) => l - a * Math.max(-1, Math.min(k(n) - 3, Math.min(9 - k(n), 1)));
+ const toHex = (x: number) => Math.round(x * 255).toString(16).padStart(2, '0');
+ return `#${toHex(f(0))}${toHex(f(8))}${toHex(f(4))}`;
+}
+
+const PRACTICE_AREA_KEYWORDS = [
+ 'Personal Injury','Family Law','Divorce','Criminal Defense','Estate Planning',
+ 'Wills','Trusts','Probate','Real Estate','Business Law','Corporate','Employment',
+ 'Immigration','Bankruptcy','Tax','Intellectual Property','Patent','Trademark',
+ 'Civil Litigation','Workers Compensation','Medical Malpractice','Wrongful Death',
+ 'Class Action','Securities','Mergers','Entertainment','Insurance Defense',
+ 'Construction','Environmental','Elder Law','DUI','Traffic',
+];
+
+function extractPracticeAreas(body: string): string[] {
+ const found = new Set<string>();
+ const lower = body.toLowerCase();
+ for (const kw of PRACTICE_AREA_KEYWORDS) {
+ if (lower.includes(kw.toLowerCase())) found.add(kw);
+ if (found.size >= 6) break;
+ }
+ return [...found];
+}
+
+function extractAttorneys($: ReturnType<typeof load>): string[] {
+ const cands = new Set<string>();
+ // Look for elements that suggest "Attorney/Partner/Counsel" near a name
+ $('h2, h3, h4, .attorney, .lawyer, .partner, .name, .team-member').each((_, el) => {
+ const t = $(el).text().trim();
+ if (!t) return;
+ // 2-4 capitalized words, optional middle initial, optional comma+suffix
+ const m = t.match(/^([A-Z][a-z]+(?:\s+[A-Z]\.?)?(?:\s+[A-Z][a-z]+){1,3})(?:,?\s*(?:Esq\.?|JD|J\.D\.|Attorney|Partner))?\b/);
+ if (m && m[1].length < 60) cands.add(m[1].trim());
+ });
+ return [...cands].slice(0, 8);
+}
+
+async function fetchSiteIntel(firm: Firm): Promise<SiteIntel | null> {
+ if (!firm.website) return null;
+ let url: URL;
+ try { url = new URL(firm.website); } catch { return null; }
+ try {
+ const res = await fetchCompliant(url.href, { timeoutMs: SITE_INTEL_TIMEOUT_MS });
+ if (!res || res.status >= 400) return null;
+ const html = await res.text();
+ const $ = load(html);
+ const title = ($('title').first().text() || '').trim().slice(0, 200) || null;
+ const h1 = ($('h1').first().text() || '').trim().slice(0, 200) || null;
+ const metaDesc = ($('meta[name="description"]').attr('content') || $('meta[property="og:description"]').attr('content') || '').trim().slice(0, 300) || null;
+ $('script, style, noscript, header nav, footer').remove();
+ const body = ($('main').text() || $('body').text() || '')
+ .replace(/\s+/g, ' ').trim().slice(0, 2000) || null;
+ const practice = body ? extractPracticeAreas(body) : [];
+ const attorneys = extractAttorneys($);
+ return { title, h1, metaDesc, body, practice, attorneys };
+ } catch { return null; }
+}
+
+async function ensureSiteIntel(firm: Firm): Promise<Firm> {
+ // Skip if cached and fresh
+ if (firm.site_intel_at) {
+ const ageDays = (Date.now() - firm.site_intel_at.getTime()) / 86400000;
+ if (ageDays < SITE_INTEL_MAX_AGE_DAYS) return firm;
+ }
+ const intel = await fetchSiteIntel(firm);
+ if (!intel) {
+ // Mark attempted so we don't retry on every variant
+ await query(`UPDATE organizations SET site_intel_at = NOW() WHERE id = $1`, [firm.id]);
+ return firm;
+ }
+ await query(`
+ UPDATE organizations
+ SET site_intel_title=$2, site_intel_h1=$3, site_intel_meta_desc=$4,
+ site_intel_body=$5, site_intel_practice=$6, site_intel_attorneys=$7,
+ site_intel_at=NOW()
+ WHERE id=$1
+ `, [firm.id, intel.title, intel.h1, intel.metaDesc, intel.body, intel.practice, intel.attorneys]);
+ return {
+ ...firm,
+ site_intel_title: intel.title,
+ site_intel_h1: intel.h1,
+ site_intel_meta_desc: intel.metaDesc,
+ site_intel_body: intel.body,
+ site_intel_practice: intel.practice,
+ site_intel_attorneys: intel.attorneys,
+ site_intel_at: new Date(),
+ };
+}
+
+async function loadFirms(): Promise<Firm[]> {
+ const where: string[] = [`o.type='law_firm'`, `o.website IS NOT NULL`];
+ const params: any[] = [];
+ if (LA_COUNTY) {
+ params.push(LA_COUNTY_CITIES.map(s => s.toLowerCase()));
+ where.push(`LOWER(o.city) = ANY($${params.length}::text[])`);
+ } else if (CITY) {
+ params.push(CITY.toLowerCase());
+ where.push(`(LOWER(o.city) = $${params.length} OR LOWER(o.neighborhood) = $${params.length})`);
+ }
+ if (ZIPS.length) {
+ params.push(ZIPS);
+ where.push(`o.zip = ANY($${params.length}::text[])`);
+ }
+ if (ID_MOD > 0) {
+ params.push(ID_MOD); params.push(ID_REM);
+ where.push(`(o.id % $${params.length - 1}) = $${params.length}`);
+ }
+ const lim = LIMIT > 0 ? `LIMIT ${LIMIT}` : '';
+ const r = await query<Firm>(`
+ SELECT o.id, o.name, o.city, o.address, o.zip, o.phone, o.website,
+ o.practice_areas,
+ o.site_intel_title, o.site_intel_h1, o.site_intel_meta_desc,
+ o.site_intel_body, o.site_intel_practice, o.site_intel_attorneys,
+ o.site_intel_at,
+ latest_audit.primary_color AS audit_primary_color,
+ latest_audit.palette AS audit_palette
+ FROM organizations o
+ LEFT JOIN LATERAL (
+ SELECT primary_color, palette
+ FROM site_audits sa
+ WHERE sa.organization_id = o.id
+ ORDER BY sa.audited_at DESC
+ LIMIT 1
+ ) latest_audit ON TRUE
+ WHERE ${where.join(' AND ')}
+ ORDER BY o.id
+ ${lim}
+ `, params);
+ return r.rows;
+}
+
+async function alreadyRendered(firmId: number, variant: string): Promise<boolean> {
+ if (RE_RENDER) return false;
+ const r = await query<{ id: number }>(
+ `SELECT id FROM site_mockups WHERE organization_id = $1 AND variant = $2 LIMIT 1`,
+ [firmId, variant]);
+ return (r.rowCount || 0) > 0;
+}
+
+// ─── LLM bespoke generation ─────────────────────────────────────────────────
+
+const LLM_SYSTEM = `You are a senior web designer. Output ONLY a complete, valid <!doctype html> page — no preamble, no markdown fences, no commentary, just the HTML. The page must be a single self-contained file with all CSS in one <style> tag in the head. No external assets except Google Fonts via <link>.`;
+
+// Each firm gets 3 LLM-generated variants, each with a distinct aesthetic + palette + template.
+const AESTHETICS = [
+ {
+ key: 'x1', label: 'Editorial Modern',
+ surface: 'light cream (#faf8f3) with charcoal (#1a1814) text',
+ typo: 'serif display (Cormorant Garamond, Playfair, EB Garamond) for headlines, Inter sans for body',
+ flavor: 'magazine/editorial — long-form intro paragraphs, generous tracking, all-caps eyebrows, pull quotes, hairline rules between sections',
+ },
+ {
+ key: 'x2', label: 'Bold Confident',
+ surface: 'high-contrast white (#ffffff) base with one bold black/accent statement section in middle',
+ typo: 'extra-bold geometric sans (Inter 700-900, or Geist Bold) for headlines, Inter 400 for body',
+ flavor: 'confident — oversized headline (90-120px), zero ornament, square-bracket tags, button-as-logo, tight grid with thick rule borders',
+ },
+ {
+ key: 'x3', label: 'Web3 Immersive',
+ surface: 'deep slate (#0b0d12) with subtle radial-gradient hero glow + glassmorphism header (backdrop-filter blur 12px)',
+ typo: 'Inter or Geist throughout, weight 600 headlines, letter-spacing -0.02em',
+ flavor: 'modern dark — hover-lift cards, 1px hairline rgba(255,255,255,0.08) borders, gradient-edge CTAs, 8-14px rounded corners, 21st.dev/shadcn-style segmented controls and glass cards',
+ },
+] as const;
+type Aesthetic = typeof AESTHETICS[number];
+
+function pickAccent(firm: Firm, variantIndex: number): string {
+ // Prefer the firm's actual site primary_color (from site_audits) on x3 (web3 vibe matches dark sites);
+ // otherwise generate a per-firm-per-variant unique HSL from the firm.id.
+ if (variantIndex === 2 && firm.audit_primary_color && /^#[0-9a-f]{6}$/i.test(firm.audit_primary_color)) {
+ return firm.audit_primary_color;
+ }
+ return uniqueAccent(firm.id * 7 + variantIndex * 113).hex;
+}
+
+function pickTemplateForVariant(firmId: number, variantIndex: number): Template | null {
+ if (TEMPLATES.length === 0) return null;
+ // Different template for each (firm, variant) so all 3 variants are structurally distinct
+ return TEMPLATES[(firmId * 3 + variantIndex * 5) % TEMPLATES.length];
+}
+
+function llmPrompt(f: Firm, variantIndex: number, aesthetic: Aesthetic, accent: string): string {
+ const city = f.city || 'California';
+ const tpl = pickTemplateForVariant(f.id, variantIndex);
+ const tplBody = tpl ? loadTemplateBody(tpl) : '';
+ const tplBlock = tpl && tplBody
+ ? `\n\nSTRUCTURAL TEMPLATE — emulate the SECTION ORDERING + LAYOUT DENSITY of this top-rated (audit ${tpl.score}/100) LA law firm:\nFirm: ${tpl.name} Site: ${tpl.website}\n--- TEMPLATE BODY (HTML excerpt — for structure inspiration ONLY, do NOT copy text or branding) ---\n${tplBody}\n--- END TEMPLATE ---\n\nFollow the same number/order of sections, the same visual hierarchy, and the same component density. IGNORE the template's color palette and typography — apply the aesthetic specified below instead.\n`
+ : '';
+
+ // Build the real-content block from cached site intel
+ const intelLines: string[] = [];
+ if (f.site_intel_title) intelLines.push(`- Page title: "${f.site_intel_title}"`);
+ if (f.site_intel_h1) intelLines.push(`- H1 on their actual site: "${f.site_intel_h1}"`);
+ if (f.site_intel_meta_desc) intelLines.push(`- Their meta description: "${f.site_intel_meta_desc}"`);
+ if (f.site_intel_practice && f.site_intel_practice.length) intelLines.push(`- Practice areas mentioned on their site: ${f.site_intel_practice.join(', ')}`);
+ if (f.site_intel_attorneys && f.site_intel_attorneys.length) intelLines.push(`- Attorneys named on their site: ${f.site_intel_attorneys.slice(0, 6).join(', ')}`);
+ if (f.site_intel_body) intelLines.push(`- About-text excerpt from their site (use this to write copy that sounds like THEM): "${f.site_intel_body.slice(0, 800)}"`);
+ const intelBlock = intelLines.length
+ ? `\nREAL CONTENT FROM THEIR ACTUAL WEBSITE (use this — do not invent generic copy):\n${intelLines.join('\n')}\n`
+ : `\nNo cached site content available — write reasonable, firm-specific copy based on the name "${f.name}" and city "${city}". Do NOT use placeholder text like "Our team of attorneys".\n`;
+
+ const practiceForSeo = (f.site_intel_practice && f.site_intel_practice.length ? f.site_intel_practice : (f.practice_areas || [])).slice(0, 4).join(', ') || 'legal services';
+
+ return `Design a UNIQUE law-firm landing page for: ${f.name}
+Variant: ${aesthetic.key} — ${aesthetic.label}
+Firm city: ${city}
+Phone: ${f.phone || '(310) 555-0100'}
+Address: ${f.address || `${city}, CA`}
+Accent color (use this EXACT hex everywhere accent is needed): ${accent}
+${intelBlock}
+AESTHETIC for this variant — match strictly:
+- Surface: ${aesthetic.surface}
+- Typography: ${aesthetic.typo}
+- Visual flavor: ${aesthetic.flavor}
+- Use ONLY the accent color ${accent} for highlights — no other vibrant colors. Neutrals/grays/black/white are unlimited.
+- 21st.dev / shadcn-influenced component patterns where they fit the aesthetic: pill badges with subtle border, segmented filter controls, ghost-bordered buttons, kbd-style chips for stats, divider lines between rows.
+- Hero headline 60-110px (depending on aesthetic), tight tracking
+- Generous section padding (80-140px), single-column 768px breakpoint
+
+REQUIRED — these MUST appear in the <head> (proper SEO):
+- <title>${f.name} — ${practiceForSeo} | ${city}</title>
+- <meta name="description" content="..."> — ONE sentence, 140-160 chars, mentions firm name + city + 1-2 practice areas, written in firm's voice
+- <meta property="og:title">, <meta property="og:description">, <meta name="robots" content="index,follow">
+
+Required sections, in this order:
+1. Sticky banner (1px accent border-bottom): "MOCKUP — DETAILS PENDING VERIFICATION" — uppercase, 10px, opacity 0.6
+2. Header: firm name as text logo, 4 nav links (Practice, Attorneys, Results, Contact), 1 CTA button
+3. Hero: headline that uses real site H1 or echoes the firm's actual messaging — NOT generic "Trusted Legal Counsel". Sub-headline 1 sentence. Two CTAs.
+4. Trust band: 4 distinctive metrics relevant to THIS firm (years, cases, recovery, reviews) — vary numbers per firm so they're not all "30+ Years"
+5. Practice areas: render the practice areas FROM THE REAL SITE (or firm DB) as 4-6 cards with firm-specific descriptions. If only 3 known, render 3 — don't fabricate.
+6. Attorneys / People section: if site_intel attorneys are present, list them with placeholder bio lines; else a single bio block for the firm
+7. Process / approach: 3 steps written in firm-specific language
+8. Contact: phone (large), address, form (name/email/matter/message + accent CTA)
+9. Footer: hairline border, copyright + state-bar advertising notice, address
+
+CRITICAL — DO NOT:
+- Use placeholder/lorem text or generic "Our team of attorneys is dedicated to..."
+- Use a different accent color than ${accent}
+- Add inline images, pattern backgrounds, or external assets other than Google Fonts <link>
+- Output anything before <!doctype html> or after </html>
+
+Self-contained: ALL CSS in one <style> tag. Mobile-responsive. Validates as HTML5.${tplBlock}
+
+Output the full HTML now, starting with <!doctype html>:`;
+}
+
+function stripCodeFences(s: string): string {
+ // Some models wrap in ```html ... ```, strip if present
+ const fenced = s.match(/```(?:html)?\s*\n([\s\S]*?)\n```/);
+ if (fenced) return fenced[1].trim();
+ return s.trim();
+}
+
+async function generateLLMHtml(f: Firm, variantIndex: number, aesthetic: Aesthetic, accent: string, signal: AbortSignal): Promise<{ html: string; ms: number } | null> {
+ const t0 = Date.now();
+ try {
+ const r = await fetch(`${OLLAMA_URL}/api/generate`, {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json' },
+ signal,
+ body: JSON.stringify({
+ model: LLM_MODEL,
+ system: LLM_SYSTEM,
+ prompt: llmPrompt(f, variantIndex, aesthetic, accent),
+ stream: false,
+ // higher temp + per-firm seed → variation across firms.
+ // num_ctx=8192 keeps RAM bounded; default would be 128K for gemma3 → OOM on 24GB Macs.
+ options: { temperature: 0.7, top_p: 0.92, num_predict: 3500, num_ctx: 8192, seed: f.id * 100 + variantIndex },
+ }),
+ });
+ if (!r.ok) { console.warn(`[llm] http ${r.status} for firm ${f.id}`); return null; }
+ const j = await r.json() as { response?: string };
+ const raw = (j.response || '').trim();
+ if (!raw) return null;
+ const html = stripCodeFences(raw);
+ if (!html.toLowerCase().includes('<!doctype') && !html.toLowerCase().includes('<html')) {
+ console.warn(`[llm] firm ${f.id} no <html> in output (${html.length} chars)`);
+ return null;
+ }
+ return { html, ms: Date.now() - t0 };
+ } catch (e) {
+ if ((e as any).name === 'AbortError') return null;
+ console.warn(`[llm] err firm ${f.id}: ${(e as Error).message}`);
+ return null;
+ }
+}
+
+// ─── render an HTML string to a PNG via Playwright ──────────────────────────
+
+async function snapshotHtml(browser: Browser, html: string, outPng: string, opts: { fullPage?: boolean } = {}): Promise<void> {
+ const ctx = await browser.newContext({ viewport: { width: 1280, height: 1700 } });
+ const page = await ctx.newPage();
+ try {
+ await page.setContent(html, { waitUntil: 'load', timeout: 20000 });
+ await page.waitForTimeout(800); // settle fonts
+ await page.screenshot({ path: outPng, type: 'png', fullPage: opts.fullPage ?? false });
+ } finally {
+ await ctx.close().catch(() => {});
+ }
+}
+
+// ─── persist a single mockup ────────────────────────────────────────────────
+
+async function persistMockup(firmId: number, variant: string, label: string, htmlPath: string, pngPath: string, model: string | null, ms: number | null): Promise<void> {
+ await query(`
+ INSERT INTO site_mockups (organization_id, variant, template_label, screenshot_path, html_path, llm_model, gen_ms)
+ VALUES ($1, $2, $3, $4, $5, $6, $7)
+ ON CONFLICT (organization_id, variant) DO UPDATE SET
+ template_label = EXCLUDED.template_label,
+ screenshot_path = EXCLUDED.screenshot_path,
+ html_path = EXCLUDED.html_path,
+ llm_model = EXCLUDED.llm_model,
+ gen_ms = EXCLUDED.gen_ms,
+ generated_at = NOW()
+ `, [firmId, variant, label, pngPath, htmlPath, model, ms]);
+}
+
+// ─── render one firm — all 4 variants ───────────────────────────────────────
+
+async function renderFirm(browser: Browser, firmIn: Firm): Promise<{ done: number; skipped: number; errs: number }> {
+ const out = { done: 0, skipped: 0, errs: 0 };
+
+ // Fetch + cache real-site intel ONCE per firm (used by all 3 variants)
+ const firm = await ensureSiteIntel(firmIn);
+
+ for (let i = 0; i < AESTHETICS.length; i++) {
+ const aesthetic = AESTHETICS[i];
+ const variantKey = aesthetic.key;
+ if (await alreadyRendered(firm.id, variantKey)) { out.skipped++; continue; }
+ if (SKIP_LLM) { out.skipped++; continue; }
+
+ const accent = pickAccent(firm, i);
+ const ac = new AbortController();
+ const timer = setTimeout(() => ac.abort(), LLM_TIMEOUT_MS);
+ try {
+ const r = await generateLLMHtml(firm, i, aesthetic, accent, ac.signal);
+ if (r) {
+ const htmlPath = path.join(MOCKUP_DIR, `firm-${firm.id}-${variantKey}.html`);
+ const pngPath = path.join(MOCKUP_DIR, `firm-${firm.id}-${variantKey}.png`);
+ fs.writeFileSync(htmlPath, r.html);
+ await snapshotHtml(browser, r.html, pngPath);
+ await persistMockup(
+ firm.id, variantKey,
+ `${aesthetic.label} (${LLM_MODEL}, ${accent})`,
+ `/mockups/firm-${firm.id}-${variantKey}.html`,
+ `/mockups/firm-${firm.id}-${variantKey}.png`,
+ LLM_MODEL, r.ms,
+ );
+ out.done++;
+ } else {
+ out.errs++;
+ }
+ } catch (e) {
+ out.errs++;
+ console.error(`[mockup] firm ${firm.id} variant ${variantKey} err: ${(e as Error).message}`);
+ } finally {
+ clearTimeout(timer);
+ }
+ }
+ return out;
+}
+
+// ─── main ──────────────────────────────────────────────────────────────────
+
+async function main() {
+ fs.mkdirSync(MOCKUP_DIR, { recursive: true });
+ const firms = await loadFirms();
+ if (firms.length === 0) { console.log('[mockup] no firms match'); await pool.end(); return; }
+
+ const scope = LA_COUNTY ? 'LA County' : (CITY ? `city=${CITY}` : (ZIPS.length ? `zips=${ZIPS.join(',')}` : 'all'));
+ console.log(`[mockup] rendering ${firms.length} firms (scope: ${scope}) llm=${SKIP_LLM ? 'OFF' : LLM_MODEL} re-render=${RE_RENDER} → ${MOCKUP_DIR}`);
+
+ const browser = await chromium.launch({ headless: true });
+ let done = 0, skipped = 0, errs = 0;
+ try {
+ for (let i = 0; i < firms.length; i++) {
+ const firm = firms[i];
+ const t0 = Date.now();
+ const r = await renderFirm(browser, firm);
+ done += r.done; skipped += r.skipped; errs += r.errs;
+ const dt = ((Date.now() - t0) / 1000).toFixed(1);
+ console.log(`[mockup] ${i + 1}/${firms.length} #${firm.id} ${firm.name.slice(0, 38).padEnd(38)} done=${r.done} skip=${r.skipped} err=${r.errs} ${dt}s`);
+ }
+ } finally {
+ await browser.close();
+ }
+ console.log(`[mockup] complete. done=${done} skipped=${skipped} errs=${errs}`);
+ await pool.end();
+}
+
+main().catch(async err => {
+ console.error('[mockup] fatal:', err);
+ try { await pool.end(); } catch {}
+ process.exit(1);
+});
diff --git a/src/enrich/site_audit.ts b/src/enrich/site_audit.ts
new file mode 100644
index 0000000..74b41ef
--- /dev/null
+++ b/src/enrich/site_audit.ts
@@ -0,0 +1,306 @@
+/**
+ * Site audit — for each firm with a website, fetch the homepage in headless
+ * Chromium, capture a 1280x800 screenshot, extract the dominant color palette,
+ * scan the DOM for marketing-quality signals, compute a 0-100 score, generate
+ * concrete suggestions, and persist all of it to the site_audits table.
+ *
+ * NOTE on "traffic ranking": real traffic numbers require paid APIs (SimilarWeb,
+ * SEMrush, Ahrefs). This module computes a *marketing health score* from
+ * publicly observable on-page signals — a defensible proxy for "is this site
+ * doing real marketing work" but NOT a substitute for actual visitor counts.
+ *
+ * CLI:
+ * npx tsx src/enrich/site_audit.ts # full pass
+ * npx tsx src/enrich/site_audit.ts --smoke # first 5 firms
+ * npx tsx src/enrich/site_audit.ts --limit 50
+ * npx tsx src/enrich/site_audit.ts --re-audit # re-audit firms even if recent
+ */
+import 'dotenv/config';
+import { chromium, Browser } from 'playwright';
+import sharp from 'sharp';
+import fs from 'node:fs';
+import path from 'node:path';
+import { pool, query } from '../db/pool.ts';
+
+const SCREENSHOT_DIR = path.resolve(process.cwd(), 'public/screenshots');
+const USER_AGENT = process.env.USER_AGENT
+ || 'LawyerDirectoryBuilder/0.1 (research; contact: steveabramsdesigns@gmail.com)';
+const CONCURRENCY = Number(process.env.AUDIT_CONCURRENCY || 3);
+
+const argv = process.argv.slice(2);
+const SMOKE = argv.includes('--smoke');
+const RE_AUDIT = argv.includes('--re-audit');
+const LIMIT_IDX = argv.indexOf('--limit');
+const LIMIT = LIMIT_IDX >= 0 ? Number(argv[LIMIT_IDX + 1]) : (SMOKE ? 5 : 0);
+
+type Firm = { id: number; name: string; website: string };
+
+type Signals = {
+ status_code: number | null;
+ final_url: string;
+ has_https: boolean;
+ has_favicon: boolean;
+ has_og_image: boolean;
+ has_meta_viewport: boolean;
+ has_analytics: boolean;
+ has_h1: boolean;
+ has_schema_org: boolean;
+ has_phone_visible: boolean;
+ page_size_bytes: number;
+ load_time_ms: number;
+ font_count: number;
+ image_count: number;
+ link_count: number;
+};
+
+async function loadFirms(): Promise<Firm[]> {
+ const lim = LIMIT > 0 ? `LIMIT ${LIMIT}` : '';
+ const reAuditFilter = RE_AUDIT
+ ? ''
+ : `AND NOT EXISTS (SELECT 1 FROM site_audits a WHERE a.organization_id = o.id AND a.audited_at > NOW() - INTERVAL '7 days' AND a.status_code BETWEEN 200 AND 399)`;
+ const r = await query<Firm>(`
+ SELECT o.id, o.name, o.website
+ FROM organizations o
+ WHERE o.type='law_firm' AND o.website IS NOT NULL
+ ${reAuditFilter}
+ ORDER BY o.id
+ ${lim}
+ `);
+ return r.rows;
+}
+
+async function extractPalette(pngBuffer: Buffer): Promise<{ primary: string; palette: Array<{hex:string;r:number;g:number;b:number;fraction:number}> }> {
+ // Downsample to 80x50 (4000 px), bucket colors into a 16-step grid (4096 buckets),
+ // count frequencies, return top 6 by fraction.
+ const { data, info } = await sharp(pngBuffer)
+ .resize(80, 50, { fit: 'cover' })
+ .removeAlpha()
+ .raw()
+ .toBuffer({ resolveWithObject: true });
+ const total = info.width * info.height;
+ const buckets = new Map<string, { r: number; g: number; b: number; n: number }>();
+ for (let i = 0; i < data.length; i += 3) {
+ const r = data[i], g = data[i + 1], b = data[i + 2];
+ // Skip near-white & near-black to surface brand colors
+ const max = Math.max(r, g, b), min = Math.min(r, g, b);
+ if (max > 245 && min > 240) continue;
+ if (max < 18) continue;
+ const key = `${r >> 4}-${g >> 4}-${b >> 4}`;
+ const cur = buckets.get(key);
+ if (cur) { cur.r += r; cur.g += g; cur.b += b; cur.n++; }
+ else buckets.set(key, { r, g, b, n: 1 });
+ }
+ const sorted = [...buckets.values()].sort((a, b) => b.n - a.n).slice(0, 6).map(c => {
+ const r = Math.round(c.r / c.n), g = Math.round(c.g / c.n), b = Math.round(c.b / c.n);
+ return {
+ hex: '#' + [r, g, b].map(v => v.toString(16).padStart(2, '0')).join(''),
+ r, g, b, fraction: c.n / total,
+ };
+ });
+ return { primary: sorted[0]?.hex || '#000000', palette: sorted };
+}
+
+function scoreAndSuggest(s: Signals, firmName: string): { score: number; suggestions: string[] } {
+ const sug: string[] = [];
+ let score = 0;
+ // HTTPS — table stakes
+ if (s.has_https) score += 15;
+ else sug.push('Move site to HTTPS (free via Let\'s Encrypt) — visitors and Google penalize HTTP.');
+ // Mobile
+ if (s.has_meta_viewport) score += 10;
+ else sug.push('Add `<meta name="viewport" content="width=device-width">` for mobile rendering.');
+ // OG image (social sharing)
+ if (s.has_og_image) score += 10;
+ else sug.push('Add an Open Graph image (og:image) so links shared on LinkedIn/Facebook show a preview.');
+ // Analytics
+ if (s.has_analytics) score += 10;
+ else sug.push('Install GA4, Plausible, or Fathom — you can\'t improve traffic you\'re not measuring.');
+ // Favicon
+ if (s.has_favicon) score += 5;
+ else sug.push('Add a favicon — bare-tab pages look unfinished.');
+ // Content
+ if (s.has_h1) score += 5;
+ else sug.push('Add a clear H1 heading — current page has none, hurts SEO.');
+ if (s.has_schema_org) score += 10;
+ else sug.push('Add Schema.org LegalService / Attorney structured data — Google rich-results eligibility.');
+ if (s.has_phone_visible) score += 5;
+ else sug.push('Surface the firm phone number above the fold — call-driven conversions need one click.');
+ // Page facts
+ if (s.page_size_bytes > 0 && s.page_size_bytes < 2_000_000) score += 10;
+ else if (s.page_size_bytes >= 2_000_000) sug.push(`Page is ${(s.page_size_bytes/1e6).toFixed(1)} MB — compress images / lazy-load.`);
+ if (s.load_time_ms > 0 && s.load_time_ms < 3000) score += 10;
+ else if (s.load_time_ms >= 3000) sug.push(`Load time ${(s.load_time_ms/1000).toFixed(1)}s — visitors bounce after 3s.`);
+ if (s.font_count >= 2 && s.font_count <= 4) score += 5;
+ else if (s.font_count > 4) sug.push(`${s.font_count} font families loaded — consolidate to 2-3.`);
+ if (s.image_count >= 3) score += 5;
+ // Cap at 100
+ return { score: Math.min(100, score), suggestions: sug };
+}
+
+async function auditOne(browser: Browser, firm: Firm): Promise<{ ok: boolean; signals?: Signals; primary?: string; palette?: any; score?: number; suggestions?: string[]; screenshot?: string; error?: string }> {
+ const ctx = await browser.newContext({
+ userAgent: USER_AGENT,
+ viewport: { width: 1280, height: 800 },
+ ignoreHTTPSErrors: true,
+ });
+ const page = await ctx.newPage();
+ let bytesIn = 0;
+ page.on('response', async r => {
+ try {
+ const len = r.headers()['content-length'];
+ if (len) bytesIn += parseInt(len, 10);
+ } catch {}
+ });
+ try {
+ const t0 = Date.now();
+ let resp;
+ try {
+ resp = await page.goto(firm.website, { waitUntil: 'load', timeout: 20000 });
+ } catch (e) {
+ return { ok: false, error: `goto: ${(e as Error).message.slice(0, 120)}` };
+ }
+ const load_time_ms = Date.now() - t0;
+ const status_code = resp?.status() ?? null;
+ const final_url = page.url();
+ if (!status_code || status_code >= 400) {
+ return { ok: false, signals: { status_code, final_url, has_https: false, has_favicon: false, has_og_image: false, has_meta_viewport: false, has_analytics: false, has_h1: false, has_schema_org: false, has_phone_visible: false, page_size_bytes: 0, load_time_ms, font_count: 0, image_count: 0, link_count: 0 } };
+ }
+
+ // NOTE: keep this evaluate() flat — no nested arrow helpers, no TS syntax —
+ // tsx's transform inserts a `__name` reference for sub-functions that
+ // doesn't exist in the browser context.
+ const signals = await page.evaluate(`(function(){
+ var d = document;
+ var head = d.head || d;
+ var html = (d.documentElement.outerHTML || '').toLowerCase();
+ var bodyText = (d.body && d.body.innerText) || '';
+ var fonts = {};
+ try {
+ var all = d.querySelectorAll('*');
+ for (var i = 0; i < all.length; i++) {
+ var f = ((window.getComputedStyle(all[i]).fontFamily) || '').split(',')[0].trim().replace(/['"]/g, '').toLowerCase();
+ if (f) fonts[f] = 1;
+ }
+ } catch (e) {}
+ var fontCount = Object.keys(fonts).length;
+ var hasGA =
+ /google-analytics\\.com\\/(g|gtag)|googletagmanager\\.com\\/gtag\\/js|plausible\\.io\\/js|fathom\\.tech|piwik\\.|matomo\\./i.test(html)
+ || !!head.querySelector('script[src*="googletagmanager"]')
+ || !!head.querySelector('script[src*="plausible"]')
+ || !!head.querySelector('script[src*="fathom"]');
+ var hasSchema =
+ html.indexOf('schema.org/legalservice') >= 0
+ || html.indexOf('schema.org/attorney') >= 0
+ || html.indexOf('schema.org/lawyer') >= 0
+ || html.indexOf('"@type":"legalservice"') >= 0
+ || html.indexOf('"@type":"attorney"') >= 0
+ || html.indexOf('"@type":"lawyer"') >= 0;
+ return {
+ has_favicon: !!head.querySelector('link[rel*="icon"]'),
+ has_og_image: !!head.querySelector('meta[property="og:image"]'),
+ has_meta_viewport: !!head.querySelector('meta[name="viewport"]'),
+ has_analytics: hasGA,
+ has_h1: !!d.querySelector('h1'),
+ has_schema_org: hasSchema,
+ has_phone_visible: /\\b\\(?\\d{3}\\)?[\\s.\\-]?\\d{3}[\\s.\\-]?\\d{4}\\b/.test(bodyText),
+ font_count: fontCount,
+ image_count: d.querySelectorAll('img').length,
+ link_count: d.querySelectorAll('a[href]').length,
+ };
+ })()`) as any;
+
+ const png = await page.screenshot({ type: 'png', fullPage: false });
+ const filename = `firm-${firm.id}.png`;
+ fs.mkdirSync(SCREENSHOT_DIR, { recursive: true });
+ fs.writeFileSync(path.join(SCREENSHOT_DIR, filename), png);
+ const { primary, palette } = await extractPalette(png);
+
+ const fullSignals: Signals = {
+ status_code,
+ final_url,
+ has_https: final_url.startsWith('https://'),
+ ...signals,
+ page_size_bytes: bytesIn,
+ load_time_ms,
+ };
+ const { score, suggestions } = scoreAndSuggest(fullSignals, firm.name);
+ return { ok: true, signals: fullSignals, primary, palette, score, suggestions, screenshot: `/screenshots/${filename}` };
+ } finally {
+ await ctx.close().catch(() => {});
+ }
+}
+
+async function persist(firmId: number, url: string, r: Awaited<ReturnType<typeof auditOne>>) {
+ const s = r.signals;
+ await query(`
+ INSERT INTO site_audits (
+ organization_id, url, final_url, status_code, screenshot_path,
+ primary_color, palette,
+ has_https, has_favicon, has_og_image, has_meta_viewport, has_analytics,
+ has_h1, has_schema_org, has_phone_visible,
+ page_size_bytes, load_time_ms, font_count, image_count, link_count,
+ marketing_score, suggestions
+ ) VALUES (
+ $1,$2,$3,$4,$5,
+ $6,$7::jsonb,
+ $8,$9,$10,$11,$12,
+ $13,$14,$15,
+ $16,$17,$18,$19,$20,
+ $21,$22
+ )
+ `, [
+ firmId, url, s?.final_url || null, s?.status_code || null, r.screenshot || null,
+ r.primary || null, r.palette ? JSON.stringify(r.palette) : null,
+ s?.has_https ?? null, s?.has_favicon ?? null, s?.has_og_image ?? null, s?.has_meta_viewport ?? null, s?.has_analytics ?? null,
+ s?.has_h1 ?? null, s?.has_schema_org ?? null, s?.has_phone_visible ?? null,
+ s?.page_size_bytes ?? null, s?.load_time_ms ?? null, s?.font_count ?? null, s?.image_count ?? null, s?.link_count ?? null,
+ r.score ?? null, r.suggestions || null,
+ ]);
+}
+
+async function main() {
+ const firms = await loadFirms();
+ if (firms.length === 0) { console.log('[audit] no firms to audit'); await pool.end(); return; }
+
+ console.log(`[audit] auditing ${firms.length} firms (concurrency=${CONCURRENCY}, screenshots → ${SCREENSHOT_DIR})…`);
+ const browser = await chromium.launch({ headless: true });
+
+ try {
+ let seen = 0, ok = 0, errs = 0;
+ let cursor = 0;
+ const workers = Array.from({ length: CONCURRENCY }, async () => {
+ while (true) {
+ const idx = cursor++;
+ if (idx >= firms.length) break;
+ const firm = firms[idx];
+ seen++;
+ try {
+ const r = await auditOne(browser, firm);
+ await persist(firm.id, firm.website, r);
+ if (r.ok) {
+ ok++;
+ console.log(`[audit] ✓ #${firm.id} ${firm.name.slice(0,40).padEnd(40)} ${r.primary} score=${r.score}`);
+ } else {
+ console.log(`[audit] ✗ #${firm.id} ${firm.name.slice(0,40).padEnd(40)} ${r.error || 'no-200'}`);
+ }
+ } catch (e) {
+ errs++;
+ console.error(`[audit] err #${firm.id}: ${(e as Error).message}`);
+ }
+ if (seen % 25 === 0) console.log(`[audit] ${seen}/${firms.length} ok=${ok} errs=${errs}`);
+ }
+ });
+ await Promise.all(workers);
+
+ console.log(`[audit] done. seen=${seen} ok=${ok} errs=${errs}`);
+ } finally {
+ await browser.close();
+ }
+ await pool.end();
+}
+
+main().catch(async (err) => {
+ console.error('[audit] fatal:', err);
+ try { await pool.end(); } catch {}
+ process.exit(1);
+});
diff --git a/src/ingest/calbar_full_scrape.ts b/src/ingest/calbar_full_scrape.ts
new file mode 100644
index 0000000..3394a01
--- /dev/null
+++ b/src/ingest/calbar_full_scrape.ts
@@ -0,0 +1,342 @@
+/**
+ * CA State Bar — full attorney scrape via sequential Detail/{barNumber} URLs.
+ *
+ * Discovery: the SPA-shell-only theory in docs/blocked_sources.md was wrong.
+ * The Detail/{n} URLs render the full attorney profile inline in the initial
+ * HTML (no JS hydration needed). Plain fetch + regex parsing works.
+ *
+ * Scope: bar numbers 29960 → 360000 (per Steve, ~330k attorneys statewide).
+ * Polite cadence ~4 RPS aggregate (concurrency 5, ~200ms per worker).
+ *
+ * Persistence:
+ * - professionals (one row per bar_number, UNIQUE constraint dedups)
+ * - organizations (firm extracted from address line, type='law_firm')
+ * - professional_locations (links pro → org by bar_number + address)
+ * - phones (phone_type='office', source_url=detail page)
+ * - scrape_jobs.checkpoint = { last_id: <int> } for resume
+ *
+ * NOT extracted: email (page deliberately scatters 20 decoy emails behind
+ * inline `<span id="eN">` elements — disambiguation needs JS rendering, which
+ * would 30× the runtime). Skipped on purpose.
+ *
+ * CLI:
+ * npx tsx src/ingest/calbar_full_scrape.ts # full pass
+ * npx tsx src/ingest/calbar_full_scrape.ts --start 29960 --end 30100 # range
+ * npx tsx src/ingest/calbar_full_scrape.ts --resume # use checkpoint
+ * npx tsx src/ingest/calbar_full_scrape.ts --smoke # 10 attorneys
+ */
+import 'dotenv/config';
+import { pool, query, withTx } from '../db/pool.ts';
+
+const SOURCE_NAME = 'CA State Bar — License Search (Detail page scrape)';
+const BASE_URL = 'https://apps.calbar.ca.gov/attorney/Licensee/Detail';
+const USER_AGENT = process.env.USER_AGENT
+ || 'LawyerDirectoryBuilder/0.1 (research; contact: steveabramsdesigns@gmail.com)';
+const CONCURRENCY = Number(process.env.CALBAR_CONCURRENCY || 5);
+const PER_REQ_DELAY_MS = Number(process.env.CALBAR_DELAY_MS || 250);
+
+const argv = process.argv.slice(2);
+const SMOKE = argv.includes('--smoke');
+const RESUME = argv.includes('--resume');
+const START_IDX = argv.indexOf('--start');
+const END_IDX = argv.indexOf('--end');
+const START = START_IDX >= 0 ? Number(argv[START_IDX + 1]) : 29960;
+const END = END_IDX >= 0 ? Number(argv[END_IDX + 1]) : (SMOKE ? START + 9 : 360000);
+
+type Parsed = {
+ bar_number: string;
+ full_name: string | null;
+ first_name: string | null;
+ last_name: string | null;
+ middle_name: string | null;
+ license_status: string | null; // Active, Inactive, Disbarred, Resigned, etc.
+ firm_name: string | null; // First chunk of address up to first comma
+ address: string | null; // Address minus firm prefix
+ city: string | null;
+ state: string | null;
+ zip: string | null;
+ phone: string | null;
+};
+
+function decodeHtml(s: string): string {
+ return s.replace(/&/g, '&').replace(/ /g, ' ').replace(/./g, '.').replace(/@/g, '@')
+ .replace(/"/g, '"').replace(/'/g, "'").replace(/<[^>]+>/g, '').replace(/\s+/g, ' ').trim();
+}
+
+function parseDetail(html: string, barNumber: number): Parsed | null {
+ // Sanity-check the page actually has an attorney record (vs 404 / "no record found")
+ if (!/Attorney Profile|License Status/i.test(html)) return null;
+ if (/no record|not found|404/i.test(html.slice(0, 4000))) return null;
+
+ // Name + bar number e.g. "Sandra J. Shapiro" #29960
+ // The header pattern: <h2>Sandra J. Shapiro</h2> followed by #29960
+ // Loose regex — capture the visible name above "#<barNumber>"
+ const nameRx = new RegExp(`([A-Z][A-Za-z'.,\\-\\s]{1,80}?)\\s*#\\s*${barNumber}\\b`);
+ const nameM = html.match(nameRx);
+ let full_name: string | null = nameM ? decodeHtml(nameM[1]).replace(/\s+/g, ' ').trim() : null;
+ if (full_name && (full_name.length > 80 || /[<>]/.test(full_name))) full_name = null;
+
+ // Status e.g. "License Status: <span ...> Active </span>"
+ // Active attorneys' badges sometimes have empty inner text — assume Active if
+ // the page exists and no other status was captured.
+ let license_status: string | null = null;
+ const statusM = html.match(/License Status:[\s\S]{0,400}?<span[^>]*>([^<]+)<\/span>/i);
+ if (statusM) license_status = decodeHtml(statusM[1]).trim() || null;
+ if (!license_status) license_status = 'Active';
+
+ // Address line. Sample: "Address: Law Office of Sandra J Shapiro, 1700 Toledo Ave, Burlingame, CA 94010-5845"
+ let firm_name: string | null = null;
+ let address: string | null = null;
+ let city: string | null = null;
+ let state: string | null = null;
+ let zip: string | null = null;
+ const addrM = html.match(/Address:\s*([^<\n]+?)(?=<|\n|Phone:|Fax:|Email:)/i);
+ if (addrM) {
+ const full = decodeHtml(addrM[1]).trim();
+ // Strip trailing "Public Email Address" or junk
+ const cleaned = full.replace(/\s*Public Email Address\s*$/i, '').replace(/\s+/g, ' ').trim();
+ const parts = cleaned.split(',').map(p => p.trim()).filter(Boolean);
+ if (parts.length >= 3) {
+ // Last part is "CA 94010-5845" or "CA 94010"
+ const lastM = parts[parts.length - 1].match(/^([A-Z]{2})\s+(\d{5}(?:-\d{4})?)/);
+ if (lastM) { state = lastM[1]; zip = lastM[2].slice(0, 5); }
+ city = parts[parts.length - 2] || null;
+ // Identify the firm-name prefix:
+ // - if first part doesn't contain a number, treat it as firm name, rest as address
+ // - if first part starts with a digit, probably no firm name (firm is solo, just street)
+ if (parts.length >= 4 && !/^\d/.test(parts[0])) {
+ firm_name = parts[0];
+ address = parts.slice(1, parts.length - 2).join(', ');
+ } else {
+ address = parts.slice(0, parts.length - 2).join(', ');
+ }
+ } else {
+ address = cleaned;
+ }
+ }
+
+ // Phone — page is CRLF; capture digits up to whitespace
+ let phone: string | null = null;
+ const phoneM = html.match(/Phone:\s*([0-9()\-.\s]{10,25})/i);
+ if (phoneM) {
+ const digits = phoneM[1].replace(/\D/g, '');
+ if (digits.length === 10) phone = `(${digits.slice(0,3)}) ${digits.slice(3,6)}-${digits.slice(6)}`;
+ else if (digits.length === 11 && digits.startsWith('1')) {
+ const d = digits.slice(1);
+ phone = `(${d.slice(0,3)}) ${d.slice(3,6)}-${d.slice(6)}`;
+ }
+ }
+
+ // First/last/middle from full_name
+ let first_name: string | null = null, last_name: string | null = null, middle_name: string | null = null;
+ if (full_name) {
+ const tokens = full_name.replace(/\./g, '').split(/\s+/).filter(Boolean);
+ if (tokens.length >= 2) {
+ first_name = tokens[0];
+ last_name = tokens[tokens.length - 1];
+ if (tokens.length > 2) middle_name = tokens.slice(1, -1).join(' ');
+ } else if (tokens.length === 1) {
+ last_name = tokens[0];
+ }
+ }
+
+ return { bar_number: String(barNumber), full_name, first_name, last_name, middle_name, license_status, firm_name, address, city, state, zip, phone };
+}
+
+async function fetchOne(barNumber: number): Promise<Parsed | null> {
+ const { fetch } = await import('undici');
+ const r = await fetch(`${BASE_URL}/${barNumber}`, {
+ headers: { 'User-Agent': USER_AGENT, Accept: 'text/html' },
+ signal: AbortSignal.timeout(15000),
+ });
+ if (r.status === 404) return null;
+ if (!r.ok) {
+ if (r.status === 429) throw new Error('CALBAR_429');
+ return null;
+ }
+ const html = await r.text();
+ return parseDetail(html, barNumber);
+}
+
+async function ensureSource(): Promise<number> {
+ await query(`
+ INSERT INTO sources (source_name, source_type, base_url, terms_notes, allowed_method, rate_limit_rps)
+ VALUES ($1, 'official_registry', $2,
+ 'Public records by statute. Polite-use ~4 rps. Detail/{n} renders inline HTML.',
+ 'crawl', 4.0)
+ ON CONFLICT (source_name) DO NOTHING
+ `, [SOURCE_NAME, BASE_URL]);
+ const r = await query<{ id: number }>(`SELECT id FROM sources WHERE source_name = $1`, [SOURCE_NAME]);
+ return r.rows[0].id;
+}
+
+async function startJob(sourceId: number, label: string, checkpoint: any): Promise<number> {
+ const r = await query<{ id: number }>(`
+ INSERT INTO scrape_jobs (source_id, job_label, status, started_at, checkpoint)
+ VALUES ($1, $2, 'running', NOW(), $3::jsonb) RETURNING id
+ `, [sourceId, label, JSON.stringify(checkpoint)]);
+ return r.rows[0].id;
+}
+
+async function updateCheckpoint(jobId: number, lastId: number, kept: number, missed: number) {
+ await query(`UPDATE scrape_jobs SET checkpoint = $2::jsonb, records_inserted = $3, records_skipped = $4 WHERE id = $1`,
+ [jobId, JSON.stringify({ last_id: lastId, kept, missed }), kept, missed]);
+}
+
+async function finishJob(jobId: number, fields: Record<string, unknown>) {
+ const sets: string[] = []; const params: unknown[] = []; let i = 1;
+ for (const [k, v] of Object.entries(fields)) { sets.push(`${k} = $${i++}`); params.push(v); }
+ sets.push(`finished_at = NOW()`); params.push(jobId);
+ await query(`UPDATE scrape_jobs SET ${sets.join(', ')} WHERE id = $${i}`, params);
+}
+
+async function loadResumeStart(): Promise<number | null> {
+ // Take the highest last_id we've checkpointed across ANY prior run of this
+ // source (running, failed, aborted, or completed but partial). Last-write
+ // wins.
+ const r = await query<{ checkpoint: any }>(`
+ SELECT checkpoint FROM scrape_jobs
+ WHERE source_id = (SELECT id FROM sources WHERE source_name = $1)
+ AND checkpoint IS NOT NULL
+ ORDER BY (checkpoint->>'last_id')::int DESC NULLS LAST
+ LIMIT 1
+ `, [SOURCE_NAME]);
+ const last = r.rows[0]?.checkpoint?.last_id;
+ return typeof last === 'number' ? last + 1 : null;
+}
+
+async function persist(p: Parsed, sourceUrl: string) {
+ await withTx(async (client) => {
+ // Upsert professional by bar_number
+ const proR = await client.query<{ id: number }>(`
+ INSERT INTO professionals (full_name, first_name, last_name, middle_name, bar_number, license_status, source_confidence_score, confidence_tier)
+ VALUES ($1, $2, $3, $4, $5, $6, 0.95, 'high')
+ ON CONFLICT (bar_number) DO UPDATE SET
+ full_name = COALESCE(EXCLUDED.full_name, professionals.full_name),
+ first_name = COALESCE(EXCLUDED.first_name, professionals.first_name),
+ last_name = COALESCE(EXCLUDED.last_name, professionals.last_name),
+ middle_name = COALESCE(EXCLUDED.middle_name, professionals.middle_name),
+ license_status = COALESCE(EXCLUDED.license_status, professionals.license_status),
+ updated_at = NOW()
+ RETURNING id
+ `, [p.full_name, p.first_name, p.last_name, p.middle_name, p.bar_number, p.license_status]);
+ const proId = proR.rows[0].id;
+
+ // Phone
+ if (p.phone) {
+ await client.query(`
+ INSERT INTO phones (professional_id, phone, phone_type, source_url, last_verified_at)
+ VALUES ($1, $2, 'office', $3, NOW())
+ ON CONFLICT DO NOTHING
+ `, [proId, p.phone, sourceUrl]);
+ }
+
+ // Firm — only create org if firm_name is reasonable (not solo + plain address)
+ let orgId: number | null = null;
+ if (p.firm_name && p.firm_name.length >= 4 && !/^\d/.test(p.firm_name)) {
+ const addrLine = p.address || null;
+ // Look up or insert org by name + city
+ const found = await client.query<{ id: number }>(
+ `SELECT id FROM organizations WHERE type='law_firm' AND LOWER(name) = LOWER($1) AND LOWER(COALESCE(city,'')) = LOWER(COALESCE($2,'')) LIMIT 1`,
+ [p.firm_name, p.city || '']);
+ if (found.rowCount && found.rowCount > 0) {
+ orgId = found.rows[0].id;
+ } else {
+ const ins = await client.query<{ id: number }>(`
+ INSERT INTO organizations (name, type, address, city, state, zip, source_url)
+ VALUES ($1, 'law_firm', $2, $3, $4, $5, $6) RETURNING id
+ `, [p.firm_name, addrLine, p.city, p.state, p.zip, sourceUrl]);
+ orgId = ins.rows[0].id;
+ }
+ }
+
+ // Location link
+ if (orgId) {
+ const fullAddr = [p.address, p.city, p.state, p.zip].filter(Boolean).join(', ');
+ await client.query(`
+ INSERT INTO professional_locations (professional_id, organization_id, address, phone, source_url, is_primary)
+ VALUES ($1, $2, $3, $4, $5, true)
+ ON CONFLICT (professional_id, organization_id) WHERE organization_id IS NOT NULL DO UPDATE SET
+ address = COALESCE(EXCLUDED.address, professional_locations.address),
+ phone = COALESCE(EXCLUDED.phone, professional_locations.phone),
+ source_url = COALESCE(EXCLUDED.source_url, professional_locations.source_url),
+ last_verified_at = NOW()
+ `, [proId, orgId, fullAddr || null, p.phone, sourceUrl]);
+ }
+ });
+}
+
+async function main() {
+ const sourceId = await ensureSource();
+ let start = START;
+ if (RESUME) {
+ const resume = await loadResumeStart();
+ if (resume && resume > start) start = resume;
+ }
+ const total = END - start + 1;
+ console.log(`[calbar] scraping bar #${start} → #${END} (${total.toLocaleString()} attorneys, conc=${CONCURRENCY})…`);
+ const jobId = await startJob(sourceId, `calbar:detail:${start}-${END}`, { last_id: start - 1, kept: 0, missed: 0 });
+
+ let cursor = start;
+ let kept = 0, missed = 0, errs = 0, last_id = start - 1;
+ const lastIds = new Set<number>();
+
+ async function processOne(barNumber: number) {
+ const sourceUrl = `${BASE_URL}/${barNumber}`;
+ try {
+ const p = await fetchOne(barNumber);
+ if (!p || !p.full_name) { missed++; return; }
+ await persist(p, sourceUrl);
+ kept++;
+ } catch (e) {
+ const msg = (e as Error).message;
+ errs++;
+ if (msg === 'CALBAR_429') {
+ console.error('[calbar] hit 429 — backing off 60s');
+ await new Promise(r => setTimeout(r, 60000));
+ }
+ } finally {
+ lastIds.add(barNumber);
+ }
+ }
+
+ const workers = Array.from({ length: CONCURRENCY }, async () => {
+ while (true) {
+ const id = cursor++;
+ if (id > END) break;
+ await processOne(id);
+ await new Promise(r => setTimeout(r, PER_REQ_DELAY_MS));
+ // Periodic progress + checkpoint
+ if (lastIds.size >= 200) {
+ const max = Math.max(...lastIds);
+ lastIds.clear();
+ last_id = max;
+ await updateCheckpoint(jobId, last_id, kept, missed);
+ const done = max - start + 1;
+ const pct = ((done / total) * 100).toFixed(1);
+ console.log(`[calbar] #${max} done=${done.toLocaleString()}/${total.toLocaleString()} (${pct}%) kept=${kept} missed=${missed} errs=${errs}`);
+ }
+ }
+ });
+
+ try {
+ await Promise.all(workers);
+ } finally {
+ await finishJob(jobId, {
+ status: 'completed',
+ records_found: kept + missed,
+ records_inserted: kept,
+ records_skipped: missed,
+ error_message: errs > 0 ? `${errs} per-id errors` : null,
+ });
+ }
+
+ console.log(`[calbar] done. kept=${kept} missed=${missed} errs=${errs}`);
+ await pool.end();
+}
+
+main().catch(async (err) => {
+ console.error('[calbar] fatal:', err);
+ try { await pool.end(); } catch {}
+ process.exit(1);
+});
diff --git a/src/ingest/here_discover.ts b/src/ingest/here_discover.ts
new file mode 100644
index 0000000..ee94382
--- /dev/null
+++ b/src/ingest/here_discover.ts
@@ -0,0 +1,308 @@
+/**
+ * HERE Discover API enricher.
+ *
+ * For each existing law firm in the DB, query HERE's /v1/discover endpoint with
+ * the firm name + city + lat/lng (when available) and category=legal-services,
+ * then upsert phone / website / email rows from the returned record.
+ *
+ * Free tier: 30,000 transactions/month at platform.here.com (no card required).
+ * - 8,244 firms ≈ 27% of one month's quota
+ * - Polite cadence ~4 rps → ~35 min for the full pass
+ *
+ * Match policy: accept the top hit when its title fuzzy-matches our firm name.
+ * "Fuzzy" = lowercased, suffix-stripped, then either substring containment or
+ * Levenshtein distance ≤ 2 of normalized form.
+ *
+ * Env: HERE_API_KEY
+ *
+ * CLI:
+ * npx tsx src/ingest/here_discover.ts # full pass
+ * npx tsx src/ingest/here_discover.ts --smoke # first 5 firms only
+ * npx tsx src/ingest/here_discover.ts --limit 100 # cap at N firms
+ * npx tsx src/ingest/here_discover.ts --only-missing # skip firms that already have phone+website
+ */
+import 'dotenv/config';
+import { pool, query, withTx } from '../db/pool.ts';
+
+const SOURCE_NAME = 'HERE Discover API';
+const DISCOVER_URL = 'https://discover.search.hereapi.com/v1/discover';
+
+const API_KEY = process.env.HERE_API_KEY;
+const RATE_DELAY_MS = Number(process.env.HERE_DELAY_MS || 250);
+const USER_AGENT = process.env.USER_AGENT
+ || 'LawyerDirectoryBuilder/0.1 (research; contact: steveabramsdesigns@gmail.com)';
+
+type DiscoverItem = {
+ title: string;
+ id: string;
+ position?: { lat: number; lng: number };
+ address?: { label?: string; city?: string; postalCode?: string; state?: string; street?: string; houseNumber?: string };
+ contacts?: Array<{ phone?: Array<{ value: string }>; www?: Array<{ value: string }>; email?: Array<{ value: string }> }>;
+ categories?: Array<{ id: string; name: string; primary?: boolean }>;
+ distance?: number;
+};
+
+type Firm = {
+ id: number;
+ name: string;
+ address: string | null;
+ city: string | null;
+ zip: string | null;
+ lat: number | null;
+ lng: number | null;
+ has_phone: boolean;
+ has_website: boolean;
+};
+
+const argv = process.argv.slice(2);
+const SMOKE = argv.includes('--smoke');
+const ONLY_MISSING = argv.includes('--only-missing');
+const LIMIT_IDX = argv.indexOf('--limit');
+const LIMIT = LIMIT_IDX >= 0 ? Number(argv[LIMIT_IDX + 1]) : (SMOKE ? 5 : 0);
+
+function clean(s: string | undefined | null) {
+ if (!s) return null;
+ const t = String(s).trim();
+ return t.length === 0 ? null : t;
+}
+
+function normName(s: string): string {
+ return s.toLowerCase()
+ .replace(/[.,'"]/g, '')
+ .replace(/\b(llp|llc|p\.?c\.?|pllc|pa|chartered|inc|ltd|esq|attorney at law|law office of|law offices of|law firm)\b/g, '')
+ .replace(/&/g, 'and')
+ .replace(/\s+/g, ' ')
+ .trim();
+}
+
+// Quick distance-1 Levenshtein-ish containment test: accept if either string
+// fully contains the other after normalization, or shares a 4+ word prefix.
+function nameMatches(ours: string, theirs: string): boolean {
+ const a = normName(ours);
+ const b = normName(theirs);
+ if (!a || !b) return false;
+ if (a === b) return true;
+ if (a.includes(b) || b.includes(a)) return true;
+ const aw = a.split(' ').filter(Boolean);
+ const bw = b.split(' ').filter(Boolean);
+ if (aw.length >= 2 && bw.length >= 2) {
+ const lead = Math.min(2, aw.length, bw.length);
+ let shared = 0;
+ for (let i = 0; i < lead; i++) if (aw[i] === bw[i]) shared++;
+ if (shared >= 2) return true;
+ }
+ return false;
+}
+
+function normPhone(raw: string): string | null {
+ const digits = raw.replace(/\D/g, '');
+ if (digits.length === 11 && digits.startsWith('1')) {
+ const d = digits.slice(1);
+ return `(${d.slice(0,3)}) ${d.slice(3,6)}-${d.slice(6)}`;
+ }
+ if (digits.length === 10) {
+ return `(${digits.slice(0,3)}) ${digits.slice(3,6)}-${digits.slice(6)}`;
+ }
+ return null;
+}
+
+async function ensureSource(): Promise<number> {
+ await query(`
+ INSERT INTO sources (source_name, source_type, base_url, terms_notes, allowed_method, rate_limit_rps)
+ VALUES ($1, 'api', $2,
+ 'HERE Discover API. Free tier 30k transactions/month. ToS allows storage. Polite-use ~4 rps.',
+ 'api', 4.0)
+ ON CONFLICT (source_name) DO NOTHING
+ `, [SOURCE_NAME, DISCOVER_URL]);
+ const r = await query<{ id: number }>(`SELECT id FROM sources WHERE source_name = $1`, [SOURCE_NAME]);
+ return r.rows[0].id;
+}
+
+async function startJob(sourceId: number, label: string): Promise<number> {
+ const r = await query<{ id: number }>(`
+ INSERT INTO scrape_jobs (source_id, job_label, status, started_at)
+ VALUES ($1, $2, 'running', NOW()) RETURNING id
+ `, [sourceId, label]);
+ return r.rows[0].id;
+}
+
+async function finishJob(jobId: number, fields: Record<string, unknown>) {
+ const sets: string[] = [];
+ const params: unknown[] = [];
+ let i = 1;
+ for (const [k, v] of Object.entries(fields)) { sets.push(`${k} = $${i++}`); params.push(v); }
+ sets.push(`finished_at = NOW()`);
+ params.push(jobId);
+ await query(`UPDATE scrape_jobs SET ${sets.join(', ')} WHERE id = $${i}`, params);
+}
+
+async function loadFirms(): Promise<Firm[]> {
+ const where = ONLY_MISSING
+ ? `AND (
+ o.id NOT IN (SELECT organization_id FROM phones)
+ OR o.id NOT IN (SELECT organization_id FROM emails WHERE organization_id IS NOT NULL)
+ OR o.website IS NULL
+ )`
+ : '';
+ const lim = LIMIT > 0 ? `LIMIT ${LIMIT}` : '';
+ const r = await query<Firm>(`
+ SELECT
+ o.id, o.name, o.address, o.city, o.zip, o.lat, o.lng,
+ EXISTS(SELECT 1 FROM phones p WHERE p.organization_id = o.id) AS has_phone,
+ o.website IS NOT NULL AS has_website
+ FROM organizations o
+ WHERE o.type = 'law_firm'
+ AND o.address IS NOT NULL
+ ${where}
+ ORDER BY o.id
+ ${lim}
+ `);
+ return r.rows;
+}
+
+async function fetchDiscover(firm: Firm): Promise<DiscoverItem | null> {
+ const { fetch } = await import('undici');
+ const params = new URLSearchParams({
+ q: `${firm.name} ${firm.city || 'Los Angeles'}`,
+ limit: '3',
+ apiKey: API_KEY!,
+ });
+ if (firm.lat != null && firm.lng != null) {
+ params.set('at', `${firm.lat},${firm.lng}`);
+ } else {
+ // LA County centroid as a fallback search bias.
+ params.set('at', '34.0522,-118.2437');
+ }
+ // Restrict to legal-services category (HERE category id 600-6500).
+ params.set('categoryFilter', '600-6500');
+ const url = `${DISCOVER_URL}?${params.toString()}`;
+
+ const r = await fetch(url, {
+ headers: { 'User-Agent': USER_AGENT, Accept: 'application/json' },
+ signal: AbortSignal.timeout(20000),
+ });
+ if (!r.ok) {
+ if (r.status === 429) throw new Error('HERE_429_RATE_LIMIT');
+ if (r.status === 401 || r.status === 403) throw new Error(`HERE_AUTH_${r.status}`);
+ return null;
+ }
+ const j = await r.json() as { items?: DiscoverItem[] };
+ if (!j.items || j.items.length === 0) return null;
+
+ for (const item of j.items) {
+ if (nameMatches(firm.name, item.title)) return item;
+ }
+ return null;
+}
+
+async function applyMatch(firm: Firm, item: DiscoverItem, sourceId: number): Promise<{ phones: number; emails: number; site: boolean }> {
+ const sourceUrl = `https://wego.here.com/p/${item.id}`;
+ const phones: string[] = [];
+ const emails: string[] = [];
+ let website: string | null = null;
+
+ for (const c of item.contacts || []) {
+ for (const p of c.phone || []) { const np = normPhone(p.value); if (np) phones.push(np); }
+ for (const w of c.www || []) { if (!website) website = clean(w.value); }
+ for (const e of c.email || []) { const ne = clean(e.value); if (ne) emails.push(ne.toLowerCase()); }
+ }
+
+ let phoneCount = 0, emailCount = 0;
+ let siteSet = false;
+
+ await withTx(async (client) => {
+ if (website && !firm.has_website) {
+ await client.query(
+ `UPDATE organizations SET website = $1, updated_at = NOW() WHERE id = $2 AND website IS NULL`,
+ [website, firm.id]);
+ siteSet = true;
+ }
+ for (const p of phones) {
+ const ins = await client.query(
+ `INSERT INTO phones (organization_id, phone, phone_type, source_url, last_verified_at)
+ VALUES ($1, $2, 'office', $3, NOW())
+ ON CONFLICT DO NOTHING`,
+ [firm.id, p, sourceUrl]);
+ phoneCount += ins.rowCount || 0;
+ }
+ for (const e of emails) {
+ const ins = await client.query(
+ `INSERT INTO emails (organization_id, email, email_type, source_url, last_verified_at)
+ VALUES ($1, $2, 'office', $3, NOW())
+ ON CONFLICT DO NOTHING`,
+ [firm.id, e, sourceUrl]);
+ emailCount += ins.rowCount || 0;
+ }
+ // Provenance.
+ await client.query(`
+ INSERT INTO raw_records (source_id, source_url, entity_type, entity_id, raw_json, fetched_at, hash)
+ VALUES ($1, $2, 'organization', $3, $4::jsonb, NOW(), encode(sha256($4::bytea), 'hex'))
+ ON CONFLICT (source_id, hash) DO NOTHING
+ `, [sourceId, sourceUrl, firm.id, JSON.stringify(item)]);
+ });
+
+ return { phones: phoneCount, emails: emailCount, site: siteSet };
+}
+
+async function main() {
+ if (!API_KEY) {
+ console.error('[here] HERE_API_KEY not set in .env — sign up at https://platform.here.com');
+ process.exit(2);
+ }
+
+ const sourceId = await ensureSource();
+ const label = SMOKE ? 'here:discover:smoke' : (LIMIT > 0 ? `here:discover:limit-${LIMIT}` : 'here:discover:la-firms');
+ const jobId = await startJob(sourceId, label);
+
+ const firms = await loadFirms();
+ console.log(`[here] enriching ${firms.length} firms…`);
+
+ let seen = 0, matched = 0, phonesNew = 0, emailsNew = 0, sitesNew = 0, errs = 0;
+ try {
+ for (const firm of firms) {
+ seen++;
+ try {
+ const item = await fetchDiscover(firm);
+ if (item) {
+ matched++;
+ const r = await applyMatch(firm, item, sourceId);
+ phonesNew += r.phones;
+ emailsNew += r.emails;
+ if (r.site) sitesNew++;
+ }
+ } catch (e) {
+ errs++;
+ const msg = (e as Error).message;
+ if (msg === 'HERE_429_RATE_LIMIT') {
+ console.error(`[here] hit 429 — backing off 30s`);
+ await new Promise(r => setTimeout(r, 30000));
+ } else if (msg.startsWith('HERE_AUTH_')) {
+ console.error(`[here] auth failure: ${msg} — aborting`);
+ throw e;
+ } else {
+ console.error(`[here] firm=${firm.id} err: ${msg}`);
+ }
+ }
+ if (seen % 100 === 0) console.log(`[here] ${seen}/${firms.length} matched=${matched} phones+=${phonesNew} sites+=${sitesNew} emails+=${emailsNew}`);
+ await new Promise(r => setTimeout(r, RATE_DELAY_MS));
+ }
+ } finally {
+ await finishJob(jobId, {
+ status: errs > firms.length / 2 ? 'failed' : 'completed',
+ records_found: matched,
+ records_inserted: phonesNew + emailsNew,
+ records_updated: sitesNew,
+ records_skipped: seen - matched,
+ error_message: errs > 0 ? `${errs} per-firm errors` : null,
+ });
+ }
+
+ console.log(`[here] done. seen=${seen} matched=${matched} phones+=${phonesNew} sites+=${sitesNew} emails+=${emailsNew} errs=${errs}`);
+ await pool.end();
+}
+
+main().catch(async (err) => {
+ console.error('[here] fatal:', err);
+ try { await pool.end(); } catch {}
+ process.exit(1);
+});
diff --git a/src/scripts/click_through.ts b/src/scripts/click_through.ts
new file mode 100644
index 0000000..c1316c6
--- /dev/null
+++ b/src/scripts/click_through.ts
@@ -0,0 +1,131 @@
+/**
+ * Click-through smoke test against the LIVE site at https://lawyers.agentabrams.com.
+ *
+ * Walks through the three primary flows:
+ * 1. Landing → "Find a lawyer" → fill form → submit → thanks
+ * 2. Landing → "List your firm" → /signup with autocomplete + submit
+ * 3. Admin login → /admin → /admin/leads → status change
+ *
+ * Screenshots written to tmp/click-through/ ; an inline report is printed.
+ */
+import { chromium } from 'playwright';
+import fs from 'node:fs';
+import path from 'node:path';
+
+const BASE = process.env.CLICK_BASE || 'https://lawyers.agentabrams.com';
+const ADMIN_EMAIL = process.env.CLICK_ADMIN_EMAIL || '';
+const ADMIN_PASS = process.env.CLICK_ADMIN_PASS || '';
+if (!ADMIN_EMAIL || !ADMIN_PASS) {
+ console.error('[click-through] CLICK_ADMIN_EMAIL and CLICK_ADMIN_PASS must be set in env');
+ process.exit(2);
+}
+const OUT = path.resolve(process.cwd(), 'tmp/click-through');
+fs.mkdirSync(OUT, { recursive: true });
+
+type Step = { name: string; ok: boolean; note?: string; shot?: string };
+const steps: Step[] = [];
+
+async function main() {
+ const browser = await chromium.launch({ headless: true });
+ try {
+ const ctx = await browser.newContext({ viewport: { width: 1280, height: 900 } });
+ const page = await ctx.newPage();
+
+ // ── Flow 1: consumer lead funnel ─────────────────────────────────────────
+ try {
+ await page.goto(BASE + '/', { waitUntil: 'load', timeout: 20000 });
+ const title = await page.title();
+ const hasHero = await page.locator('h1').first().isVisible();
+ const findCta = page.locator('a[href="/find-a-lawyer"]').first();
+ const ctaHref = await findCta.getAttribute('href');
+ await page.screenshot({ path: path.join(OUT, '01-landing.png') });
+ steps.push({ name: 'Landing renders', ok: hasHero && title.length > 0, note: `title="${title}" cta=${ctaHref}`, shot: '01-landing.png' });
+
+ await findCta.click();
+ await page.waitForURL('**/find-a-lawyer', { timeout: 10000 });
+ await page.screenshot({ path: path.join(OUT, '02-find-form.png') });
+ steps.push({ name: 'Click "Find a lawyer" → /find-a-lawyer', ok: page.url().includes('/find-a-lawyer'), shot: '02-find-form.png' });
+
+ await page.fill('input[name="full_name"]', 'Smoke Test User');
+ await page.fill('input[name="email"]', `smoke-${Date.now()}@test.com`);
+ await page.fill('input[name="phone"]', '(415) 555-0100');
+ await page.selectOption('select[name="practice_area"]', 'personal_injury');
+ await page.selectOption('select[name="urgency"]', 'within_week');
+ await page.fill('input[name="zip"]', '90210');
+ await page.fill('textarea[name="case_description"]', 'Smoke test from the click-through script');
+ await page.screenshot({ path: path.join(OUT, '03-form-filled.png') });
+ await page.click('button[type="submit"]');
+ await page.waitForURL('**/find-a-lawyer/thanks', { timeout: 10000 });
+ await page.screenshot({ path: path.join(OUT, '04-thanks.png') });
+ steps.push({ name: 'Submit lead form → /thanks page', ok: page.url().includes('/thanks'), shot: '04-thanks.png' });
+ } catch (e) {
+ steps.push({ name: 'Lead funnel', ok: false, note: (e as Error).message.slice(0, 200) });
+ }
+
+ // ── Flow 2: signup form + autocomplete ───────────────────────────────────
+ try {
+ await page.goto(BASE + '/signup', { waitUntil: 'load' });
+ await page.screenshot({ path: path.join(OUT, '05-signup.png') });
+ const fields = await Promise.all([
+ page.locator('input[name="firm_name"]').isVisible(),
+ page.locator('input[name="website"]').isVisible(),
+ page.locator('input[name="full_name"]').isVisible(),
+ page.locator('input[name="email"]').isVisible(),
+ page.locator('input[name="phone"]').isVisible(),
+ ]);
+ const allVisible = fields.every(Boolean);
+ steps.push({ name: '/signup shows all 5 large fields', ok: allVisible, note: `firm/site/name/email/phone visible: ${fields.join(',')}`, shot: '05-signup.png' });
+
+ // Autocomplete: type 'Emrani' and wait for dropdown
+ await page.fill('input#full_name', 'Emrani');
+ await page.waitForSelector('.ac-row', { timeout: 4000 });
+ const acCount = await page.locator('.ac-row').count();
+ await page.screenshot({ path: path.join(OUT, '06-autocomplete.png') });
+ await page.locator('.ac-row').first().click();
+ const barFilled = await page.inputValue('input#bar_number');
+ const firmFilled = await page.inputValue('input#firm_name');
+ await page.screenshot({ path: path.join(OUT, '07-autocomplete-picked.png') });
+ steps.push({ name: 'Autocomplete pick auto-fills bar# + firm', ok: barFilled.length > 0 && firmFilled.length > 0, note: `bar="${barFilled}" firm="${firmFilled}" rows=${acCount}`, shot: '07-autocomplete-picked.png' });
+ } catch (e) {
+ steps.push({ name: 'Signup autocomplete', ok: false, note: (e as Error).message.slice(0, 200) });
+ }
+
+ // ── Flow 3: admin login + leads queue ────────────────────────────────────
+ try {
+ await page.goto(BASE + '/login', { waitUntil: 'load' });
+ await page.fill('input[name="email"]', ADMIN_EMAIL);
+ await page.fill('input[name="password"]', ADMIN_PASS);
+ await page.click('button[type="submit"]');
+ await page.waitForURL(/\/dashboard/, { timeout: 10000 });
+ await page.screenshot({ path: path.join(OUT, '08-dashboard.png') });
+ steps.push({ name: 'Admin login → /dashboard', ok: page.url().includes('/dashboard'), shot: '08-dashboard.png' });
+
+ await page.goto(BASE + '/admin', { waitUntil: 'load' });
+ await page.screenshot({ path: path.join(OUT, '09-admin-users.png') });
+ const userCount = await page.locator('table tbody tr').count();
+ steps.push({ name: '/admin shows user table', ok: userCount > 0, note: `rows=${userCount}`, shot: '09-admin-users.png' });
+
+ await page.goto(BASE + '/admin/leads', { waitUntil: 'load' });
+ await page.screenshot({ path: path.join(OUT, '10-admin-leads.png') });
+ const leadCount = await page.locator('table tbody tr').count();
+ steps.push({ name: '/admin/leads shows the smoke lead', ok: leadCount > 0, note: `leads=${leadCount}`, shot: '10-admin-leads.png' });
+ } catch (e) {
+ steps.push({ name: 'Admin flow', ok: false, note: (e as Error).message.slice(0, 200) });
+ }
+
+ } finally {
+ await browser.close();
+ }
+
+ // ── Report ───────────────────────────────────────────────────────────────
+ const pass = steps.filter(s => s.ok).length;
+ console.log(`\n══ click-through report ══ ${pass}/${steps.length} pass\n`);
+ for (const s of steps) {
+ const mark = s.ok ? '✓' : '✗';
+ console.log(`${mark} ${s.name}${s.note ? ' · ' + s.note : ''}${s.shot ? ' → ' + s.shot : ''}`);
+ }
+ console.log(`\nScreenshots: ${OUT}/`);
+ if (pass < steps.length) process.exit(1);
+}
+
+main().catch(e => { console.error('fatal:', e); process.exit(1); });
diff --git a/src/scripts/create_admin.ts b/src/scripts/create_admin.ts
new file mode 100644
index 0000000..231eaf8
--- /dev/null
+++ b/src/scripts/create_admin.ts
@@ -0,0 +1,36 @@
+/**
+ * Seed an admin user. Re-runs are idempotent — if email exists, password is
+ * reset and role is forced to admin.
+ *
+ * CREATE_ADMIN_PASSWORD='...' npm run create-admin -- --email steve@designerwallcoverings.com
+ */
+import 'dotenv/config';
+import { pool, query } from '../db/pool.ts';
+import { hashPassword } from '../server/auth.ts';
+
+function arg(name: string): string | undefined {
+ const i = process.argv.indexOf('--' + name);
+ return i >= 0 ? process.argv[i + 1] : undefined;
+}
+
+(async () => {
+ const email = arg('email');
+ const password = process.env.CREATE_ADMIN_PASSWORD || process.env.ADMIN_PASSWORD;
+ const fullName = arg('name') || 'Steve Abrams';
+ if (!email || !password) {
+ console.error('usage: CREATE_ADMIN_PASSWORD=<p> create-admin --email <e> [--name <n>]');
+ process.exit(2);
+ }
+ const hash = await hashPassword(password);
+ const r = await query<{ id: number; created: boolean }>(`
+ INSERT INTO app_users (email, password_hash, full_name, role, tier, plan, status)
+ VALUES (LOWER($1), $2, $3, 'admin', 'admin', 'pro', 'active')
+ ON CONFLICT (email) DO UPDATE SET
+ password_hash = EXCLUDED.password_hash,
+ full_name = COALESCE(app_users.full_name, EXCLUDED.full_name),
+ role = 'admin', tier = 'admin', status = 'active', updated_at = NOW()
+ RETURNING id, (xmax = 0) AS created
+ `, [email, hash, fullName]);
+ console.log(`✓ admin ${r.rows[0].created ? 'created' : 'updated'}: ${email} (id=${r.rows[0].id})`);
+ await pool.end();
+})().catch(async (e) => { console.error(e); try { await pool.end(); } catch {} process.exit(1); });
diff --git a/src/scripts/probe_calbar.ts b/src/scripts/probe_calbar.ts
new file mode 100644
index 0000000..9dc2340
--- /dev/null
+++ b/src/scripts/probe_calbar.ts
@@ -0,0 +1,72 @@
+/**
+ * Network probe — open one CalBar Detail/{id} page in Playwright and log every
+ * XHR/fetch request and response so we can identify the data endpoint that
+ * powers the SPA. Once found we'll replay it via plain fetch for all ~330k
+ * bar numbers without keeping a browser session per request.
+ */
+import { chromium } from 'playwright';
+
+const SAMPLE_BAR_NUMBER = process.argv[2] || '29960';
+const URL = `https://apps.calbar.ca.gov/attorney/Licensee/Detail/${SAMPLE_BAR_NUMBER}`;
+
+(async () => {
+ const browser = await chromium.launch({ headless: true });
+ try {
+ const ctx = await browser.newContext({
+ userAgent: 'LawyerDirectoryBuilder/0.1 (research; contact: steveabramsdesigns@gmail.com)',
+ });
+ const page = await ctx.newPage();
+
+ const requests: Array<{ url: string; method: string; resourceType: string; headers: Record<string,string> }> = [];
+ page.on('request', req => {
+ const u = req.url();
+ if (u.includes('calbar') || u.includes('apps.calbar')) {
+ requests.push({ url: u, method: req.method(), resourceType: req.resourceType(), headers: req.headers() });
+ }
+ });
+
+ const responses: Array<{ url: string; status: number; ct: string; bodyPreview: string; bodyLen: number }> = [];
+ page.on('response', async res => {
+ const u = res.url();
+ if (!u.includes('calbar') && !u.includes('apps.calbar')) return;
+ const ct = res.headers()['content-type'] || '';
+ if (!ct.includes('json') && !ct.includes('xml') && !ct.includes('javascript')) return;
+ try {
+ const body = await res.text();
+ responses.push({
+ url: u, status: res.status(), ct,
+ bodyPreview: body.slice(0, 600),
+ bodyLen: body.length,
+ });
+ } catch {}
+ });
+
+ console.log(`probing ${URL}…`);
+ await page.goto(URL, { waitUntil: 'networkidle', timeout: 30000 });
+ await page.waitForTimeout(2000); // catch lazy XHRs
+
+ console.log('\n=== JSON/XML/JS responses (calbar domain) ===');
+ for (const r of responses) {
+ console.log(`\n${r.status} ${r.url}`);
+ console.log(` content-type: ${r.ct} size: ${r.bodyLen}`);
+ console.log(` body preview:\n ${r.bodyPreview.replace(/\n/g, '\n ').slice(0, 500)}`);
+ }
+
+ console.log('\n=== content of the rendered page ===');
+ const title = await page.title();
+ console.log(`title: ${title}`);
+ // Grab the innerText of the rendered detail panel if present
+ const text = await page.evaluate(`(function(){
+ var sel = ['#moduleAttorneySearchList', '#detail', '.lic-info', '.attorney-detail', 'main'];
+ for (var i=0;i<sel.length;i++) {
+ var el = document.querySelector(sel[i]);
+ if (el && el.innerText) return { selector: sel[i], text: el.innerText.slice(0, 800) };
+ }
+ return { selector: 'body', text: (document.body.innerText||'').slice(0, 800) };
+ })()`) as any;
+ console.log(`from "${text.selector}":\n${text.text}`);
+
+ } finally {
+ await browser.close();
+ }
+})().catch(e => { console.error('probe fail:', e); process.exit(1); });
diff --git a/src/scripts/run_overnight.ts b/src/scripts/run_overnight.ts
index 7ad05d4..05d8ba8 100644
--- a/src/scripts/run_overnight.ts
+++ b/src/scripts/run_overnight.ts
@@ -1,17 +1,25 @@
/**
- * Overnight orchestrator — runs the importers in sequence, checkpointing progress
- * via scrape_jobs.checkpoint so a kill mid-run resumes cleanly on the next start.
+ * Overnight orchestrator — runs every importer + enricher in sequence.
*
- * Sources wired:
- * 1. OSM Overpass refresh (LA County office=lawyer / amenity=lawyer)
- * 2. Wikidata SPARQL (notable LA firms — BigLaw filler)
- * 3. Aggregation pass (recompute attorney_count + concentration views)
+ * Sources wired (in order):
+ * 1. OSM Overpass refresh (LA County office=lawyer / amenity=lawyer)
+ * 2. Wikidata SPARQL (notable LA firms — BigLaw filler)
+ * 3. LA City Business Licenses (Socrata, NAICS 5411)
+ * 4. OSM name-match (broader name-based query)
+ * 5. Wikipedia firm articles (LA County law-firm category)
+ * 6. Wikipedia lawyer biographies (LA-county lawyers → professionals table)
+ * 7. Aggregation pass (firm_size_band recompute)
*
- * Sources blocked by anti-bot (NOT run; documented separately):
- * - CA State Bar attorney search → SPA, requires Playwright (see docs/blocked_sources.md)
+ * Enrichers (in order, AFTER importers so they have fresh records):
+ * 8. Nominatim geocoder (fill lat/lng for new addresses)
+ * 9. DNS website discovery (candidate-domain check + name verify)
+ * 10. Firm-website contact pages (regex phone/email out of /contact pages)
+ *
+ * Blocked sources (NOT run; documented in docs/blocked_sources.md):
+ * - CA State Bar attorney search → SPA, requires Playwright
* - CA SOS bizfile → POST endpoint requires recaptcha token
*
- * Loop policy: run once, exit 0. Wrap with `pm2`/`watch`/cron if continuous mode is desired.
+ * Loop policy: run once, exit 0. Wrap with launchd if continuous mode is desired.
*/
import 'dotenv/config';
import { spawn } from 'node:child_process';
@@ -76,11 +84,20 @@ async function summary() {
async function main() {
out('=== overnight run start ===');
- const code1 = await run('osm-overpass', 'src/ingest/osm_overpass.ts');
- const code2 = await run('wikidata', 'src/ingest/wikidata.ts');
+ const codes: Record<string, number> = {};
+ codes.osm = await run('osm-overpass', 'src/ingest/osm_overpass.ts');
+ codes.wd = await run('wikidata', 'src/ingest/wikidata.ts');
+ codes.lacity = await run('la-city-licenses', 'src/ingest/la_city_business_licenses.ts');
+ codes.osmName = await run('osm-name-search', 'src/ingest/osm_name_search.ts');
+ codes.wikiFirms = await run('wikipedia-infobox', 'src/ingest/wikipedia_infobox.ts');
+ codes.wikiPros = await run('wikipedia-lawyers', 'src/ingest/wikipedia_lawyers.ts');
await aggregationPass();
+ codes.geocode = await run('nominatim-geocoder', 'src/enrich/geocode_nominatim.ts');
+ codes.discover = await run('discover-websites', 'src/enrich/discover_websites.ts');
+ codes.contacts = await run('firm-website-contacts','src/enrich/firm_website_contacts.ts');
await summary();
- out(`=== overnight run end (osm=${code1}, wd=${code2}) ===`);
+ const summary_line = Object.entries(codes).map(([k, v]) => `${k}=${v}`).join(' ');
+ out(`=== overnight run end (${summary_line}) ===`);
await pool.end();
log.end();
}
diff --git a/src/scripts/setup_stripe_lawyer_pro_price.ts b/src/scripts/setup_stripe_lawyer_pro_price.ts
new file mode 100644
index 0000000..55ce3e7
--- /dev/null
+++ b/src/scripts/setup_stripe_lawyer_pro_price.ts
@@ -0,0 +1,75 @@
+/**
+ * One-shot setup: create (or reuse) the Stripe Product + recurring Price
+ * for the Counsel & Bar Lawyer Pro $29/mo subscription tier and print the
+ * price ID. Caller saves it via the secrets-manager CLI as
+ * STRIPE_PRICE_LAWYER_PRO_MONTHLY.
+ *
+ * Idempotent: if a "Counsel & Bar — Lawyer Pro" product already exists with
+ * an active monthly USD price at $29, we reuse it. Otherwise we create both.
+ *
+ * npx tsx src/scripts/setup_stripe_lawyer_pro_price.ts
+ */
+import 'dotenv/config';
+import Stripe from 'stripe';
+
+const PRODUCT_NAME = 'Counsel & Bar — Lawyer Pro';
+const UNIT_AMOUNT = 2900; // $29.00 USD
+const CURRENCY = 'usd';
+const INTERVAL: 'month' | 'year' = 'month';
+
+const SK = process.env.STRIPE_SECRET_KEY;
+if (!SK || !SK.startsWith('sk_')) {
+ console.error('STRIPE_SECRET_KEY missing or invalid in .env');
+ process.exit(2);
+}
+const stripe = new Stripe(SK);
+
+function fmtCents(c: number): string { return `$${(c / 100).toFixed(2)}`; }
+
+(async () => {
+ // 1. Find or create the Product
+ let product: Stripe.Product | null = null;
+ for await (const p of stripe.products.list({ active: true, limit: 100 })) {
+ if (p.name === PRODUCT_NAME) { product = p; break; }
+ }
+ if (!product) {
+ console.log(`creating product "${PRODUCT_NAME}"…`);
+ product = await stripe.products.create({
+ name: PRODUCT_NAME,
+ description: 'Lawyer-tier subscription on Counsel & Bar (lawyers.agentabrams.com). Includes profile claim, review responses, client DMs, lead alerts.',
+ metadata: { sku: 'lawyer-pro-monthly', site: 'lawyers.agentabrams.com' },
+ });
+ } else {
+ console.log(`found existing product ${product.id} (${product.name})`);
+ }
+
+ // 2. Find an existing matching active Price, else create one
+ const prices = await stripe.prices.list({ product: product.id, active: true, limit: 100 });
+ let price: Stripe.Price | null = null;
+ for (const pr of prices.data) {
+ if (pr.unit_amount === UNIT_AMOUNT && pr.currency === CURRENCY && pr.recurring?.interval === INTERVAL) {
+ price = pr; break;
+ }
+ }
+ if (!price) {
+ console.log(`creating recurring price ${fmtCents(UNIT_AMOUNT)} ${CURRENCY.toUpperCase()} / ${INTERVAL}…`);
+ price = await stripe.prices.create({
+ product: product.id,
+ unit_amount: UNIT_AMOUNT,
+ currency: CURRENCY,
+ recurring: { interval: INTERVAL },
+ lookup_key: 'lawyer_pro_monthly',
+ metadata: { tier: 'lawyer', sku: 'lawyer-pro-monthly' },
+ });
+ } else {
+ console.log(`found existing price ${price.id} (${fmtCents(price.unit_amount!)} ${price.currency.toUpperCase()} / ${price.recurring?.interval})`);
+ }
+
+ console.log('\n=== Stripe Lawyer Pro price ready ===');
+ console.log('product id: ', product.id);
+ console.log('price id: ', price.id);
+ console.log('amount: ', fmtCents(price.unit_amount!), price.currency.toUpperCase(), '/', price.recurring?.interval);
+ console.log('lookup key: ', price.lookup_key || '(none)');
+ console.log('\nNext step (run from your shell):');
+ console.log(` node ~/Projects/secrets-manager/cli.js add STRIPE_PRICE_LAWYER_PRO_MONTHLY '${price.id}'\n`);
+})().catch(e => { console.error('setup failed:', e); process.exit(1); });
diff --git a/src/scripts/setup_stripe_tier_webhook.ts b/src/scripts/setup_stripe_tier_webhook.ts
new file mode 100644
index 0000000..d6c2329
--- /dev/null
+++ b/src/scripts/setup_stripe_tier_webhook.ts
@@ -0,0 +1,57 @@
+/**
+ * One-shot setup: create the Stripe webhook endpoint for the Lawyer Pro
+ * subscription tier (handled by src/server/billing.ts at /webhooks/stripe-tier)
+ * and print the signing secret.
+ *
+ * This is a SEPARATE Stripe endpoint from the original $99 EZ Upgrade webhook
+ * (which lives at /webhooks/stripe and is set up by setup_stripe_webhook.ts).
+ * Each Stripe endpoint has its own signing secret — sharing one secret would
+ * cause one of the two webhook handlers to silently reject every event.
+ *
+ * Idempotent: if a webhook for /webhooks/stripe-tier already exists, we delete
+ * it first so the new endpoint comes back with a fresh signing secret.
+ *
+ * npx tsx src/scripts/setup_stripe_tier_webhook.ts
+ */
+import 'dotenv/config';
+import Stripe from 'stripe';
+
+const URL = 'https://lawyers.agentabrams.com/webhooks/stripe-tier';
+const EVENTS: Stripe.WebhookEndpointCreateParams.EnabledEvent[] = [
+ 'customer.subscription.created',
+ 'customer.subscription.updated',
+ 'customer.subscription.deleted',
+ 'invoice.payment_failed',
+];
+
+const SK = process.env.STRIPE_SECRET_KEY;
+if (!SK || !SK.startsWith('sk_')) {
+ console.error('STRIPE_SECRET_KEY missing or invalid in .env');
+ process.exit(2);
+}
+const stripe = new Stripe(SK);
+
+(async () => {
+ // Find any existing endpoint at this URL — delete to force fresh secret
+ const existing = await stripe.webhookEndpoints.list({ limit: 100 });
+ for (const e of existing.data) {
+ if (e.url === URL) {
+ console.log(`found existing tier webhook ${e.id} — deleting to refresh signing secret`);
+ await stripe.webhookEndpoints.del(e.id);
+ }
+ }
+
+ const created = await stripe.webhookEndpoints.create({
+ url: URL,
+ enabled_events: EVENTS,
+ description: 'Counsel & Bar — Lawyer Pro $29/mo subscription tier (lawyers.agentabrams.com/webhooks/stripe-tier)',
+ });
+
+ console.log('\n=== tier webhook endpoint created ===');
+ console.log('id: ', created.id);
+ console.log('url: ', created.url);
+ console.log('events: ', created.enabled_events.length, '·', created.enabled_events.join(', '));
+ console.log('signing secret:', created.secret);
+ console.log('\nNext step (run from your shell):');
+ console.log(` node ~/Projects/secrets-manager/cli.js add STRIPE_TIER_WEBHOOK_SECRET '${created.secret}'\n`);
+})().catch(e => { console.error('setup failed:', e); process.exit(1); });
diff --git a/src/scripts/setup_stripe_webhook.ts b/src/scripts/setup_stripe_webhook.ts
new file mode 100644
index 0000000..cb336a8
--- /dev/null
+++ b/src/scripts/setup_stripe_webhook.ts
@@ -0,0 +1,52 @@
+/**
+ * One-shot setup: create (or refresh) the Stripe webhook endpoint for
+ * lawyers.agentabrams.com and print the signing secret. Caller saves the
+ * secret via the secrets-manager CLI.
+ *
+ * Idempotent: if a webhook for the same URL already exists, we delete it
+ * first so the new endpoint comes back with a fresh signing secret.
+ *
+ * npx tsx src/scripts/setup_stripe_webhook.ts
+ */
+import 'dotenv/config';
+import Stripe from 'stripe';
+
+const URL = 'https://lawyers.agentabrams.com/webhooks/stripe';
+const EVENTS: Stripe.WebhookEndpointCreateParams.EnabledEvent[] = [
+ 'checkout.session.completed',
+ 'checkout.session.expired',
+ 'charge.refunded',
+ 'payment_intent.payment_failed',
+];
+
+const SK = process.env.STRIPE_SECRET_KEY;
+if (!SK || !SK.startsWith('sk_')) {
+ console.error('STRIPE_SECRET_KEY missing or invalid in .env');
+ process.exit(2);
+}
+const stripe = new Stripe(SK);
+
+(async () => {
+ // Find any existing endpoint at this URL — delete to force fresh secret
+ const existing = await stripe.webhookEndpoints.list({ limit: 100 });
+ for (const e of existing.data) {
+ if (e.url === URL) {
+ console.log(`found existing webhook ${e.id} — deleting to refresh signing secret`);
+ await stripe.webhookEndpoints.del(e.id);
+ }
+ }
+
+ const created = await stripe.webhookEndpoints.create({
+ url: URL,
+ enabled_events: EVENTS,
+ description: 'Counsel & Bar — lawyers.agentabrams.com production',
+ });
+
+ console.log('\n=== webhook endpoint created ===');
+ console.log('id: ', created.id);
+ console.log('url: ', created.url);
+ console.log('events: ', created.enabled_events.length);
+ console.log('signing secret:', created.secret);
+ console.log('\nNext step (run from your shell):');
+ console.log(` node ~/Projects/secrets-manager/cli.js add STRIPE_WEBHOOK_SECRET '${created.secret}'\n`);
+})().catch(e => { console.error('setup failed:', e); process.exit(1); });
← ade90a7 Counsel & Bar visual loop: 21 iterations of Aston Martin DB1
·
back to Lawyer Directory Builder
·
Add 7 Express route modules: auth, admin, billing, community 4704a6f →