`professional-directory/review.md`

34 lines
## Snapshot
- Compliance-first LA professional/small-business directory with ingest, crawl, enrich, dedupe, API/admin agents.
- Stack: Node.js, Express 5, PostgreSQL, `pg`, shell scripts, PM2; raw HTML/JS admin UI with Leaflet CDN.
- How it's run: PM2 via [ecosystem.config.js](/Users/stevestudio2/Projects/professional-directory/ecosystem.config.js:5), plus manual `npm`/shell scripts.
- Status: live: commit on 2026-04-30; PM2 production config present.

## Top Risks (P0 — fix this week)
- [agents/api-agent/server.js](/Users/stevestudio2/Projects/professional-directory/agents/api-agent/server.js:18), [agents/ingest-agent/server.js](/Users/stevestudio2/Projects/professional-directory/agents/ingest-agent/server.js:47): no auth on mutating/control endpoints; localhost CSRF or any proxy exposure can trigger imports, opt-outs, or CRM edits. Add shared auth middleware, Origin checks, and protect all `POST` routes.
- [agents/ingest-agent/server.js](/Users/stevestudio2/Projects/professional-directory/agents/ingest-agent/server.js:55), [agents/crawler-agent/server.js](/Users/stevestudio2/Projects/professional-directory/agents/crawler-agent/server.js:48): request body can override child-process env, including DB/source URLs. Remove arbitrary env merging; allowlist only safe knobs like `LIMIT`.
- [agents/api-agent/server.js](/Users/stevestudio2/Projects/professional-directory/agents/api-agent/server.js:166): API references `suggested_domain`, `domain_available`, `starred`, `contacted_at`, `is_chain`, `lead_score`; schema lacks them at [db/schema.sql](/Users/stevestudio2/Projects/professional-directory/db/schema.sql:63). Fresh deploy breaks org routes. Add a migration or remove those fields.

## Notable Issues (P1 — fix this month)
- [scripts/run-all-night.sh](/Users/stevestudio2/Projects/professional-directory/scripts/run-all-night.sh:29): failed stages deliberately continue and final output still says complete. Make dependency stages fail-fast or emit a nonzero final status with failed stage list.
- [agents/ingest-agent/importers/npi.js](/Users/stevestudio2/Projects/professional-directory/agents/ingest-agent/importers/npi.js:189): reruns insert duplicate `professional_locations` and `phones`. Add unique constraints plus `ON CONFLICT` upserts.
- [agents/ingest-agent/importers/npi.js](/Users/stevestudio2/Projects/professional-directory/agents/ingest-agent/importers/npi.js:346): row-level DB failures are counted as skipped and job still finishes `done`. If error rate crosses threshold, mark job failed.
- [scripts/dedupe.js](/Users/stevestudio2/Projects/professional-directory/scripts/dedupe.js:80): org merge updates/deletes multiple tables without a transaction. Wrap each duplicate group in `withTx`.
- [agents/api-agent/server.js](/Users/stevestudio2/Projects/professional-directory/agents/api-agent/server.js:727): arbitrary `website` values from imported data become clickable `href`s. Normalize/allowlist `http:`/`https:` before rendering/exporting.
- [agents/shared/compliance.js](/Users/stevestudio2/Projects/professional-directory/agents/shared/compliance.js:58): rate limiter is in-memory and non-atomic per process. Multiple child jobs can exceed host limits; use a shared queue/lock.

## Polish (P2 — when convenient)
- [package.json](/Users/stevestudio2/Projects/professional-directory/package.json:15): no `test` or smoke verification script despite complex importers. Add schema/API smoke tests at minimum.
- [package.json](/Users/stevestudio2/Projects/professional-directory/package.json:7): workspaces list agent packages, but only `agents/shared/package.json` exists. Either add package manifests or simplify workspaces.
- [agents/shared/db.js](/Users/stevestudio2/Projects/professional-directory/agents/shared/db.js:32): query errors log SQL snippets but not params; debugging bad imports will be slow. Log sanitized params/job labels.
- [.env](/Users/stevestudio2/Projects/professional-directory/.env:13): local weak auth values exist but are unused. Either wire them in or remove them to avoid false security.

## Strengths
- SQL is mostly parameterized; no obvious SQL injection in inspected request filters.
- Clear provenance model: `raw_records`, `source_url`, `scrape_jobs`, opt-out fields are first-class.
- Services bind to `127.0.0.1`, reducing accidental public exposure if no proxy is added.

## Open questions for the owner
- Is this ever exposed through Nginx, Tailscale Funnel, Cloudflare Tunnel, or SSH port forwarding?
- Does the live database already have uncommitted columns for the sales workflow?
- Should this be a doctors directory, a small-business prospecting CRM, or both?