`PLAN.md`

63 lines
# PLAN — normalized cross-source flyer index (TK-10708)

## Position (why this exists next to re-flyer-aggregator)

`re-flyer-aggregator` already DISCOVERS + GENERATES flyers, but its outputs live in three
different shapes (a deal-scoped PG view `deal_assets_v`, a broker-scoped `property-flyers.json`,
and loose `out/spotlight-*.html` files). There is no single normalized, deduped index in the
schema TK-10708 asks for. `re-flyers/` is that thin **unifier** — it READS the existing outputs
(no re-scrape, $0 local) and normalizes them into one feed + a sort/density viewer. It does not
duplicate the discovery/generation engine.

## Aggregation architecture (5 bullets)

1. **Collect (read-only) from 3 existing producers** — Source A `property-flyers.json` (broker
   OMs/brochures, Tier-1), Source B reflyers PG `external_marketing_asset` (classified deal
   assets), Source C `out/spotlight-*.html` (Tier-2 self-generated recaps). `scripts/build-seed.mjs`
   is the collector; it re-runs any time to refresh.
2. **Normalize to one schema** —
   `{ source_build, listing_id, title, address, flyer_url, generated_at, asset_type,
   rights_basis, tier, dedup_key }`. `source_build ∈ {broker-site, usre-deal, rentv-spotlight}`
   maps each row to its producing RE surface; `asset_type` uses the aggregator's controlled
   vocabulary (`offering_memorandum`, `marketing_flyer`, `property_spotlight`, `property_brief`,
   `market_report`, `deal_recap`).
3. **Dedup strategy** — `dedup_key = sha1(lower(flyer_url || title))[:12]`. Same PDF/landing URL
   discovered under two sources collapses to one row (first writer wins). URL is the strong key;
   title is the fallback for self-generated recaps that have no external URL. Future upgrade:
   dedup by subject (`county_fips + ain` / `doc_number`) once broker rows carry a resolved parcel.
4. **Compliance carried through, never bypassed** — every row keeps `rights_basis` + `tier`. The
   viewer flags GATED/Tier-3 rows (dashed, 🔒, link suppressed) so they're visible but not
   clickable-to-fetch. The index references URLs only — it **never** downloads or re-hosts a file.
   Promoting anything into a live consumer (usre/CRCP/RENTV) stays GATED to Steve.
5. **Serve + view** — `server.js` exposes `GET /api/flyers` (serves `data/flyers.json`) behind
   basic-auth `admin/DW2024!` on an OS-assigned free port (no permanent port bound).
   `public/index.html` is the grid with the mandatory **sort** (Newest / Source / Type / Tier /
   Title A→Z) + **density slider** (2–7 cols) + source filter + search, all localStorage-persisted
   per Steve's standing grid rule.

## Schema (the normalized row)

| field | source | notes |
|---|---|---|
| `source_build` | derived | `broker-site` \| `usre-deal` \| `rentv-spotlight` |
| `listing_id` | source | deal doc-id for recaps; null for broker/deal rows until parcel-resolved |
| `title` | source | property/brochure name or recap `<title>` |
| `address` | source | null today (broker flyers carry a name; upgrade = parcel join) |
| `flyer_url` | source | PDF/landing URL, or `file://` for local Tier-2 recap |
| `generated_at` | source | `found_at` / `last_verified_at` / file mtime |
| `asset_type` | source | controlled vocab (above) |
| `rights_basis` | source | `first_party_broker` \| `self_generated` \| `GATED_needs_review` \| … |
| `tier` | source | 1 \| 2 \| 3 |
| `dedup_key` | derived | sha1 collapse key |

## Roadmap

1. **Done (this scaffold):** collector, normalized `data/flyers.json` (224 rows), `/api/flyers`,
   sort+density viewer, RESEARCH/PLAN.
2. **Next (safe, local):** resolve `address` + `listing_id` by joining broker flyers to usre
   parcels (subject-level dedup); a scheduled `build-seed` refresh; pull in the aggregator's
   `deal_assets_v` directly from usre for the richer deal join.
3. **Gated (needs Steve):** wire this feed into CRCP/RENTV/usre as a live consumer surface; any
   Kamatera deploy; promote validated rows into usre; any Tier-3 marketplace access or PDF re-host;
   any send-to-list. None of these are executed here.