← back to Govarbitrage
docs/ARCHITECTURE.md
200 lines
# GovArbitrage — Architecture
GovArbitrage is a Next.js 16 App-Router application over a Prisma/Postgres data
model. Its core is a set of **deterministic engines** that turn a single
AI-supplied retail anchor into a full valuation, cost, and scoring picture for
each auction listing. AI is confined to product *identification*; everything else
is reproducible math.
---
## Data-flow diagram
```
┌─────────────────────────────────────────────┐
Import source │ GovDeals / Public Surplus / GSA / county / │
(auction site, │ state / university / Municibid / Bid4Assets │
CSV, extension)│ / CSV / browser extension │
└───────────────────────┬─────────────────────┘
│ create
▼
┌───────────────┐
│ Listing row │ (Prisma)
└───────┬───────┘
│ runResearch(listingId)
▼
┌────────────────────────────────────────────────────────────┐
│ Research pipeline (src/pipeline/research.ts) │
│ │
│ identify ──► value ──► cost ──► score ──► persist │
│ │ │ │ │ │
│ ai.ts valuation costs scoring │
│ (Ollama- .ts .ts + .ts (+ freight.ts, │
│ first) freight demand.ts heuristic) │
└───────────────────────────┬────────────────────────────────┘
│ upsert (single transaction)
▼
┌──────────────┬───────────────┬───────────────┬────────────────┐
│ Research │ CostBreakdown │ Score[] (×9) │ Comparable[] │
│ row │ row │ + ListingEvent│ │
└──────┬───────┴───────┬───────┴───────┬────────┴────────────────┘
│ │ │
▼ ▼ ▼
┌───────────────────────────────────────────────────────────────┐
│ Read layer: src/lib/listings.ts (flatten) · dashboard.ts │
└───────────────────────────┬───────────────────────────────────┘
▼
┌────────────────────────────────────────┐
│ Dashboard (page.tsx + listings-table) │
│ · API routes (/api/listings, /[id]) │
│ · Listing detail (/listings/[id]) │
└────────────────────────────────────────┘
```
---
## Layers
### 1. Data model (`prisma/schema.prisma`)
Money is stored as `Decimal(14,2)`; scores are `Float` 0..100; probabilities are
`Float` 0..1.
| Model | Role |
| -------------------- | ---------------------------------------------------------------------------------------- |
| **Listing** | The auction lot. Source + `sourceAuctionId` (unique together), title/description/category, AI-identified attributes (manufacturer, model, serial, sku, year), condition, quantity, weight/dimensions, location, live auction state (`currentBid`, `bidCount`, `closingAt`, `imageUrls`), `identifiedBy`/`identifyRaw`, and a `researchStatus`. Owns all the relations below. |
| **Research** | 1:1 valuation output — retail estimates, used-market spread, channel values (wholesale/liquidation/sell-today), 7/30/90-day horizon values, expected sale price, probability of sale, days-until-sold, confidence, sources JSON, and a summary. |
| **CostBreakdown** | 1:1 cost sheet — 20+ line items plus the rollups `expectedReturns`, `totalInvestment`, `expectedNetProfit`, `roi`, `annualizedReturn`, and `recommendedMaxBid`; `assumptions` JSON records the basis for each estimate. |
| **Score** | One row **per scoring profile** (unique `[listingId, profile]`) — the profile `value`, the seven component sub-scores, `risk`, `dropShip`, a plain-English `explanation`, and structured `factors`. |
| **Comparable** | Sold / active / retail comps backing a valuation (`kind`, title, price, url, source, soldAt). |
| **BuyerInterestPage**| Compliant, contingent-only interest page for a listing (slug, headline, `contingent` default true, `published`, standing `disclaimer`). |
| **BuyerLead** | A captured buyer lead with an optional **contingent** offer (never ownership). |
| **AuctionOutcome** | Tracks the lifecycle: WATCHING → BID_PLACED → WON/LOST/ABANDONED, plus max bid set, won price, actual sale. |
| **Note** | Freeform analyst notes on a listing. |
| **ListingEvent** | Typed, append-only event log (IMPORTED, RESEARCH_STARTED/COMPLETED/FAILED, SCORED, PRICE_CHANGED, NOTE_ADDED, BUYER_LEAD_ADDED, OUTCOME_CHANGED, MANUAL_EDIT). |
| **SavedView** | A persisted dashboard configuration (filters, sort, columns, pinned, grouping) as JSON. |
| **User** | Auth principal (email, passwordHash, role ADMIN/ANALYST/VIEWER). |
| **ApiCredential** | Per-user provider credential stored **AES-256-GCM encrypted at rest** (ciphertext/iv/authTag), never returned to the client. |
| **AuditLog** | Privileged-action audit trail (action, entity, entityId, ip, meta). |
### 2. Deterministic engines (`src/engines`)
Constants for every engine live in `src/engines/constants.ts` (condition→retail
factors, repair factors, per-source buyer premiums, fee rates, prep-cost
assumptions, freight threshold, target ROI). Shared types live in
`src/engines/types.ts`. Each engine has a colocated Vitest test
(`*.test.ts`).
- **`valuation.ts`** — `computeValuation()` derives the full price ladder from a
single **retail anchor** (`newRetail`) plus item facts. Per-unit retail figures
(new / replacement / average), a used-market spread anchored on
`CONDITION_RETAIL_FACTOR`, channel values (wholesale = 0.6× used, liquidation =
0.42×, sell-today = 1.1× liquidation), demand-scaled time-horizon values
(7/30/90-day), a demand-weighted **expected sale price** (scaled by quantity),
**probability of sale** (demand + condition), **days-until-sold** (demand-driven),
and a **confidence score** built from identification confidence + number of
comparables.
- **`costs.ts`** — `computeCosts()` computes 20+ line items split into
bid-independent `fixedCosts()` (freight/shipping via `freight.ts`, insurance,
packing, pickup labor, testing, repairs, certification, storage, photography,
listing labor, plus resale-side marketplace + payment fees) and bid-dependent
costs (buyer's premium by source, sales tax). Rolls up to `expectedReturns`,
`totalInvestment`, `expectedNetProfit`, `roi`, and `annualizedReturn` (ROI
compounded over the holding period). Then **back-solves `recommendedMaxBid`** —
the highest bid that still clears `TARGET_ROI` given all bid-independent costs
and resale fees — and records its `assumptions`.
- **`scoring.ts`** — `computeScores()` first computes **seven component
sub-scores** via `computeComponents()` (arbitrage from ROI with diminishing
returns, demand, velocity, logistics, condition, competition, buyer readiness),
plus `computeRisk()` (LOW/MEDIUM/HIGH) and `computeDropShip()`
(EASY/MODERATE/DIFFICULT/INFEASIBLE). It then produces **nine named profiles**
(`OVERALL_OPPORTUNITY`, `BEST_ARBITRAGE`, `QUICK_FLIP`, `COLLECTOR`,
`LOCAL_PICKUP`, `EASY_FREIGHT`, `PARTS_ONLY`, `HIGH_CONFIDENCE`, `HIGH_PROFIT`),
each a weighted blend of the components plus profile-specific adjustments, and
each carrying a **plain-English `explanation`** naming its top drivers, ROI,
time-to-sell, risk, and drop-ship rating.
- **`freight.ts`** — `estimateFreight()` maps total lot weight to a logistics
cost: parcel tiers below `FREIGHT_THRESHOLD_LBS` (150 lb), LTL freight (base +
per-hundredweight) above it. Returns whether the item ships as freight and a
human-readable `basis`.
- **`demand.ts`** — a deterministic, keyword-driven demand + retail-floor
heuristic. Matches listing text against category signals (electronics, medical,
lab, industrial, office/furniture, cameras, networking, tools, vehicles) and a
premium-brand list. Used both as the **fallback when no AI model is reachable**
and as a sanity floor for AI estimates.
### 3. Research pipeline (`src/pipeline/research.ts`)
`runResearch(listingId, opts)` orchestrates the whole flow and is **idempotent**
(all writes are upserts, safe to re-run):
1. Marks the listing `IN_PROGRESS`.
2. Runs the demand heuristic on the listing text.
3. **Identity + anchor resolution** with priority *overrides → AI → heuristic*.
When `opts.useAI` is set, calls `identifyProduct()`; AI's `estimatedNewRetail`
is blended against the heuristic floor and its demand/confidence are clamped.
4. **Valuation** → `computeValuation()`.
5. **Costs** → `computeCosts()` (winning bid modeled as the current bid; local
pickup inferred from weight + auction terms via `inferLocalPickup()`).
6. **Scores** → `computeScores()`.
7. **Persist** in a single `prisma.$transaction`: upserts `Research`,
`CostBreakdown`, replaces `Comparable[]`, upserts all nine `Score` rows,
updates the listing to `COMPLETE` (optionally writing back identity), and
appends a `RESEARCH_COMPLETED` `ListingEvent`.
Returns a compact summary (`expectedNetProfit`, `roi`, `overallScore`,
`identifiedBy`). The seeder (`prisma/seed.ts`) drives this pipeline with
deterministic anchors + comparables to produce realistic sample data.
### 4. AI layer (`src/lib/ai.ts`)
Local-first product identification. `identifyProduct()` tries the Ollama vision
model (when images are supplied), then the Ollama text model, then — only if
`AI_PROVIDER=anthropic` and `ANTHROPIC_API_KEY` is set — an Anthropic Claude
fallback. All calls fail soft (return `null`), letting the pipeline fall back to
the heuristic. `ollamaReachable()` powers health checks. See the README's
["How the AI layer works"](../README.md#how-the-ai-layer-works) section.
### 5. API routes (`src/app/api/listings`)
- **`route.ts`** — `GET /api/listings`, parses query params into `QueryParams`
and delegates to `queryListings()`; `dynamic = "force-dynamic"`.
- **`[id]/route.ts`** — `GET /api/listings/[id]`, returns the full listing with
all relations (research, costBreakdown, scores, comparables, notes, events,
buyerLeads, buyerPage, outcome) or 404.
The read/query logic lives in `src/lib/listings.ts`: `flattenListing()`
denormalizes `Listing + Research + CostBreakdown + chosen Score` into a flat
`ListingRow`, and `queryListings()` applies filtering (source/condition/category/
search/closing-soon), per-profile score selection, risk post-filtering, sorting,
and pagination. See [API.md](./API.md).
### 6. UI (`src/app`, `src/components`)
- **`src/app/page.tsx`** — the dashboard. Server component: fetches
`getDashboardStats()` (`src/lib/dashboard.ts`) for the summary cards
(`DashboardCards`) and renders the client `ListingsTable`.
- **`src/components/listings-table.tsx`** — the dense **TanStack Table**. ~40
sortable/hideable columns, a profile selector that re-ranks server-side,
server-side search + source/condition/risk filters (debounced fetch to
`/api/listings`), column-visibility menu, CSV export, sticky first column, and
row-click into the detail view.
- **`src/app/listings/[id]/page.tsx`** — the listing detail. Server component
rendering the image gallery, headline metrics, the full valuation ladder, the
line-item cost breakdown + profit rollup, per-profile score **explanations**,
comparables, auction terms + risk, the compliant buyer-interest section, and
event history.
- **`src/components/ui/*`** — shadcn/ui-style primitives (`card`, `badge`,
`button`, `input`) built on `class-variance-authority` + `tailwind-merge`.
### Configuration notes
`next.config.ts` pins the Turbopack workspace root, marks `@prisma/client`,
`bullmq`, and `ioredis` as `serverExternalPackages`, and allows remote images
from any host. Tailwind v4 is wired via `postcss.config.mjs`
(`@tailwindcss/postcss`).