← back to Govarbitrage
README.md
227 lines
# GovArbitrage
**GovArbitrage** discovers government-surplus auction listings, researches and
values them, scores them for resale arbitrage, and surfaces the best
opportunities on a dense analyst dashboard.
It answers one question for every lot: *"If I win this at the current bid, what
does it cost me all-in, what can I resell it for, how fast, and how much do I
actually make?"* — then ranks every listing across nine named opportunity
profiles so you can sort by whatever kind of flip you're hunting.
The heavy lifting is done by **deterministic, explainable engines** (valuation,
costs, scoring, freight, demand). AI is used only for the fuzzy step —
identifying what a product actually *is* from a messy auction title — and it runs
**local-first on Ollama at $0/query**, with an opt-in Anthropic fallback.
---
## Features
- **Listing intake** from multiple government auction sources (GovDeals, Public
Surplus, GSA Auctions, county/state/university surplus, Municibid, Bid4Assets,
CSV import, browser-extension capture).
- **AI product identification** — infers manufacturer, model, category, and a
new-retail anchor from the listing title/description (local Ollama text +
vision models; Anthropic opt-in fallback; deterministic heuristic floor).
- **Full valuation ladder** — from a single retail anchor + condition + demand,
derives new/replacement/average retail, a used-market spread, wholesale /
liquidation / sell-today channel values, 7- / 30- / 90-day time-horizon values,
expected sale price, probability of sale, days-until-sold, and a confidence
score.
- **20+ line-item cost breakdown** — buyer's premium, sales tax, shipping,
freight, insurance, packing, pickup labor, testing, repairs, certification,
marketplace + payment fees, storage, photography, listing labor — rolled up
into total investment, expected net profit, ROI, and annualized return.
- **Recommended max bid** — back-solved from a target ROI given all
bid-independent costs and resale fees, so you know the ceiling before you bid.
- **Nine scoring profiles** — Overall Opportunity, Best Arbitrage, Quick Flip,
Collector, Local Pickup, Easy Freight, Parts Only, High Confidence, High
Profit — each built from seven component sub-scores, each with a plain-English
explanation of *why* it scored the way it did, plus risk and drop-ship
feasibility ratings.
- **Dense TanStack dashboard** — ~40 sortable/hideable columns, per-profile
re-ranking, server-side search + filtering (source / condition / risk /
closing-soon), CSV export, and a live countdown to auction close.
- **Listing detail view** — valuation ladder, full cost breakdown, per-profile
score explanations, comparables, auction terms, and event history.
- **Compliant buyer workflow** — contingent-offer interest pages and leads that
never represent ownership before the auction is won (see
[Compliance](#compliance)).
- **Audit trail & secrets** — every listing carries a typed event log; API
credentials submitted to `POST /api/credentials` are encrypted at rest with
AES-256-GCM (`src/lib/crypto.ts`) and never returned to the client; an
`AuditLog` records privileged actions. Import/admin write endpoints are guarded
by an optional `IMPORT_TOKEN` header.
### Security status (honest)
**Implemented & verified:**
- **Session authentication** — email/password login (`/login`), scrypt-hashed
passwords (`src/lib/password.ts`), signed JWT session cookies (`jose`,
`src/lib/session.ts`), and **Edge middleware** (`src/middleware.ts`) that gates
every page + API route. Unauthenticated pages redirect to `/login`; APIs return
401. Public exceptions: `/login`, `/api/auth/*`, the `/b/<slug>` buyer pages, and
the anonymous `buyer-lead` submission endpoint.
- **Roles** — `ADMIN` / `ANALYST` / `VIEWER` on the `User` model; `hasRole()` helper
for role checks; login/logout/failed-login written to `AuditLog`.
- **AES-256-GCM secret encryption** for stored provider keys (`src/lib/crypto.ts`).
- **Write-guard** — import/admin endpoints accept a valid session **or** the
machine `x-import-token` header (`IMPORT_TOKEN`), enforced in middleware and again
in-handler (`requireWrite`).
- **Zod** request validation on auth + write endpoints.
Demo users (password `changeme`): `admin@govarbitrage.local` (ADMIN),
`analyst@govarbitrage.local` (ANALYST).
**Remaining follow-on:** request **rate-limiting** (not yet implemented) — add
before exposing the login + public buyer-lead endpoints to the open internet.
---
## Tech stack
| Layer | Technology |
| ---------------- | ----------------------------------------------------------------- |
| Framework | **Next.js 16** (App Router, server components, Turbopack) |
| Language | **TypeScript** 5.7 |
| UI runtime | **React 19** |
| ORM | **Prisma 6** (`@prisma/client` 6.19) |
| Database | **PostgreSQL 14** |
| Styling | **Tailwind CSS v4** (`@tailwindcss/postcss`) |
| Components | shadcn/ui-style primitives (`class-variance-authority`, `clsx`, `tailwind-merge`, `lucide-react`) |
| Data grid | **TanStack Table** (`@tanstack/react-table` 8) |
| Validation | **Zod** |
| CSV | `csv-parse` |
| Testing | **Vitest** (engine unit tests) + **Playwright** (e2e) |
| Background jobs | **BullMQ + Redis** (optional; direct/in-process mode without it) |
| Local AI | **Ollama** (text + vision models) |
| Cloud AI (opt-in)| Anthropic Claude (fallback only) |
---
## Prerequisites
- **Node.js 20+**
- **PostgreSQL** running locally (a `govarbitrage` database — Postgres 14+)
- **Ollama** running locally for AI identification (optional; the app falls back
to a deterministic heuristic when no model is reachable). Pull the configured
models, e.g. `ollama pull qwen3:14b` and `ollama pull qwen2.5vl:7b`.
- **Redis** (optional) only if you want BullMQ-backed background research.
---
## Setup
```bash
# 1. Install dependencies
npm install
# 2. Create the database
createdb govarbitrage
# 3. Configure environment
cp .env.example .env
# Edit .env if your Postgres user / connection differs.
# 4. Create the schema (no migration history needed for local dev)
npx prisma db push
# 5. Seed realistic sample listings + full research/valuation/scoring
npm run db:seed
# 6. Run the dev server
npm run dev
```
The app runs on **http://localhost:3000** by default.
---
## npm scripts
| Script | Command | What it does |
| ------------------- | ------------------------------ | ----------------------------------------------------- |
| `npm run dev` | `next dev` | Start the dev server (http://localhost:3000) |
| `npm run build` | `prisma generate && next build`| Generate the Prisma client, then production build |
| `npm run start` | `next start` | Serve the production build |
| `npm run lint` | `eslint .` | Lint the codebase |
| `npm run typecheck` | `tsc --noEmit` | Type-check without emitting |
| `npm run test` | `vitest run` | Run the engine unit tests once |
| `npm run test:watch`| `vitest` | Run tests in watch mode |
| `npm run db:generate` | `prisma generate` | (Re)generate the Prisma client |
| `npm run db:push` | `prisma db push` | Push the schema to the database (no migration files) |
| `npm run db:migrate`| `prisma migrate dev` | Create + apply a dev migration |
| `npm run db:seed` | `tsx prisma/seed.ts` | Seed sample listings + run the research pipeline |
| `npm run worker` | `tsx src/worker/index.ts` | Run the BullMQ research worker (optional; requires Redis) |
| `npm run e2e` | `playwright test` | Run Playwright end-to-end tests |
> The research worker (`npm run worker`) is the BullMQ-backed background runner.
> When `REDIS_URL` is unset, research runs in direct/in-process mode instead
> (invoked directly through `src/pipeline/research.ts`), which is fine at this
> scale.
---
## Environment variables
From `.env.example`:
| Variable | Default (example) | Purpose |
| --------------------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| `DATABASE_URL` | `postgresql://<user>@localhost:5432/govarbitrage?schema=public` | Postgres connection string. |
| `OLLAMA_BASE_URL` | `http://localhost:11434` | Local Ollama endpoint. |
| `OLLAMA_TEXT_MODEL` | `qwen3:14b` | Ollama model for text-based product identification. |
| `OLLAMA_VISION_MODEL` | `qwen2.5vl:7b` | Ollama vision model (used when listing images are supplied). |
| `ANTHROPIC_API_KEY` | *(blank)* | Cloud fallback key. Leave blank to stay 100% local / $0. |
| `AI_PROVIDER` | `ollama` | `ollama` (default, local, free) or `anthropic` (opt-in, paid). |
| `REDIS_URL` | *(blank)* | Optional. Set to enable BullMQ background jobs; unset = in-process. |
| `AUTH_SECRET` | `dev-only-change-me` | Session/auth secret. Change in production. |
| `ENCRYPTION_KEY` | 32-byte hex | AES-256-GCM key for encrypting stored API credentials at rest. |
| `NODE_ENV` | `development` | Node environment. |
---
## How the AI layer works
Product identification is the one genuinely fuzzy step in the pipeline, so it's
the only place AI is used. It runs **local-first**:
1. **Ollama text model** (`OLLAMA_TEXT_MODEL`, default `qwen3:14b`) — the default
path. Runs on this machine, `$0`/query. If listing images are supplied, the
**vision model** (`OLLAMA_VISION_MODEL`, default `qwen2.5vl:7b`) is tried
first.
2. **Anthropic Claude** — an **opt-in** fallback, used only when
`AI_PROVIDER=anthropic` *and* `ANTHROPIC_API_KEY` is set, and only if the
local model didn't return a result.
3. **Deterministic heuristic** — if no model is reachable at all, the pipeline
falls back to the keyword-driven demand/retail heuristic in
`src/engines/demand.ts`. This keeps the app fully functional and reproducible
offline; the AI estimate is also *blended* with the heuristic floor to avoid
wild outliers.
Everything downstream of identification (valuation, costs, scoring) is
deterministic math, so the numbers are reproducible and auditable rather than a
black box.
---
## Compliance
GovArbitrage's buyer workflow is deliberately constrained so you never
misrepresent ownership of an item you have not yet won:
- Buyer-interest pages and leads are **contingent by default**
(`contingent = true`) and carry a standing disclaimer:
> *"This item is not yet owned. Any offer is contingent on winning the
> government auction and is non-binding until the auction is won and the item
> is in hand."*
- The workflow **only accepts contingent offers** and **never represents
ownership before an auction is won**. Detail views label the buyer section
explicitly: *"Buyer Interest (contingent — no ownership represented pre-win)."*
This is a hard product rule, not a setting.