← back to Cli Printing Press
feat(pipeline): add Proof-of-Behavior verification phase
efaec84bd19b81e42291da3b06fe902324e5ceee · 2026-03-26 23:00:46 -0700 · Matt Van Horn
Replace string-matching checks with behavioral verification that traces
actual data flow in generated CLIs. Catches hallucinated API endpoints,
dead flags, ghost SQLite tables, orphan FTS indexes, and auth mismatches
before shipping.
- verify.go: PathProof, FlagProof, PipelineProof, AuthProof, CompileGate
- verify_report.go: VerificationReport with PASS/WARN/FAIL verdict
- remediate.go: safe auto-fix (deletion-only) with compile gate + revert
- fullrun.go: wired as Step 5.5 between dogfood and scorecard
- 11 tests covering all hallucination types found in Discord/Notion/Linear
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Files touched
A docs/plans/2026-03-26-fix-printing-press-proof-of-behavior-verification-plan.mdM internal/pipeline/fullrun.goA internal/pipeline/remediate.goA internal/pipeline/verify.goA internal/pipeline/verify_report.goA internal/pipeline/verify_test.go
Diff
commit efaec84bd19b81e42291da3b06fe902324e5ceee
Author: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Date: Thu Mar 26 23:00:46 2026 -0700
feat(pipeline): add Proof-of-Behavior verification phase
Replace string-matching checks with behavioral verification that traces
actual data flow in generated CLIs. Catches hallucinated API endpoints,
dead flags, ghost SQLite tables, orphan FTS indexes, and auth mismatches
before shipping.
- verify.go: PathProof, FlagProof, PipelineProof, AuthProof, CompileGate
- verify_report.go: VerificationReport with PASS/WARN/FAIL verdict
- remediate.go: safe auto-fix (deletion-only) with compile gate + revert
- fullrun.go: wired as Step 5.5 between dogfood and scorecard
- 11 tests covering all hallucination types found in Discord/Notion/Linear
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
...ng-press-proof-of-behavior-verification-plan.md | 346 +++++++++++++
internal/pipeline/fullrun.go | 68 +++
internal/pipeline/remediate.go | 267 ++++++++++
internal/pipeline/verify.go | 541 ++++++++++++++++++++
internal/pipeline/verify_report.go | 264 ++++++++++
internal/pipeline/verify_test.go | 556 +++++++++++++++++++++
6 files changed, 2042 insertions(+)
diff --git a/docs/plans/2026-03-26-fix-printing-press-proof-of-behavior-verification-plan.md b/docs/plans/2026-03-26-fix-printing-press-proof-of-behavior-verification-plan.md
new file mode 100644
index 00000000..17f2b06f
--- /dev/null
+++ b/docs/plans/2026-03-26-fix-printing-press-proof-of-behavior-verification-plan.md
@@ -0,0 +1,346 @@
+---
+title: "fix: Add Proof-of-Behavior Verification Phase to Printing Press"
+type: fix
+status: active
+date: 2026-03-26
+---
+
+# fix: Add Proof-of-Behavior Verification Phase to Printing Press
+
+## Overview
+
+The printing press generates CLIs that score Grade A on its own scorecard but fail on the first real API call. The root cause is Goodhart's Law: the scorecard measures **string presence** ("does sync.go contain `readCache`?") not **behavior** ("does sync.go actually call `UpsertMessage` on a table that search.go queries?"). Claude, optimizing for the scorecard, writes dead code containing trigger strings.
+
+Evidence from 3 shipped CLIs:
+- **Discord CLI**: Scored 96/110, but sync.go hit flat paths (`GET /messages`) that don't exist in Discord's guild-scoped API. Store created domain tables nothing populated. Auth sent `Bearer` instead of `Bot`.
+- **Notion CLI**: Scored 79/100, but `members_fts` FTS5 table was populated during sync but never queried by any command. Empty `if flags.timeout > 0 {}` block exists purely to satisfy dead-code checker. `rateLimitErr()` called in forecast.go on a non-rate-limit error path.
+- **Linear CLI**: Scored 96/110, but `--csv` and `--stdin` flags registered and never checked. Helper functions existed purely to trigger scorecard matches.
+
+## Problem Statement
+
+The current pipeline has verification scattered across three places that don't talk to each other:
+
+| Location | What it checks | Why it fails |
+|----------|---------------|-------------|
+| `validate.go` (7 gates) | `go build`, `go vet`, binary runs, `--help`/`version`/`doctor` produce output | Only checks compilation - doesn't verify behavior |
+| `dogfood.go` (5 checks) | Path regex, auth format, dead flags/functions, pipeline integrity | String-matching: checks if `UpsertX` appears in source, not if it's on a live code path |
+| `scorecard.go` Tier 2 (6 dims) | PathValidity, AuthProtocol, DataPipeline, SyncCorrectness, TypeFidelity, DeadCode | Same string-matching problem. Plus scoring 0-10 makes failures gradual when they should be binary |
+
+**The missing piece**: No check verifies that the **data actually flows**. You can have:
+- A `UpsertMessage()` method in store.go (scorecard: +2 points)
+- A `sync.go` file that imports store (scorecard: +2 points)
+- But sync.go calls generic `Upsert()`, not `UpsertMessage()` (scorecard doesn't catch this)
+- So the domain tables are ghost tables - created but never populated
+
+## Proposed Solution
+
+Replace the scattered checks with a single **Proof-of-Behavior** phase that runs as a hard gate after Phase 4 (GOAT Build) and before Phase 5 (Final Scorecard). This phase produces a structured `VerificationReport` that the scorecard consumes.
+
+### Architecture
+
+```
+Phase 4 (GOAT Build)
+ |
+ v
+Phase 4.7: PROOF OF BEHAVIOR <-- NEW
+ |-- 4.7a: Compile Gate (go build + go vet)
+ |-- 4.7b: Path Proof (dry-run URL validation against spec)
+ |-- 4.7c: Flag Proof (every registered flag referenced in commands)
+ |-- 4.7d: Pipeline Proof (every domain table has WRITE + READ + SEARCH paths)
+ |-- 4.7e: Auth Proof (auth header format matches spec's securitySchemes)
+ |-- 4.7f: Verdict (PASS/WARN/FAIL with structured report)
+ |-- 4.7g: Auto-Remediation (safe deletions only, then re-verify)
+ |
+ v
+Phase 5 (Final Scorecard - consumes VerificationReport)
+```
+
+### Key Design Decisions
+
+1. **Hard gate, not a score.** The current scorecard gives 0-5 points for dead code. This means 3 dead flags still passes. The Proof-of-Behavior phase is pass/fail: zero dead flags, zero ghost tables, zero hallucinated paths, or it doesn't ship.
+
+2. **Grep-based analysis, not `golang.org/x/tools/cmd/deadcode`.** The external `deadcode` tool uses Rapid Type Analysis from `main()`. In a Cobra app, all RunE functions appear unreachable because Cobra dispatches via runtime reflection. The existing grep-based approach in `scorecard.go:1031-1064` is Cobra-aware and purpose-built for this codebase.
+
+3. **Auto-fix is deletion-only.** Safe auto-fixes: delete dead functions, delete unwired flags, delete ghost tables. Unsafe operations (wiring `sync.go` to call `UpsertMessage` instead of generic `Upsert`) require re-running Phase 4's GOAT Build, not a patch. The line between "fix" and "re-generate" is: if you need to write new code, it's re-generation.
+
+4. **Compile gate after every auto-fix.** After removing dead code, `go build ./... && go vet ./...` must pass before re-verification. If auto-fix breaks compilation, auto-fix is reverted and the failure is reported as FAIL (not WARN).
+
+5. **User-edited code protection.** Files without the `// Code generated by CLI Printing Press` header are never auto-fixed. Files with uncommitted modifications (checked via `git diff`) are flagged but not touched.
+
+## Technical Approach
+
+### Phase 4.7a: Compile Gate
+
+```go
+// internal/pipeline/verify.go
+func (v *Verifier) CompileGate() error {
+ // Same 3 gates as validate.go but re-run after GOAT Build
+ // 1. go mod tidy
+ // 2. go vet ./...
+ // 3. go build ./...
+ // If any fail, Phase 4.7 is FAIL - no point checking behavior
+}
+```
+
+### Phase 4.7b: Path Proof
+
+For each generated command file in `internal/cli/`:
+1. Extract the URL path from the source code (regex: `path\s*[:=]\s*"([^"]+)"` or `c\.(Get|Post|Patch|Delete|Put)\s*\(\s*"([^"]+)"`)
+2. Normalize path params: replace `/guilds/123/channels` with `/guilds/{guild_id}/channels`
+3. Look up the normalized path in the original OpenAPI spec's `paths` object
+4. If the path doesn't exist in the spec: **CRITICAL** - hallucinated endpoint
+
+```go
+// internal/pipeline/verify.go
+type PathProofResult struct {
+ Command string // e.g., "guilds channels list"
+ ExtractedURL string // e.g., "/guilds/{guild_id}/channels"
+ Method string // e.g., "GET"
+ InSpec bool
+ SpecPath string // matched spec path, empty if not found
+}
+
+func (v *Verifier) PathProof(specPaths map[string]map[string]bool) []PathProofResult {
+ // 1. Glob internal/cli/*.go (exclude root.go, helpers.go, doctor.go, auth.go, version.go)
+ // 2. For each file, extract path assignments via regex
+ // 3. Skip files with no path (local-only commands: search, sql, activity, stale, etc.)
+ // 4. Normalize extracted path (replace literal IDs with {param} placeholders)
+ // 5. Match against specPaths map
+ // 6. Return results with InSpec=false for hallucinated paths
+}
+```
+
+**GraphQL exception**: For GraphQL APIs (detected by single `/graphql` endpoint in spec), skip path validation entirely. Instead, validate that query strings in generated code reference real schema types (parse `.graphql` query files against schema).
+
+**Local-command exception**: Commands that only query local SQLite (search, sql, activity, stale, trends, patterns, forecast, health, similar, bottleneck, status) don't have paths. Skip them. Detection: file has no `c.Get/Post/Patch/Delete/Put` calls.
+
+### Phase 4.7c: Flag Proof
+
+For every persistent flag registered in `root.go`:
+1. Parse the flag field name from `rootCmd.PersistentFlags().XxxVar(&flags.fieldName, ...)`
+2. Search all files in `internal/cli/*.go` EXCLUDING `root.go` for `flags.fieldName`
+3. Also search `internal/client/client.go` for `f.fieldName` or `flags.fieldName` (catches flags passed through newClient)
+4. If a flag has zero references outside its declaration: **dead flag**
+
+```go
+type FlagProofResult struct {
+ FlagName string // e.g., "dryRun"
+ CLIName string // e.g., "--dry-run"
+ RefCount int // references in command files + client
+ DeadFlag bool
+}
+```
+
+**Indirect usage rule**: A flag counts as "used" if:
+- It appears in a command file's RunE function (`flags.dryRun`)
+- OR it's assigned to the client config struct in `newClient()` AND the client field is checked in `client.go`
+- OR it's checked in `helpers.go` AND the helper is called from a command file
+
+This catches the `flags.timeout -> c.Timeout -> client uses it` pattern.
+
+### Phase 4.7d: Pipeline Proof (the critical one)
+
+For every domain-specific SQLite table in `store.go`:
+
+```go
+type PipelineProofResult struct {
+ TableName string // e.g., "messages"
+ HasWrite bool // sync.go calls UpsertMessage or store.UpsertMessage
+ WritePath string // e.g., "sync.go:142"
+ HasRead bool // any command file queries this table
+ ReadPath string // e.g., "activity.go:58"
+ HasSearch bool // if FTS5 table exists, a SearchX() is called
+ SearchPath string // e.g., "search.go:54"
+ FTSExists bool // FTS5 virtual table exists for this entity
+ GhostTable bool // table created but never populated
+ OrphanFTS bool // FTS5 exists but SearchX never called
+}
+```
+
+Detection algorithm:
+1. Parse `store.go` for `CREATE TABLE` statements -> extract table names
+2. Parse `store.go` for `CREATE VIRTUAL TABLE ... USING fts5` -> extract FTS table names and their source tables
+3. For each domain table (exclude `sync_state`, `resources`, `*_fts`):
+ a. WRITE: grep `internal/cli/*.go` for `UpsertTableName` or `store.UpsertTableName` or `db.UpsertTableName`
+ b. READ: grep `internal/cli/*.go` for `FROM tablename` or `JOIN tablename` or table-specific query methods
+ c. SEARCH: if FTS table exists, grep for `SearchTableName` or `tablename_fts MATCH`
+4. A table with HasWrite=false is a **ghost table** (created but never populated)
+5. A table with HasWrite=true but HasRead=false is an **orphan table** (populated but never queried)
+6. An FTS table with no SearchX calls is an **orphan index**
+
+**Exempt tables**: `sync_state` (internal metadata - read/written by sync only), `resources` (generic fallback - acceptable if domain tables exist), `*_fts` (populated by triggers, read by search).
+
+### Phase 4.7e: Auth Proof
+
+Compare the generated auth header against the spec's `securitySchemes`:
+
+```go
+type AuthProofResult struct {
+ SpecFormat string // e.g., "Bot {token}" or "Bearer {token}"
+ GeneratedFormat string // extracted from client.go
+ Match bool
+ EnvVarCorrect bool // env var name matches spec convention
+}
+```
+
+Detection:
+1. Parse spec's `securitySchemes` for auth type (apiKey, http bearer, http basic, oauth2)
+2. Parse `internal/client/client.go` for the Authorization header value format
+3. Compare: if spec says `Bot {token}` but code sends `Bearer {token}`, that's a **critical mismatch** (every request will 401)
+
+### Phase 4.7f: Verdict
+
+```go
+type VerificationReport struct {
+ Paths []PathProofResult
+ Flags []FlagProofResult
+ Pipeline []PipelineProofResult
+ Auth AuthProofResult
+
+ // Computed
+ HallucinatedPaths int // paths not in spec
+ DeadFlags int
+ GhostTables int
+ OrphanFTS int
+ AuthMismatch bool
+
+ Verdict string // "PASS", "WARN", "FAIL"
+ // FAIL if: any hallucinated paths, auth mismatch, or ghost tables on primary entities
+ // WARN if: dead flags > 0, orphan FTS, ghost tables on support entities only
+ // PASS if: everything clean
+}
+```
+
+**FAIL** = hard stop. Cannot proceed to Phase 5. Must fix before shipping.
+**WARN** = proceed but flag in final report. These are quality issues, not correctness issues.
+**PASS** = clean. Full confidence in generated CLI.
+
+### Phase 4.7g: Auto-Remediation
+
+Only runs if verdict is WARN or FAIL. Only does **safe deletions**:
+
+| Issue | Auto-Fix | Safety |
+|-------|----------|--------|
+| Dead flag in root.go | Remove the `PersistentFlags().XxxVar` line + struct field | Safe - compilation proves correctness |
+| Dead function in helpers.go | Remove the function | Safe - if anything called it, `go build` will fail |
+| Ghost table (never populated) | Remove CREATE TABLE from store.go | Safe - nothing references it |
+| Orphan FTS (never searched) | Remove CREATE VIRTUAL TABLE + triggers | Safe - nothing queries it |
+| Hallucinated path | **DO NOT AUTO-FIX** - flag for GOAT Build re-run | Unsafe - needs domain knowledge |
+| Auth mismatch | **DO NOT AUTO-FIX** - flag for manual review | Unsafe - wrong fix = all requests fail |
+| Broken data pipeline (sync calls generic Upsert) | **DO NOT AUTO-FIX** - flag for GOAT Build re-run | Unsafe - needs code generation |
+
+After auto-fix:
+1. Run `go build ./... && go vet ./...` - if fails, revert all auto-fixes and report FAIL
+2. Re-run all proofs (4.7b-4.7e)
+3. If still FAIL after auto-fix: report as FAIL with "auto-fix insufficient, manual intervention required"
+4. Maximum 1 auto-fix cycle (no recursive fixing)
+
+### Anti-Gaming Rules
+
+These rules go in the SKILL.md to prevent Claude from hallucinating to optimize for the new checks:
+
+```
+PROOF-OF-BEHAVIOR RULES:
+
+1. NEVER add a function to helpers.go that isn't called by a real command.
+ The Proof-of-Behavior phase traces every function to a call site.
+
+2. NEVER register a flag in root.go that no command checks.
+ The flag proof traces every flag to a usage site in RunE code.
+
+3. NEVER create a SQLite table that sync doesn't populate.
+ The pipeline proof traces every table to a UpsertX call in sync.go.
+
+4. NEVER create an FTS5 index that no search command queries.
+ The pipeline proof traces every FTS5 table to a SearchX call.
+
+5. NEVER add an empty if-block to satisfy dead-code detection.
+ `if flags.timeout > 0 { // already applied }` is still dead code.
+ The flag must be READ (its value influences control flow or output).
+
+6. NEVER call a function in the wrong context to satisfy wiring checks.
+ `rateLimitErr()` called on a non-rate-limit path is a hallucination.
+ The function must be called where its name implies it should be called.
+```
+
+## Acceptance Criteria
+
+### Must Have (FAIL gate)
+- [ ] Zero hallucinated API paths (every extracted URL exists in the spec)
+- [ ] Zero auth mismatches (generated auth format matches spec's securitySchemes)
+- [ ] Zero ghost tables on primary entities (every primary domain table has a WRITE path)
+- [ ] Compile gate passes after auto-remediation
+- [ ] Structured `VerificationReport` produced as JSON + markdown
+
+### Should Have (WARN gate)
+- [ ] Zero dead flags (every root.go flag referenced in command or client code)
+- [ ] Zero dead functions (every helpers.go function called from a command file)
+- [ ] Zero orphan FTS indexes (every FTS5 table queried by a search command)
+- [ ] Auto-remediation deletes dead flags/functions/tables and re-verifies
+
+### Nice to Have
+- [ ] GraphQL query field validation (for non-REST APIs)
+- [ ] User-edit protection (skip auto-fix on files without generated-code header)
+- [ ] Scorecard Tier 2 consumes VerificationReport instead of running its own checks
+
+## System-Wide Impact
+
+- **Interaction graph**: Phase 4.7 reads the OpenAPI spec (loaded in Phase 0), reads generated source files (from Phase 2-4), and produces a VerificationReport consumed by Phase 5's scorecard. Auto-remediation modifies generated files, requiring re-compilation.
+- **Error propagation**: If auto-fix breaks compilation, all fixes are reverted atomically (git stash or copy-restore). No partial fix state.
+- **State lifecycle risks**: Auto-fix could delete a function that a user manually added after generation. Mitigated by the generated-code header check.
+- **API surface parity**: The `printing-press scorecard` command should gain a `--verify` flag that runs the Proof-of-Behavior phase standalone.
+
+## Implementation Phases
+
+### Phase 1: Verification Engine (core)
+- [ ] `internal/pipeline/verify.go` - `Verifier` struct with `PathProof()`, `FlagProof()`, `PipelineProof()`, `AuthProof()`
+- [ ] `internal/pipeline/verify_report.go` - `VerificationReport` struct with JSON/markdown serialization
+- [ ] Wire into `fullrun.go` between GOAT Build and Final Scorecard
+- [ ] Tests: `verify_test.go` with fixtures from Discord/Notion/Linear CLIs
+
+### Phase 2: Auto-Remediation
+- [ ] `internal/pipeline/remediate.go` - safe deletion functions (remove dead flag, remove dead function, remove ghost table)
+- [ ] Compile gate after remediation
+- [ ] Re-verification loop (max 1 cycle)
+- [ ] Tests: `remediate_test.go` with before/after fixtures
+
+### Phase 3: Scorecard Integration
+- [ ] Replace `scoreDeadCode()`, `scorePathValidity()`, `scoreDataPipelineIntegrity()` with VerificationReport consumption
+- [ ] Add `--verify` flag to `printing-press scorecard` command
+- [ ] Update SKILL.md Phase 4.5/4.6 to reference Phase 4.7
+
+### Phase 4: Anti-Gaming Rules
+- [ ] Add Proof-of-Behavior rules to SKILL.md
+- [ ] Add rule 5 detection: empty if-blocks that reference a flag but don't use its value
+- [ ] Add rule 6 detection: function called in semantically wrong context (heuristic: function name implies error type but caller doesn't handle that error type)
+
+## Success Metrics
+
+| Metric | Before | Target |
+|--------|--------|--------|
+| Hallucinated paths caught before ship | 0% (found manually after) | 100% |
+| Dead flags in shipped CLIs | 3-5 per CLI | 0 |
+| Ghost tables in shipped CLIs | 1-2 per CLI | 0 |
+| Auth mismatches caught before ship | 0% (found on first API call) | 100% |
+| Time to verify | N/A (manual) | < 30 seconds per CLI |
+
+## Dependencies & Risks
+
+- **Risk**: Auto-fix could be too aggressive and delete functions that are "dead" only because the call site hasn't been generated yet (in a multi-phase build). Mitigation: Auto-fix only runs AFTER the GOAT Build phase completes.
+- **Risk**: Path normalization (replacing literal IDs with `{param}`) could fail on unusual URL patterns. Mitigation: Use the spec's path templates as the source of truth, match generated paths against them with regex.
+- **Risk**: False positives on "indirect" flag usage through struct assignment chains. Mitigation: Follow the assignment chain through `newClient()` and the client struct.
+
+## Sources & References
+
+### Internal References
+- Existing dogfood validator: `internal/pipeline/dogfood.go`
+- Existing scorecard Tier 2: `internal/pipeline/scorecard.go:800-1080`
+- Existing self-improvement loop: `internal/pipeline/selfimprove.go`
+- Existing quality gates: `internal/generator/validate.go`
+- Discord CLI hallucination evidence: `docs/plans/2026-03-26-feat-printing-press-v2-anti-hallucination-overhaul-plan.md`
+
+### External References
+- Go deadcode tool (why we DON'T use it for Cobra apps): https://go.dev/blog/deadcode
+- kin-openapi spec validation: https://github.com/getkin/kin-openapi
+- oapi-codegen compile-time interface proof pattern: https://github.com/oapi-codegen/oapi-codegen
+- Speakeasy's generate-compile-test loop: https://www.speakeasy.com/docs/sdks/sdk-contract-testing/bootstrapping-test-generation
diff --git a/internal/pipeline/fullrun.go b/internal/pipeline/fullrun.go
index 8f452b4d..749a2c0c 100644
--- a/internal/pipeline/fullrun.go
+++ b/internal/pipeline/fullrun.go
@@ -39,6 +39,11 @@ type FullRunResult struct {
Dogfood *DogfoodReport
DogfoodError string
+ // Step 5.5: Verification
+ Verification *VerificationReport
+ VerificationError string
+ Remediation *RemediationResult
+
// Step 5: Scorecard
Scorecard *Scorecard
ScorecardError string
@@ -142,6 +147,35 @@ func MakeBestCLI(apiName, level, specFlag, specURL, outputDir, pressBinary strin
result.Dogfood = dogfood
}
+ // Step 5.5: Proof of Behavior Verification
+ verSpecPath := ""
+ if specFlag == "--spec" {
+ verSpecPath = specURL
+ }
+ verReport, verErr := RunVerification(outputDir, verSpecPath)
+ if verErr != nil {
+ result.VerificationError = verErr.Error()
+ result.Errors = append(result.Errors, fmt.Sprintf("verification: %v", verErr))
+ } else {
+ result.Verification = verReport
+
+ // Auto-remediate if WARN or FAIL
+ if verReport.Verdict != "PASS" {
+ remResult, remErr := Remediate(outputDir, verReport)
+ if remErr != nil {
+ result.Errors = append(result.Errors, fmt.Sprintf("remediation: %v", remErr))
+ } else {
+ result.Remediation = remResult
+
+ // Re-verify after remediation
+ reVerReport, reVerErr := RunVerification(outputDir, verSpecPath)
+ if reVerErr == nil {
+ result.Verification = reVerReport
+ }
+ }
+ }
+ }
+
// Step 6: Scorecard
scorecard, scErr := RunScorecard(outputDir, pipelineDir, "")
if scErr != nil {
@@ -332,6 +366,24 @@ func PrintComparisonTable(results []*FullRunResult) string {
return r.Dogfood.Verdict
})
+ // Verification
+ writeRow(&b, "Verification", results, func(r *FullRunResult) string {
+ if r.Verification == nil {
+ return "n/a"
+ }
+ summary := r.Verification.Verdict
+ if r.Verification.HallucinatedPaths > 0 {
+ summary += fmt.Sprintf(" %dp", r.Verification.HallucinatedPaths)
+ }
+ if r.Verification.DeadFlags > 0 {
+ summary += fmt.Sprintf(" %df", r.Verification.DeadFlags)
+ }
+ if r.Verification.GhostTables > 0 {
+ summary += fmt.Sprintf(" %dg", r.Verification.GhostTables)
+ }
+ return summary
+ })
+
// LLM Polish
writeRow(&b, "LLM Polish", results, func(r *FullRunResult) string {
if r.PolishResult == nil {
@@ -506,6 +558,22 @@ func GenerateLearningsPlan(results []*FullRunResult, outputPath string) error {
}
b.WriteString("\n")
+ // Verification summary
+ b.WriteString("## Verification Summary\n\n")
+ for _, r := range results {
+ if r.Verification != nil {
+ b.WriteString(fmt.Sprintf("- **%s** - %s (paths:%d flags:%d ghost:%d fts:%d)\n",
+ r.APIName, r.Verification.Verdict,
+ r.Verification.HallucinatedPaths,
+ r.Verification.DeadFlags,
+ r.Verification.GhostTables,
+ r.Verification.OrphanFTS))
+ } else {
+ b.WriteString(fmt.Sprintf("- **%s** - not run (%s)\n", r.APIName, r.VerificationError))
+ }
+ }
+ b.WriteString("\n")
+
// Next steps
b.WriteString("## Next Steps\n\n")
b.WriteString("1. Fix the highest-priority template gaps listed above\n")
diff --git a/internal/pipeline/remediate.go b/internal/pipeline/remediate.go
new file mode 100644
index 00000000..cdfc78f0
--- /dev/null
+++ b/internal/pipeline/remediate.go
@@ -0,0 +1,267 @@
+package pipeline
+
+import (
+ "context"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "regexp"
+ "strings"
+ "time"
+)
+
+// RemediationResult summarizes what auto-remediation fixed (or reverted).
+type RemediationResult struct {
+ FlagsRemoved []string `json:"flags_removed"`
+ FuncsRemoved []string `json:"funcs_removed"`
+ TablesRemoved []string `json:"tables_removed"`
+ FTSRemoved []string `json:"fts_removed"`
+ CompileAfterFix bool `json:"compile_after_fix"`
+ Reverted bool `json:"reverted"`
+}
+
+type fileBackup struct {
+ path string
+ content []byte
+}
+
+func backupFiles(paths []string) ([]fileBackup, error) {
+ var backups []fileBackup
+ for _, p := range paths {
+ data, err := os.ReadFile(p)
+ if os.IsNotExist(err) {
+ continue
+ }
+ if err != nil {
+ return nil, fmt.Errorf("backing up %s: %w", p, err)
+ }
+ backups = append(backups, fileBackup{path: p, content: data})
+ }
+ return backups, nil
+}
+
+func restoreBackups(backups []fileBackup) error {
+ for _, b := range backups {
+ if err := os.WriteFile(b.path, b.content, 0o644); err != nil {
+ return fmt.Errorf("restoring %s: %w", b.path, err)
+ }
+ }
+ return nil
+}
+
+// Remediate applies safe auto-remediation (deletions only) based on a verification report.
+// If the compile gate fails after fixes, all changes are reverted.
+func Remediate(dir string, report *VerificationReport) (*RemediationResult, error) {
+ if report == nil {
+ return &RemediationResult{}, nil
+ }
+
+ var deadFlags []string
+ for _, f := range report.Flags {
+ if f.DeadFlag || f.References == 0 {
+ deadFlags = append(deadFlags, f.FlagName)
+ }
+ }
+
+ var ghostTables []string
+ var orphanFTS []string
+ for _, p := range report.Pipeline {
+ if p.GhostTable || !p.HasWrite {
+ ghostTables = append(ghostTables, p.TableName)
+ }
+ if p.OrphanFTS || (p.HasFTS && !p.HasSearch) {
+ orphanFTS = append(orphanFTS, p.TableName+"_fts")
+ }
+ }
+
+ if len(deadFlags) == 0 && len(ghostTables) == 0 && len(orphanFTS) == 0 {
+ return &RemediationResult{}, nil
+ }
+
+ filesToBackup := []string{
+ filepath.Join(dir, "internal", "cli", "root.go"),
+ filepath.Join(dir, "internal", "cli", "helpers.go"),
+ filepath.Join(dir, "internal", "store", "store.go"),
+ }
+
+ backups, err := backupFiles(filesToBackup)
+ if err != nil {
+ return nil, fmt.Errorf("creating backups: %w", err)
+ }
+
+ result := &RemediationResult{}
+
+ if len(deadFlags) > 0 {
+ if err := removeDeadFlags(dir, deadFlags); err != nil {
+ _ = restoreBackups(backups)
+ return nil, fmt.Errorf("removing dead flags: %w", err)
+ }
+ result.FlagsRemoved = deadFlags
+ }
+
+ if len(ghostTables) > 0 {
+ if err := removeGhostTables(dir, ghostTables); err != nil {
+ _ = restoreBackups(backups)
+ return nil, fmt.Errorf("removing ghost tables: %w", err)
+ }
+ result.TablesRemoved = ghostTables
+ }
+
+ if len(orphanFTS) > 0 {
+ if err := removeOrphanFTS(dir, orphanFTS); err != nil {
+ _ = restoreBackups(backups)
+ return nil, fmt.Errorf("removing orphan FTS: %w", err)
+ }
+ result.FTSRemoved = orphanFTS
+ }
+
+ if err := compileGateCheck(dir); err != nil {
+ _ = restoreBackups(backups)
+ result.CompileAfterFix = false
+ result.Reverted = true
+ return result, fmt.Errorf("compile gate failed after remediation, changes reverted: %w", err)
+ }
+
+ result.CompileAfterFix = true
+ return result, nil
+}
+
+func removeDeadFlags(dir string, deadFlags []string) error {
+ rootPath := filepath.Join(dir, "internal", "cli", "root.go")
+ data, err := os.ReadFile(rootPath)
+ if os.IsNotExist(err) {
+ return nil
+ }
+ if err != nil {
+ return fmt.Errorf("reading root.go: %w", err)
+ }
+
+ content := string(data)
+
+ for _, flagName := range deadFlags {
+ regLine := regexp.MustCompile(`(?m)^[^\n]*&flags\.` + regexp.QuoteMeta(flagName) + `\b[^\n]*\n`)
+ content = regLine.ReplaceAllString(content, "")
+
+ structField := regexp.MustCompile(`(?m)^\s+` + regexp.QuoteMeta(flagName) + `\s+\S+[^\n]*\n`)
+ content = structField.ReplaceAllString(content, "")
+ }
+
+ return os.WriteFile(rootPath, []byte(content), 0o644)
+}
+
+func removeDeadFunctions(dir string, deadFuncs []string) error {
+ helpersPath := filepath.Join(dir, "internal", "cli", "helpers.go")
+ data, err := os.ReadFile(helpersPath)
+ if os.IsNotExist(err) {
+ return nil
+ }
+ if err != nil {
+ return fmt.Errorf("reading helpers.go: %w", err)
+ }
+
+ lines := strings.Split(string(data), "\n")
+
+ for _, funcName := range deadFuncs {
+ funcSig := "func " + funcName + "("
+ startIdx := -1
+ for i, line := range lines {
+ if strings.Contains(line, funcSig) {
+ startIdx = i
+ for startIdx > 0 && strings.HasPrefix(strings.TrimSpace(lines[startIdx-1]), "//") {
+ startIdx--
+ }
+ break
+ }
+ }
+ if startIdx == -1 {
+ continue
+ }
+
+ braceCount := 0
+ endIdx := -1
+ for i := startIdx; i < len(lines); i++ {
+ braceCount += strings.Count(lines[i], "{") - strings.Count(lines[i], "}")
+ if braceCount > 0 || strings.Contains(lines[i], "{") {
+ if braceCount == 0 && i > startIdx {
+ endIdx = i
+ break
+ }
+ }
+ }
+ if endIdx == -1 {
+ continue
+ }
+
+ lines = append(lines[:startIdx], lines[endIdx+1:]...)
+ }
+
+ return os.WriteFile(helpersPath, []byte(strings.Join(lines, "\n")), 0o644)
+}
+
+func removeGhostTables(dir string, ghostTables []string) error {
+ storePath := filepath.Join(dir, "internal", "store", "store.go")
+ data, err := os.ReadFile(storePath)
+ if os.IsNotExist(err) {
+ return nil
+ }
+ if err != nil {
+ return fmt.Errorf("reading store.go: %w", err)
+ }
+
+ content := string(data)
+
+ for _, tableName := range ghostTables {
+ createRe := regexp.MustCompile(`(?is)CREATE TABLE\s+(?:IF NOT EXISTS\s+)?` + regexp.QuoteMeta(tableName) + `\s*\(.*?\);`)
+ content = createRe.ReplaceAllString(content, "")
+
+ indexRe := regexp.MustCompile(`(?im)^[^\n]*CREATE\s+(?:UNIQUE\s+)?INDEX\s+(?:IF NOT EXISTS\s+)?\w+\s+ON\s+` + regexp.QuoteMeta(tableName) + `\b[^;]*;[^\n]*\n?`)
+ content = indexRe.ReplaceAllString(content, "")
+ }
+
+ return os.WriteFile(storePath, []byte(content), 0o644)
+}
+
+func removeOrphanFTS(dir string, orphanTables []string) error {
+ storePath := filepath.Join(dir, "internal", "store", "store.go")
+ data, err := os.ReadFile(storePath)
+ if os.IsNotExist(err) {
+ return nil
+ }
+ if err != nil {
+ return fmt.Errorf("reading store.go: %w", err)
+ }
+
+ content := string(data)
+
+ for _, ftsName := range orphanTables {
+ virtualRe := regexp.MustCompile(`(?is)CREATE VIRTUAL TABLE\s+(?:IF NOT EXISTS\s+)?` + regexp.QuoteMeta(ftsName) + `\s+USING\s+fts5\s*\(.*?\);`)
+ content = virtualRe.ReplaceAllString(content, "")
+
+ baseName := strings.TrimSuffix(ftsName, "_fts")
+
+ triggerRe := regexp.MustCompile(`(?is)CREATE TRIGGER\s+(?:IF NOT EXISTS\s+)?\w+\s+AFTER\s+(?:INSERT|DELETE|UPDATE)\s+ON\s+` + regexp.QuoteMeta(baseName) + `\b.*?END\s*;`)
+ content = triggerRe.ReplaceAllString(content, "")
+ }
+
+ return os.WriteFile(storePath, []byte(content), 0o644)
+}
+
+func compileGateCheck(dir string) error {
+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+ defer cancel()
+
+ build := exec.CommandContext(ctx, "go", "build", "./...")
+ build.Dir = dir
+ if out, err := build.CombinedOutput(); err != nil {
+ return fmt.Errorf("go build failed: %w\n%s", err, out)
+ }
+
+ vet := exec.CommandContext(ctx, "go", "vet", "./...")
+ vet.Dir = dir
+ if out, err := vet.CombinedOutput(); err != nil {
+ return fmt.Errorf("go vet failed: %w\n%s", err, out)
+ }
+
+ return nil
+}
diff --git a/internal/pipeline/verify.go b/internal/pipeline/verify.go
new file mode 100644
index 00000000..3738ac15
--- /dev/null
+++ b/internal/pipeline/verify.go
@@ -0,0 +1,541 @@
+package pipeline
+
+import (
+ "context"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "regexp"
+ "strings"
+ "time"
+ "unicode"
+)
+
+type Verifier struct {
+ Dir string
+ SpecPath string
+ spec *openAPISpec
+}
+
+type PathProofResult struct {
+ File string `json:"file"`
+ Command string `json:"command"`
+ Path string `json:"path"`
+ ExtractedURL string `json:"extracted_url"`
+ Method string `json:"method"`
+ Valid bool `json:"valid"`
+ InSpec bool `json:"in_spec"`
+ SpecPath string `json:"spec_path,omitempty"`
+}
+
+type FlagProofResult struct {
+ Flag string `json:"flag"`
+ FlagName string `json:"flag_name"`
+ CLIName string `json:"cli_name"`
+ References int `json:"references"`
+ RefCount int `json:"ref_count"`
+ DeadFlag bool `json:"dead_flag"`
+}
+
+type PipelineProofResult struct {
+ Table string `json:"table"`
+ TableName string `json:"table_name"`
+ Columns int `json:"columns"`
+ HasWrite bool `json:"has_write"`
+ WritePath string `json:"write_path,omitempty"`
+ HasRead bool `json:"has_read"`
+ ReadPath string `json:"read_path,omitempty"`
+ HasSearch bool `json:"has_search"`
+ SearchPath string `json:"search_path,omitempty"`
+ HasFTS bool `json:"has_fts"`
+ FTSExists bool `json:"fts_exists"`
+ GhostTable bool `json:"ghost_table"`
+ OrphanFTS bool `json:"orphan_fts"`
+}
+
+type AuthProofResult struct {
+ SpecScheme string `json:"spec_scheme"`
+ SpecFormat string `json:"spec_format"`
+ GeneratedScheme string `json:"generated_scheme"`
+ GeneratedFormat string `json:"generated_format"`
+ Match bool `json:"match"`
+ Mismatch bool `json:"mismatch"`
+ EnvVarCorrect bool `json:"env_var_correct"`
+ Detail string `json:"detail"`
+}
+
+func NewVerifier(dir, specPath string) (*Verifier, error) {
+ v := &Verifier{
+ Dir: dir,
+ SpecPath: specPath,
+ }
+ if specPath != "" {
+ loaded, err := loadDogfoodOpenAPISpec(specPath)
+ if err != nil {
+ return nil, fmt.Errorf("loading spec: %w", err)
+ }
+ v.spec = loaded
+ }
+ return v, nil
+}
+
+func (v *Verifier) CompileGate() error {
+ if v == nil {
+ return fmt.Errorf("nil verifier")
+ }
+
+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+ defer cancel()
+
+ build := exec.CommandContext(ctx, "go", "build", "./...")
+ build.Dir = v.Dir
+ if out, err := build.CombinedOutput(); err != nil {
+ return fmt.Errorf("go build failed: %w\n%s", err, out)
+ }
+
+ vet := exec.CommandContext(ctx, "go", "vet", "./...")
+ vet.Dir = v.Dir
+ if out, err := vet.CombinedOutput(); err != nil {
+ return fmt.Errorf("go vet failed: %w\n%s", err, out)
+ }
+
+ return nil
+}
+
+var (
+ verifyPathAssignRe = regexp.MustCompile(`(?m)\bpath\s*[:=]\s*"([^"]+)"`)
+ verifyClientCallRe = regexp.MustCompile(`c\.(Get|Post|Patch|Delete|Put)\s*\(\s*"([^"]+)"`)
+)
+
+var verifyInfraFiles = map[string]struct{}{
+ "root.go": {},
+ "helpers.go": {},
+ "doctor.go": {},
+ "auth.go": {},
+ "version.go": {},
+ "export.go": {},
+ "import.go": {},
+ "search.go": {},
+ "sync.go": {},
+ "tail.go": {},
+ "analytics.go": {},
+ "workflow.go": {},
+}
+
+func (v *Verifier) PathProof() []PathProofResult {
+ if v == nil {
+ return nil
+ }
+
+ cliDir := filepath.Join(v.Dir, "internal", "cli")
+ files := listGoFiles(cliDir)
+ if len(files) == 0 {
+ return nil
+ }
+
+ var specPatterns []*regexp.Regexp
+ var specKeys []string
+ if v.spec != nil && len(v.spec.Paths) > 0 {
+ specPatterns = compileSpecPathPatterns(v.spec.Paths)
+ specKeys = make([]string, 0, len(v.spec.Paths))
+ for k := range v.spec.Paths {
+ specKeys = append(specKeys, k)
+ }
+ }
+
+ var results []PathProofResult
+
+ for _, file := range files {
+ base := filepath.Base(file)
+ if _, infra := verifyInfraFiles[base]; infra {
+ continue
+ }
+
+ content, err := os.ReadFile(file)
+ if err != nil {
+ continue
+ }
+ src := string(content)
+ command := strings.TrimSuffix(base, ".go")
+
+ pathMatches := verifyPathAssignRe.FindAllStringSubmatch(src, -1)
+ clientMatches := verifyClientCallRe.FindAllStringSubmatch(src, -1)
+
+ if len(pathMatches) == 0 && len(clientMatches) == 0 {
+ continue
+ }
+
+ for _, m := range pathMatches {
+ url := m[1]
+ r := PathProofResult{
+ File: base,
+ Command: command,
+ Path: url,
+ ExtractedURL: url,
+ }
+ if len(specPatterns) > 0 {
+ r.InSpec = pathMatchesSpec(url, specPatterns)
+ r.Valid = r.InSpec
+ if r.InSpec {
+ r.SpecPath = findMatchingSpecPath(url, specKeys)
+ }
+ } else {
+ r.Valid = true
+ }
+ results = append(results, r)
+ }
+
+ for _, m := range clientMatches {
+ method := strings.ToUpper(m[1])
+ url := m[2]
+ r := PathProofResult{
+ File: base,
+ Command: command,
+ Path: url,
+ ExtractedURL: url,
+ Method: method,
+ }
+ if len(specPatterns) > 0 {
+ r.InSpec = pathMatchesSpec(url, specPatterns)
+ r.Valid = r.InSpec
+ if r.InSpec {
+ r.SpecPath = findMatchingSpecPath(url, specKeys)
+ }
+ } else {
+ r.Valid = true
+ }
+ results = append(results, r)
+ }
+ }
+
+ return results
+}
+
+func (v *Verifier) FlagProof() []FlagProofResult {
+ if v == nil {
+ return nil
+ }
+
+ rootData, err := os.ReadFile(filepath.Join(v.Dir, "internal", "cli", "root.go"))
+ if err != nil {
+ return nil
+ }
+
+ fieldRe := regexp.MustCompile(`&flags\.(\w+)`)
+ matches := fieldRe.FindAllStringSubmatch(string(rootData), -1)
+ if len(matches) == 0 {
+ return nil
+ }
+
+ seen := make(map[string]struct{})
+ var fields []string
+ for _, m := range matches {
+ if _, ok := seen[m[1]]; !ok {
+ seen[m[1]] = struct{}{}
+ fields = append(fields, m[1])
+ }
+ }
+
+ cliFiles := listGoFiles(filepath.Join(v.Dir, "internal", "cli"))
+ var cliSources []string
+ for _, file := range cliFiles {
+ if filepath.Base(file) == "root.go" {
+ continue
+ }
+ data, err := os.ReadFile(file)
+ if err != nil {
+ continue
+ }
+ cliSources = append(cliSources, string(data))
+ }
+
+ var clientSource string
+ clientData, err := os.ReadFile(filepath.Join(v.Dir, "internal", "client", "client.go"))
+ if err == nil {
+ clientSource = string(clientData)
+ }
+
+ var results []FlagProofResult
+ for _, field := range fields {
+ refCount := 0
+ needle := "flags." + field
+ for _, src := range cliSources {
+ refCount += strings.Count(src, needle)
+ }
+ if clientSource != "" {
+ refCount += strings.Count(clientSource, "f."+field)
+ refCount += strings.Count(clientSource, field)
+ }
+
+ results = append(results, FlagProofResult{
+ Flag: field,
+ FlagName: field,
+ CLIName: camelToKebab(field),
+ References: refCount,
+ RefCount: refCount,
+ DeadFlag: refCount == 0,
+ })
+ }
+
+ return results
+}
+
+var (
+ createTableRe = regexp.MustCompile(`CREATE TABLE\s+(?:IF NOT EXISTS\s+)?(\w+)\s*\(`)
+ createVirtualTableRe = regexp.MustCompile(`CREATE VIRTUAL TABLE\s+(?:IF NOT EXISTS\s+)?(\w+)\s+USING\s+fts5`)
+)
+
+var exemptTables = map[string]struct{}{
+ "sync_state": {},
+ "resources": {},
+}
+
+func (v *Verifier) PipelineProof() []PipelineProofResult {
+ if v == nil {
+ return nil
+ }
+
+ storeData, err := os.ReadFile(filepath.Join(v.Dir, "internal", "store", "store.go"))
+ if err != nil {
+ return nil
+ }
+ storeSrc := string(storeData)
+
+ tableMatches := createTableRe.FindAllStringSubmatch(storeSrc, -1)
+ ftsMatches := createVirtualTableRe.FindAllStringSubmatch(storeSrc, -1)
+
+ ftsSet := make(map[string]struct{})
+ for _, m := range ftsMatches {
+ ftsSet[m[1]] = struct{}{}
+ }
+
+ cliFiles := listGoFiles(filepath.Join(v.Dir, "internal", "cli"))
+ var cliSources []string
+ for _, file := range cliFiles {
+ data, err := os.ReadFile(file)
+ if err != nil {
+ continue
+ }
+ cliSources = append(cliSources, string(data))
+ }
+ allCLI := strings.Join(cliSources, "\n")
+ allCLILower := strings.ToLower(allCLI)
+
+ var results []PipelineProofResult
+
+ for _, m := range tableMatches {
+ tableName := m[1]
+
+ if _, exempt := exemptTables[tableName]; exempt {
+ continue
+ }
+ if strings.HasSuffix(tableName, "_fts") {
+ continue
+ }
+
+ pascal := pascalCase(tableName)
+ ftsName := tableName + "_fts"
+ _, ftsExists := ftsSet[ftsName]
+
+ columnCount := countTableColumns(storeSrc, tableName)
+
+ r := PipelineProofResult{
+ Table: tableName,
+ TableName: tableName,
+ Columns: columnCount,
+ FTSExists: ftsExists,
+ HasFTS: ftsExists,
+ }
+
+ upsertNeedle := "Upsert" + pascal
+ insertNeedle := strings.ToLower("INSERT INTO " + tableName)
+ insertReplaceNeedle := strings.ToLower("INSERT OR REPLACE INTO " + tableName)
+ if strings.Contains(allCLI, upsertNeedle) {
+ r.HasWrite = true
+ r.WritePath = upsertNeedle
+ } else if strings.Contains(allCLILower, insertNeedle) || strings.Contains(allCLILower, insertReplaceNeedle) {
+ r.HasWrite = true
+ r.WritePath = "INSERT INTO " + tableName
+ }
+
+ fromNeedle := strings.ToLower("FROM " + tableName)
+ joinNeedle := strings.ToLower("JOIN " + tableName)
+ getNeedle := "Get" + pascal
+ queryNeedle := "Query" + pascal
+ if strings.Contains(allCLILower, fromNeedle) || strings.Contains(allCLILower, joinNeedle) {
+ r.HasRead = true
+ r.ReadPath = "FROM/JOIN " + tableName
+ } else if strings.Contains(allCLI, getNeedle) || strings.Contains(allCLI, queryNeedle) {
+ r.HasRead = true
+ r.ReadPath = getNeedle
+ }
+
+ if ftsExists {
+ searchNeedle := "Search" + pascal
+ ftsMatchNeedle := strings.ToLower(ftsName + " MATCH")
+ if strings.Contains(allCLI, searchNeedle) || strings.Contains(allCLILower, ftsMatchNeedle) {
+ r.HasSearch = true
+ r.SearchPath = searchNeedle
+ }
+ }
+
+ r.GhostTable = !r.HasWrite
+ r.OrphanFTS = ftsExists && !r.HasSearch
+
+ results = append(results, r)
+ }
+
+ return results
+}
+
+func (v *Verifier) AuthProof() AuthProofResult {
+ if v == nil {
+ return AuthProofResult{Match: true, Detail: "nil verifier"}
+ }
+
+ result := AuthProofResult{
+ Match: true,
+ Detail: "no recognized auth scheme in spec",
+ }
+
+ if v.spec == nil {
+ result.Detail = "spec not provided; auth check skipped"
+ return result
+ }
+
+ expectedPrefix := ""
+ for name, scheme := range v.spec.Components.SecuritySchemes {
+ if strings.Contains(strings.ToLower(name), "bot") {
+ schemeName := name + ` scheme (expects "Bot " prefix)`
+ result.SpecFormat = schemeName
+ result.SpecScheme = schemeName
+ expectedPrefix = "Bot "
+ break
+ }
+ if strings.EqualFold(scheme.Type, "http") && strings.EqualFold(scheme.Scheme, "bearer") {
+ schemeName := `http bearer scheme (expects "Bearer " prefix)`
+ result.SpecFormat = schemeName
+ result.SpecScheme = schemeName
+ expectedPrefix = "Bearer "
+ }
+ }
+
+ clientData, err := os.ReadFile(filepath.Join(v.Dir, "internal", "client", "client.go"))
+ if err != nil {
+ result.Match = false
+ result.Mismatch = true
+ result.Detail = fmt.Sprintf("failed to read client.go: %v", err)
+ return result
+ }
+
+ clientSource := string(clientData)
+ switch {
+ case strings.Contains(clientSource, `"Bot "`):
+ result.GeneratedFormat = "Bot "
+ result.GeneratedScheme = "Bot "
+ case strings.Contains(clientSource, `"Bearer "`):
+ result.GeneratedFormat = "Bearer "
+ result.GeneratedScheme = "Bearer "
+ default:
+ result.GeneratedFormat = "unknown"
+ result.GeneratedScheme = "unknown"
+ }
+
+ envVarRe := regexp.MustCompile(`os\.Getenv\("([^"]+)"\)`)
+ envMatches := envVarRe.FindAllStringSubmatch(clientSource, -1)
+ if len(envMatches) > 0 {
+ for _, m := range envMatches {
+ envName := m[1]
+ if strings.HasSuffix(envName, "_TOKEN") || strings.HasSuffix(envName, "_API_KEY") || strings.HasSuffix(envName, "_KEY") {
+ result.EnvVarCorrect = true
+ break
+ }
+ }
+ }
+
+ if expectedPrefix == "" {
+ result.Detail = "no bot/bearer scheme detected in spec"
+ return result
+ }
+
+ result.Match = result.GeneratedFormat == expectedPrefix
+ result.Mismatch = !result.Match
+ if result.Match {
+ result.Detail = fmt.Sprintf("spec and generated client both use %q", strings.TrimSpace(expectedPrefix))
+ } else {
+ result.Detail = fmt.Sprintf("spec expects %q but generated client uses %q", strings.TrimSpace(expectedPrefix), strings.TrimSpace(result.GeneratedFormat))
+ }
+
+ return result
+}
+
+func findMatchingSpecPath(path string, specKeys []string) string {
+ paramRe := regexp.MustCompile(`\\\{[^/]+\\\}`)
+ for _, key := range specKeys {
+ quoted := regexp.QuoteMeta(key)
+ regex := "^" + paramRe.ReplaceAllString(quoted, `[^/]+`) + "$"
+ re, err := regexp.Compile(regex)
+ if err != nil {
+ continue
+ }
+ if re.MatchString(path) {
+ return key
+ }
+ }
+ return ""
+}
+
+func pascalCase(tableName string) string {
+ parts := strings.Split(tableName, "_")
+ var b strings.Builder
+ for _, part := range parts {
+ if part == "" {
+ continue
+ }
+ singular := part
+ if len(singular) > 1 && strings.HasSuffix(singular, "s") {
+ singular = singular[:len(singular)-1]
+ }
+ runes := []rune(singular)
+ runes[0] = unicode.ToUpper(runes[0])
+ b.WriteString(string(runes))
+ }
+ return b.String()
+}
+
+func countTableColumns(storeSrc, tableName string) int {
+ re := regexp.MustCompile(`(?is)CREATE TABLE\s+(?:IF NOT EXISTS\s+)?` + regexp.QuoteMeta(tableName) + `\s*\((.*?)\);`)
+ match := re.FindStringSubmatch(storeSrc)
+ if match == nil {
+ return 0
+ }
+ columns := 0
+ for _, line := range strings.Split(match[1], "\n") {
+ line = strings.TrimSpace(strings.TrimSuffix(line, ","))
+ if line == "" {
+ continue
+ }
+ upper := strings.ToUpper(line)
+ if strings.HasPrefix(upper, "PRIMARY KEY") || strings.HasPrefix(upper, "FOREIGN KEY") || strings.HasPrefix(upper, "UNIQUE") || strings.HasPrefix(upper, "CONSTRAINT") || strings.HasPrefix(upper, "CHECK") {
+ continue
+ }
+ columns++
+ }
+ return columns
+}
+
+func camelToKebab(s string) string {
+ var b strings.Builder
+ for i, r := range s {
+ if unicode.IsUpper(r) {
+ if i > 0 {
+ b.WriteByte('-')
+ }
+ b.WriteRune(unicode.ToLower(r))
+ } else {
+ b.WriteRune(r)
+ }
+ }
+ return b.String()
+}
diff --git a/internal/pipeline/verify_report.go b/internal/pipeline/verify_report.go
new file mode 100644
index 00000000..68ec0546
--- /dev/null
+++ b/internal/pipeline/verify_report.go
@@ -0,0 +1,264 @@
+package pipeline
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "path/filepath"
+ "strings"
+)
+
+// VerificationReport holds the results of a Proof of Behavior verification run.
+type VerificationReport struct {
+ Dir string `json:"dir"`
+ SpecPath string `json:"spec_path,omitempty"`
+ Paths []PathProofResult `json:"paths"`
+ Flags []FlagProofResult `json:"flags"`
+ Pipeline []PipelineProofResult `json:"pipeline"`
+ Auth AuthProofResult `json:"auth"`
+
+ HallucinatedPaths int `json:"hallucinated_paths"`
+ DeadFlags int `json:"dead_flags"`
+ DeadFunctions int `json:"dead_functions"`
+ GhostTables int `json:"ghost_tables"`
+ OrphanFTS int `json:"orphan_fts"`
+ AuthMismatch bool `json:"auth_mismatch"`
+
+ Verdict string `json:"verdict"`
+ Issues []string `json:"issues,omitempty"`
+}
+
+// RunVerification runs the full Proof of Behavior verification suite against a generated CLI.
+func RunVerification(dir, specPath string) (*VerificationReport, error) {
+ v, err := NewVerifier(dir, specPath)
+ if err != nil {
+ return nil, fmt.Errorf("creating verifier: %w", err)
+ }
+
+ if err := v.CompileGate(); err != nil {
+ report := &VerificationReport{
+ Dir: dir,
+ SpecPath: specPath,
+ Verdict: "FAIL",
+ Issues: []string{fmt.Sprintf("compile gate failed: %s", err)},
+ }
+ if writeErr := writeReport(dir, report); writeErr != nil {
+ return report, fmt.Errorf("writing report after compile failure: %w", writeErr)
+ }
+ return report, nil
+ }
+
+ paths := v.PathProof()
+ flags := v.FlagProof()
+ pipeline := v.PipelineProof()
+ auth := v.AuthProof()
+
+ report := &VerificationReport{
+ Dir: dir,
+ SpecPath: specPath,
+ Paths: paths,
+ Flags: flags,
+ Pipeline: pipeline,
+ Auth: auth,
+ }
+
+ // Compute summary counts.
+ for _, p := range paths {
+ if !p.Valid {
+ report.HallucinatedPaths++
+ }
+ }
+ for _, f := range flags {
+ if f.References == 0 {
+ report.DeadFlags++
+ }
+ }
+ for _, p := range pipeline {
+ if !p.HasWrite {
+ report.GhostTables++
+ }
+ if p.HasFTS && !p.HasSearch {
+ report.OrphanFTS++
+ }
+ }
+ report.AuthMismatch = auth.Mismatch
+
+ report.Issues = collectVerificationIssues(report)
+ report.Verdict = deriveVerificationVerdict(report)
+
+ if err := writeReport(dir, report); err != nil {
+ return report, fmt.Errorf("writing verification report: %w", err)
+ }
+
+ return report, nil
+}
+
+// deriveVerificationVerdict determines PASS, WARN, or FAIL from the report.
+func deriveVerificationVerdict(report *VerificationReport) string {
+ // FAIL conditions.
+ if report.HallucinatedPaths > 0 {
+ return "FAIL"
+ }
+ if report.AuthMismatch {
+ return "FAIL"
+ }
+ for _, p := range report.Pipeline {
+ if !p.HasWrite && p.Columns >= 5 {
+ return "FAIL"
+ }
+ }
+
+ // WARN conditions.
+ if report.DeadFlags > 0 {
+ return "WARN"
+ }
+ if report.OrphanFTS > 0 {
+ return "WARN"
+ }
+ for _, p := range report.Pipeline {
+ if !p.HasWrite && p.Columns < 5 {
+ return "WARN"
+ }
+ }
+
+ return "PASS"
+}
+
+// collectVerificationIssues generates human-readable issue strings for each problem found.
+func collectVerificationIssues(report *VerificationReport) []string {
+ var issues []string
+
+ for _, p := range report.Paths {
+ if !p.Valid {
+ issues = append(issues, fmt.Sprintf("hallucinated path: %s (not in spec)", p.Path))
+ }
+ }
+ for _, f := range report.Flags {
+ if f.References == 0 {
+ issues = append(issues, fmt.Sprintf("dead flag: %s (0 references)", f.Flag))
+ }
+ }
+ for _, p := range report.Pipeline {
+ if !p.HasWrite {
+ issues = append(issues, fmt.Sprintf("ghost table: %s (no WRITE path)", p.Table))
+ }
+ if p.HasFTS && !p.HasSearch {
+ issues = append(issues, fmt.Sprintf("orphan FTS: %s_fts (no search command queries it)", p.Table))
+ }
+ }
+ if report.Auth.Mismatch {
+ issues = append(issues, fmt.Sprintf("auth mismatch: spec expects %s, generated uses %s", report.Auth.SpecScheme, report.Auth.GeneratedScheme))
+ }
+
+ return issues
+}
+
+// Markdown generates a human-readable markdown report.
+func (r *VerificationReport) Markdown() string {
+ var b strings.Builder
+
+ b.WriteString("# Proof of Behavior Report\n\n")
+ b.WriteString(fmt.Sprintf("**Verdict: %s**\n\n", r.Verdict))
+
+ // Path Proof section.
+ validPaths := 0
+ for _, p := range r.Paths {
+ if p.Valid {
+ validPaths++
+ }
+ }
+ b.WriteString("## Path Proof\n")
+ b.WriteString(fmt.Sprintf("Tested: %d | Valid: %d | Hallucinated: %d\n\n", len(r.Paths), validPaths, r.HallucinatedPaths))
+
+ if r.HallucinatedPaths > 0 {
+ b.WriteString("| Path | Status |\n")
+ b.WriteString("|------|--------|\n")
+ for _, p := range r.Paths {
+ if !p.Valid {
+ b.WriteString(fmt.Sprintf("| `%s` | INVALID |\n", p.Path))
+ }
+ }
+ b.WriteString("\n")
+ }
+
+ // Flag Proof section.
+ b.WriteString("## Flag Proof\n")
+ b.WriteString(fmt.Sprintf("Total: %d | Dead: %d\n\n", len(r.Flags), r.DeadFlags))
+
+ if r.DeadFlags > 0 {
+ for _, f := range r.Flags {
+ if f.References == 0 {
+ b.WriteString(fmt.Sprintf("- `%s` (0 references)\n", f.Flag))
+ }
+ }
+ b.WriteString("\n")
+ }
+
+ // Pipeline Proof section.
+ b.WriteString("## Pipeline Proof\n")
+ b.WriteString(fmt.Sprintf("Tables: %d | Ghost: %d | Orphan FTS: %d\n\n", len(r.Pipeline), r.GhostTables, r.OrphanFTS))
+
+ if len(r.Pipeline) > 0 {
+ b.WriteString("| Table | WRITE | READ | SEARCH |\n")
+ b.WriteString("|-------|-------|------|--------|\n")
+ for _, p := range r.Pipeline {
+ w := boolMark(p.HasWrite)
+ rd := boolMark(p.HasRead)
+ s := boolMark(p.HasSearch)
+ b.WriteString(fmt.Sprintf("| %s | %s | %s | %s |\n", p.Table, w, rd, s))
+ }
+ b.WriteString("\n")
+ }
+
+ // Auth Proof section.
+ b.WriteString("## Auth Proof\n")
+ b.WriteString(fmt.Sprintf("Spec: %s | Generated: %s | Match: %t\n\n", r.Auth.SpecScheme, r.Auth.GeneratedScheme, !r.Auth.Mismatch))
+
+ // Issues section.
+ if len(r.Issues) > 0 {
+ b.WriteString("## Issues\n")
+ for _, issue := range r.Issues {
+ b.WriteString(fmt.Sprintf("- %s\n", issue))
+ }
+ b.WriteString("\n")
+ }
+
+ return b.String()
+}
+
+// LoadVerificationReport loads a verification report from verification-report.json in dir.
+func LoadVerificationReport(dir string) (*VerificationReport, error) {
+ path := filepath.Join(dir, "verification-report.json")
+ data, err := os.ReadFile(path)
+ if err != nil {
+ return nil, fmt.Errorf("reading verification report: %w", err)
+ }
+
+ var report VerificationReport
+ if err := json.Unmarshal(data, &report); err != nil {
+ return nil, fmt.Errorf("parsing verification report: %w", err)
+ }
+
+ return &report, nil
+}
+
+func writeReport(dir string, report *VerificationReport) error {
+ data, err := json.MarshalIndent(report, "", " ")
+ if err != nil {
+ return fmt.Errorf("marshaling report: %w", err)
+ }
+
+ path := filepath.Join(dir, "verification-report.json")
+ if err := os.WriteFile(path, data, 0o644); err != nil {
+ return fmt.Errorf("writing %s: %w", path, err)
+ }
+
+ return nil
+}
+
+func boolMark(v bool) string {
+ if v {
+ return "Y"
+ }
+ return "-"
+}
diff --git a/internal/pipeline/verify_test.go b/internal/pipeline/verify_test.go
new file mode 100644
index 00000000..a7490591
--- /dev/null
+++ b/internal/pipeline/verify_test.go
@@ -0,0 +1,556 @@
+package pipeline
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+func setupVerifierDirs(t *testing.T, dir string) {
+ t.Helper()
+ require.NoError(t, os.MkdirAll(filepath.Join(dir, "internal", "cli"), 0o755))
+ require.NoError(t, os.MkdirAll(filepath.Join(dir, "internal", "client"), 0o755))
+ require.NoError(t, os.MkdirAll(filepath.Join(dir, "internal", "store"), 0o755))
+}
+
+func TestPathProof_DetectsHallucinatedEndpoint(t *testing.T) {
+ dir := t.TempDir()
+ setupVerifierDirs(t, dir)
+
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "users_get.go"), `package cli
+func usersGet() {
+ path = "/users/{id}"
+}
+`)
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "bogus_get.go"), `package cli
+func bogusGet() {
+ path = "/bogus/endpoint"
+}
+`)
+
+ specPath := filepath.Join(dir, "spec.json")
+ writeTestFile(t, specPath, `{
+ "paths": {
+ "/users/{user_id}": {}
+ },
+ "components": { "securitySchemes": {} }
+}`)
+
+ v, err := NewVerifier(dir, specPath)
+ require.NoError(t, err)
+
+ results := v.PathProof()
+ require.Len(t, results, 2)
+
+ var validCount, invalidCount int
+ for _, r := range results {
+ if r.InSpec {
+ validCount++
+ } else {
+ invalidCount++
+ assert.Equal(t, "/bogus/endpoint", r.Path)
+ }
+ }
+ assert.Equal(t, 1, validCount, "one path should be in spec")
+ assert.Equal(t, 1, invalidCount, "one path should be hallucinated")
+}
+
+func TestPathProof_SkipsLocalCommands(t *testing.T) {
+ dir := t.TempDir()
+ setupVerifierDirs(t, dir)
+
+ // These are in verifyInfraFiles and should be skipped
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "search.go"), `package cli
+func runSearch() {
+ query := "hello"
+ _ = query
+}
+`)
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "sync.go"), `package cli
+func runSync() {
+ state := "ready"
+ _ = state
+}
+`)
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "analytics.go"), `package cli
+func runAnalytics() {
+ count := 42
+ _ = count
+}
+`)
+
+ specPath := filepath.Join(dir, "spec.json")
+ writeTestFile(t, specPath, `{
+ "paths": { "/users/{id}": {} },
+ "components": { "securitySchemes": {} }
+}`)
+
+ v, err := NewVerifier(dir, specPath)
+ require.NoError(t, err)
+
+ results := v.PathProof()
+ assert.Empty(t, results, "infra/local files should be skipped, no false positives")
+}
+
+func TestFlagProof_DetectsDeadFlags(t *testing.T) {
+ dir := t.TempDir()
+ setupVerifierDirs(t, dir)
+
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "root.go"), `package cli
+type rootFlags struct {
+ jsonOutput bool
+ csvOutput bool
+ dryRun bool
+}
+func initFlags(flags *rootFlags) {
+ _ = &flags.jsonOutput
+ _ = &flags.csvOutput
+ _ = &flags.dryRun
+}
+`)
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "users_list.go"), `package cli
+func usersList() {
+ flags.jsonOutput = true
+}
+`)
+ // client.go uses "dryRun" (lowercase d) to match the field name exactly
+ writeTestFile(t, filepath.Join(dir, "internal", "client", "client.go"), `package client
+type Client struct {
+ dryRun bool
+}
+func (c *Client) Do() {
+ if c.dryRun {
+ return
+ }
+}
+`)
+
+ v, err := NewVerifier(dir, "")
+ require.NoError(t, err)
+
+ results := v.FlagProof()
+ require.Len(t, results, 3)
+
+ flagMap := make(map[string]FlagProofResult)
+ for _, r := range results {
+ flagMap[r.Flag] = r
+ }
+
+ assert.False(t, flagMap["jsonOutput"].DeadFlag, "jsonOutput is used in CLI")
+ assert.True(t, flagMap["csvOutput"].DeadFlag, "csvOutput has no references")
+ // FlagProof counts strings.Count(clientSource, field) where field="dryRun"
+ assert.False(t, flagMap["dryRun"].DeadFlag, "dryRun is referenced in client.go")
+}
+
+func TestFlagProof_IndirectUsageThroughClient(t *testing.T) {
+ dir := t.TempDir()
+ setupVerifierDirs(t, dir)
+
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "root.go"), `package cli
+type rootFlags struct {
+ timeout int
+}
+func initFlags(flags *rootFlags) {
+ _ = &flags.timeout
+}
+`)
+ // No CLI file references flags.timeout, but client.go has "timeout" substring
+ writeTestFile(t, filepath.Join(dir, "internal", "client", "client.go"), `package client
+type Client struct {
+ timeout int
+}
+func (c *Client) SetTimeout(t int) {
+ c.timeout = t
+}
+`)
+
+ v, err := NewVerifier(dir, "")
+ require.NoError(t, err)
+
+ results := v.FlagProof()
+ require.Len(t, results, 1)
+ assert.Equal(t, "timeout", results[0].Flag)
+ assert.False(t, results[0].DeadFlag, "timeout is referenced indirectly through client.go")
+ assert.Greater(t, results[0].RefCount, 0)
+}
+
+func TestPipelineProof_DetectsGhostTable(t *testing.T) {
+ dir := t.TempDir()
+ setupVerifierDirs(t, dir)
+
+ writeTestFile(t, filepath.Join(dir, "internal", "store", "store.go"), "package store\n"+
+ "func schema() string {\n"+
+ "\treturn `\n"+
+ "\t\tCREATE TABLE IF NOT EXISTS messages (\n"+
+ "\t\t\tid TEXT PRIMARY KEY,\n"+
+ "\t\t\tchannel_id TEXT NOT NULL,\n"+
+ "\t\t\tauthor TEXT NOT NULL,\n"+
+ "\t\t\tcontent TEXT NOT NULL,\n"+
+ "\t\t\ttimestamp TEXT NOT NULL,\n"+
+ "\t\t\tdata JSON NOT NULL\n"+
+ "\t\t);\n"+
+ "\t\tCREATE TABLE IF NOT EXISTS sync_state (\n"+
+ "\t\t\tentity_type TEXT PRIMARY KEY,\n"+
+ "\t\t\tlast_sync_at TEXT NOT NULL,\n"+
+ "\t\t\tcursor TEXT\n"+
+ "\t\t);\n"+
+ "\t`\n"+
+ "}\n")
+
+ // sync.go calls generic Upsert, NOT UpsertMessage
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "sync.go"), `package cli
+func runSync() {
+ _ = Upsert("something")
+}
+`)
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "search.go"), `package cli
+func runSearch() {
+ _ = SearchMessage("query")
+}
+`)
+
+ v, err := NewVerifier(dir, "")
+ require.NoError(t, err)
+
+ results := v.PipelineProof()
+ require.Len(t, results, 1, "sync_state is exempt, only messages should appear")
+
+ msg := results[0]
+ assert.Equal(t, "messages", msg.TableName)
+ assert.True(t, msg.GhostTable, "messages has no write path (UpsertMessage not called)")
+ assert.GreaterOrEqual(t, msg.Columns, 5)
+}
+
+func TestPipelineProof_DetectsOrphanFTS(t *testing.T) {
+ dir := t.TempDir()
+ setupVerifierDirs(t, dir)
+
+ writeTestFile(t, filepath.Join(dir, "internal", "store", "store.go"), "package store\n"+
+ "func schema() string {\n"+
+ "\treturn `\n"+
+ "\t\tCREATE TABLE IF NOT EXISTS messages (\n"+
+ "\t\t\tid TEXT PRIMARY KEY,\n"+
+ "\t\t\tchannel_id TEXT NOT NULL,\n"+
+ "\t\t\tauthor TEXT NOT NULL,\n"+
+ "\t\t\tcontent TEXT NOT NULL,\n"+
+ "\t\t\ttimestamp TEXT NOT NULL,\n"+
+ "\t\t\tdata JSON NOT NULL\n"+
+ "\t\t);\n"+
+ "\t\tCREATE VIRTUAL TABLE IF NOT EXISTS messages_fts USING fts5(content, content='messages');\n"+
+ "\t`\n"+
+ "}\n")
+
+ // sync.go writes to messages via UpsertMessage
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "sync.go"), `package cli
+func runSync() {
+ _ = UpsertMessage("data")
+}
+`)
+ // But NO command calls SearchMessage
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "list.go"), `package cli
+func runList() {
+ _ = GetMessage("id")
+}
+`)
+
+ v, err := NewVerifier(dir, "")
+ require.NoError(t, err)
+
+ results := v.PipelineProof()
+ require.Len(t, results, 1)
+
+ msg := results[0]
+ assert.Equal(t, "messages", msg.TableName)
+ assert.True(t, msg.HasWrite, "UpsertMessage is called")
+ assert.False(t, msg.GhostTable, "write path exists")
+ assert.True(t, msg.FTSExists, "messages_fts exists")
+ assert.True(t, msg.OrphanFTS, "no command calls SearchMessage")
+}
+
+func TestPipelineProof_HealthyPipeline(t *testing.T) {
+ dir := t.TempDir()
+ setupVerifierDirs(t, dir)
+
+ writeTestFile(t, filepath.Join(dir, "internal", "store", "store.go"), "package store\n"+
+ "func schema() string {\n"+
+ "\treturn `\n"+
+ "\t\tCREATE TABLE IF NOT EXISTS messages (\n"+
+ "\t\t\tid TEXT PRIMARY KEY,\n"+
+ "\t\t\tcontent TEXT NOT NULL,\n"+
+ "\t\t\tdata JSON NOT NULL\n"+
+ "\t\t);\n"+
+ "\t`\n"+
+ "}\n")
+
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "sync.go"), `package cli
+func runSync() {
+ _ = UpsertMessage("data")
+}
+`)
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "search.go"), `package cli
+func runSearch() {
+ _ = SearchMessage("query")
+}
+`)
+
+ v, err := NewVerifier(dir, "")
+ require.NoError(t, err)
+
+ results := v.PipelineProof()
+ require.Len(t, results, 1)
+
+ msg := results[0]
+ assert.False(t, msg.GhostTable, "write path exists")
+ assert.False(t, msg.OrphanFTS, "no FTS table, so no orphan")
+}
+
+func TestAuthProof_DetectsMismatch(t *testing.T) {
+ dir := t.TempDir()
+ setupVerifierDirs(t, dir)
+
+ writeTestFile(t, filepath.Join(dir, "internal", "client", "client.go"), `package client
+func authHeader(token string) string {
+ return "Bearer " + token
+}
+`)
+
+ specPath := filepath.Join(dir, "spec.json")
+ writeTestFile(t, specPath, `{
+ "paths": {},
+ "components": {
+ "securitySchemes": {
+ "BotToken": {
+ "type": "http",
+ "scheme": "bearer"
+ }
+ }
+ }
+}`)
+
+ v, err := NewVerifier(dir, specPath)
+ require.NoError(t, err)
+
+ result := v.AuthProof()
+ assert.False(t, result.Match, "Bearer in client vs BotToken scheme should mismatch")
+ assert.True(t, result.Mismatch)
+ assert.Contains(t, result.GeneratedScheme, "Bearer")
+}
+
+func TestRunVerification_FullIntegration(t *testing.T) {
+ dir := t.TempDir()
+ setupVerifierDirs(t, dir)
+
+ // Hallucinated path
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "users_get.go"), `package cli
+func usersGet() {
+ path = "/users/{id}"
+}
+`)
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "bogus_get.go"), `package cli
+func bogusGet() {
+ path = "/bogus/hallucinated"
+}
+`)
+
+ // Dead flag
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "root.go"), `package cli
+type rootFlags struct {
+ deadFlag bool
+}
+func initFlags(flags *rootFlags) {
+ _ = &flags.deadFlag
+}
+`)
+
+ // Ghost table
+ writeTestFile(t, filepath.Join(dir, "internal", "store", "store.go"), "package store\n"+
+ "func schema() string {\n"+
+ "\treturn `\n"+
+ "\t\tCREATE TABLE IF NOT EXISTS messages (\n"+
+ "\t\t\tid TEXT PRIMARY KEY,\n"+
+ "\t\t\tchannel_id TEXT NOT NULL,\n"+
+ "\t\t\tauthor TEXT NOT NULL,\n"+
+ "\t\t\tcontent TEXT NOT NULL,\n"+
+ "\t\t\ttimestamp TEXT NOT NULL,\n"+
+ "\t\t\tdata JSON NOT NULL\n"+
+ "\t\t);\n"+
+ "\t`\n"+
+ "}\n")
+
+ // No UpsertMessage or SearchMessage calls
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "list.go"), `package cli
+func runList() {
+ _ = "nothing relevant"
+}
+`)
+
+ writeTestFile(t, filepath.Join(dir, "internal", "client", "client.go"), `package client
+func authHeader(token string) string {
+ return "Bearer " + token
+}
+`)
+
+ specPath := filepath.Join(dir, "spec.json")
+ writeTestFile(t, specPath, `{
+ "paths": {
+ "/users/{user_id}": {}
+ },
+ "components": { "securitySchemes": {} }
+}`)
+
+ // We can't run RunVerification because it calls CompileGate (go build).
+ // Instead, test the individual proofs and verdict derivation.
+ v, err := NewVerifier(dir, specPath)
+ require.NoError(t, err)
+
+ paths := v.PathProof()
+ flags := v.FlagProof()
+ pipeline := v.PipelineProof()
+ auth := v.AuthProof()
+
+ report := &VerificationReport{
+ Dir: dir,
+ SpecPath: specPath,
+ Paths: paths,
+ Flags: flags,
+ Pipeline: pipeline,
+ Auth: auth,
+ }
+
+ for _, p := range paths {
+ if !p.Valid {
+ report.HallucinatedPaths++
+ }
+ }
+ for _, f := range flags {
+ if f.References == 0 {
+ report.DeadFlags++
+ }
+ }
+ for _, p := range pipeline {
+ if !p.HasWrite {
+ report.GhostTables++
+ }
+ if p.HasFTS && !p.HasSearch {
+ report.OrphanFTS++
+ }
+ }
+ report.AuthMismatch = auth.Mismatch
+
+ report.Issues = collectVerificationIssues(report)
+ report.Verdict = deriveVerificationVerdict(report)
+
+ assert.Equal(t, "FAIL", report.Verdict)
+ assert.Equal(t, 1, report.HallucinatedPaths)
+ assert.Equal(t, 1, report.DeadFlags)
+ assert.Equal(t, 1, report.GhostTables)
+ assert.GreaterOrEqual(t, len(report.Issues), 3)
+}
+
+func TestRemediate_RemovesDeadFlags(t *testing.T) {
+ dir := t.TempDir()
+ setupVerifierDirs(t, dir)
+
+ writeTestFile(t, filepath.Join(dir, "internal", "cli", "root.go"), `package cli
+type rootFlags struct {
+ jsonOutput bool
+ deadFlag bool
+}
+func initFlags(flags *rootFlags) {
+ _ = &flags.jsonOutput
+ _ = &flags.deadFlag
+}
+`)
+
+ report := &VerificationReport{
+ Flags: []FlagProofResult{
+ {Flag: "deadFlag", FlagName: "deadFlag", DeadFlag: true, References: 0},
+ },
+ }
+
+ // Remediate will try compileGateCheck which will fail (no go.mod), but
+ // the dead flag removal itself should happen. We check the file content
+ // before the compile gate reverts it by inspecting the removeDeadFlags function directly.
+ err := removeDeadFlags(dir, []string{"deadFlag"})
+ require.NoError(t, err)
+
+ data, err := os.ReadFile(filepath.Join(dir, "internal", "cli", "root.go"))
+ require.NoError(t, err)
+ content := string(data)
+
+ assert.NotContains(t, content, "&flags.deadFlag")
+ assert.NotContains(t, content, "deadFlag")
+ assert.Contains(t, content, "&flags.jsonOutput", "jsonOutput should remain")
+
+ _ = report // used to set up context
+}
+
+func TestDeriveVerificationVerdict(t *testing.T) {
+ tests := []struct {
+ name string
+ report *VerificationReport
+ want string
+ }{
+ {
+ name: "hallucinated paths -> FAIL",
+ report: &VerificationReport{
+ HallucinatedPaths: 1,
+ },
+ want: "FAIL",
+ },
+ {
+ name: "auth mismatch -> FAIL",
+ report: &VerificationReport{
+ AuthMismatch: true,
+ },
+ want: "FAIL",
+ },
+ {
+ name: "ghost table with 5+ columns -> FAIL",
+ report: &VerificationReport{
+ Pipeline: []PipelineProofResult{
+ {TableName: "messages", HasWrite: false, Columns: 6},
+ },
+ },
+ want: "FAIL",
+ },
+ {
+ name: "dead flags only -> WARN",
+ report: &VerificationReport{
+ DeadFlags: 2,
+ },
+ want: "WARN",
+ },
+ {
+ name: "orphan FTS -> WARN",
+ report: &VerificationReport{
+ OrphanFTS: 1,
+ },
+ want: "WARN",
+ },
+ {
+ name: "ghost table with <5 columns -> WARN",
+ report: &VerificationReport{
+ Pipeline: []PipelineProofResult{
+ {TableName: "small", HasWrite: false, Columns: 3},
+ },
+ },
+ want: "WARN",
+ },
+ {
+ name: "all clean -> PASS",
+ report: &VerificationReport{},
+ want: "PASS",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ got := deriveVerificationVerdict(tt.report)
+ assert.Equal(t, tt.want, got)
+ })
+ }
+}
← 345b0753 docs: update README with v2 overhaul, agent-first features,
·
back to Cli Printing Press
·
feat(templates): discrawl-inspired sync performance upgrades 8e926036 →