← back to Cli Printing Press
feat(cli): add --data-source flag for live/local/auto read resolution (#119)
b88aa97b027755a4582e0024c3f52ac7cfe28922 · 2026-04-03 21:01:37 -0700 · Trevin Chow
Every generated CLI now gets a --data-source auto|live|local flag that
governs all read commands. Default (auto) hits the live API and falls
back to local SQLite on network failure with an honest warning. Local
mode reads synced data directly. Live mode errors on network failure.
Provenance metadata (source, sync timestamp, fallback reason) appears
on every response — JSON wraps in {"results":...,"meta":{...}}, human
output prints a one-line summary to stderr.
Machine changes (every future CLI benefits):
- root.go.tmpl: --data-source flag + validation
- data_source.go.tmpl: NEW vision-gated template with resolveRead(),
resolvePaginatedRead(), network error detection, store fallback
- command_endpoint.go.tmpl: GET commands call resolveRead() when
HasStore is true, provenance wraps JSON, all output flags preserved
- command_promoted.go.tmpl: same treatment for promoted aliases
- search.go.tmpl: tries API search endpoint (GET or POST) when
available, falls back to local FTS with provenance
- helpers.go.tmpl: DataProvenance struct, printProvenance(),
wrapWithProvenance(), consolidated defaultDBPath()
- profiler.go: extracts SearchEndpointPath, SearchQueryParam,
SearchEndpointMethod from detected search endpoints
- generator.go: flows search endpoint fields into visionData,
adds HasStore to endpoint/promoted template data
- DB path consolidated to ~/.local/share/ across all templates
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Files touched
A docs/plans/2026-04-03-004-feat-data-source-resolution-plan.mdM internal/generator/generator.goM internal/generator/generator_test.goM internal/generator/templates/analytics.go.tmplM internal/generator/templates/channel_workflow.go.tmplM internal/generator/templates/command_endpoint.go.tmplM internal/generator/templates/command_promoted.go.tmplA internal/generator/templates/data_source.go.tmplM internal/generator/templates/helpers.go.tmplM internal/generator/templates/insights/health_score.go.tmplM internal/generator/templates/insights/similar.go.tmplM internal/generator/templates/mcp_tools.go.tmplM internal/generator/templates/root.go.tmplM internal/generator/templates/search.go.tmplM internal/generator/templates/sync.go.tmplM internal/generator/templates/workflows/pm_load.go.tmplM internal/generator/templates/workflows/pm_orphans.go.tmplM internal/generator/templates/workflows/pm_stale.go.tmplM internal/profiler/profiler.go
Diff
commit b88aa97b027755a4582e0024c3f52ac7cfe28922
Author: Trevin Chow <trevin@trevinchow.com>
Date: Fri Apr 3 21:01:37 2026 -0700
feat(cli): add --data-source flag for live/local/auto read resolution (#119)
Every generated CLI now gets a --data-source auto|live|local flag that
governs all read commands. Default (auto) hits the live API and falls
back to local SQLite on network failure with an honest warning. Local
mode reads synced data directly. Live mode errors on network failure.
Provenance metadata (source, sync timestamp, fallback reason) appears
on every response — JSON wraps in {"results":...,"meta":{...}}, human
output prints a one-line summary to stderr.
Machine changes (every future CLI benefits):
- root.go.tmpl: --data-source flag + validation
- data_source.go.tmpl: NEW vision-gated template with resolveRead(),
resolvePaginatedRead(), network error detection, store fallback
- command_endpoint.go.tmpl: GET commands call resolveRead() when
HasStore is true, provenance wraps JSON, all output flags preserved
- command_promoted.go.tmpl: same treatment for promoted aliases
- search.go.tmpl: tries API search endpoint (GET or POST) when
available, falls back to local FTS with provenance
- helpers.go.tmpl: DataProvenance struct, printProvenance(),
wrapWithProvenance(), consolidated defaultDBPath()
- profiler.go: extracts SearchEndpointPath, SearchQueryParam,
SearchEndpointMethod from detected search endpoints
- generator.go: flows search endpoint fields into visionData,
adds HasStore to endpoint/promoted template data
- DB path consolidated to ~/.local/share/ across all templates
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
...6-04-03-004-feat-data-source-resolution-plan.md | 404 +++++++++++++++++++++
internal/generator/generator.go | 59 ++-
internal/generator/generator_test.go | 14 +-
internal/generator/templates/analytics.go.tmpl | 4 +-
.../generator/templates/channel_workflow.go.tmpl | 9 +-
.../generator/templates/command_endpoint.go.tmpl | 38 ++
.../generator/templates/command_promoted.go.tmpl | 37 ++
internal/generator/templates/data_source.go.tmpl | 203 +++++++++++
internal/generator/templates/helpers.go.tmpl | 62 ++++
.../templates/insights/health_score.go.tmpl | 7 +-
.../generator/templates/insights/similar.go.tmpl | 7 +-
internal/generator/templates/mcp_tools.go.tmpl | 2 +
internal/generator/templates/root.go.tmpl | 8 +
internal/generator/templates/search.go.tmpl | 174 +++++++--
internal/generator/templates/sync.go.tmpl | 4 +-
.../generator/templates/workflows/pm_load.go.tmpl | 7 +-
.../templates/workflows/pm_orphans.go.tmpl | 7 +-
.../generator/templates/workflows/pm_stale.go.tmpl | 7 +-
internal/profiler/profiler.go | 33 ++
19 files changed, 983 insertions(+), 103 deletions(-)
diff --git a/docs/plans/2026-04-03-004-feat-data-source-resolution-plan.md b/docs/plans/2026-04-03-004-feat-data-source-resolution-plan.md
new file mode 100644
index 00000000..85f99993
--- /dev/null
+++ b/docs/plans/2026-04-03-004-feat-data-source-resolution-plan.md
@@ -0,0 +1,404 @@
+---
+title: "feat: Data Source Resolution — Live-First with Honest Local Fallback"
+type: feat
+status: completed
+date: 2026-04-03
+---
+
+# Data Source Resolution — Live-First with Honest Local Fallback
+
+## Overview
+
+Add a unified `--data-source auto|live|local` flag to every generated CLI so that all read commands share the same data resolution model: default to live API, fall back to local SQLite honestly on network failure, and always tell the user where results came from. Today, regular commands always hit the API (no fallback), and search always hits local SQLite (no live option). These two paths have zero coordination. This change makes every read command data-source-aware with provenance metadata on every response.
+
+## Problem Frame
+
+Generated CLIs have two disconnected data paths:
+
+1. **Regular commands** (`links list`, `orders get`) always hit the live API. If the network is down, they fail. There is no way to use locally synced data.
+2. **`search`** always hits local SQLite FTS5. If sync hasn't run, it errors. If sync ran last week, it silently shows stale data. If the API has a search endpoint, the search command ignores it.
+
+No command tells the user whether data is live or cached, how old cached data is, or why a particular path was chosen. Agents calling `search --json` get results with no freshness metadata — they cannot reason about trustworthiness.
+
+The fundamental issue: **no shared model for "where does data come from."**
+
+## Requirements Trace
+
+- R1. Every read command defaults to live API calls — no behavior change for existing users on non-search commands. Search behavior intentionally changes: it now tries the API search endpoint when available (R8), which is an improvement over the previous local-only behavior.
+- R2. `--data-source local` works for any read command after `sync`, not just search
+- R3. `--data-source auto` falls back to local on network failure with clear warning and sync timestamp
+- R4. Provenance metadata (source, sync age, fallback reason) appears on every response — human and JSON
+- R5. Write commands always hit the API regardless of `--data-source`
+- R6. `sync` command unchanged — remains the only path to populate local data
+- R7. No TTL, no auto-sync, no read-through caching, no silent fallbacks
+- R8. When API has a search endpoint and mode is auto/live, `search` hits the API endpoint
+- R9. When API has no search endpoint, `search` uses local FTS with an explanation of why
+
+## Scope Boundaries
+
+- **Not changing sync.** Sync stays explicit, user-initiated, identical to today.
+- **Not adding read-through caching.** Regular commands do NOT populate the local database as a side effect. However, if local data exists from an explicit `sync`, `auto` mode may read it on network failure. This is fallback usage of pre-synced data, not read-through caching.
+- **Not adding deletion detection.** `sync --full` remains the workaround for deleted-upstream records.
+- **Not adding per-resource data source config.** V1 uses a single flag for all resources.
+- **Not changing write commands.** POST/PUT/PATCH/DELETE always hit the API.
+
+## Context & Research
+
+### Relevant Code and Patterns
+
+- `internal/generator/templates/root.go.tmpl` — rootFlags struct definition, persistent flag registration, `newClient()` helper (line 151)
+- `internal/generator/templates/command_endpoint.go.tmpl` — per-endpoint command generation; GET commands call `c.Get(path, params)` at line 86, POST/PUT/PATCH/DELETE use `c.Post()`/`c.Put()`/etc.
+- `internal/generator/templates/search.go.tmpl` — always queries local SQLite via `db.Search{{pascal .Name}}()`, no live option, no provenance
+- `internal/generator/templates/store.go.tmpl` — `GetSyncState()` (line 385), `GetLastSyncedAt()` (line 417), `SaveSyncState()` (line 374) already exist
+- `internal/generator/templates/client.go.tmpl` — pure HTTP client with `Get()`, `Post()`, etc. Adaptive rate limiter.
+- `internal/generator/templates/helpers.go.tmpl` — shared helper functions, output formatting, error classification
+- `internal/generator/templates/channel_workflow.go.tmpl` — `defaultDBPath()` at line 200, workflow archive/status commands
+- `internal/profiler/profiler.go` — `hasSearchEndpoint` detected at line 133 but only used to gate `NeedsSearch`; `SyncableResource` struct has `{Name, Path}`
+- `internal/generator/schema_builder.go` — `TableDef` with `FTS5` bool, FTS5 tables get `Search{{pascal .Name}}()` methods
+- `internal/generator/generator.go` — template data assembly, vision feature selection
+
+### Institutional Learnings
+
+- **DB path inconsistency** (Postman Explore retro): `defaultDBPath()` in `channel_workflow.go.tmpl` uses `~/.config/`, but search/sync used `~/.local/share/`. All local data access must go through a single canonical path. This change should consolidate `defaultDBPath()` into `helpers.go.tmpl`.
+- **Store template search gap** (5 retros): The store template gates FTS5 Search methods on `{{if .FTS5}}` but the profiler underdetects searchable fields. The generated `Search{{pascal .Name}}()` methods are present for high-gravity tables — we can rely on them for local reads.
+- **Verify gap for local-data commands** (Postman Explore Run 2 retro): Verify has no mechanism to distinguish "needs local data" commands from "hits live API" commands. The `--data-source` flag gives verify a way to test local-data paths.
+
+## Key Technical Decisions
+
+- **Resolution logic lives in a separate vision-gated template (`data_source.go.tmpl`), not in helpers**: The HTTP client stays a pure HTTP client. `resolveRead()` lives in a new `data_source.go.tmpl` that is only rendered when `VisionSet.Store` is true — following the same pattern as `search.go.tmpl` and `sync.go.tmpl`. This avoids a compilation failure: `helpers.go.tmpl` is rendered for ALL CLIs unconditionally, but `resolveRead()` imports the `store` package which only exists when `VisionSet.Store` is true. CLIs without a store continue calling `c.Get()` directly (zero change from today). The endpoint template conditionally calls `resolveRead()` when the data source template exists.
+
+- **GET commands call `resolveRead()` instead of `c.Get()` directly (when store is available)**: The endpoint template for GET commands swaps `c.Get(path, params)` for `resolveRead(c, flags, resourceType, isList, path, params)` when `VisionSet.Store` is true. The function takes an `isList bool` parameter (set at generation time based on whether the endpoint has pagination) to distinguish list vs get-by-ID operations — avoids fragile runtime path parsing. Write commands are unchanged.
+
+- **Provenance always wraps JSON output**: A `DataProvenance` struct (`source string`, `syncedAt *time.Time`, `reason string`) travels with results. JSON output always wraps in `{"results": [...], "meta": {...}}` — including for successful live calls. This is a breaking change to JSON output shape but there are no existing users yet. Human output prints a one-line provenance to stderr (stdout stays clean for piping).
+
+- **`defaultDBPath()` moves to `helpers.go.tmpl` with canonical path `~/.local/share/`**: Consolidates the DB path in one place. The canonical path is `~/.local/share/<cli-name>/data.db` (XDG compliant, where sync already writes). ALL templates that reference DB paths must be updated: `channel_workflow.go.tmpl` (currently `~/.config/`), `search.go.tmpl` (currently hardcodes `~/.local/share/`), workflow and insight templates (currently `~/.config/`). The `~/.config/` path is eliminated entirely.
+
+- **Search endpoint detection flows into template data**: The profiler already detects `hasSearchEndpoint`. We add `SearchEndpointPath` and `SearchQueryParam` to the template data so the search template can conditionally hit the API. For live search response normalization, use the same unwrapping heuristic that `paginatedGet()` already uses — walk known wrapper paths (`data`, `results`, `items`) to extract the result array.
+
+- **`store.List()` output is normalized to match API response shape**: `store.List()` returns `[]json.RawMessage` while API responses may be wrapped in `{"data": [...]}` etc. `resolveRead()` marshals `[]json.RawMessage` from the store into a single `json.RawMessage` array. Local-mode results will NOT include API envelope fields (pagination metadata, totals) — this is documented.
+
+## Open Questions
+
+### Resolved During Planning
+
+- **Where does the resolution function live?** In a new `data_source.go.tmpl` that is only rendered when `VisionSet.Store` is true. NOT in `helpers.go.tmpl` — helpers is rendered unconditionally for all CLIs, but `resolveRead()` imports the `store` package which only exists when VisionSet.Store is true. The separate template follows the same pattern as `search.go.tmpl` and `sync.go.tmpl`.
+
+- **How do GET endpoint commands read local data?** Via `store.List()` and `store.Get()` which already exist. `resolveRead()` takes an `isList bool` parameter (set at generation time based on whether the endpoint has pagination) to choose `List()` vs `Get()`. The store's `[]json.RawMessage` from `List()` is marshaled into a single `json.RawMessage` array for compatibility with the downstream rendering pipeline.
+
+- **Does `--data-source` affect non-GET read operations?** No. Only `GET` and `HEAD` methods. The endpoint template already branches on method — the data source branch nests inside the GET branch only.
+
+- **How does the search command know the API search endpoint path?** The profiler extracts it at generation time and passes it via the `visionData` struct in `generator.go`. The search template conditionally includes the live-search code path when `SearchEndpointPath` is non-empty.
+
+- **How is the JSON output shape changing?** Every `--json` response now wraps in `{"results": [...], "meta": {...}}` — including default auto+live. This is a breaking change accepted because there are no existing users. The envelope is always present for consistency — agents always know where provenance lives.
+
+- **How does `resolveRead()` handle paginated GET commands?** When mode is `local`, skip `paginatedGet()` entirely and call `store.List()` directly (local store has all synced data). When mode is `live` or `auto`, use `paginatedGet()` as today. When `auto` + network error on first page, fall back to `store.List()`.
+
+- **Which DB path is canonical?** `~/.local/share/<cli-name>/data.db` (XDG compliant, where sync already writes). The `~/.config/<name>/store.db` path in `channel_workflow.go.tmpl` and workflow/insight templates is eliminated. All templates updated to use `defaultDBPath()` from helpers.
+
+- **How does `resolveRead()` know the resource type?** The endpoint template passes `.ResourceName` (the spec resource map key, e.g., "links", "domains"). The sync command upserts using `SyncableResource.Name` from the profiler. These should match but must be verified — add a test that sync writes with a resource type retrievable by `resolveRead()`.
+
+- **How are API search results normalized?** Use the same unwrapping heuristic `paginatedGet()` uses — walk known wrapper paths (`data`, `results`, `items`) to extract the result array. If none match, treat the response as a bare array.
+
+### Deferred to Implementation
+
+- **Exact conditional import mechanics for endpoint templates.** The endpoint template needs `{{if}}` guards around the `store` import and `resolveRead()` call, gated on whether the VisionSet includes Store. The exact template conditionals depend on what fields are available in the endpoint template data — may need to add a `HasStore bool` to the endpoint data struct.
+
+## High-Level Technical Design
+
+> *This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.*
+
+```
+Command (GET endpoint)
+ │
+ ├─ flags.dataSource == "live" or ("auto" default)?
+ │ │
+ │ ├─ Call c.Get(path, params)
+ │ │ │
+ │ │ ├─ Success → return results, provenance{source:"live"}
+ │ │ │
+ │ │ └─ Network error + mode == "auto"?
+ │ │ │
+ │ │ ├─ Open store, query local data
+ │ │ │ │
+ │ │ │ ├─ Data exists → return results, provenance{source:"local", reason:"api_unreachable", syncedAt:...}
+ │ │ │ │
+ │ │ │ └─ No data → error: "API unreachable and no local data. Run 'sync' first."
+ │ │ │
+ │ │ └─ mode == "live"? → error: "API unreachable."
+ │ │
+ │ └─ (Non-network error) → return error as usual
+ │
+ └─ flags.dataSource == "local"?
+ │
+ ├─ Open store, query local data
+ │ │
+ │ ├─ Data exists → return results, provenance{source:"local", reason:"user_requested", syncedAt:...}
+ │ │
+ │ └─ No data → error: "No local data. Run 'sync' first."
+ │
+ └─ (Store not initialized) → error with hint to sync
+
+
+Search command:
+ │
+ ├─ mode == "local"?
+ │ └─ Use FTS5 (existing behavior + provenance)
+ │
+ ├─ API has search endpoint?
+ │ │
+ │ ├─ Yes → Call c.Get(searchPath, {queryParam: query})
+ │ │ │
+ │ │ ├─ Success → return results, provenance{source:"live"}
+ │ │ │
+ │ │ └─ Network error + mode == "auto"?
+ │ │ └─ Fall back to FTS5 + provenance{reason:"api_unreachable"}
+ │ │
+ │ └─ No → Use FTS5 + provenance{reason:"no_search_endpoint"}
+ │
+ └─ mode == "live" + no search endpoint?
+ └─ Use FTS5 + provenance{reason:"no_search_endpoint"} with explanation
+```
+
+## Implementation Units
+
+- [ ] **Unit 1: Add `--data-source` flag and provenance types**
+
+**Goal:** Add the `dataSource` field to `rootFlags`, register the `--data-source` persistent flag, define the `DataProvenance` struct, and add provenance output helpers.
+
+**Requirements:** R1, R4, R7
+
+**Dependencies:** None
+
+**Files:**
+- Modify: `internal/generator/templates/root.go.tmpl`
+- Modify: `internal/generator/templates/helpers.go.tmpl`
+- Test: `internal/generator/generator_test.go` (verify generated code compiles with new flag)
+
+**Approach:**
+- Add `dataSource string` to `rootFlags` struct in `root.go.tmpl`
+- Register `--data-source` as a persistent string flag with default `"auto"` and allowed values `auto|live|local`
+- In `PersistentPreRunE`, validate the flag value (must be auto, live, or local)
+- Define `DataProvenance` struct in `helpers.go.tmpl`: `Source string`, `SyncedAt *time.Time`, `Reason string`, `ResourceType string`
+- Add `printProvenance(cmd, provenance)` helper that writes the one-line stderr message (e.g., "3 results (live)" or "3 results (cached, synced 2 hours ago)")
+- Add `wrapWithProvenance(data, provenance)` helper that wraps JSON results in `{"results": [...], "meta": {...}}` when `--json` is set
+- Move `defaultDBPath()` from `channel_workflow.go.tmpl` to `helpers.go.tmpl` with canonical path `~/.local/share/<cli-name>/data.db` (XDG compliant, where sync already writes). Update ALL templates that reference DB paths: `channel_workflow.go.tmpl` (remove its copy), `search.go.tmpl` (replace hardcoded path at line 66), `sync.go.tmpl` (replace hardcoded path), and any workflow/insight templates using `~/.config/`
+
+**Patterns to follow:**
+- Existing flag registration pattern in `root.go.tmpl` (lines 50-69)
+- Existing `PersistentPreRunE` validation for `--agent` flag (lines 71-88)
+- Existing helper functions in `helpers.go.tmpl` (e.g., `classifyAPIError`, `wantsHumanTable`)
+
+**Test scenarios:**
+- Happy path: generated CLI compiles with `--data-source` flag present, `--help` shows the flag with correct description
+- Happy path: `--data-source auto` is the default when flag is not explicitly set
+- Edge case: `--data-source invalid` rejected in PersistentPreRunE with clear error
+- Happy path: `printProvenance()` formats "3 results (live)" for live source
+- Happy path: `printProvenance()` formats "3 results (cached, synced 2 hours ago)" with relative time
+- Happy path: `wrapWithProvenance()` produces `{"results": [...], "meta": {"source": "live"}}` JSON envelope
+- Edge case: `wrapWithProvenance()` includes `synced_at` and `reason` only when source is "local"
+
+**Verification:**
+- Generate a test CLI, build it, confirm `--data-source` flag appears in `--help`
+- Confirm `--data-source invalid` produces a validation error
+
+---
+
+- [ ] **Unit 2: Add `resolveRead()` in new `data_source.go.tmpl`**
+
+**Goal:** Implement the core resolution logic in a new vision-gated template that dispatches reads to either the HTTP client or local store based on the `--data-source` flag value, network availability, and local data existence.
+
+**Requirements:** R1, R2, R3, R7
+
+**Dependencies:** Unit 1
+
+**Files:**
+- Create: `internal/generator/templates/data_source.go.tmpl`
+- Modify: `internal/generator/generator.go` (add `data_source.go.tmpl` to vision-gated rendering, only when `VisionSet.Store` is true)
+- Test: `internal/generator/generator_test.go`
+
+**Approach:**
+- Create `data_source.go.tmpl` in the `cli` package. This template is only rendered when `VisionSet.Store` is true (following the pattern of `search.go.tmpl`, `sync.go.tmpl`). CLIs without a store never get this file — they call `c.Get()` directly.
+- Add `resolveRead()` function with signature: `client *client.Client`, `flags *rootFlags`, `resourceType string`, `isList bool`, `path string`, `params map[string]string` → returns `(json.RawMessage, DataProvenance, error)`
+- The `isList` parameter is set at generation time based on whether the endpoint has `.Endpoint.Pagination`. This avoids fragile runtime path parsing to distinguish list vs get-by-ID.
+- Resolution logic:
+ - If `dataSource == "local"`: open store via `defaultDBPath()`, call `store.List(resourceType, limit)` when `isList` or `store.Get(resourceType, id)` when not. Marshal `[]json.RawMessage` from `List()` into a single `json.RawMessage` array. If no data, return error with sync hint. Attach provenance with `reason: "user_requested"`.
+ - If `dataSource == "live"`: call `c.Get(path, params)`. On success, return with `source: "live"`. On network error, return the error directly (no fallback).
+ - If `dataSource == "auto"` (default): call `c.Get(path, params)`. On success, return with `source: "live"`. On network error, try the store. If store has data, return it with `source: "local"`, `reason: "api_unreachable"`, and the sync timestamp. If store has no data, return error: "API unreachable and no local data."
+- Also add `resolvePaginatedRead()` for paginated GET commands: when `local`, skip `paginatedGet()` and call `store.List()` directly. When `live`/`auto`, delegate to `paginatedGet()`. When `auto` + network error, fall back to `store.List()`.
+- Network error detection: check for `*net.OpError`, `*url.Error`, DNS errors, connection refused, and timeout errors. HTTP 4xx/5xx errors are NOT network errors — they propagate normally.
+- Store is opened lazily — only when local data is actually needed.
+- The function queries existing `store.GetSyncState(resourceType)` to get the sync timestamp for provenance.
+
+**Patterns to follow:**
+- Existing `classifyAPIError()` pattern in helpers for error inspection
+- Existing `store.Open()` / `store.List()` / `store.Get()` API
+
+**Test scenarios:**
+- Happy path: `auto` mode with live API reachable → returns live data with `source: "live"`
+- Happy path: `auto` mode with network error and local data exists → returns local data with `source: "local"`, `reason: "api_unreachable"`, populated `synced_at`
+- Edge case: `auto` mode with network error and no local data → returns error mentioning both API unreachability and sync hint
+- Happy path: `live` mode with live API reachable → returns live data
+- Error path: `live` mode with network error → returns error "API unreachable" (no fallback)
+- Happy path: `local` mode with synced data → returns local data with `reason: "user_requested"`
+- Error path: `local` mode with no synced data → returns error "No local data. Run 'sync' first."
+- Edge case: HTTP 401/403/500 errors are NOT treated as network errors — they propagate as API errors even in `auto` mode
+- Edge case: Store DB file doesn't exist → graceful handling (no panic, clear error)
+
+**Verification:**
+- Generated CLI compiles with `resolveRead()` function present
+- Unit tests cover all branches of the resolution matrix
+
+---
+
+- [ ] **Unit 3: Update GET command template for data source resolution**
+
+**Goal:** Modify the endpoint command template so that GET commands call `resolveRead()` instead of `c.Get()` directly, enabling live/local/auto data source behavior.
+
+**Requirements:** R1, R2, R3, R5
+
+**Dependencies:** Unit 1, Unit 2
+
+**Files:**
+- Modify: `internal/generator/templates/command_endpoint.go.tmpl`
+- Modify: `internal/generator/templates/command_promoted.go.tmpl` (promoted commands also call `c.Get()` and `paginatedGet()` directly — lines 54, 70)
+- Test: `internal/generator/generator_test.go`
+
+**Approach:**
+- In the `{{if or (eq .Endpoint.Method "GET") (eq .Endpoint.Method "HEAD")}}` branch (line 68), replace the direct `c.Get(path, params)` call with `resolveRead(c, &flags, "{{resourceName .}}", path, params)`
+- Apply the same swap in `command_promoted.go.tmpl` for its GET branch (line 70) and `paginatedGet()` call (line 54), so promoted aliases respect `--data-source` identically to their non-promoted equivalents
+- The `resourceName` template function maps the endpoint's parent resource to the store's resource type name (same normalization sync uses)
+- Add conditional `store` import when the VisionSet includes Store/Sync features
+- After `resolveRead()` returns, call `printProvenance()` for human output (stderr) and `wrapWithProvenance()` for JSON output
+- Write commands (POST, PUT, PATCH, DELETE) remain unchanged — they continue calling `c.Post()` etc. directly
+- The `paginatedGet()` helper also needs wrapping — when `--data-source local`, pagination is not needed (local store returns all matching rows)
+
+**Patterns to follow:**
+- Existing method branching in `command_endpoint.go.tmpl` (line 68-153)
+- Existing `wantsHumanTable()` output mode detection pattern
+
+**Test scenarios:**
+- Happy path: generated GET command with `--data-source auto` (default) calls API and returns live data — no behavior change from today
+- Happy path: generated GET command with `--data-source local` returns locally synced data
+- Happy path: generated POST command ignores `--data-source` flag entirely — always hits API
+- Integration: GET command with `--data-source auto` and simulated network error falls back to local data with provenance warning on stderr
+- Edge case: paginated GET with `--all` flag and `--data-source local` returns all local data without pagination calls
+- Happy path: `--json` output includes `meta.source` field for GET commands
+
+**Verification:**
+- Generate a CLI from a known spec, confirm GET commands show provenance in both table and JSON modes
+- Confirm POST/PUT/PATCH/DELETE commands are unchanged
+
+---
+
+- [ ] **Unit 4: Expose search endpoint details in profiler**
+
+**Goal:** When the profiler detects a search endpoint, extract and expose the endpoint path and query parameter name so the search template can conditionally hit the API.
+
+**Requirements:** R8, R9
+
+**Dependencies:** None (can run in parallel with Units 1-3)
+
+**Files:**
+- Modify: `internal/profiler/profiler.go`
+- Modify: `internal/generator/generator.go` (extend the `visionData` struct to include `SearchEndpointPath` and `SearchQueryParam`, populated from `g.profile`)
+- Test: `internal/profiler/profiler_test.go`
+
+**Approach:**
+- Add `SearchEndpointPath string` and `SearchQueryParam string` fields to `APIProfile`
+- When `hasSearchEndpoint` is true (line 133), also record the endpoint's path and identify the query parameter (look for params named `q`, `query`, `search`, `keyword`, `term` — pick the first match)
+- Extend the `visionData` struct in `generator.go` (around line 413) to include `SearchEndpointPath` and `SearchQueryParam`, sourced from `g.profile`. The existing pipeline does NOT automatically flow APIProfile fields into visionData — the struct must be manually extended.
+
+**Patterns to follow:**
+- Existing search endpoint detection at line 130-134 of profiler.go
+- Existing `SyncableResource` struct pattern for exposing detected data to templates
+
+**Test scenarios:**
+- Happy path: API with `/search` endpoint and `q` query param → `SearchEndpointPath` = "/search", `SearchQueryParam` = "q"
+- Happy path: API with `/users/search` endpoint and `query` param → correctly extracted
+- Edge case: API with no search endpoint → both fields are empty strings
+- Edge case: API with search endpoint but no recognizable query param → `SearchEndpointPath` set, `SearchQueryParam` defaults to "q"
+- Happy path: multiple search endpoints (e.g., `/users/search` and `/orders/search`) → picks the most general one (shortest path or root-level)
+
+**Verification:**
+- Profile a spec with a known search endpoint, confirm fields are populated
+- Profile a spec without a search endpoint, confirm fields are empty
+
+---
+
+- [ ] **Unit 5: Update search template for live/local routing and provenance**
+
+**Goal:** Modify the search command to try the API search endpoint (when available) before falling back to local FTS, and add provenance metadata to all search output.
+
+**Requirements:** R4, R8, R9
+
+**Dependencies:** Unit 1, Unit 4
+
+**Files:**
+- Modify: `internal/generator/templates/search.go.tmpl`
+- Test: `internal/generator/generator_test.go`
+
+**Approach:**
+- When `SearchEndpointPath` is non-empty in template data AND `dataSource` is `auto` or `live`:
+ - Call `c.Get(searchEndpointPath, map[string]string{searchQueryParam: query})`
+ - Normalize the response: unwrap known envelope paths (`data`, `results`, `items`) to extract the result array — same heuristic `paginatedGet()` uses
+ - On success: return results with `source: "live"`
+ - On network error + `auto` mode: fall back to local FTS with `reason: "api_unreachable"`
+ - On `live` mode + no network: return error
+- When `SearchEndpointPath` is empty (API has no search endpoint):
+ - `auto` and `live` modes both use local FTS with `reason: "no_search_endpoint"` and print explanation: "This API has no search endpoint. Searching local data."
+ - `local` mode uses local FTS with `reason: "user_requested"`
+- Add provenance to all output paths — both JSON (`meta` envelope) and human (stderr line)
+- Remove the `Args: cobra.ExactArgs(1)` constraint and add `if len(args) == 0 { return cmd.Help() }` for verify compatibility (same pattern used to fix tail in the dub-pp-cli polish)
+
+**Patterns to follow:**
+- Existing search command structure in `search.go.tmpl`
+- Existing `resolveRead()` pattern from Unit 2 for live/local dispatching
+- Provenance helpers from Unit 1
+
+**Test scenarios:**
+- Happy path: API with search endpoint, `auto` mode → hits API, returns live results with `source: "live"`
+- Happy path: API with search endpoint, `local` mode → uses local FTS, returns with `reason: "user_requested"`
+- Happy path: API without search endpoint, `auto` mode → uses local FTS with `reason: "no_search_endpoint"`, stderr shows explanation
+- Error path: API with search endpoint, `live` mode, network error → returns error
+- Happy path: API with search endpoint, `auto` mode, network error → falls back to local FTS with `reason: "api_unreachable"`
+- Happy path: `--json` output wraps results in `{"results": [...], "meta": {"source": "...", ...}}`
+- Edge case: local FTS with no synced data → error: "No local data. Run 'sync' first."
+- Happy path: human output shows "3 results (live)" or "3 results (cached, synced 2 hours ago)" on stderr
+
+**Verification:**
+- Generate a CLI from a spec with a search endpoint, confirm search tries API first
+- Generate a CLI from a spec without a search endpoint, confirm search uses local FTS with explanation
+- Confirm JSON output includes provenance metadata in all cases
+
+---
+
+**Deferred: Persistent data-source config default.** Adding `data_source` to the config file and env var override is a convenience feature that serves no stated requirement. The `--data-source` flag (Unit 1) delivers all required functionality. Config persistence can be added as a follow-up if users request it.
+
+## System-Wide Impact
+
+- **Interaction graph:** The `resolveRead()` helper sits between all GET command handlers and the client/store. It is the single point of control for data source decisions. The search command has its own resolution path (because it may use API search endpoints), but follows the same provenance pattern.
+- **Error propagation:** Network errors in `auto` mode are caught and trigger fallback — they do NOT propagate as command errors. HTTP 4xx/5xx errors always propagate. In `live` mode, network errors propagate as errors.
+- **State lifecycle risks:** No new mutable state. The local SQLite database is read-only from the perspective of `resolveRead()`. Only `sync` writes to it. No cache invalidation, no write-through.
+- **API surface parity:** The `--data-source` flag applies to all read commands uniformly. Write commands are unaffected. The flag is a root-level persistent flag so it works identically across all subcommands.
+- **Integration coverage:** The key integration scenario is: generate a CLI → run `sync` → verify `--data-source local` returns synced data → verify `--data-source auto` with network down falls back to local with provenance.
+- **Unchanged invariants:** `sync` command behavior is unchanged. Write command behavior is unchanged. The HTTP client API is unchanged. The store schema is unchanged (only new query methods added). The profiler's `NeedsSearch` gate is unchanged.
+
+## Risks & Dependencies
+
+| Risk | Mitigation |
+|------|------------|
+| Store-less CLIs break if resolveRead() is in helpers.go | Resolved: `resolveRead()` lives in `data_source.go.tmpl`, only rendered when `VisionSet.Store` is true. CLIs without stores never see this code. |
+| JSON output breaking change: wrapping in `{"results": [...], "meta": {...}}` | Accepted: always wrap, no existing users. Document in generated README. Envelope is always present for agent consistency. |
+| Template complexity: GET branch becomes significantly more complex | Keep `resolveRead()` and `resolvePaginatedRead()` as single well-tested helpers in `data_source.go.tmpl`. The per-command template change is a one-line swap. |
+| Local data shape differs from API response shape | `resolveRead()` marshals `[]json.RawMessage` from `store.List()` into a single JSON array. Local results omit API envelope fields (pagination metadata, totals). Documented as expected behavior. |
+| DB path inconsistency across templates | Resolved: canonical path is `~/.local/share/<cli-name>/data.db`. All templates updated in Unit 1 to use `defaultDBPath()` from helpers. |
+| Sync timestamp accuracy: `last_synced_at` may not reflect actual data freshness if sync was partial | Sync already tracks per-resource timestamps. `resolveRead()` uses the resource-specific timestamp, not a global one. |
+| Resource type name mismatch between endpoint templates and sync | Add test verifying that `.ResourceName` from the endpoint template matches `SyncableResource.Name` from the profiler for the same resource. |
+
+## Sources & References
+
+- Related code: `internal/generator/templates/` (all template files listed above)
+- Related code: `internal/profiler/profiler.go` (APIProfile, NeedsSearch, hasSearchEndpoint)
+- Related code: `internal/generator/schema_builder.go` (TableDef, FTS5 decisions)
+- Retro: `docs/retros/2026-03-30-postman-explore-retro.md` (DB path inconsistency finding)
+- Retro: `docs/retros/2026-04-03-postman-explore-run2-retro.md` (verify gap for local-data commands)
diff --git a/internal/generator/generator.go b/internal/generator/generator.go
index 5bfb3f8f..5c367d76 100644
--- a/internal/generator/generator.go
+++ b/internal/generator/generator.go
@@ -210,6 +210,17 @@ func (g *Generator) Generate() error {
}
}
+ // Early profiling: compute VisionSet before endpoint rendering so
+ // templates can check HasStore for data source resolution.
+ if g.VisionSet.IsZero() {
+ g.profile = profiler.Profile(g.Spec)
+ plan := g.profile.ToVisionaryPlan(g.Spec.Name)
+ g.VisionSet = SelectVisionTemplates(plan)
+ }
+ if g.profile == nil {
+ g.profile = profiler.Profile(g.Spec)
+ }
+
// Generate single files
singleFiles := map[string]string{
"main.go.tmpl": filepath.Join("cmd", naming.CLI(g.Spec.Name), "main.go"),
@@ -279,6 +290,7 @@ func (g *Generator) Generate() error {
CommandPath string
EndpointName string
Endpoint spec.Endpoint
+ HasStore bool
*spec.APISpec
}{
ResourceName: name,
@@ -286,6 +298,7 @@ func (g *Generator) Generate() error {
CommandPath: name,
EndpointName: eName,
Endpoint: endpoint,
+ HasStore: g.VisionSet.Store,
APISpec: g.Spec,
}
epPath := filepath.Join("internal", "cli", name+"_"+eName+".go")
@@ -321,6 +334,7 @@ func (g *Generator) Generate() error {
CommandPath string
EndpointName string
Endpoint spec.Endpoint
+ HasStore bool
*spec.APISpec
}{
ResourceName: subName,
@@ -328,6 +342,7 @@ func (g *Generator) Generate() error {
CommandPath: name + " " + subName,
EndpointName: eName,
Endpoint: endpoint,
+ HasStore: g.VisionSet.Store,
APISpec: g.Spec,
}
epPath := filepath.Join("internal", "cli", name+"_"+subName+"_"+eName+".go")
@@ -367,16 +382,7 @@ func (g *Generator) Generate() error {
}
}
- // Vision features: profile the API and render selected templates
- if g.VisionSet.IsZero() {
- // Auto-profile if no explicit vision set provided
- g.profile = profiler.Profile(g.Spec)
- plan := g.profile.ToVisionaryPlan(g.Spec.Name)
- g.VisionSet = SelectVisionTemplates(plan)
- }
- if g.profile == nil {
- g.profile = profiler.Profile(g.Spec)
- }
+ // Vision features: profile already computed in early profiling above
schema := BuildSchema(g.Spec)
// Create store directory if needed
@@ -412,16 +418,22 @@ func (g *Generator) Generate() error {
visionData := struct {
*spec.APISpec
- SyncableResources []profiler.SyncableResource
- SearchableFields map[string][]string
- Tables []TableDef
- Pagination profiler.PaginationProfile
+ SyncableResources []profiler.SyncableResource
+ SearchableFields map[string][]string
+ Tables []TableDef
+ Pagination profiler.PaginationProfile
+ SearchEndpointPath string
+ SearchQueryParam string
+ SearchEndpointMethod string
}{
- APISpec: g.Spec,
- SyncableResources: g.profile.SyncableResources,
- SearchableFields: g.profile.SearchableFields,
- Tables: schema,
- Pagination: g.profile.Pagination,
+ APISpec: g.Spec,
+ SyncableResources: g.profile.SyncableResources,
+ SearchableFields: g.profile.SearchableFields,
+ Tables: schema,
+ Pagination: g.profile.Pagination,
+ SearchEndpointPath: g.profile.SearchEndpointPath,
+ SearchQueryParam: g.profile.SearchQueryParam,
+ SearchEndpointMethod: g.profile.SearchEndpointMethod,
}
for _, tmplName := range g.VisionSet.TemplateNames() {
@@ -441,6 +453,13 @@ func (g *Generator) Generate() error {
}
}
+ // Render data source resolution template when store is enabled
+ if g.VisionSet.Store {
+ if err := g.renderTemplate("data_source.go.tmpl", filepath.Join("internal", "cli", "data_source.go"), visionData); err != nil {
+ return fmt.Errorf("rendering data_source: %w", err)
+ }
+ }
+
// Render workflow template when store is enabled (root.go registers it conditionally on VisionSet.Store)
if g.VisionSet.Store {
workflowData := struct {
@@ -513,12 +532,14 @@ func (g *Generator) Generate() error {
ResourceName string
EndpointName string
Endpoint spec.Endpoint
+ HasStore bool
*spec.APISpec
}{
PromotedName: pc.PromotedName,
ResourceName: pc.ResourceName,
EndpointName: pc.EndpointName,
Endpoint: pc.Endpoint,
+ HasStore: g.VisionSet.Store,
APISpec: g.Spec,
}
promotedPath := filepath.Join("internal", "cli", "promoted_"+pc.PromotedName+".go")
diff --git a/internal/generator/generator_test.go b/internal/generator/generator_test.go
index 3df8fe33..d30cf246 100644
--- a/internal/generator/generator_test.go
+++ b/internal/generator/generator_test.go
@@ -22,9 +22,9 @@ func TestGenerateProjectsCompile(t *testing.T) {
specPath string
expectedFiles int
}{
- {name: "stytch", specPath: filepath.Join("..", "..", "testdata", "stytch.yaml"), expectedFiles: 30},
- {name: "clerk", specPath: filepath.Join("..", "..", "testdata", "clerk.yaml"), expectedFiles: 35},
- {name: "loops", specPath: filepath.Join("..", "..", "testdata", "loops.yaml"), expectedFiles: 33},
+ {name: "stytch", specPath: filepath.Join("..", "..", "testdata", "stytch.yaml"), expectedFiles: 31},
+ {name: "clerk", specPath: filepath.Join("..", "..", "testdata", "clerk.yaml"), expectedFiles: 36},
+ {name: "loops", specPath: filepath.Join("..", "..", "testdata", "loops.yaml"), expectedFiles: 34},
}
for _, tt := range tests {
@@ -445,16 +445,18 @@ func TestGeneratedOutput_MutatingCommandsHaveEnvelope(t *testing.T) {
assert.Contains(t, content, `envelope["success"] = false`)
}
-func TestGeneratedOutput_GetCommandsLackEnvelope(t *testing.T) {
+func TestGeneratedOutput_GetCommandsLackMutationEnvelope(t *testing.T) {
t.Parallel()
outputDir := generatePetstore(t)
- // GET command should NOT have confirmation envelope
+ // GET command should NOT have the mutation-style confirmation envelope
+ // (action/resource/status/success fields). It MAY have provenance wrapping
+ // via wrapWithProvenance when HasStore is true.
getGo, err := os.ReadFile(filepath.Join(outputDir, "internal", "cli", "pet_get-by-id.go"))
require.NoError(t, err)
content := string(getGo)
- assert.NotContains(t, content, "envelope")
+ assert.NotContains(t, content, `"action"`)
assert.NotContains(t, content, "statusCode")
}
diff --git a/internal/generator/templates/analytics.go.tmpl b/internal/generator/templates/analytics.go.tmpl
index 5e18a2bb..d86c511b 100644
--- a/internal/generator/templates/analytics.go.tmpl
+++ b/internal/generator/templates/analytics.go.tmpl
@@ -7,7 +7,6 @@ import (
"encoding/json"
"fmt"
"os"
- "path/filepath"
"sort"
"{{modulePath}}/internal/store"
@@ -35,8 +34,7 @@ Data must be synced first with the sync command.`,
{{.Name}}-pp-cli analytics --type messages --group-by channel_id --limit 10 --json`,
RunE: func(cmd *cobra.Command, args []string) error {
if dbPath == "" {
- home, _ := os.UserHomeDir()
- dbPath = filepath.Join(home, ".local", "share", "{{.Name}}-pp-cli", "data.db")
+ dbPath = defaultDBPath("{{.Name}}-pp-cli")
}
db, err := store.Open(dbPath)
diff --git a/internal/generator/templates/channel_workflow.go.tmpl b/internal/generator/templates/channel_workflow.go.tmpl
index 5a8f2ee5..fa194c3d 100644
--- a/internal/generator/templates/channel_workflow.go.tmpl
+++ b/internal/generator/templates/channel_workflow.go.tmpl
@@ -6,8 +6,6 @@ package cli
import (
"encoding/json"
"fmt"
- "os"
- "path/filepath"
"time"
"{{modulePath}}/internal/store"
@@ -137,7 +135,7 @@ and full resync. After archiving, use 'search' for instant full-text search.`,
},
}
- cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.config/{{.Name}}-pp-cli/store.db)")
+ cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.local/share/{{.Name}}-pp-cli/data.db)")
cmd.Flags().BoolVar(&full, "full", false, "Full re-archive (ignore previous sync state)")
return cmd
@@ -197,7 +195,4 @@ func newWorkflowStatusCmd(flags *rootFlags) *cobra.Command {
return cmd
}
-func defaultDBPath(name string) string {
- home, _ := os.UserHomeDir()
- return filepath.Join(home, ".config", name, "store.db")
-}
+// defaultDBPath is defined in helpers.go
diff --git a/internal/generator/templates/command_endpoint.go.tmpl b/internal/generator/templates/command_endpoint.go.tmpl
index c5ddef78..90b0fb29 100644
--- a/internal/generator/templates/command_endpoint.go.tmpl
+++ b/internal/generator/templates/command_endpoint.go.tmpl
@@ -67,6 +67,15 @@ func new{{camel .FuncPrefix}}{{camel .EndpointName}}Cmd(flags *rootFlags) *cobra
{{- if or (eq .Endpoint.Method "GET") (eq .Endpoint.Method "HEAD")}}
{{- if .Endpoint.Pagination}}
+{{- if .HasStore}}
+ data, prov, err := resolvePaginatedRead(c, flags, "{{lower .ResourceName}}", path, map[string]string{
+{{- range .Endpoint.Params}}
+{{- if and (not .Positional) (not .PathParam)}}
+ "{{.Name}}": fmt.Sprintf("%v", flag{{camel .Name}}),
+{{- end}}
+{{- end}}
+ }, flagAll, "{{.Endpoint.Pagination.CursorParam}}", "{{.Endpoint.Pagination.NextCursorPath}}", "{{.Endpoint.Pagination.HasMoreField}}")
+{{- else}}
data, err := paginatedGet(c, path, map[string]string{
{{- range .Endpoint.Params}}
{{- if and (not .Positional) (not .PathParam)}}
@@ -74,6 +83,7 @@ func new{{camel .FuncPrefix}}{{camel .EndpointName}}Cmd(flags *rootFlags) *cobra
{{- end}}
{{- end}}
}, flagAll, "{{.Endpoint.Pagination.CursorParam}}", "{{.Endpoint.Pagination.NextCursorPath}}", "{{.Endpoint.Pagination.HasMoreField}}")
+{{- end}}
{{- else}}
params := map[string]string{}
{{- range .Endpoint.Params}}
@@ -83,8 +93,12 @@ func new{{camel .FuncPrefix}}{{camel .EndpointName}}Cmd(flags *rootFlags) *cobra
}
{{- end}}
{{- end}}
+{{- if .HasStore}}
+ data, prov, err := resolveRead(c, flags, "{{lower .ResourceName}}", {{if .Endpoint.Pagination}}true{{else}}false{{end}}, path, params)
+{{- else}}
data, err := c.Get(path, params)
{{- end}}
+{{- end}}
{{- else if eq .Endpoint.Method "POST"}}
var body map[string]any
if stdinBody {
@@ -220,6 +234,30 @@ func new{{camel .FuncPrefix}}{{camel .EndpointName}}Cmd(flags *rootFlags) *cobra
}
{{- end}}
{{- if or (eq .Endpoint.Method "GET") (eq .Endpoint.Method "HEAD")}}
+{{- if .HasStore}}
+ // Print provenance to stderr for human-facing output
+ {
+ var countItems []json.RawMessage
+ _ = json.Unmarshal(data, &countItems)
+ printProvenance(cmd, len(countItems), prov)
+ }
+ // For JSON output, wrap with provenance envelope before passing through flags
+ if flags.asJSON || !isTerminal(cmd.OutOrStdout()) {
+ filtered := data
+ if flags.compact {
+ filtered = compactFields(filtered)
+ }
+ if flags.selectFields != "" {
+ filtered = filterFields(filtered, flags.selectFields)
+ }
+ wrapped, wrapErr := wrapWithProvenance(filtered, prov)
+ if wrapErr != nil {
+ return wrapErr
+ }
+ return printOutput(cmd.OutOrStdout(), wrapped, true)
+ }
+ // For all other output modes (table, csv, plain, quiet), use the standard pipeline
+{{- end}}
if wantsHumanTable(cmd.OutOrStdout(), flags) {
var items []map[string]any
if json.Unmarshal(data, &items) == nil && len(items) > 0 {
diff --git a/internal/generator/templates/command_promoted.go.tmpl b/internal/generator/templates/command_promoted.go.tmpl
index d92a0473..f4b8df48 100644
--- a/internal/generator/templates/command_promoted.go.tmpl
+++ b/internal/generator/templates/command_promoted.go.tmpl
@@ -51,6 +51,15 @@ func new{{camel .PromotedName}}PromotedCmd(flags *rootFlags) *cobra.Command {
{{- end}}
{{- if .Endpoint.Pagination}}
+{{- if .HasStore}}
+ data, prov, err := resolvePaginatedRead(c, flags, "{{lower .ResourceName}}", path, map[string]string{
+{{- range .Endpoint.Params}}
+{{- if not .Positional}}
+ "{{.Name}}": fmt.Sprintf("%v", flag{{camel .Name}}),
+{{- end}}
+{{- end}}
+ }, flagAll, "{{.Endpoint.Pagination.CursorParam}}", "{{.Endpoint.Pagination.NextCursorPath}}", "{{.Endpoint.Pagination.HasMoreField}}")
+{{- else}}
data, err := paginatedGet(c, path, map[string]string{
{{- range .Endpoint.Params}}
{{- if not .Positional}}
@@ -58,6 +67,7 @@ func new{{camel .PromotedName}}PromotedCmd(flags *rootFlags) *cobra.Command {
{{- end}}
{{- end}}
}, flagAll, "{{.Endpoint.Pagination.CursorParam}}", "{{.Endpoint.Pagination.NextCursorPath}}", "{{.Endpoint.Pagination.HasMoreField}}")
+{{- end}}
{{- else}}
params := map[string]string{}
{{- range .Endpoint.Params}}
@@ -67,12 +77,39 @@ func new{{camel .PromotedName}}PromotedCmd(flags *rootFlags) *cobra.Command {
}
{{- end}}
{{- end}}
+{{- if .HasStore}}
+ data, prov, err := resolveRead(c, flags, "{{lower .ResourceName}}", {{if .Endpoint.Pagination}}true{{else}}false{{end}}, path, params)
+{{- else}}
data, err := c.Get(path, params)
+{{- end}}
{{- end}}
if err != nil {
return classifyAPIError(err)
}
+{{- if .HasStore}}
+ // Print provenance to stderr
+ {
+ var countItems []json.RawMessage
+ _ = json.Unmarshal(data, &countItems)
+ printProvenance(cmd, len(countItems), prov)
+ }
+ // For JSON output, wrap with provenance envelope
+ if flags.asJSON || !isTerminal(cmd.OutOrStdout()) {
+ filtered := data
+ if flags.compact {
+ filtered = compactFields(filtered)
+ }
+ if flags.selectFields != "" {
+ filtered = filterFields(filtered, flags.selectFields)
+ }
+ wrapped, wrapErr := wrapWithProvenance(filtered, prov)
+ if wrapErr != nil {
+ return wrapErr
+ }
+ return printOutput(cmd.OutOrStdout(), wrapped, true)
+ }
+{{- end}}
if wantsHumanTable(cmd.OutOrStdout(), flags) {
var items []map[string]any
if json.Unmarshal(data, &items) == nil && len(items) > 0 {
diff --git a/internal/generator/templates/data_source.go.tmpl b/internal/generator/templates/data_source.go.tmpl
new file mode 100644
index 00000000..f557e8de
--- /dev/null
+++ b/internal/generator/templates/data_source.go.tmpl
@@ -0,0 +1,203 @@
+// Copyright {{currentYear}} {{.Owner}}. Licensed under Apache-2.0. See LICENSE.
+// Generated by CLI Printing Press (https://github.com/mvanhorn/cli-printing-press). DO NOT EDIT.
+
+package cli
+
+import (
+ "encoding/json"
+ "fmt"
+ "net"
+ "net/url"
+ "os"
+ "strings"
+ "time"
+
+ "{{modulePath}}/internal/client"
+ "{{modulePath}}/internal/store"
+)
+
+// isNetworkError returns true for errors caused by network connectivity issues
+// (DNS, connection refused, timeout). HTTP 4xx/5xx errors are NOT network errors.
+func isNetworkError(err error) bool {
+ if err == nil {
+ return false
+ }
+ var urlErr *url.Error
+ if As(err, &urlErr) {
+ // url.Error wraps the underlying network error
+ err = urlErr.Err
+ }
+ var netErr *net.OpError
+ if As(err, &netErr) {
+ return true
+ }
+ var dnsErr *net.DNSError
+ if As(err, &dnsErr) {
+ return true
+ }
+ // Check for common network error strings
+ msg := err.Error()
+ return strings.Contains(msg, "connection refused") ||
+ strings.Contains(msg, "no such host") ||
+ strings.Contains(msg, "network is unreachable") ||
+ strings.Contains(msg, "i/o timeout") ||
+ strings.Contains(msg, "TLS handshake timeout")
+}
+
+// openStoreForRead opens the local SQLite store for reading.
+// Returns nil, nil if the database file does not exist (no sync has been run).
+func openStoreForRead(cliName string) (*store.Store, error) {
+ dbPath := defaultDBPath(cliName)
+ if _, err := os.Stat(dbPath); os.IsNotExist(err) {
+ return nil, nil
+ }
+ return store.Open(dbPath)
+}
+
+// localProvenance builds a DataProvenance for local data reads.
+func localProvenance(db *store.Store, resourceType, reason string) DataProvenance {
+ prov := DataProvenance{
+ Source: "local",
+ Reason: reason,
+ ResourceType: resourceType,
+ }
+ _, lastSynced, _, err := db.GetSyncState(resourceType)
+ if err == nil && !lastSynced.IsZero() {
+ prov.SyncedAt = &lastSynced
+ }
+ return prov
+}
+
+// resolveRead dispatches a GET request to either the live API or local store
+// based on the --data-source flag. Returns the response data and provenance metadata.
+//
+// Parameters:
+// - c: the HTTP client for live API calls
+// - flags: root flags containing dataSource setting
+// - resourceType: the store resource type name (e.g., "links", "domains")
+// - isList: true for list endpoints, false for get-by-ID endpoints
+// - path: the API path (e.g., "/links" or "/links/abc123")
+// - params: query parameters for the API call
+func resolveRead(c *client.Client, flags *rootFlags, resourceType string, isList bool, path string, params map[string]string) (json.RawMessage, DataProvenance, error) {
+ switch flags.dataSource {
+ case "local":
+ return resolveLocal(resourceType, isList, path, params, "user_requested")
+
+ case "live":
+ data, err := c.Get(path, params)
+ if err != nil {
+ return nil, DataProvenance{}, err
+ }
+ return data, DataProvenance{Source: "live"}, nil
+
+ default: // "auto"
+ data, err := c.Get(path, params)
+ if err == nil {
+ return data, DataProvenance{Source: "live"}, nil
+ }
+ if !isNetworkError(err) {
+ // HTTP 4xx/5xx errors propagate — not a fallback case
+ return nil, DataProvenance{}, err
+ }
+ // Network error — try local fallback
+ localData, prov, localErr := resolveLocal(resourceType, isList, path, params, "api_unreachable")
+ if localErr != nil {
+ return nil, DataProvenance{}, fmt.Errorf("API unreachable and no local data. Run '{{.Name}}-pp-cli sync' to enable offline access.\n\nOriginal error: %w", err)
+ }
+ return localData, prov, nil
+ }
+}
+
+// resolvePaginatedRead dispatches a paginated GET request to either the live API
+// or local store. When local, skips pagination and returns all synced data.
+func resolvePaginatedRead(c *client.Client, flags *rootFlags, resourceType string, path string, params map[string]string, fetchAll bool, cursorParam, nextCursorPath, hasMoreField string) (json.RawMessage, DataProvenance, error) {
+ switch flags.dataSource {
+ case "local":
+ return resolveLocal(resourceType, true, path, params, "user_requested")
+
+ case "live":
+ data, err := paginatedGet(c, path, params, fetchAll, cursorParam, nextCursorPath, hasMoreField)
+ if err != nil {
+ return nil, DataProvenance{}, err
+ }
+ return data, DataProvenance{Source: "live"}, nil
+
+ default: // "auto"
+ data, err := paginatedGet(c, path, params, fetchAll, cursorParam, nextCursorPath, hasMoreField)
+ if err == nil {
+ return data, DataProvenance{Source: "live"}, nil
+ }
+ if !isNetworkError(err) {
+ return nil, DataProvenance{}, err
+ }
+ localData, prov, localErr := resolveLocal(resourceType, true, path, params, "api_unreachable")
+ if localErr != nil {
+ return nil, DataProvenance{}, fmt.Errorf("API unreachable and no local data. Run '{{.Name}}-pp-cli sync' to enable offline access.\n\nOriginal error: %w", err)
+ }
+ return localData, prov, nil
+ }
+}
+
+// resolveLocal reads data from the local SQLite store.
+// Note: local reads return ALL synced data for the resource type. Endpoint-specific
+// filters (query params, path scoping like /teams/{id}/users) are NOT applied locally.
+// The provenance metadata includes "unscoped":true when params were present but not applied.
+func resolveLocal(resourceType string, isList bool, path string, params map[string]string, reason string) (json.RawMessage, DataProvenance, error) {
+ db, err := openStoreForRead("{{.Name}}-pp-cli")
+ if err != nil {
+ return nil, DataProvenance{}, fmt.Errorf("opening local database: %w\nRun '{{.Name}}-pp-cli sync' first.", err)
+ }
+ if db == nil {
+ return nil, DataProvenance{}, fmt.Errorf("no local data. Run '{{.Name}}-pp-cli sync' first")
+ }
+ defer db.Close()
+
+ prov := localProvenance(db, resourceType, reason)
+
+ // Warn if endpoint had filters that local reads can't reproduce
+ if len(params) > 0 {
+ fmt.Fprintf(os.Stderr, "warning: local data is unfiltered — endpoint filters are not applied to cached data\n")
+ }
+
+ if isList {
+ raw, err := db.List(resourceType, 0) // 0 = no limit, return all synced data
+ if err != nil {
+ return nil, DataProvenance{}, fmt.Errorf("querying local store: %w", err)
+ }
+ // Filter out empty/invalid records (empty arrays, null, whitespace-only)
+ // that can end up in the store from pagination boundary artifacts.
+ var items []json.RawMessage
+ for _, r := range raw {
+ trimmed := strings.TrimSpace(string(r))
+ if trimmed == "" || trimmed == "null" || trimmed == "[]" || trimmed == "{}" {
+ continue
+ }
+ items = append(items, r)
+ }
+ if len(items) == 0 {
+ return nil, DataProvenance{}, fmt.Errorf("no local data for %q. Run '{{.Name}}-pp-cli sync' first", resourceType)
+ }
+ // Marshal []json.RawMessage into a single JSON array
+ data, err := json.Marshal(items)
+ if err != nil {
+ return nil, DataProvenance{}, fmt.Errorf("marshaling local data: %w", err)
+ }
+ return data, prov, nil
+ }
+
+ // Get by ID — extract the last path segment as the ID
+ parts := strings.Split(strings.TrimRight(path, "/"), "/")
+ id := parts[len(parts)-1]
+
+ item, err := db.Get(resourceType, id)
+ if err != nil {
+ return nil, DataProvenance{}, fmt.Errorf("querying local store: %w", err)
+ }
+ if item == nil {
+ return nil, DataProvenance{}, fmt.Errorf("resource %q with ID %q not found in local store. Run '{{.Name}}-pp-cli sync' first", resourceType, id)
+ }
+ return item, prov, nil
+}
+
+// Ensure time import is used (compilation guard).
+var _ = time.Now
diff --git a/internal/generator/templates/helpers.go.tmpl b/internal/generator/templates/helpers.go.tmpl
index 6effd2ad..893c2b5c 100644
--- a/internal/generator/templates/helpers.go.tmpl
+++ b/internal/generator/templates/helpers.go.tmpl
@@ -9,12 +9,14 @@ import (
"fmt"
"io"
"os"
+ "path/filepath"
{{- if and .Auth.Type (ne .Auth.Type "none")}}
"regexp"
{{- end}}
"sort"
"strings"
"text/tabwriter"
+ "time"
"unicode"
"github.com/spf13/cobra"
@@ -1057,3 +1059,63 @@ func findField(obj map[string]any, names ...string) string {
}
return ""
}
+
+// DataProvenance describes where data came from and when it was last synced.
+type DataProvenance struct {
+ Source string `json:"source"` // "live" or "local"
+ SyncedAt *time.Time `json:"synced_at,omitempty"` // when local data was last synced
+ Reason string `json:"reason,omitempty"` // why local was used: "user_requested", "api_unreachable", "no_search_endpoint"
+ ResourceType string `json:"resource_type,omitempty"` // which resource type was queried
+}
+
+// printProvenance writes a one-line provenance message to stderr.
+func printProvenance(cmd *cobra.Command, count int, prov DataProvenance) {
+ if prov.Source == "live" {
+ fmt.Fprintf(cmd.ErrOrStderr(), "%d results (live)\n", count)
+ return
+ }
+ age := "unknown"
+ if prov.SyncedAt != nil {
+ d := time.Since(*prov.SyncedAt)
+ switch {
+ case d < time.Minute:
+ age = "just now"
+ case d < time.Hour:
+ age = fmt.Sprintf("%d minutes ago", int(d.Minutes()))
+ case d < 24*time.Hour:
+ age = fmt.Sprintf("%d hours ago", int(d.Hours()))
+ default:
+ age = fmt.Sprintf("%d days ago", int(d.Hours()/24))
+ }
+ }
+ prefix := ""
+ if prov.Reason == "api_unreachable" {
+ prefix = "API unreachable. "
+ }
+ fmt.Fprintf(cmd.ErrOrStderr(), "%s%d results (cached, synced %s)\n", prefix, count, age)
+}
+
+// wrapWithProvenance wraps JSON data in a provenance envelope: {"results": ..., "meta": {...}}.
+func wrapWithProvenance(data json.RawMessage, prov DataProvenance) (json.RawMessage, error) {
+ meta := map[string]any{"source": prov.Source}
+ if prov.SyncedAt != nil {
+ meta["synced_at"] = prov.SyncedAt.UTC().Format(time.RFC3339)
+ }
+ if prov.Reason != "" {
+ meta["reason"] = prov.Reason
+ }
+ if prov.ResourceType != "" {
+ meta["resource_type"] = prov.ResourceType
+ }
+ envelope := map[string]any{
+ "results": json.RawMessage(data),
+ "meta": meta,
+ }
+ return json.Marshal(envelope)
+}
+
+// defaultDBPath returns the canonical path for the local SQLite database.
+func defaultDBPath(name string) string {
+ home, _ := os.UserHomeDir()
+ return filepath.Join(home, ".local", "share", name, "data.db")
+}
diff --git a/internal/generator/templates/insights/health_score.go.tmpl b/internal/generator/templates/insights/health_score.go.tmpl
index 2aa76dbb..91bed7d1 100644
--- a/internal/generator/templates/insights/health_score.go.tmpl
+++ b/internal/generator/templates/insights/health_score.go.tmpl
@@ -6,8 +6,6 @@ package cli
import (
"encoding/json"
"fmt"
- "os"
- "path/filepath"
"time"
"{{modulePath}}/internal/store"
@@ -33,8 +31,7 @@ A score of 100 means all items are fresh, assigned, and actively worked on.`,
{{.Name}}-pp-cli health --json`,
RunE: func(cmd *cobra.Command, args []string) error {
if dbPath == "" {
- home, _ := os.UserHomeDir()
- dbPath = filepath.Join(home, ".config", "{{.Name}}-pp-cli", "store.db")
+ dbPath = defaultDBPath("{{.Name}}-pp-cli")
}
db, err := store.Open(dbPath)
@@ -164,7 +161,7 @@ A score of 100 means all items are fresh, assigned, and actively worked on.`,
},
}
- cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.config/{{.Name}}-pp-cli/store.db)")
+ cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.local/share/{{.Name}}-pp-cli/data.db)")
return cmd
}
diff --git a/internal/generator/templates/insights/similar.go.tmpl b/internal/generator/templates/insights/similar.go.tmpl
index b31468aa..355ecf0c 100644
--- a/internal/generator/templates/insights/similar.go.tmpl
+++ b/internal/generator/templates/insights/similar.go.tmpl
@@ -6,8 +6,6 @@ package cli
import (
"encoding/json"
"fmt"
- "os"
- "path/filepath"
"{{modulePath}}/internal/store"
"github.com/spf13/cobra"
@@ -36,8 +34,7 @@ work items, tickets, or tasks.`,
itemID := args[0]
if dbPath == "" {
- home, _ := os.UserHomeDir()
- dbPath = filepath.Join(home, ".config", "{{.Name}}-pp-cli", "store.db")
+ dbPath = defaultDBPath("{{.Name}}-pp-cli")
}
db, err := store.Open(dbPath)
@@ -162,7 +159,7 @@ work items, tickets, or tasks.`,
},
}
- cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.config/{{.Name}}-pp-cli/store.db)")
+ cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.local/share/{{.Name}}-pp-cli/data.db)")
cmd.Flags().IntVar(&limit, "limit", 10, "Maximum number of similar items to show")
return cmd
diff --git a/internal/generator/templates/mcp_tools.go.tmpl b/internal/generator/templates/mcp_tools.go.tmpl
index 1bd539d1..3bbe74ca 100644
--- a/internal/generator/templates/mcp_tools.go.tmpl
+++ b/internal/generator/templates/mcp_tools.go.tmpl
@@ -284,6 +284,8 @@ func dbPath() string {
home, _ := os.UserHomeDir()
return filepath.Join(home, ".local", "share", "{{.Name}}-pp-cli", "data.db")
}
+// Note: MCP tools use their own dbPath() because they are in a separate package (main, not cli).
+// The CLI's defaultDBPath() in the cli package uses the same canonical path.
{{- if .VisionSet.Sync}}
diff --git a/internal/generator/templates/root.go.tmpl b/internal/generator/templates/root.go.tmpl
index d211c26d..0fdc7a85 100644
--- a/internal/generator/templates/root.go.tmpl
+++ b/internal/generator/templates/root.go.tmpl
@@ -32,6 +32,7 @@ type rootFlags struct {
configPath string
timeout time.Duration
rateLimit float64
+ dataSource string
}
// Execute runs the CLI in non-interactive mode: never prompts, all values via flags or stdin.
@@ -62,6 +63,7 @@ func Execute() error {
rootCmd.PersistentFlags().BoolVar(&noColor, "no-color", false, "Disable colored output")
rootCmd.PersistentFlags().BoolVar(&humanFriendly, "human-friendly", false, "Enable colored output and rich formatting")
rootCmd.PersistentFlags().BoolVar(&flags.agent, "agent", false, "Set all agent-friendly defaults (--json --compact --no-input --no-color --yes)")
+ rootCmd.PersistentFlags().StringVar(&flags.dataSource, "data-source", "auto", "Data source for read commands: auto (live with local fallback), live (API only), local (synced data only)")
{{- if eq .SpecSource "sniffed"}}
rootCmd.PersistentFlags().Float64Var(&flags.rateLimit, "rate-limit", 2, "Max requests per second (0 to disable, default 2 for sniffed APIs)")
{{- else}}
@@ -86,6 +88,12 @@ func Execute() error {
noColor = true
}
}
+ switch flags.dataSource {
+ case "auto", "live", "local":
+ // valid
+ default:
+ return fmt.Errorf("invalid --data-source value %q: must be auto, live, or local", flags.dataSource)
+ }
return nil
}
diff --git a/internal/generator/templates/search.go.tmpl b/internal/generator/templates/search.go.tmpl
index 65626478..a28ac71a 100644
--- a/internal/generator/templates/search.go.tmpl
+++ b/internal/generator/templates/search.go.tmpl
@@ -6,8 +6,6 @@ package cli
import (
"encoding/json"
"fmt"
- "os"
- "path/filepath"
"strings"
"{{modulePath}}/internal/store"
@@ -38,6 +36,30 @@ func isNilOrEmpty(raw json.RawMessage) bool {
return true
}
+{{- /* extractSearchResults tries common response wrapper paths to extract result arrays */}}
+
+// extractSearchResults unwraps API search responses by checking common envelope paths.
+func extractSearchResults(data json.RawMessage) []json.RawMessage {
+ // Try direct array first
+ var items []json.RawMessage
+ if json.Unmarshal(data, &items) == nil {
+ return items
+ }
+ // Try common wrapper paths: data, results, items
+ var wrapped map[string]json.RawMessage
+ if json.Unmarshal(data, &wrapped) == nil {
+ for _, key := range []string{"data", "results", "items", "records", "entries"} {
+ if inner, ok := wrapped[key]; ok {
+ if json.Unmarshal(inner, &items) == nil {
+ return items
+ }
+ }
+ }
+ }
+ // Return as single-item array
+ return []json.RawMessage{data}
+}
+
func newSearchCmd(flags *rootFlags) *cobra.Command {
var resourceType string
var limit int
@@ -45,25 +67,73 @@ func newSearchCmd(flags *rootFlags) *cobra.Command {
cmd := &cobra.Command{
Use: "search <query>",
- Short: "Full-text search across locally synced data",
- Long: `Search locally cached data using FTS5 full-text search.
-Data must be synced first with the sync command. Searches are instant
-(millisecond-fast) since they query local SQLite, not the remote API.`,
- Example: ` # Search all synced data
+ Short: "Full-text search across synced data or live API",
+ Long: `Search data using FTS5 full-text search on locally synced data,
+or hit the API's search endpoint when available.
+
+In auto mode (default): uses the API search endpoint if the API has one,
+otherwise searches local data. Falls back to local on network failure.
+In live mode: uses the API search endpoint only.
+In local mode: searches locally synced data only.`,
+ Example: ` # Search (uses API endpoint if available, local FTS otherwise)
{{.Name}}-pp-cli search "error timeout"
- # Search a specific resource type
- {{.Name}}-pp-cli search "payment failed" --type transactions
+ # Force local search only
+ {{.Name}}-pp-cli search "payment failed" --data-source local
+
+ # Search a specific resource type locally
+ {{.Name}}-pp-cli search "critical" --type transactions --data-source local
# JSON output for piping
{{.Name}}-pp-cli search "critical" --json --limit 20`,
- Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
+ if len(args) == 0 {
+ return cmd.Help()
+ }
query := args[0]
+{{- if .SearchEndpointPath}}
+ // This API has a search endpoint: {{.SearchEndpointMethod}} {{.SearchEndpointPath}}
+ if flags.dataSource != "local" {
+ c, err := flags.newClient()
+ if err != nil {
+ return err
+ }
+{{- if eq .SearchEndpointMethod "POST"}}
+ data, _, getErr := c.Post("{{.SearchEndpointPath}}", map[string]any{
+ "{{.SearchQueryParam}}": query,
+ })
+{{- else}}
+ data, getErr := c.Get("{{.SearchEndpointPath}}", map[string]string{
+ "{{.SearchQueryParam}}": query,
+ })
+{{- end}}
+ if getErr == nil {
+ // Live search succeeded
+ results := extractSearchResults(data)
+ prov := DataProvenance{Source: "live"}
+ return outputSearchResults(cmd, flags, results, limit, prov)
+ }
+ // Check if it's a network error for auto-mode fallback
+ if flags.dataSource == "live" || !isNetworkError(getErr) {
+ return classifyAPIError(getErr)
+ }
+ // auto mode + network error: fall through to local FTS
+ fmt.Fprintf(cmd.ErrOrStderr(), "API unreachable, falling back to local search.\n")
+ }
+{{- else}}
+ // This API has no search endpoint.
+ if flags.dataSource == "live" {
+ return fmt.Errorf("this API has no search endpoint. Use --data-source local or --data-source auto to search locally synced data")
+ }
+ if flags.dataSource == "auto" {
+ fmt.Fprintf(cmd.ErrOrStderr(), "This API has no search endpoint. Searching local data.\n")
+ }
+{{- end}}
+
+ // Local FTS search
if dbPath == "" {
- home, _ := os.UserHomeDir()
- dbPath = filepath.Join(home, ".local", "share", "{{.Name}}-pp-cli", "data.db")
+ dbPath = defaultDBPath("{{.Name}}-pp-cli")
}
db, err := store.Open(dbPath)
@@ -82,8 +152,6 @@ Data must be synced first with the sync command. Searches are instant
{{- end}}
case "":
// Search all FTS-enabled tables individually to avoid duplicates.
- // Do NOT use db.Search() here — it queries resources_fts which
- // overlaps with per-table FTS indexes and causes duplicate results.
seen := make(map[string]bool)
_ = seen // prevent unused error when no FTS tables exist
{{- range .Tables}}
@@ -111,35 +179,19 @@ Data must be synced first with the sync command. Searches are instant
return fmt.Errorf("search failed: %w", err)
}
- // Filter out entries with nil or empty identifier fields.
- filtered := make([]json.RawMessage, 0, len(results))
- for _, r := range results {
- if !isNilOrEmpty(r) {
- filtered = append(filtered, r)
- }
- }
- results = filtered
-
- // Enforce limit across aggregated results.
- if len(results) > limit {
- results = results[:limit]
- }
-
- if len(results) == 0 {
- fmt.Fprintf(os.Stderr, "No results for %q\n", query)
- return nil
+ reason := "user_requested"
+{{- if .SearchEndpointPath}}
+ if flags.dataSource == "auto" {
+ reason = "api_unreachable"
}
-
- if flags.asJSON {
- enc := json.NewEncoder(os.Stdout)
- enc.SetIndent("", " ")
- return enc.Encode(results)
+{{- else}}
+ if flags.dataSource != "local" {
+ reason = "no_search_endpoint"
}
+{{- end}}
+ prov := localProvenance(db, "search", reason)
- for _, r := range results {
- fmt.Println(string(r))
- }
- return nil
+ return outputSearchResults(cmd, flags, results, limit, prov)
},
}
@@ -149,3 +201,45 @@ Data must be synced first with the sync command. Searches are instant
return cmd
}
+
+// outputSearchResults filters, counts, and outputs search results with provenance.
+func outputSearchResults(cmd *cobra.Command, flags *rootFlags, results []json.RawMessage, limit int, prov DataProvenance) error {
+ // Filter out entries with nil or empty identifier fields.
+ filtered := make([]json.RawMessage, 0, len(results))
+ for _, r := range results {
+ if !isNilOrEmpty(r) {
+ filtered = append(filtered, r)
+ }
+ }
+ results = filtered
+
+ // Enforce limit across aggregated results.
+ if len(results) > limit {
+ results = results[:limit]
+ }
+
+ if len(results) == 0 {
+ fmt.Fprintf(cmd.ErrOrStderr(), "No results (source: %s)\n", prov.Source)
+ return nil
+ }
+
+ // Print provenance to stderr for human output
+ printProvenance(cmd, len(results), prov)
+
+ if flags.asJSON || !isTerminal(cmd.OutOrStdout()) {
+ data, err := json.Marshal(results)
+ if err != nil {
+ return err
+ }
+ wrapped, err := wrapWithProvenance(json.RawMessage(data), prov)
+ if err != nil {
+ return err
+ }
+ return printOutput(cmd.OutOrStdout(), wrapped, true)
+ }
+
+ for _, r := range results {
+ fmt.Fprintln(cmd.OutOrStdout(), string(r))
+ }
+ return nil
+}
diff --git a/internal/generator/templates/sync.go.tmpl b/internal/generator/templates/sync.go.tmpl
index bccbd434..e62ac85e 100644
--- a/internal/generator/templates/sync.go.tmpl
+++ b/internal/generator/templates/sync.go.tmpl
@@ -7,7 +7,6 @@ import (
"encoding/json"
"fmt"
"os"
- "path/filepath"
"regexp"
"strconv"
"strings"
@@ -62,8 +61,7 @@ Once synced, use the 'search' command for instant full-text search.`,
c.NoCache = true
if dbPath == "" {
- home, _ := os.UserHomeDir()
- dbPath = filepath.Join(home, ".local", "share", "{{.Name}}-pp-cli", "data.db")
+ dbPath = defaultDBPath("{{.Name}}-pp-cli")
}
db, err := store.Open(dbPath)
diff --git a/internal/generator/templates/workflows/pm_load.go.tmpl b/internal/generator/templates/workflows/pm_load.go.tmpl
index c2c2b18e..7cc7a0a8 100644
--- a/internal/generator/templates/workflows/pm_load.go.tmpl
+++ b/internal/generator/templates/workflows/pm_load.go.tmpl
@@ -6,8 +6,6 @@ package cli
import (
"encoding/json"
"fmt"
- "os"
- "path/filepath"
"sort"
"{{modulePath}}/internal/store"
@@ -33,8 +31,7 @@ person. Helps identify overloaded team members and unbalanced workload.`,
{{.Name}}-pp-cli load --json`,
RunE: func(cmd *cobra.Command, args []string) error {
if dbPath == "" {
- home, _ := os.UserHomeDir()
- dbPath = filepath.Join(home, ".config", "{{.Name}}-pp-cli", "store.db")
+ dbPath = defaultDBPath("{{.Name}}-pp-cli")
}
db, err := store.Open(dbPath)
@@ -179,7 +176,7 @@ person. Helps identify overloaded team members and unbalanced workload.`,
},
}
- cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.config/{{.Name}}-pp-cli/store.db)")
+ cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.local/share/{{.Name}}-pp-cli/data.db)")
cmd.Flags().IntVar(&limit, "limit", 50, "Maximum entries to show")
return cmd
diff --git a/internal/generator/templates/workflows/pm_orphans.go.tmpl b/internal/generator/templates/workflows/pm_orphans.go.tmpl
index 16e9b5bc..a8acd381 100644
--- a/internal/generator/templates/workflows/pm_orphans.go.tmpl
+++ b/internal/generator/templates/workflows/pm_orphans.go.tmpl
@@ -6,8 +6,6 @@ package cli
import (
"encoding/json"
"fmt"
- "os"
- "path/filepath"
"strings"
"{{modulePath}}/internal/store"
@@ -30,8 +28,7 @@ such as assignee, project, priority, or labels. Useful for triaging unowned work
{{.Name}}-pp-cli orphans --json`,
RunE: func(cmd *cobra.Command, args []string) error {
if dbPath == "" {
- home, _ := os.UserHomeDir()
- dbPath = filepath.Join(home, ".config", "{{.Name}}-pp-cli", "store.db")
+ dbPath = defaultDBPath("{{.Name}}-pp-cli")
}
db, err := store.Open(dbPath)
@@ -154,7 +151,7 @@ such as assignee, project, priority, or labels. Useful for triaging unowned work
},
}
- cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.config/{{.Name}}-pp-cli/store.db)")
+ cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.local/share/{{.Name}}-pp-cli/data.db)")
cmd.Flags().IntVar(&limit, "limit", 50, "Maximum items to show")
return cmd
diff --git a/internal/generator/templates/workflows/pm_stale.go.tmpl b/internal/generator/templates/workflows/pm_stale.go.tmpl
index 24ea4ea6..da5be4eb 100644
--- a/internal/generator/templates/workflows/pm_stale.go.tmpl
+++ b/internal/generator/templates/workflows/pm_stale.go.tmpl
@@ -6,8 +6,6 @@ package cli
import (
"encoding/json"
"fmt"
- "os"
- "path/filepath"
"time"
"{{modulePath}}/internal/store"
@@ -38,8 +36,7 @@ the specified number of days. Useful for identifying forgotten or blocked work.`
{{.Name}}-pp-cli stale --days 30 --json`,
RunE: func(cmd *cobra.Command, args []string) error {
if dbPath == "" {
- home, _ := os.UserHomeDir()
- dbPath = filepath.Join(home, ".config", "{{.Name}}-pp-cli", "store.db")
+ dbPath = defaultDBPath("{{.Name}}-pp-cli")
}
db, err := store.Open(dbPath)
@@ -169,7 +166,7 @@ the specified number of days. Useful for identifying forgotten or blocked work.`
cmd.Flags().IntVar(&days, "days", 30, "Number of days without update to consider stale")
cmd.Flags().StringVar(&team, "team", "", "Filter by team identifier")
- cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.config/{{.Name}}-pp-cli/store.db)")
+ cmd.Flags().StringVar(&dbPath, "db", "", "Database path (default: ~/.local/share/{{.Name}}-pp-cli/data.db)")
cmd.Flags().IntVar(&limit, "limit", 50, "Maximum items to show")
return cmd
diff --git a/internal/profiler/profiler.go b/internal/profiler/profiler.go
index 8a765a91..0d05e4bc 100644
--- a/internal/profiler/profiler.go
+++ b/internal/profiler/profiler.go
@@ -69,6 +69,15 @@ type APIProfile struct {
SyncableResources []SyncableResource
SearchableFields map[string][]string
+ // SearchEndpointPath is the API path for live search (e.g., "/search", "/users/search").
+ // Empty if the API has no search endpoint.
+ SearchEndpointPath string
+ // SearchQueryParam is the query parameter name for the search endpoint (e.g., "q", "query").
+ // Defaults to "q" if a search endpoint exists but no recognized param is found.
+ SearchQueryParam string
+ // SearchEndpointMethod is the HTTP method for the search endpoint (GET or POST).
+ SearchEndpointMethod string
+
Domain DomainSignals
Pagination PaginationProfile
}
@@ -131,6 +140,30 @@ func Profile(s *spec.APISpec) *APIProfile {
if containsAny(endpointNameLower, []string{"search"}) || containsAny(pathLower, []string{"search"}) {
hasSearchEndpoint = true
+ // Prefer shorter/more general search paths (e.g., /search over /users/search)
+ if p.SearchEndpointPath == "" || len(endpoint.Path) < len(p.SearchEndpointPath) {
+ p.SearchEndpointPath = endpoint.Path
+ p.SearchEndpointMethod = strings.ToUpper(endpoint.Method)
+ // Find the query parameter for the search endpoint
+ p.SearchQueryParam = "q" // default
+ for _, param := range endpoint.Params {
+ pname := strings.ToLower(param.Name)
+ if pname == "q" || pname == "query" || pname == "search" || pname == "keyword" || pname == "term" {
+ p.SearchQueryParam = param.Name
+ break
+ }
+ }
+ // For POST search endpoints, check body params too
+ if p.SearchEndpointMethod == "POST" {
+ for _, param := range endpoint.Body {
+ pname := strings.ToLower(param.Name)
+ if pname == "q" || pname == "query" || pname == "search" || pname == "keyword" || pname == "term" {
+ p.SearchQueryParam = param.Name
+ break
+ }
+ }
+ }
+ }
}
if containsAny(pathLower, []string{"webhook", "event", "callback", "notification"}) {
p.HasRealtime = true
← ff465aac fix(skills): publish skill specifies full PR URL format
·
back to Cli Printing Press
·
feat(cli): search body construction, README website links, a 90cf5844 →