← back to Cli Printing Press
fix(cli): sync path resolution for non-paginated list endpoints
8763a05fc175510dce9636ce516bba626866e15a · 2026-04-03 23:05:10 -0700 · Trevin Chow
The profiler only added endpoints to SyncableResources when
endpoint.Pagination != nil. APIs like Steam Web that return data
arrays without explicit pagination metadata were never syncable,
causing sync to fall back to "GET /<resource-name>" and 404.
Fix: move the syncable map population outside the pagination gate.
All list endpoints detected by isListEndpoint() are now syncable,
with a guard excluding paths containing "{" (unfilled path params).
Non-paginated endpoints sync as single-page fetches.
This unblocks the entire offline/local stack for non-REST APIs:
sync, transcendence commands, --data-source local, and data pipeline
scoring.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Files touched
A docs/plans/2026-04-03-006-fix-sync-path-resolution-plan.mdM internal/profiler/profiler.goM internal/profiler/profiler_test.go
Diff
commit 8763a05fc175510dce9636ce516bba626866e15a
Author: Trevin Chow <trevin@trevinchow.com>
Date: Fri Apr 3 23:05:10 2026 -0700
fix(cli): sync path resolution for non-paginated list endpoints
The profiler only added endpoints to SyncableResources when
endpoint.Pagination != nil. APIs like Steam Web that return data
arrays without explicit pagination metadata were never syncable,
causing sync to fall back to "GET /<resource-name>" and 404.
Fix: move the syncable map population outside the pagination gate.
All list endpoints detected by isListEndpoint() are now syncable,
with a guard excluding paths containing "{" (unfilled path params).
Non-paginated endpoints sync as single-page fetches.
This unblocks the entire offline/local stack for non-REST APIs:
sync, transcendence commands, --data-source local, and data pipeline
scoring.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
...2026-04-03-006-fix-sync-path-resolution-plan.md | 94 ++++++++++++++++++++++
internal/profiler/profiler.go | 41 +++++++++-
internal/profiler/profiler_test.go | 5 +-
3 files changed, 135 insertions(+), 5 deletions(-)
diff --git a/docs/plans/2026-04-03-006-fix-sync-path-resolution-plan.md b/docs/plans/2026-04-03-006-fix-sync-path-resolution-plan.md
new file mode 100644
index 00000000..a87be157
--- /dev/null
+++ b/docs/plans/2026-04-03-006-fix-sync-path-resolution-plan.md
@@ -0,0 +1,94 @@
+---
+title: "fix: Sync path resolution for non-paginated list endpoints"
+type: fix
+status: active
+date: 2026-04-03
+---
+
+# Fix Sync Path Resolution for Non-Paginated List Endpoints
+
+## Overview
+
+The profiler only adds endpoints to `SyncableResources` when `endpoint.Pagination != nil`. APIs like Steam Web that return data arrays without explicit pagination metadata are never added to the syncable map. The sync template's `syncResourcePath()` then falls back to `"/" + resource`, producing 404s. Fix: expand syncable detection to include all list endpoints, not just paginated ones.
+
+## Problem Frame
+
+The sync template already has the correct path mapping infrastructure (`syncResourcePath()` reads from a map populated by `{{range .SyncableResources}}`). The profiler already detects list endpoints via `isListEndpoint()`. But the connection between them is gated on `endpoint.Pagination != nil`:
+
+```
+isListEndpoint() → yes → endpoint.Pagination != nil? → yes → add to syncable
+ → NO → skip (BUG)
+```
+
+This means ~10-15% of APIs (non-REST or no-pagination) have empty syncable maps, breaking sync, transcendence commands, `--data-source local`, and data pipeline scoring.
+
+## Requirements Trace
+
+- R1. List endpoints without explicit pagination metadata are added to `SyncableResources`
+- R2. `syncResourcePath()` returns the spec's actual path for all syncable resources
+- R3. Sync fetches non-paginated endpoints once (no cursor loop) without error
+- R4. Endpoints requiring unfilled path parameters (e.g., `{steamid}`) are excluded from syncable
+
+## Scope Boundaries
+
+- **Not changing the sync template.** The template already handles non-paginated resources correctly — it breaks out of the pagination loop when results < page size.
+- **Not adding path parameter substitution.** Endpoints like `/users/{id}/games` that require a known ID remain unsyncable. That's a separate feature.
+- **Not changing pagination detection.** The profiler's pagination detection is correct — this fix is about syncable detection, not pagination detection.
+
+## Key Technical Decisions
+
+- **Move `syncable[resourceName] = endpoint.Path` outside the pagination gate**: The simplest fix. List endpoints detected by `isListEndpoint()` should be syncable regardless of whether they have pagination metadata. The sync template's pagination loop already handles single-page responses gracefully.
+
+- **Exclude endpoints with unfilled path parameters**: If `endpoint.Path` contains `{`, the endpoint requires a runtime parameter the sync command can't provide. Skip these.
+
+## Implementation Units
+
+- [ ] **Unit 1: Expand syncable resource detection in the profiler**
+
+**Goal:** Add all list endpoints (not just paginated ones) to `SyncableResources`, excluding those with path parameters.
+
+**Requirements:** R1, R2, R3, R4
+
+**Dependencies:** None
+
+**Files:**
+- Modify: `internal/profiler/profiler.go`
+- Test: `internal/profiler/profiler_test.go`
+
+**Approach:**
+- In the `walk` function, move the `syncable[resourceName] = endpoint.Path` assignment out of the `endpoint.Pagination != nil` block and into the `isListEndpoint()` block
+- Preserve the existing shortest-path guard: only set `syncable[resourceName]` if the key is absent or the new path is shorter
+- Add a guard: skip if `strings.Contains(endpoint.Path, "{")` — these require path params sync can't provide
+- Keep the enum expansion logic inside the pagination block (it depends on pagination for page-through)
+- The pagination-specific counter (`p.ListEndpoints++`) stays inside the pagination block — it counts paginated endpoints specifically
+- Update the `SyncableResource` doc comment from "paginated list sync" to "list sync (paginated or single-page)"
+
+**Patterns to follow:**
+- Existing `isListEndpoint()` detection at `profiler.go`
+- Existing `syncable` map population pattern
+
+**Test scenarios:**
+- Happy path: spec with paginated list endpoints → syncable resources populated with correct paths (existing behavior preserved)
+- Happy path: spec with non-paginated list endpoints (like Steam's `/ISteamApps/GetAppList/v2/`) → added to syncable with correct path
+- Edge case: endpoint with path parameter (`/users/{id}/games`) → excluded from syncable
+- Edge case: multiple list endpoints for same resource → shortest path wins (existing behavior)
+- Integration: generate a CLI from Steam spec → `defaultSyncResources()` includes Steam interfaces, `syncResourcePath()` returns correct paths
+
+**Verification:**
+- `go test ./internal/profiler/...` passes
+- Generate a CLI from the Steam spec → `syncResourcePath("isteam-apps")` returns `/ISteamApps/GetAppList/v2/` not `/isteam-apps`
+- Sync on a Steam CLI fetches from the correct API path
+
+## Risks & Dependencies
+
+| Risk | Mitigation |
+|------|------------|
+| Non-paginated endpoints may return very large responses | The sync template already has a `limit` param in the request — if the API ignores it, the response may be large but won't loop. Acceptable for a sync operation. |
+| Some list endpoints may not be useful to sync (e.g., internal/admin endpoints) | The profiler's `isListEndpoint()` is already conservative. Adding path-param exclusion further narrows the set. |
+
+## Sources & References
+
+- Related code: `internal/profiler/profiler.go` (lines 237-264, syncable population)
+- Related code: `internal/generator/templates/sync.go.tmpl` (lines 441-454, syncResourcePath)
+- Retro: Steam run 4 retro (sync path issues)
+- Retro: Steam run 2 retro (sync correctness findings)
diff --git a/internal/profiler/profiler.go b/internal/profiler/profiler.go
index 47ce2bf2..ba5b2819 100644
--- a/internal/profiler/profiler.go
+++ b/internal/profiler/profiler.go
@@ -52,7 +52,7 @@ type SearchBodyField struct {
Required bool `json:"required"`
}
-// SyncableResource describes a resource that supports paginated list sync.
+// SyncableResource describes a resource that supports list sync (paginated or single-page).
type SyncableResource struct {
Name string
Path string
@@ -237,6 +237,17 @@ func Profile(s *spec.APISpec) *APIProfile {
if isListEndpoint(endpointName, endpoint) {
listCapableGETs++
listResources[resourceName] = struct{}{}
+
+ // Add to syncable if the endpoint can be fetched without runtime context.
+ // Exclude: (a) paths with unfilled params like {steamid}
+ // (b) endpoints with required non-pagination query params (scoped lists
+ // like GetFriendList?steamid=REQUIRED that need a parent ID)
+ if !strings.Contains(endpoint.Path, "{") && !hasRequiredScopeParams(endpoint) {
+ if existing, ok := syncable[resourceName]; !ok || len(endpoint.Path) < len(existing) {
+ syncable[resourceName] = endpoint.Path
+ }
+ }
+
if endpoint.Pagination != nil {
p.ListEndpoints++
@@ -255,8 +266,8 @@ func Profile(s *spec.APISpec) *APIProfile {
syncable[expandedName] = expandedPath
}
} else {
- // Standard: pick the shortest path for determinism when
- // multiple paginated list endpoints exist for the same resource.
+ // Paginated endpoints override the path set above — they have
+ // richer pagination support for full data retrieval.
if existing, ok := syncable[resourceName]; !ok || len(endpoint.Path) < len(existing) {
syncable[resourceName] = endpoint.Path
}
@@ -537,6 +548,30 @@ func dataFit(v bool) int {
return 1
}
+// hasRequiredScopeParams returns true if the endpoint has required query parameters
+// that aren't pagination-related. These are "scoped list" endpoints (e.g., GetFriendList
+// requires steamid) that can't be synced without runtime context.
+func hasRequiredScopeParams(endpoint spec.Endpoint) bool {
+ paginationParams := map[string]bool{
+ "limit": true, "per_page": true, "page_size": true, "pageSize": true, "first": true, "count": true, "max_results": true,
+ "after": true, "cursor": true, "page_token": true, "offset": true, "page": true, "before": true, "starting_after": true,
+ "since": true, "updated_after": true, "modified_since": true, "since_id": true,
+ "key": true, "format": true, // auth and format params, not scope
+ }
+ for _, param := range endpoint.Params {
+ if param.Required && !param.Positional && !param.PathParam {
+ if !paginationParams[param.Name] && !paginationParams[strings.ToLower(param.Name)] {
+ // Enum params with 2+ values are handled by enum expansion, not scope
+ if len(param.Enum) >= 2 {
+ continue
+ }
+ return true
+ }
+ }
+ }
+ return false
+}
+
func isListEndpoint(name string, endpoint spec.Endpoint) bool {
if strings.ToUpper(endpoint.Method) != "GET" {
return false
diff --git a/internal/profiler/profiler_test.go b/internal/profiler/profiler_test.go
index 50183ae8..bbcc878a 100644
--- a/internal/profiler/profiler_test.go
+++ b/internal/profiler/profiler_test.go
@@ -105,12 +105,13 @@ func TestProfileEnumExpansion(t *testing.T) {
syncPaths[sr.Name] = sr.Path
}
- // 5 resources: 4 from enum expansion + 1 from teams
- assert.Len(t, profile.SyncableResources, 5)
+ // 6 resources: 4 from enum expansion + networkentity itself + teams
+ assert.Len(t, profile.SyncableResources, 6)
assert.Contains(t, syncNames, "collection")
assert.Contains(t, syncNames, "workspace")
assert.Contains(t, syncNames, "api")
assert.Contains(t, syncNames, "flow")
+ assert.Contains(t, syncNames, "networkentity")
assert.Contains(t, syncNames, "team")
// Expanded paths include the enum value as a query param
← 729ad07d fix(cli): use catalog Homepage for README website link, remo
·
back to Cli Printing Press
·
fix(skills): publish skill checks for merged PRs before reus 360db307 →