fix(profiler): improve HighVolume and NeedsSearch heuristics

a783ac12ca60843c5231dd82cdfba38f375f2170 · 2026-03-25 21:26:19 -0700 · Matt Van Horn

HighVolume now triggers on 5+ paginated endpoints (absolute threshold)
in addition to the >50% ratio check. Discord has 18 paginated
endpoints across 230 total - the ratio was low but the absolute
count is clearly high-volume.

NeedsSearch now checks the ratio of search endpoints to list
resources instead of boolean existence. Discord has 3 niche search
endpoints (thread, member, message) but 10+ list resources without
search - local FTS5 is still valuable.

Removed debug test file.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Files touched

M internal/profiler/profiler.go

Diff

commit a783ac12ca60843c5231dd82cdfba38f375f2170
Author: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Date:   Wed Mar 25 21:26:19 2026 -0700

    fix(profiler): improve HighVolume and NeedsSearch heuristics
    
    HighVolume now triggers on 5+ paginated endpoints (absolute threshold)
    in addition to the >50% ratio check. Discord has 18 paginated
    endpoints across 230 total - the ratio was low but the absolute
    count is clearly high-volume.
    
    NeedsSearch now checks the ratio of search endpoints to list
    resources instead of boolean existence. Discord has 3 niche search
    endpoints (thread, member, message) but 10+ list resources without
    search - local FTS5 is still valuable.
    
    Removed debug test file.
    
    Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 internal/profiler/profiler.go | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/internal/profiler/profiler.go b/internal/profiler/profiler.go
index 263b44f6..472278b6 100644
--- a/internal/profiler/profiler.go
+++ b/internal/profiler/profiler.go
@@ -137,9 +137,16 @@ func Profile(s *spec.APISpec) *APIProfile {
 		p.OfflineValuable = p.ReadRatio > 0.6
 	}
 	if listCapableGETs > 0 {
-		p.HighVolume = float64(p.ListEndpoints)/float64(listCapableGETs) > 0.5
+		paginationRatio := float64(p.ListEndpoints) / float64(listCapableGETs)
+		// HighVolume: either >50% of list endpoints are paginated, or 5+ paginated endpoints exist
+		p.HighVolume = paginationRatio > 0.5 || p.ListEndpoints >= 5
 	}
-	p.NeedsSearch = len(listResources) >= 3 && !hasSearchEndpoint
+	// NeedsSearch: 3+ list resources exist and fewer than half have dedicated search endpoints
+	searchEndpointCount := 0
+	if hasSearchEndpoint {
+		searchEndpointCount = 1 // conservative: count as 1 even if multiple search endpoints exist
+	}
+	p.NeedsSearch = len(listResources) >= 3 && float64(searchEndpointCount)/float64(len(listResources)) < 0.5
 
 	p.SyncableResources = sortedKeys(syncable)
 	for resource, fields := range searchable {

← de296d1f feat(scorecard): implement two-tier Vision scoring · back to Cli Printing Press · feat(llmpolish): add LLM Vision Synthesis for domain-aware c 611f4b67 →