← back to Cli Printing Press
fix(scorecard): handle unscored auth semantics (#29)
7a1c164697aa3c0fd98a6fb602acb2a27c6336f1 · 2026-03-28 10:27:26 -0700 · Trevin Chow
* fix(scorecard): handle unscored auth semantics
Resolve scorecard auth and spec-derived scoring semantics so missing or non-applicable evidence no longer penalizes generated CLIs. Fail fast on invalid specs, honor effective OpenAPI security semantics, preserve JSON compatibility via unscored_dimensions, and cover the edge cases with focused tier-2 tests.
* fix(dogfood): restore spec loading
Files touched
M docs/solutions/best-practices/steinberger-scorecard-scoring-architecture-2026-03-27.mdM internal/cli/scorecard.goM internal/pipeline/dogfood.goM internal/pipeline/scorecard.goM internal/pipeline/scorecard_tier2_test.goM skills/printing-press-score/SKILL.md
Diff
commit 7a1c164697aa3c0fd98a6fb602acb2a27c6336f1
Author: Trevin Chow <trevin@trevinchow.com>
Date: Sat Mar 28 10:27:26 2026 -0700
fix(scorecard): handle unscored auth semantics (#29)
* fix(scorecard): handle unscored auth semantics
Resolve scorecard auth and spec-derived scoring semantics so missing or non-applicable evidence no longer penalizes generated CLIs. Fail fast on invalid specs, honor effective OpenAPI security semantics, preserve JSON compatibility via unscored_dimensions, and cover the edge cases with focused tier-2 tests.
* fix(dogfood): restore spec loading
---
...er-scorecard-scoring-architecture-2026-03-27.md | 31 +-
internal/cli/scorecard.go | 15 +-
internal/pipeline/dogfood.go | 127 ++++++-
internal/pipeline/scorecard.go | 415 +++++++++++++++++----
internal/pipeline/scorecard_tier2_test.go | 354 +++++++++++++++++-
skills/printing-press-score/SKILL.md | 15 +-
6 files changed, 867 insertions(+), 90 deletions(-)
diff --git a/docs/solutions/best-practices/steinberger-scorecard-scoring-architecture-2026-03-27.md b/docs/solutions/best-practices/steinberger-scorecard-scoring-architecture-2026-03-27.md
index b7024c58..27441544 100644
--- a/docs/solutions/best-practices/steinberger-scorecard-scoring-architecture-2026-03-27.md
+++ b/docs/solutions/best-practices/steinberger-scorecard-scoring-architecture-2026-03-27.md
@@ -1,6 +1,7 @@
---
title: "Steinberger scorecard scoring system: architecture, conventions, and modification rules"
date: 2026-03-27
+last_updated: 2026-03-28
category: best-practices
module: internal/pipeline
problem_type: best_practice
@@ -21,6 +22,7 @@ tags:
- tier-weighting
- best-practice
- calibration
+ - unscored-dimensions
---
# Steinberger scorecard: architecture, conventions, and modification rules
@@ -71,6 +73,29 @@ Total = tier1Normalized + tier2Normalized // 0-100
Each tier contributes exactly 50 points max. If you add a Tier 1 dimension, update the `120` constant. If you add a Tier 2 dimension, update the `50` constant.
+### Unscored dimensions
+
+Some dimensions are only valid when the source of truth contains the evidence needed to judge them. For example, `PathValidity` needs OpenAPI paths and `AuthProtocol` needs `securitySchemes`.
+
+If that evidence is missing, the dimension is **unscored**, not mediocre:
+
+```go
+type Scorecard struct {
+ UnscoredDimensions []string `json:"unscored_dimensions,omitempty"`
+}
+```
+
+Render unscored dimensions as `N/A`, omit them from gap reports, and remove their max points from the denominator before normalizing the tier:
+
+```go
+tier2Max := 50
+if sc.IsDimensionUnscored("path_validity") { tier2Max -= 10 }
+if sc.IsDimensionUnscored("auth_protocol") { tier2Max -= 10 }
+tier2Normalized := (tier2Raw * 50) / tier2Max
+```
+
+Do **not** encode "missing evidence" as a midpoint like `5/10`. A midpoint looks neutral in code review but still lowers the final score because it stays inside the denominator. That turns an epistemic unknown into a real product penalty.
+
### Grade thresholds
| Grade | Threshold |
@@ -182,9 +207,11 @@ Six prefixes appear in both workflow and insight lists: `stale`, `conflicts`, `s
8. **Structural detection complements prefix matching.** Don't rely solely on filename prefixes.
-9. **Update tier constants when adding dimensions.** Tier 1 constant is `120` (12 x 10). Tier 2 constant is `50`. Both live in the `RunScorecard` function.
+9. **Unknown evidence must become `N/A`, not a midpoint.** If the spec or other authority lacks the data needed to evaluate a dimension, mark it unscored, expose it in `unscored_dimensions`, skip it in gap reports, and remove its max points from the tier denominator.
+
+10. **Update tier constants when adding dimensions.** Tier 1 constant is `120` (12 x 10). Tier 2 constant is `50`. Both live in the `RunScorecard` function. If dimensions can become unscored, adjust the runtime denominator too.
-10. **Test every scoring function independently.** Each `score*()` function should have fixture-based tests covering: high score, low score, and dimension-specific edge cases.
+11. **Test every scoring function independently.** Each `score*()` function should have fixture-based tests covering: high score, low score, dimension-specific edge cases, and unscored/unknown states for evidence-dependent dimensions.
For detailed examples of bugs caused by violating these rules, see `docs/solutions/logic-errors/scorecard-accuracy-broadened-pattern-matching-2026-03-27.md`.
diff --git a/internal/cli/scorecard.go b/internal/cli/scorecard.go
index add388cc..eb0f35b6 100644
--- a/internal/cli/scorecard.go
+++ b/internal/cli/scorecard.go
@@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"os"
+ "strings"
"github.com/mvanhorn/cli-printing-press/internal/pipeline"
"github.com/spf13/cobra"
@@ -47,6 +48,13 @@ func newScorecardCmd() *cobra.Command {
// Human-readable output
s := sc.Steinberger
+ renderScore := func(name string, score, max int) string {
+ if sc.IsDimensionUnscored(name) {
+ return "N/A"
+ }
+ return fmt.Sprintf("%d/%d", score, max)
+ }
+
fmt.Printf("Quality Scorecard: %s\n\n", sc.APIName)
fmt.Printf(" Output Modes %d/10\n", s.OutputModes)
fmt.Printf(" Auth %d/10\n", s.Auth)
@@ -61,13 +69,16 @@ func newScorecardCmd() *cobra.Command {
fmt.Printf(" Workflows %d/10\n", s.Workflows)
fmt.Printf(" Insight %d/10\n", s.Insight)
fmt.Printf("\n Domain Correctness\n")
- fmt.Printf(" Path Validity %d/10\n", s.PathValidity)
- fmt.Printf(" Auth Protocol %d/10\n", s.AuthProtocol)
+ fmt.Printf(" Path Validity %s\n", renderScore("path_validity", s.PathValidity, 10))
+ fmt.Printf(" Auth Protocol %s\n", renderScore("auth_protocol", s.AuthProtocol, 10))
fmt.Printf(" Data Pipeline Integrity %d/10\n", s.DataPipelineIntegrity)
fmt.Printf(" Sync Correctness %d/10\n", s.SyncCorrectness)
fmt.Printf(" Type Fidelity %d/5\n", s.TypeFidelity)
fmt.Printf(" Dead Code %d/5\n", s.DeadCode)
fmt.Printf("\n Total: %d/100 - Grade %s\n", s.Total, sc.OverallGrade)
+ if len(sc.UnscoredDimensions) > 0 {
+ fmt.Printf(" Note: omitted from denominator: %s\n", strings.Join(sc.UnscoredDimensions, ", "))
+ }
if len(sc.GapReport) > 0 {
fmt.Printf("\nGaps:\n")
diff --git a/internal/pipeline/dogfood.go b/internal/pipeline/dogfood.go
index e238ec1b..5584f220 100644
--- a/internal/pipeline/dogfood.go
+++ b/internal/pipeline/dogfood.go
@@ -12,6 +12,7 @@ import (
"strings"
"time"
+ openapiparser "github.com/mvanhorn/cli-printing-press/internal/openapi"
apispec "github.com/mvanhorn/cli-printing-press/internal/spec"
)
@@ -130,19 +131,135 @@ func writeDogfoodResults(report *DogfoodReport, dir string) error {
}
func loadDogfoodOpenAPISpec(specPath string) (*openAPISpec, error) {
- summary, err := loadSpecSummary(specPath)
+ data, err := os.ReadFile(specPath)
if err != nil {
- return nil, err
+ return nil, fmt.Errorf("reading spec: %w", err)
+ }
+
+ parsed, parseErr := openapiparser.ParseLenient(data)
+ if parseErr == nil {
+ return &openAPISpec{
+ Paths: collectDogfoodSpecPaths(parsed.Resources),
+ Auth: parsed.Auth,
+ }, nil
}
- if summary == nil {
- return nil, nil
+
+ summary, err := loadOpenAPISpec(specPath)
+ if err != nil {
+ return nil, parseErr
}
+
return &openAPISpec{
Paths: summary.Paths,
- Auth: summary.Auth,
+ Auth: deriveDogfoodAuth(summary),
}, nil
}
+func collectDogfoodSpecPaths(resources map[string]apispec.Resource) []string {
+ var paths []string
+ for _, resource := range resources {
+ collectDogfoodResourcePaths(resource, &paths)
+ }
+ return uniqueSorted(paths)
+}
+
+func collectDogfoodResourcePaths(resource apispec.Resource, paths *[]string) {
+ for _, endpoint := range resource.Endpoints {
+ if strings.TrimSpace(endpoint.Path) != "" {
+ *paths = append(*paths, endpoint.Path)
+ }
+ }
+ for _, subresource := range resource.SubResources {
+ collectDogfoodResourcePaths(subresource, paths)
+ }
+}
+
+func deriveDogfoodAuth(spec *openAPISpecInfo) apispec.AuthConfig {
+ if spec == nil {
+ return apispec.AuthConfig{Type: "none"}
+ }
+
+ candidateKeys := referencedDogfoodSecurityKeys(spec.SecurityRequirements)
+ if len(candidateKeys) == 0 {
+ for key := range spec.SecuritySchemes {
+ candidateKeys = append(candidateKeys, key)
+ }
+ sort.Strings(candidateKeys)
+ }
+
+ for _, key := range candidateKeys {
+ scheme, ok := spec.SecuritySchemes[key]
+ if !ok {
+ continue
+ }
+ if auth, ok := dogfoodAuthConfigForScheme(scheme); ok {
+ return auth
+ }
+ }
+
+ return apispec.AuthConfig{Type: "none"}
+}
+
+func referencedDogfoodSecurityKeys(requirements []securityRequirementSet) []string {
+ seen := make(map[string]struct{})
+ var keys []string
+ for _, requirementSet := range requirements {
+ for _, alternative := range requirementSet.Alternatives {
+ for _, key := range alternative {
+ if _, ok := seen[key]; ok {
+ continue
+ }
+ seen[key] = struct{}{}
+ keys = append(keys, key)
+ }
+ }
+ }
+ sort.Strings(keys)
+ return keys
+}
+
+func dogfoodAuthConfigForScheme(scheme openAPISecurityScheme) (apispec.AuthConfig, bool) {
+ nameLower := strings.ToLower(scheme.Key)
+ auth := apispec.AuthConfig{
+ Type: "none",
+ Scheme: scheme.Key,
+ }
+
+ switch {
+ case strings.Contains(nameLower, "bot"):
+ auth.Type = "api_key"
+ auth.Header = "Authorization"
+ auth.Format = "Bot {bot_token}"
+ return auth, true
+ case scheme.Type == "http" && scheme.Scheme == "bearer":
+ auth.Type = "bearer_token"
+ auth.Header = "Authorization"
+ return auth, true
+ case scheme.Type == "http" && scheme.Scheme == "basic":
+ auth.Type = "api_key"
+ auth.Header = "Authorization"
+ auth.Format = "Basic {username}:{password}"
+ return auth, true
+ case scheme.Type == "apikey":
+ auth.Type = "api_key"
+ auth.In = scheme.In
+ auth.Header = strings.TrimSpace(scheme.HeaderName)
+ if auth.Header == "" {
+ auth.Header = "Authorization"
+ }
+ if strings.EqualFold(auth.Header, "Authorization") && strings.Contains(nameLower, "bot") {
+ auth.Format = "Bot {bot_token}"
+ }
+ return auth, true
+ case scheme.Type == "oauth2" || scheme.Type == "openidconnect":
+ auth.Type = "bearer_token"
+ auth.Header = "Authorization"
+ return auth, true
+ default:
+ return apispec.AuthConfig{}, false
+ }
+}
+
func checkPaths(dir string, paths []string) PathCheckResult {
result := PathCheckResult{}
if len(paths) == 0 {
diff --git a/internal/pipeline/scorecard.go b/internal/pipeline/scorecard.go
index abf704f3..fd10edc4 100644
--- a/internal/pipeline/scorecard.go
+++ b/internal/pipeline/scorecard.go
@@ -6,10 +6,9 @@ import (
"os"
"path/filepath"
"regexp"
+ "slices"
"strconv"
"strings"
-
- apispec "github.com/mvanhorn/cli-printing-press/internal/spec"
)
// infraCoreFiles are CLI infrastructure files excluded from workflow/insight scoring.
@@ -29,11 +28,12 @@ var infraAllFiles = map[string]bool{
// Scorecard holds the auto-scored evaluation of a generated CLI against the Steinberger bar.
type Scorecard struct {
- APIName string `json:"api_name"`
- Steinberger SteinerScore `json:"steinberger"`
- CompetitorScores []CompScore `json:"competitor_scores"`
- OverallGrade string `json:"overall_grade"`
- GapReport []string `json:"gap_report"`
+ APIName string `json:"api_name"`
+ Steinberger SteinerScore `json:"steinberger"`
+ CompetitorScores []CompScore `json:"competitor_scores"`
+ OverallGrade string `json:"overall_grade"`
+ GapReport []string `json:"gap_report"`
+ UnscoredDimensions []string `json:"unscored_dimensions,omitempty"`
}
// SteinerScore breaks down the Steinberger bar into 11 dimensions, each 0-10.
@@ -91,8 +91,24 @@ func RunScorecard(outputDir, pipelineDir, specPath string, verifyReport *VerifyR
sc.Steinberger.Vision = scoreVision(outputDir)
sc.Steinberger.Workflows = scoreWorkflows(outputDir)
sc.Steinberger.Insight = scoreInsight(outputDir)
- sc.Steinberger.PathValidity = scorePathValidity(outputDir, specPath)
- sc.Steinberger.AuthProtocol = scoreAuthProtocol(outputDir, specPath)
+
+ spec, err := loadOpenAPISpec(specPath)
+ if err != nil {
+ return nil, err
+ }
+
+ pathValidity := evaluatePathValidity(outputDir, spec)
+ sc.Steinberger.PathValidity = pathValidity.score
+ if !pathValidity.scored {
+ sc.UnscoredDimensions = append(sc.UnscoredDimensions, "path_validity")
+ }
+
+ authProtocol := evaluateAuthProtocol(outputDir, spec)
+ sc.Steinberger.AuthProtocol = authProtocol.score
+ if !authProtocol.scored {
+ sc.UnscoredDimensions = append(sc.UnscoredDimensions, "auth_protocol")
+ }
+
sc.Steinberger.DataPipelineIntegrity = scoreDataPipelineIntegrity(outputDir)
sc.Steinberger.SyncCorrectness = scoreSyncCorrectness(outputDir)
sc.Steinberger.TypeFidelity = scoreTypeFidelity(outputDir)
@@ -129,7 +145,18 @@ func RunScorecard(outputDir, pipelineDir, specPath string, verifyReport *VerifyR
// Weighted composite: Tier 1 = 50%, Tier 2 = 50% of final 100-point scale
tier1Normalized := (tier1Raw * 50) / 120 // scale 0-120 to 0-50
- tier2Normalized := (tier2Raw * 50) / 50 // scale 0-50 to 0-50
+ tier2Max := 50
+ if sc.IsDimensionUnscored("path_validity") {
+ tier2Max -= 10
+ }
+ if sc.IsDimensionUnscored("auth_protocol") {
+ tier2Max -= 10
+ }
+
+ tier2Normalized := 0
+ if tier2Max > 0 {
+ tier2Normalized = (tier2Raw * 50) / tier2Max
+ }
sc.Steinberger.Total = tier1Normalized + tier2Normalized
if sc.Steinberger.Total > 0 {
@@ -155,7 +182,7 @@ func RunScorecard(outputDir, pipelineDir, specPath string, verifyReport *VerifyR
sc.OverallGrade = computeGrade(sc.Steinberger.Percentage)
// Gap report for dimensions below 5
- sc.GapReport = buildGapReport(sc.Steinberger)
+ sc.GapReport = buildGapReport(sc.Steinberger, sc.UnscoredDimensions)
// Competitor comparison from research.json
sc.CompetitorScores = buildCompetitorScores(sc.Steinberger.Total, pipelineDir)
@@ -168,6 +195,15 @@ func RunScorecard(outputDir, pipelineDir, specPath string, verifyReport *VerifyR
return sc, nil
}
+func (sc *Scorecard) IsDimensionUnscored(name string) bool {
+ for _, dimension := range sc.UnscoredDimensions {
+ if dimension == name {
+ return true
+ }
+ }
+ return false
+}
+
func scoreOutputModes(dir string) int {
rootContent := readFileContent(filepath.Join(dir, "internal", "cli", "root.go"))
helpersContent := readFileContent(filepath.Join(dir, "internal", "cli", "helpers.go"))
@@ -820,40 +856,126 @@ func scoreInsight(dir string) int {
}
}
+type openAPISecurityScheme struct {
+ Key string
+ Type string
+ Scheme string
+ In string
+ HeaderName string
+}
+
+type securityRequirementSet struct {
+ Alternatives [][]string
+ AllowsAnonymous bool
+}
+
type openAPISpecInfo struct {
- Paths []string
- Auth apispec.AuthConfig
+ Paths []string
+ SecuritySchemes map[string]openAPISecurityScheme
+ SecurityRequirements []securityRequirementSet
}
-func loadOpenAPISpec(specPath string) *openAPISpecInfo {
+func loadOpenAPISpec(specPath string) (*openAPISpecInfo, error) {
if specPath == "" {
- return nil
+ return nil, nil
}
- summary, err := loadSpecSummary(specPath)
- if err != nil || summary == nil {
- return nil
+ data, err := os.ReadFile(specPath)
+ if err != nil {
+ return nil, fmt.Errorf("reading spec: %w", err)
}
- return &openAPISpecInfo{
- Paths: summary.Paths,
- Auth: summary.Auth,
+
+ var raw map[string]any
+ if err := json.Unmarshal(data, &raw); err != nil {
+ return nil, fmt.Errorf("parsing spec JSON: %w", err)
}
-}
-func scorePathValidity(dir, specPath string) int {
- if specPath == "" {
- return 5
+ info := &openAPISpecInfo{
+ SecuritySchemes: make(map[string]openAPISecurityScheme),
+ }
+ if paths, ok := raw["paths"].(map[string]any); ok {
+ for path := range paths {
+ info.Paths = append(info.Paths, path)
+ }
+ slices.Sort(info.Paths)
+ }
+
+ if components, ok := raw["components"].(map[string]any); ok {
+ if securitySchemes, ok := components["securitySchemes"].(map[string]any); ok {
+ for schemeName, value := range securitySchemes {
+ scheme := openAPISecurityScheme{Key: schemeName}
+ if fields, ok := value.(map[string]any); ok {
+ scheme.Type = strings.ToLower(asString(fields["type"]))
+ scheme.Scheme = strings.ToLower(asString(fields["scheme"]))
+ scheme.In = strings.ToLower(asString(fields["in"]))
+ scheme.HeaderName = asString(fields["name"])
+ }
+ info.SecuritySchemes[schemeName] = scheme
+ }
+ }
+ }
+
+ rootSecurity, rootHasSecurity := parseSecurityRequirementSet(raw["security"])
+ foundOperation := false
+ if paths, ok := raw["paths"].(map[string]any); ok {
+ for _, pathValue := range paths {
+ pathItem, ok := pathValue.(map[string]any)
+ if !ok {
+ continue
+ }
+ for method, operationValue := range pathItem {
+ if !isHTTPMethod(method) {
+ continue
+ }
+ foundOperation = true
+ operation, ok := operationValue.(map[string]any)
+ if !ok {
+ continue
+ }
+ if requirementSet, ok := parseSecurityRequirementSet(operation["security"]); ok {
+ info.SecurityRequirements = append(info.SecurityRequirements, requirementSet)
+ continue
+ }
+ if rootHasSecurity {
+ info.SecurityRequirements = append(info.SecurityRequirements, rootSecurity)
+ }
+ }
+ }
+ }
+ if !foundOperation && rootHasSecurity {
+ info.SecurityRequirements = append(info.SecurityRequirements, rootSecurity)
}
- spec := loadOpenAPISpec(specPath)
- if spec == nil || len(spec.Paths) == 0 {
- return 5
+ for _, requirementSet := range info.SecurityRequirements {
+ for _, alternative := range requirementSet.Alternatives {
+ for _, name := range alternative {
+ if _, ok := info.SecuritySchemes[name]; !ok {
+ return nil, fmt.Errorf("spec references undefined security scheme %q", name)
+ }
+ }
+ }
+ }
+
+ return info, nil
+}
+
+type dimensionScore struct {
+ score int
+ scored bool
+}
+
+func evaluatePathValidity(dir string, spec *openAPISpecInfo) dimensionScore {
+ if spec == nil {
+ return dimensionScore{}
+ }
+ if len(spec.Paths) == 0 {
+ return dimensionScore{}
}
pathRe := regexp.MustCompile(`\bpath\s*(?::=|=|:)\s*"([^"]+)"`)
cmdFiles := sampleCommandFiles(dir, 10)
if len(cmdFiles) == 0 {
- return 0
+ return dimensionScore{scored: true}
}
total := 0
@@ -870,69 +992,190 @@ func scorePathValidity(dir, specPath string) int {
}
if total == 0 {
- return 0
+ return dimensionScore{scored: true}
}
- return (matches * 10) / total
-}
-
-func scoreAuthProtocol(dir, specPath string) int {
- if specPath == "" {
- return 5
+ return dimensionScore{
+ score: (matches * 10) / total,
+ scored: true,
}
+}
- spec := loadOpenAPISpec(specPath)
+func evaluateAuthProtocol(dir string, spec *openAPISpecInfo) dimensionScore {
if spec == nil {
- return 5
+ return dimensionScore{}
+ }
+ if len(spec.SecurityRequirements) == 0 {
+ return dimensionScore{}
}
clientContent := readFileContent(filepath.Join(dir, "internal", "client", "client.go"))
configContent := readFileContent(filepath.Join(dir, "internal", "config", "config.go"))
if clientContent == "" {
- return 0
+ return dimensionScore{scored: true}
}
- score := 0
+ totalScore := 0
+ scoredSets := 0
+ for _, requirementSet := range spec.SecurityRequirements {
+ if requirementSet.AllowsAnonymous {
+ continue
+ }
+
+ bestScore := -1
+ scoreable := false
+ for _, alternative := range requirementSet.Alternatives {
+ score, ok := scoreAuthAlternative(clientContent, configContent, spec.SecuritySchemes, alternative)
+ if !ok {
+ continue
+ }
+ scoreable = true
+ if score > bestScore {
+ bestScore = score
+ }
+ }
+ if !scoreable {
+ continue
+ }
+
+ totalScore += bestScore
+ scoredSets++
+ }
+ if scoredSets == 0 {
+ return dimensionScore{}
+ }
+ return dimensionScore{
+ score: totalScore / scoredSets,
+ scored: true,
+ }
+}
+
+func parseSecurityRequirementSet(value any) (securityRequirementSet, bool) {
+ requirements, ok := value.([]any)
+ if !ok {
+ return securityRequirementSet{}, false
+ }
+
+ set := securityRequirementSet{}
+ if len(requirements) == 0 {
+ set.AllowsAnonymous = true
+ return set, true
+ }
+
+ seenAlternatives := make(map[string]struct{})
+ for _, requirement := range requirements {
+ names, ok := requirement.(map[string]any)
+ if !ok {
+ continue
+ }
+ if len(names) == 0 {
+ set.AllowsAnonymous = true
+ continue
+ }
+
+ var alternative []string
+ for name := range names {
+ if strings.TrimSpace(name) == "" {
+ continue
+ }
+ alternative = append(alternative, name)
+ }
+ if len(alternative) == 0 {
+ set.AllowsAnonymous = true
+ continue
+ }
+ slices.Sort(alternative)
+ key := strings.Join(alternative, "\x00")
+ if _, ok := seenAlternatives[key]; ok {
+ continue
+ }
+ seenAlternatives[key] = struct{}{}
+ set.Alternatives = append(set.Alternatives, alternative)
+ }
+
+ return set, true
+}
+
+func scoreAuthAlternative(clientContent, configContent string, schemes map[string]openAPISecurityScheme, alternative []string) (int, bool) {
+ if len(alternative) == 0 {
+ return 0, false
+ }
+
+ total := 0
+ scoreableSchemes := 0
+ for _, key := range alternative {
+ scheme, ok := schemes[key]
+ if !ok {
+ continue
+ }
+ score, scoreable := scoreAuthScheme(clientContent, configContent, scheme)
+ if !scoreable {
+ continue
+ }
+ total += score
+ scoreableSchemes++
+ }
+ if scoreableSchemes == 0 {
+ return 0, false
+ }
+ return total / scoreableSchemes, true
+}
+
+func scoreAuthScheme(clientContent, configContent string, scheme openAPISecurityScheme) (int, bool) {
+ nameLower := strings.ToLower(scheme.Key)
+ headerName := "Authorization"
authHeaderMatched := false
headerNameMatched := false
queryMatched := false
envMatched := false
+ scoreable := false
+
+ if strings.EqualFold(scheme.Type, "apikey") && scheme.In == "header" && strings.TrimSpace(scheme.HeaderName) != "" {
+ headerName = scheme.HeaderName
+ }
switch {
- case strings.Contains(strings.ToLower(spec.Auth.Format), "bot "):
+ case strings.Contains(nameLower, "bot"):
+ scoreable = true
if strings.Contains(clientContent, `"Bot "`) || strings.Contains(clientContent, "`Bot `") {
authHeaderMatched = true
}
- case strings.EqualFold(spec.Auth.Type, "bearer_token"):
+ case strings.Contains(nameLower, "bearer") || (scheme.Type == "http" && scheme.Scheme == "bearer"):
+ scoreable = true
if strings.Contains(clientContent, `"Bearer "`) || strings.Contains(clientContent, "`Bearer `") {
authHeaderMatched = true
}
- case strings.Contains(strings.ToLower(spec.Auth.Format), "basic "):
+ case strings.Contains(nameLower, "basic") || (scheme.Type == "http" && scheme.Scheme == "basic"):
+ scoreable = true
if strings.Contains(clientContent, `"Basic "`) || strings.Contains(clientContent, "`Basic `") {
authHeaderMatched = true
}
+ case strings.EqualFold(scheme.Type, "apikey"):
+ scoreable = true
+ case strings.EqualFold(scheme.Type, "oauth2"), strings.EqualFold(scheme.Type, "openidconnect"):
+ scoreable = true
+ if strings.Contains(clientContent, `"Bearer "`) || strings.Contains(clientContent, "`Bearer `") {
+ authHeaderMatched = true
+ }
}
-
- headerName := spec.Auth.Header
- if headerName == "" {
- headerName = "Authorization"
+ if !scoreable {
+ return 0, false
}
+
if strings.Contains(clientContent, `Header.Set("`+headerName+`"`) ||
strings.Contains(clientContent, `Header.Add("`+headerName+`"`) {
headerNameMatched = true
}
- if strings.EqualFold(spec.Auth.In, "query") &&
- (strings.Contains(clientContent, ".Query()") || strings.Contains(clientContent, "url.Values") || strings.Contains(clientContent, "RawQuery")) {
+ if scheme.In == "query" && (strings.Contains(clientContent, ".Query()") || strings.Contains(clientContent, "url.Values") || strings.Contains(clientContent, "RawQuery")) {
queryMatched = true
}
- for _, envVar := range spec.Auth.EnvVars {
- if strings.Contains(strings.ToUpper(configContent), strings.ToUpper(envVar)) {
- envMatched = true
- break
- }
+ envNeedle := sanitizeEnvName(scheme.Key)
+ if envNeedle != "" && strings.Contains(strings.ToUpper(configContent), envNeedle) {
+ envMatched = true
}
+ score := 0
if authHeaderMatched {
score += 3
}
@@ -948,7 +1191,16 @@ func scoreAuthProtocol(dir, specPath string) int {
if score > 10 {
score = 10
}
- return score
+ return score, true
+}
+
+func isHTTPMethod(method string) bool {
+ switch strings.ToLower(method) {
+ case "get", "put", "post", "delete", "options", "head", "patch", "trace":
+ return true
+ default:
+ return false
+ }
}
func scoreDataPipelineIntegrity(dir string) int {
@@ -1388,8 +1640,12 @@ func computeGrade(percentage int) string {
}
}
-func buildGapReport(s SteinerScore) []string {
+func buildGapReport(s SteinerScore, unscored []string) []string {
var gaps []string
+ unscoredSet := make(map[string]struct{}, len(unscored))
+ for _, name := range unscored {
+ unscoredSet[name] = struct{}{}
+ }
dimensions := []struct {
name string
score int
@@ -1414,6 +1670,9 @@ func buildGapReport(s SteinerScore) []string {
{"dead_code", s.DeadCode},
}
for _, d := range dimensions {
+ if _, skip := unscoredSet[d.name]; skip {
+ continue
+ }
max := 10
if d.name == "type_fidelity" || d.name == "dead_code" {
max = 5
@@ -1472,6 +1731,9 @@ func writeScorecardMD(sc *Scorecard, pipelineDir string) error {
var b strings.Builder
b.WriteString(fmt.Sprintf("# Scorecard: %s\n\n", sc.APIName))
b.WriteString(fmt.Sprintf("**Overall Grade: %s** (%d%%)\n\n", sc.OverallGrade, sc.Steinberger.Percentage))
+ if len(sc.UnscoredDimensions) > 0 {
+ b.WriteString(fmt.Sprintf("Unscored dimensions omitted from the total denominator: %s\n\n", strings.Join(sc.UnscoredDimensions, ", ")))
+ }
// Steinberger dimensions table
b.WriteString("## Quality Dimensions\n\n")
@@ -1479,27 +1741,32 @@ func writeScorecardMD(sc *Scorecard, pipelineDir string) error {
b.WriteString("|-----------|-------|\n")
s := sc.Steinberger
dimensions := []struct {
- name string
- score int
+ name string
+ nameKey string
+ score int
}{
- {"Output Modes", s.OutputModes},
- {"Auth", s.Auth},
- {"Error Handling", s.ErrorHandling},
- {"Terminal UX", s.TerminalUX},
- {"README", s.README},
- {"Doctor", s.Doctor},
- {"Agent Native", s.AgentNative},
- {"Local Cache", s.LocalCache},
- {"Breadth", s.Breadth},
- {"Vision", s.Vision},
- {"Workflows", s.Workflows},
- {"Insight", s.Insight},
- {"Path Validity", s.PathValidity},
- {"Auth Protocol", s.AuthProtocol},
- {"Data Pipeline Integrity", s.DataPipelineIntegrity},
- {"Sync Correctness", s.SyncCorrectness},
+ {"Output Modes", "output_modes", s.OutputModes},
+ {"Auth", "auth", s.Auth},
+ {"Error Handling", "error_handling", s.ErrorHandling},
+ {"Terminal UX", "terminal_ux", s.TerminalUX},
+ {"README", "readme", s.README},
+ {"Doctor", "doctor", s.Doctor},
+ {"Agent Native", "agent_native", s.AgentNative},
+ {"Local Cache", "local_cache", s.LocalCache},
+ {"Breadth", "breadth", s.Breadth},
+ {"Vision", "vision", s.Vision},
+ {"Workflows", "workflows", s.Workflows},
+ {"Insight", "insight", s.Insight},
+ {"Path Validity", "path_validity", s.PathValidity},
+ {"Auth Protocol", "auth_protocol", s.AuthProtocol},
+ {"Data Pipeline Integrity", "data_pipeline_integrity", s.DataPipelineIntegrity},
+ {"Sync Correctness", "sync_correctness", s.SyncCorrectness},
}
for _, d := range dimensions {
+ if sc.IsDimensionUnscored(d.nameKey) {
+ b.WriteString(fmt.Sprintf("| %s | N/A |\n", d.name))
+ continue
+ }
bar := strings.Repeat("#", d.score) + strings.Repeat(".", 10-d.score)
b.WriteString(fmt.Sprintf("| %s | %d/10 %s |\n", d.name, d.score, bar))
}
diff --git a/internal/pipeline/scorecard_tier2_test.go b/internal/pipeline/scorecard_tier2_test.go
index abb5ec56..2a388a56 100644
--- a/internal/pipeline/scorecard_tier2_test.go
+++ b/internal/pipeline/scorecard_tier2_test.go
@@ -1,8 +1,10 @@
package pipeline
import (
+ "encoding/json"
"os"
"path/filepath"
+ "strings"
"testing"
"github.com/stretchr/testify/assert"
@@ -228,7 +230,9 @@ func runLinks() string {
}
}`)
- assert.Equal(t, 10, scorePathValidity(dir, specPath))
+ spec, err := loadOpenAPISpec(specPath)
+ assert.NoError(t, err)
+ assert.Equal(t, 10, evaluatePathValidity(dir, spec).score)
})
}
@@ -495,6 +499,354 @@ CREATE TABLE bookings (
})
}
+func TestRunScorecard_UnscoredSpecDimensions(t *testing.T) {
+ t.Run("no spec omits path and auth dimensions from scoring", func(t *testing.T) {
+ dir := t.TempDir()
+ writeScorecardFixture(t, dir, "internal/cli/links.go", `
+package cli
+
+func runLinks() string {
+ path := "/links"
+ return path
+}
+`)
+
+ pipelineDir := t.TempDir()
+ sc, err := RunScorecard(dir, pipelineDir, "", nil)
+ assert.NoError(t, err)
+ assert.ElementsMatch(t, []string{"path_validity", "auth_protocol"}, sc.UnscoredDimensions)
+ assert.NotContains(t, sc.GapReport, "path_validity scored 0/10 - needs improvement")
+ assert.NotContains(t, sc.GapReport, "auth_protocol scored 0/10 - needs improvement")
+ })
+
+ t.Run("missing security schemes renormalizes tier2 instead of treating auth as zero", func(t *testing.T) {
+ dir := t.TempDir()
+ writeScorecardFixture(t, dir, "internal/cli/links.go", `
+package cli
+
+func runLinks() string {
+ path := "/links"
+ return path
+}
+`)
+
+ specWithoutAuth := filepath.Join(dir, "spec-no-auth.json")
+ writeScorecardFixture(t, dir, "spec-no-auth.json", `{
+ "paths": {
+ "/links": {}
+ },
+ "components": {
+ "securitySchemes": {}
+ }
+}`)
+
+ specWithBearer := filepath.Join(dir, "spec-bearer.json")
+ writeScorecardFixture(t, dir, "spec-bearer.json", `{
+ "paths": {
+ "/links": {}
+ },
+ "security": [
+ {
+ "bearerAuth": []
+ }
+ ],
+ "components": {
+ "securitySchemes": {
+ "bearerAuth": {
+ "type": "http",
+ "scheme": "bearer"
+ }
+ }
+ }
+}`)
+
+ pipelineNoAuth := t.TempDir()
+ scNoAuth, err := RunScorecard(dir, pipelineNoAuth, specWithoutAuth, nil)
+ assert.NoError(t, err)
+ assert.Contains(t, scNoAuth.UnscoredDimensions, "auth_protocol")
+
+ pipelineBearer := t.TempDir()
+ scBearer, err := RunScorecard(dir, pipelineBearer, specWithBearer, nil)
+ assert.NoError(t, err)
+ assert.NotContains(t, scBearer.UnscoredDimensions, "auth_protocol")
+
+ assert.Equal(t, scBearer.Steinberger.PathValidity, scNoAuth.Steinberger.PathValidity)
+ assert.Equal(t, scBearer.Steinberger.AuthProtocol, 0)
+ sharedTier2Raw := scBearer.Steinberger.PathValidity +
+ scBearer.Steinberger.DataPipelineIntegrity +
+ scBearer.Steinberger.SyncCorrectness +
+ scBearer.Steinberger.TypeFidelity +
+ scBearer.Steinberger.DeadCode
+ expectedDelta := (sharedTier2Raw * 50 / 40) - (sharedTier2Raw * 50 / 50)
+ assert.Equal(t, scBearer.Steinberger.Total+expectedDelta, scNoAuth.Steinberger.Total)
+ })
+
+ t.Run("unused declared security schemes leave auth unscored", func(t *testing.T) {
+ dir := t.TempDir()
+ writeScorecardFixture(t, dir, "internal/client/client.go", `
+package client
+
+func setAuth(req interface{ Header() map[string]string }) {}
+`)
+
+ specPath := filepath.Join(dir, "spec-unused-auth.json")
+ writeScorecardFixture(t, dir, "spec-unused-auth.json", `{
+ "paths": {
+ "/links": {
+ "get": {
+ "responses": {
+ "200": { "description": "ok" }
+ }
+ }
+ }
+ },
+ "components": {
+ "securitySchemes": {
+ "bearerAuth": {
+ "type": "http",
+ "scheme": "bearer"
+ }
+ }
+ }
+}`)
+
+ pipelineDir := t.TempDir()
+ sc, err := RunScorecard(dir, pipelineDir, specPath, nil)
+ assert.NoError(t, err)
+ assert.Contains(t, sc.UnscoredDimensions, "auth_protocol")
+ })
+
+ t.Run("referenced oauth2 scheme remains scoreable", func(t *testing.T) {
+ dir := t.TempDir()
+ writeScorecardFixture(t, dir, "internal/client/client.go", `
+package client
+
+type request struct {
+ Header map[string]string
+}
+
+func setAuth(req *request, token string) {
+ req.Header = map[string]string{}
+ req.Header["Authorization"] = "Bearer " + token
+}
+`)
+
+ specPath := filepath.Join(dir, "spec-oauth.json")
+ writeScorecardFixture(t, dir, "spec-oauth.json", `{
+ "paths": {
+ "/links": {
+ "get": {
+ "security": [
+ {
+ "oauth": []
+ }
+ ],
+ "responses": {
+ "200": { "description": "ok" }
+ }
+ }
+ }
+ },
+ "components": {
+ "securitySchemes": {
+ "oauth": {
+ "type": "oauth2"
+ }
+ }
+ }
+}`)
+
+ pipelineDir := t.TempDir()
+ sc, err := RunScorecard(dir, pipelineDir, specPath, nil)
+ assert.NoError(t, err)
+ assert.NotContains(t, sc.UnscoredDimensions, "auth_protocol")
+ assert.Greater(t, sc.Steinberger.AuthProtocol, 0)
+ })
+
+ t.Run("anonymous alternative leaves auth unscored", func(t *testing.T) {
+ dir := t.TempDir()
+ writeScorecardFixture(t, dir, "internal/client/client.go", `
+package client
+
+type request struct {
+ Header map[string]string
+}
+
+func setAuth(req *request, token string) {
+ req.Header = map[string]string{}
+ req.Header["Authorization"] = "Bearer " + token
+}
+`)
+
+ specPath := filepath.Join(dir, "spec-optional-auth.json")
+ writeScorecardFixture(t, dir, "spec-optional-auth.json", `{
+ "paths": {
+ "/links": {
+ "get": {
+ "security": [
+ {},
+ {
+ "oauth": []
+ }
+ ],
+ "responses": {
+ "200": { "description": "ok" }
+ }
+ }
+ }
+ },
+ "components": {
+ "securitySchemes": {
+ "oauth": {
+ "type": "oauth2"
+ }
+ }
+ }
+}`)
+
+ pipelineDir := t.TempDir()
+ sc, err := RunScorecard(dir, pipelineDir, specPath, nil)
+ assert.NoError(t, err)
+ assert.Contains(t, sc.UnscoredDimensions, "auth_protocol")
+ })
+
+ t.Run("alternative auth schemes use best matching option", func(t *testing.T) {
+ dir := t.TempDir()
+ writeScorecardFixture(t, dir, "internal/client/client.go", `
+package client
+
+import "net/http"
+
+func setAuth(req *http.Request, token string) {
+ req.Header.Set("Authorization", "Bearer "+token)
+}
+`)
+
+ specPath := filepath.Join(dir, "spec-auth-alternatives.json")
+ writeScorecardFixture(t, dir, "spec-auth-alternatives.json", `{
+ "paths": {
+ "/links": {
+ "get": {
+ "security": [
+ {
+ "api_key": []
+ },
+ {
+ "oauth": []
+ }
+ ],
+ "responses": {
+ "200": { "description": "ok" }
+ }
+ }
+ }
+ },
+ "components": {
+ "securitySchemes": {
+ "api_key": {
+ "type": "apiKey",
+ "in": "header",
+ "name": "X-API-Key"
+ },
+ "oauth": {
+ "type": "oauth2"
+ }
+ }
+ }
+}`)
+
+ pipelineDir := t.TempDir()
+ sc, err := RunScorecard(dir, pipelineDir, specPath, nil)
+ assert.NoError(t, err)
+ assert.NotContains(t, sc.UnscoredDimensions, "auth_protocol")
+ assert.GreaterOrEqual(t, sc.Steinberger.AuthProtocol, 3)
+ })
+
+ t.Run("operation security override can make inherited auth unscored", func(t *testing.T) {
+ dir := t.TempDir()
+ writeScorecardFixture(t, dir, "internal/client/client.go", `
+package client
+
+import "net/http"
+
+func setAuth(req *http.Request, token string) {
+ req.Header.Set("Authorization", "Bearer "+token)
+}
+`)
+
+ specPath := filepath.Join(dir, "spec-root-auth-operation-anon.json")
+ writeScorecardFixture(t, dir, "spec-root-auth-operation-anon.json", `{
+ "security": [
+ {
+ "oauth": []
+ }
+ ],
+ "paths": {
+ "/links": {
+ "get": {
+ "security": [],
+ "responses": {
+ "200": { "description": "ok" }
+ }
+ },
+ "post": {
+ "security": [
+ {}
+ ],
+ "responses": {
+ "200": { "description": "ok" }
+ }
+ }
+ }
+ },
+ "components": {
+ "securitySchemes": {
+ "oauth": {
+ "type": "oauth2"
+ }
+ }
+ }
+}`)
+
+ pipelineDir := t.TempDir()
+ sc, err := RunScorecard(dir, pipelineDir, specPath, nil)
+ assert.NoError(t, err)
+ assert.Contains(t, sc.UnscoredDimensions, "auth_protocol")
+ })
+
+ t.Run("invalid spec path returns an error instead of renormalizing", func(t *testing.T) {
+ dir := t.TempDir()
+ pipelineDir := t.TempDir()
+
+ _, err := RunScorecard(dir, pipelineDir, filepath.Join(dir, "missing-spec.json"), nil)
+ assert.Error(t, err)
+ assert.Contains(t, err.Error(), "reading spec")
+ })
+
+ t.Run("json output keeps numeric fields for backward compatibility", func(t *testing.T) {
+ dir := t.TempDir()
+ writeScorecardFixture(t, dir, "internal/cli/links.go", `
+package cli
+
+func runLinks() string {
+ path := "/links"
+ return path
+}
+`)
+
+ pipelineDir := t.TempDir()
+ sc, err := RunScorecard(dir, pipelineDir, "", nil)
+ assert.NoError(t, err)
+
+ data, err := json.Marshal(sc)
+ assert.NoError(t, err)
+ body := string(data)
+ assert.True(t, strings.Contains(body, `"path_validity":0`))
+ assert.True(t, strings.Contains(body, `"auth_protocol":0`))
+ assert.True(t, strings.Contains(body, `"unscored_dimensions":["path_validity","auth_protocol"]`))
+ })
+}
+
func TestScoreWorkflows(t *testing.T) {
t.Run("counts files matching expanded prefixes", func(t *testing.T) {
dir := t.TempDir()
diff --git a/skills/printing-press-score/SKILL.md b/skills/printing-press-score/SKILL.md
index 20cca286..fc9040ac 100644
--- a/skills/printing-press-score/SKILL.md
+++ b/skills/printing-press-score/SKILL.md
@@ -79,7 +79,7 @@ For each resolved CLI directory, find the OpenAPI spec:
1. Check `<cli-dir>/spec.json` — the pipeline converts YAML specs to JSON during generation
2. If not found, scan `docs/plans/*-pipeline/state.json` files for one matching this CLI's directory. Read its `spec_path` field. If that file exists on disk, use it.
-3. If no spec found, **proceed without `--spec`**. Note to the user: "No spec found — Tier 2 (domain correctness) scores will be 0. Provide a spec path to get full scoring."
+3. If no spec found, **proceed without `--spec`**. Note to the user: "No spec found — spec-derived dimensions will be marked N/A and omitted from the denominator. Provide a spec path for full scoring."
## Step 4: Build the Binary
@@ -121,8 +121,8 @@ Parse the JSON output. The structure is:
"vision": 6,
"workflows": 3,
"insight": 5,
- "path_validity": 9,
- "auth_protocol": 8,
+ "path_validity": 0,
+ "auth_protocol": 0,
"data_pipeline_integrity": 7,
"sync_correctness": 6,
"type_fidelity": 4,
@@ -131,10 +131,13 @@ Parse the JSON output. The structure is:
"percentage": 72
},
"overall_grade": "B",
- "gap_report": ["..."]
+ "gap_report": ["..."],
+ "unscored_dimensions": ["path_validity", "auth_protocol"]
}
```
+If `unscored_dimensions` is present, those dimensions should be rendered as `N/A`, not `0/x`, and should be described as omitted from the denominator rather than as fixable CLI defects. For backward compatibility, JSON still encodes the numeric fields as `0`; consumers must use `unscored_dimensions` to distinguish `N/A` from a real zero.
+
### Compare Mode
Run **both** scorecard commands in **parallel** using two simultaneous Bash tool calls:
@@ -195,10 +198,10 @@ Gaps:
- <gap 2>
```
-If Tier 2 was skipped (no spec), add a note after the table:
+If `unscored_dimensions` is non-empty, add a note after the table:
```
-Note: Tier 2 scores are 0 — no OpenAPI spec was found. Run with a spec path for full scoring.
+Note: path_validity, auth_protocol were unscored and omitted from the denominator. Provide a spec path for full scoring.
```
### Compare Table
← 6727b860 feat(skill): add 7-principle agent build checklist and Prior
·
back to Cli Printing Press
·
feat(pipeline): move mutable runs into scoped runstate (#30) 4120dfcb →