← back to Cli Printing Press
fix(scorer): live-check treats CLI's graceful empty-handling as PASS, not FAIL (#488)
bd102cf047bb542288d26e019dde96fc20893b6b · 2026-05-01 21:35:59 -0700 · Trevin Chow
* fix(scorer): live-check treats CLI's graceful empty-handling as PASS, not FAIL (#484)
When research.json's LLM-authored novel_features[].example field uses a
placeholder slug (my-launch-slug) or one whose meaning has rotated on a
content-rotating API (Product Hunt's "notion" decaying from a current
launch to the 2014 Notion entry), the live-check sampler invokes the CLI
with that arg verbatim. A well-behaved CLI prints
"Error: <resource> not found: <input>" to stderr and exits non-zero —
the bedrock contract for "API correctly returned no record." The sampler
counted this as a CLI failure.
producthunt's first live-check showed Passed 7/12 (58%) entirely from
this misclassification: 3 of the 5 failures were CLIs reporting "post
not found: <slug>", which is the right behavior. The headline scorecard
wasn't dragged (the score caps at Passed >= 3), but the Failed count
misled reviewers into thinking the CLI was broken.
Add isGracefulEmptyResponse: when exit is non-zero, treat as PASS only
when stderr both:
1. Contains one of {not found, no results, no match, no record,
no such} — the CLI is reporting an empty-result outcome, not
a generic failure.
2. Echoes one of the user's positional args (≥ 3 chars) — filters
out generic failures like "config file not found" or
"authentication required" that say "not found" but don't reference
user input. The arg-echo is the key boundary.
Real CLI bugs whose stderr happens to use a graceful phrase but doesn't
echo input still fail. The original TestLiveCheck_FailOnExitError ("exit
5; something went wrong") still fails, as expected.
Coverage:
- TestLiveCheck_PassOnGracefulEmpty: each of 5 graceful phrases with
arg echo through the full RunLiveCheck path → Status=Pass with
reason "graceful empty: ..."
- TestLiveCheck_FailOnGenericExitErrorWithoutArgEcho: phrase-without-
echo, no-phrase-no-echo, auth-failure-with-phrase-no-echo all still
count as Failed. Preserves the original fail-on-exit contract for
non-graceful cases.
- TestIsGracefulEmptyResponse: 11 table-driven cases covering exit
code, phrase presence, arg echo, short-token threshold, flag-only
args (no positional candidates), case-insensitivity, and the
flag-value-as-positional case (--query notion → "notion").
Out of scope: the other 2 of producthunt's 5 failures hit the
outputMentionsQuery check (lookalike features by design return
DIFFERENT items than the query). That's a separate failure mode needing
research.json schema (placeholder: true) or per-feature opt-out.
Tracked as a follow-up; the issue's H1 root cause closes here.
Closes #484 (H1).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(scorer): simplify live-check graceful-empty after review
Three-agent review pass (reuse / quality / efficiency) on 48b361ab
surfaced four cleanup opportunities, all applied here:
1. Drop the dead `if exitCode == 0 { return false }` guard in
isGracefulEmptyResponse. The only caller (live_check.go:340)
invokes it inside `errors.As(runErr, &exitErr)` after a non-nil
`runErr`, so exit code is guaranteed non-zero. Removing it makes
the contract explicit: "called only on non-zero exits."
2. Replace the matched-flag loop with an early-return + a small
containsAnyOf helper. The flag pattern was a leftover from
incremental drafting; the helper reads as two flat guards.
3. Inline positionalCandidates into the arg loop. The function was
used at exactly one site, allocated an intermediate slice, and
added one pass over args. The flag-skip is two lines inline.
4. Collapse the two end-to-end tests from table-driven to one
canonical case each. TestIsGracefulEmptyResponse exercises the
helper directly with 10 boundary cases; the integration tests
only need to prove RunLiveCheck routes through the new branch.
Saves 6 subprocess spawns per test run while preserving full
branch coverage.
Also trim two redundant doc-comment paragraphs ("the second condition
is the key boundary" duplicates bullet #2; "Defensive: returns false
on exit 0" became moot once the guard was dropped).
Naming note: chose containsAnyOf rather than reusing
internal/pipeline/dogfood.go's containsAny. The existing helper has
the inverse signature (many sources, one needle) and renaming it
would touch unrelated code; the "AnyOf" suffix signals direction.
Net: -59 LoC vs the original commit. All tests still pass including
TestLiveCheck_FailOnExitError (regression check).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Files touched
M internal/pipeline/live_check.goM internal/pipeline/live_check_test.go
Diff
commit bd102cf047bb542288d26e019dde96fc20893b6b
Author: Trevin Chow <trevin@trevinchow.com>
Date: Fri May 1 21:35:59 2026 -0700
fix(scorer): live-check treats CLI's graceful empty-handling as PASS, not FAIL (#488)
* fix(scorer): live-check treats CLI's graceful empty-handling as PASS, not FAIL (#484)
When research.json's LLM-authored novel_features[].example field uses a
placeholder slug (my-launch-slug) or one whose meaning has rotated on a
content-rotating API (Product Hunt's "notion" decaying from a current
launch to the 2014 Notion entry), the live-check sampler invokes the CLI
with that arg verbatim. A well-behaved CLI prints
"Error: <resource> not found: <input>" to stderr and exits non-zero —
the bedrock contract for "API correctly returned no record." The sampler
counted this as a CLI failure.
producthunt's first live-check showed Passed 7/12 (58%) entirely from
this misclassification: 3 of the 5 failures were CLIs reporting "post
not found: <slug>", which is the right behavior. The headline scorecard
wasn't dragged (the score caps at Passed >= 3), but the Failed count
misled reviewers into thinking the CLI was broken.
Add isGracefulEmptyResponse: when exit is non-zero, treat as PASS only
when stderr both:
1. Contains one of {not found, no results, no match, no record,
no such} — the CLI is reporting an empty-result outcome, not
a generic failure.
2. Echoes one of the user's positional args (≥ 3 chars) — filters
out generic failures like "config file not found" or
"authentication required" that say "not found" but don't reference
user input. The arg-echo is the key boundary.
Real CLI bugs whose stderr happens to use a graceful phrase but doesn't
echo input still fail. The original TestLiveCheck_FailOnExitError ("exit
5; something went wrong") still fails, as expected.
Coverage:
- TestLiveCheck_PassOnGracefulEmpty: each of 5 graceful phrases with
arg echo through the full RunLiveCheck path → Status=Pass with
reason "graceful empty: ..."
- TestLiveCheck_FailOnGenericExitErrorWithoutArgEcho: phrase-without-
echo, no-phrase-no-echo, auth-failure-with-phrase-no-echo all still
count as Failed. Preserves the original fail-on-exit contract for
non-graceful cases.
- TestIsGracefulEmptyResponse: 11 table-driven cases covering exit
code, phrase presence, arg echo, short-token threshold, flag-only
args (no positional candidates), case-insensitivity, and the
flag-value-as-positional case (--query notion → "notion").
Out of scope: the other 2 of producthunt's 5 failures hit the
outputMentionsQuery check (lookalike features by design return
DIFFERENT items than the query). That's a separate failure mode needing
research.json schema (placeholder: true) or per-feature opt-out.
Tracked as a follow-up; the issue's H1 root cause closes here.
Closes #484 (H1).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(scorer): simplify live-check graceful-empty after review
Three-agent review pass (reuse / quality / efficiency) on 48b361ab
surfaced four cleanup opportunities, all applied here:
1. Drop the dead `if exitCode == 0 { return false }` guard in
isGracefulEmptyResponse. The only caller (live_check.go:340)
invokes it inside `errors.As(runErr, &exitErr)` after a non-nil
`runErr`, so exit code is guaranteed non-zero. Removing it makes
the contract explicit: "called only on non-zero exits."
2. Replace the matched-flag loop with an early-return + a small
containsAnyOf helper. The flag pattern was a leftover from
incremental drafting; the helper reads as two flat guards.
3. Inline positionalCandidates into the arg loop. The function was
used at exactly one site, allocated an intermediate slice, and
added one pass over args. The flag-skip is two lines inline.
4. Collapse the two end-to-end tests from table-driven to one
canonical case each. TestIsGracefulEmptyResponse exercises the
helper directly with 10 boundary cases; the integration tests
only need to prove RunLiveCheck routes through the new branch.
Saves 6 subprocess spawns per test run while preserving full
branch coverage.
Also trim two redundant doc-comment paragraphs ("the second condition
is the key boundary" duplicates bullet #2; "Defensive: returns false
on exit 0" became moot once the guard was dropped).
Naming note: chose containsAnyOf rather than reusing
internal/pipeline/dogfood.go's containsAny. The existing helper has
the inverse signature (many sources, one needle) and renaming it
would touch unrelated code; the "AnyOf" suffix signals direction.
Net: -59 LoC vs the original commit. All tests still pass including
TestLiveCheck_FailOnExitError (regression check).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
internal/pipeline/live_check.go | 76 ++++++++++++++++++++++++++++++++-
internal/pipeline/live_check_test.go | 83 ++++++++++++++++++++++++++++++++++++
2 files changed, 158 insertions(+), 1 deletion(-)
diff --git a/internal/pipeline/live_check.go b/internal/pipeline/live_check.go
index a1dab997..4ec6920c 100644
--- a/internal/pipeline/live_check.go
+++ b/internal/pipeline/live_check.go
@@ -338,7 +338,19 @@ func runOneFeatureCheck(cliDir, binaryPath string, f NovelFeature, timeout time.
if runErr != nil {
var exitErr *exec.ExitError
if errors.As(runErr, &exitErr) {
- return fail(fmt.Sprintf("exit %d: %s", exitErr.ExitCode(), trimOutput(stderrCap.String())))
+ stderr := stderrCap.String()
+ if isGracefulEmptyResponse(stderr, args) {
+ // CLI exited non-zero gracefully on "no record matches this
+ // input" — that's the CORRECT behavior for an unknown slug
+ // (research.json's LLM-authored example slugs decay over
+ // time on content-rotating APIs). Don't penalize a feature
+ // that handled the empty case well. See issue #484.
+ result.Status = StatusPass
+ result.Reason = "graceful empty: " + trimOutput(stderr)
+ result.OutputSample = sampleOutput(stderr)
+ return result
+ }
+ return fail(fmt.Sprintf("exit %d: %s", exitErr.ExitCode(), trimOutput(stderr)))
}
return fail("run error: " + runErr.Error())
}
@@ -615,6 +627,68 @@ func trimOutput(s string) string {
return s
}
+// gracefulEmptyPhrases is the closed vocabulary of stderr substrings that
+// signal "the CLI looked up the user's input and found no matching record."
+// Kept short and explicit to avoid false positives on generic error prose.
+// Lower-cased; matched case-insensitively against stderr.
+var gracefulEmptyPhrases = []string{
+ "not found",
+ "no results",
+ "no match",
+ "no record",
+ "no such",
+}
+
+// isGracefulEmptyResponse reports whether a non-zero exit + stderr indicates
+// the CLI gracefully handled an "input maps to no record" case rather than
+// hit an actual bug. Two conditions must both hold:
+//
+// 1. stderr contains one of gracefulEmptyPhrases.
+// 2. stderr echoes at least one of the user-supplied positional args
+// (≥ 3 chars). This is the key boundary — it filters out generic
+// failures like "config file not found" or "authentication required"
+// that happen to contain a graceful phrase but don't reference user
+// input. Flag values written as "--key value" count as positional
+// candidates because a CLI echoing a flag's value (e.g. --query notion
+// → "no match for query: notion") is still strong evidence that user
+// input was processed.
+//
+// Caller contract: invoke only on non-zero exits. See issue #484.
+func isGracefulEmptyResponse(stderr string, args []string) bool {
+ lower := strings.ToLower(stderr)
+ if !containsAnyOf(lower, gracefulEmptyPhrases) {
+ return false
+ }
+ for _, arg := range args {
+ if strings.HasPrefix(arg, "-") {
+ continue
+ }
+ a := strings.ToLower(arg)
+ if len(a) < 3 {
+ // Skip short tokens to avoid coincidental substring matches.
+ // Three chars is the same threshold outputMentionsQuery uses.
+ continue
+ }
+ if strings.Contains(lower, a) {
+ return true
+ }
+ }
+ return false
+}
+
+// containsAnyOf reports whether any of needles is a substring of s. The
+// "any-of" suffix distinguishes this from dogfood.go's containsAny, which
+// has the inverse signature ([]string sources, string needle). Caller is
+// expected to pass a pre-lowered s when matching case-insensitively.
+func containsAnyOf(s string, needles []string) bool {
+ for _, n := range needles {
+ if strings.Contains(s, n) {
+ return true
+ }
+ }
+ return false
+}
+
// InsightCapFromLiveCheck returns the maximum Insight score a CLI should
// receive given its live-check pass rate. A CLI whose flagships return
// broken output shouldn't earn a Grade A scorecard.
diff --git a/internal/pipeline/live_check_test.go b/internal/pipeline/live_check_test.go
index 723576b6..17d7b6f5 100644
--- a/internal/pipeline/live_check_test.go
+++ b/internal/pipeline/live_check_test.go
@@ -602,3 +602,86 @@ func TestDetectRawHTMLEntities_TruncatesLongMatchInMessage(t *testing.T) {
require.NotEmpty(t, msg)
require.LessOrEqual(t, len(msg), 200, "warning message should stay bounded regardless of match length")
}
+
+// TestLiveCheck_PassOnGracefulEmpty proves the wiring: when the CLI exits
+// non-zero with stderr that gracefully reports "no matching record" and
+// echoes the user's input, RunLiveCheck counts it as PASS rather than FAIL
+// (issue #484). One canonical case here; per-phrase coverage lives in
+// TestIsGracefulEmptyResponse, which exercises the helper directly.
+func TestLiveCheck_PassOnGracefulEmpty(t *testing.T) {
+ dir := t.TempDir()
+ writeStubBinary(t, dir, "stub",
+ `echo "Error: post not found: my-launch-slug" >&2; exit 1`)
+ writeTestResearchJSON(t, dir, []NovelFeature{
+ {Name: "Lookup", Command: "posts get",
+ Example: `stub posts get my-launch-slug --json`},
+ })
+ result := RunLiveCheck(LiveCheckOptions{
+ CLIDir: dir, BinaryName: "stub", Timeout: 5 * time.Second,
+ })
+ require.Equal(t, 1, result.Passed, "graceful empty should count as PASS")
+ require.Equal(t, 0, result.Failed, "graceful empty must not count as FAIL")
+ require.Contains(t, result.Features[0].Reason, "graceful empty",
+ "reason should label the graceful-empty branch so reviewers know why it passed")
+}
+
+// TestLiveCheck_FailOnGenericExitErrorWithoutArgEcho proves the negative
+// wiring: a non-zero exit whose stderr lacks the arg echo (a config error,
+// auth failure, or generic error) still counts as FAIL. One canonical case
+// here; the boundary cases live in TestIsGracefulEmptyResponse.
+func TestLiveCheck_FailOnGenericExitErrorWithoutArgEcho(t *testing.T) {
+ dir := t.TempDir()
+ writeStubBinary(t, dir, "stub", `echo "config file not found" >&2; exit 1`)
+ writeTestResearchJSON(t, dir, []NovelFeature{
+ {Name: "Lookup", Command: "posts get",
+ Example: `stub posts get my-launch-slug --json`},
+ })
+ result := RunLiveCheck(LiveCheckOptions{
+ CLIDir: dir, BinaryName: "stub", Timeout: 5 * time.Second,
+ })
+ require.Equal(t, 0, result.Passed, "phrase-without-arg-echo must NOT pass via graceful-empty")
+ require.Equal(t, 1, result.Failed)
+}
+
+// TestIsGracefulEmptyResponse exercises the helper directly across phrase
+// and arg-echo dimensions. Table-driven to keep the boundary explicit.
+// Caller contract guarantees non-zero exit, so no exit-code dimension here.
+func TestIsGracefulEmptyResponse(t *testing.T) {
+ cases := []struct {
+ name string
+ stderr string
+ args []string
+ want bool
+ }{
+ {"phrase + arg echo → graceful",
+ "Error: post not found: my-launch-slug",
+ []string{"posts", "get", "my-launch-slug"}, true},
+ {"phrase but no arg echo → not graceful (config error case)",
+ "config file not found",
+ []string{"posts", "get", "my-launch-slug"}, false},
+ {"arg echo but no phrase → not graceful",
+ "Error: 500 internal server error processing my-launch-slug",
+ []string{"posts", "get", "my-launch-slug"}, false},
+ {"no positional args → not graceful (no input to echo)",
+ "Error: post not found", []string{"posts", "list"}, false},
+ {"flag-only args → not graceful",
+ "Error: post not found", []string{"--limit", "5"}, false},
+ {"no results phrase + arg echo → graceful",
+ "no results for cursor-ide", []string{"posts", "get", "cursor-ide"}, true},
+ {"empty stderr → not graceful",
+ "", []string{"posts", "get", "my-slug"}, false},
+ {"case-insensitive phrase match",
+ "Post NOT FOUND: my-slug", []string{"posts", "get", "my-slug"}, true},
+ {"short positional args (< 3 chars) skipped to avoid coincidence",
+ "no results for go", []string{"posts", "get", "go"}, false},
+ {"flag value as arg-echo candidate is fine",
+ "no match for query: notion",
+ []string{"posts", "search", "--query", "notion"}, true},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ got := isGracefulEmptyResponse(tc.stderr, tc.args)
+ require.Equal(t, tc.want, got)
+ })
+ }
+}
← aab364be fix(generator): three template defaults polish has to fix ma
·
back to Cli Printing Press
·
chore(main): release 3.4.0 (#490) e8c0e574 →