← back to Cli Printing Press
fix(cli): dogfood agent-context discovery reads stdout only (#407)
fe1e6d77ecd61a087e6e86b4b513efc137c59ce9 · 2026-04-29 21:02:46 -0700 · Trevin Chow
* fix(cli): dogfood agent-context discovery reads stdout only
The dogfood "Examples" check runs `<cli> agent-context` and parses
the output as JSON. Previously the call went through runDogfoodCmd
which uses cmd.CombinedOutput() — fine for --help, broken for
agent-context. Any stderr emission before agent-context's JSON
write (deprecation warnings, config-load notices, panics that race
with the JSON, etc.) prefixes the JSON with text and breaks
json.Unmarshal with `invalid character 'E' looking for beginning of
value` or similar. The dogfood check then skips with a useless
"could not discover command tree" detail line.
Surfaced when polishing weather-goat: agent-context introspection
hit `invalid character 'E'` and the Examples row went from
informative to skipped on every run.
Replace the runDogfoodCmd call in discoverExampleCheckCommands with
a direct exec.CommandContext using cmd.Output() so we read stdout
only. The shared runDogfoodCmd helper continues to use
CombinedOutput; --help and similar callers still want both streams
in their output. Surface stderr in the error path via *exec.ExitError
so a real agent-context failure tells the user what broke instead
of "exit status 1".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(cli): extract runStdoutOnly helper, fix duplicate call site
The /simplify reviewer found a second caller of the same broken
pattern: pickGeneratedCommandFeatures in live_check.go also runs
agent-context via runDogfoodCmd (CombinedOutput) and parses the
result as JSON, with the same stderr-leak vulnerability. The bug
hadn't fired in a session yet but was exactly the same shape.
Extract runStdoutOnly as a sibling to runDogfoodCmd in dogfood.go.
Both agent-context callers now go through it; both wrap the returned
error with "agent-context failed: %w" for symmetric user-visible
messages. The helper keeps the timeout-then-stderr-then-bare-error
precedence the inline code had.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Files touched
M internal/pipeline/dogfood.goM internal/pipeline/dogfood_test.goM internal/pipeline/live_check.go
Diff
commit fe1e6d77ecd61a087e6e86b4b513efc137c59ce9
Author: Trevin Chow <trevin@trevinchow.com>
Date: Wed Apr 29 21:02:46 2026 -0700
fix(cli): dogfood agent-context discovery reads stdout only (#407)
* fix(cli): dogfood agent-context discovery reads stdout only
The dogfood "Examples" check runs `<cli> agent-context` and parses
the output as JSON. Previously the call went through runDogfoodCmd
which uses cmd.CombinedOutput() — fine for --help, broken for
agent-context. Any stderr emission before agent-context's JSON
write (deprecation warnings, config-load notices, panics that race
with the JSON, etc.) prefixes the JSON with text and breaks
json.Unmarshal with `invalid character 'E' looking for beginning of
value` or similar. The dogfood check then skips with a useless
"could not discover command tree" detail line.
Surfaced when polishing weather-goat: agent-context introspection
hit `invalid character 'E'` and the Examples row went from
informative to skipped on every run.
Replace the runDogfoodCmd call in discoverExampleCheckCommands with
a direct exec.CommandContext using cmd.Output() so we read stdout
only. The shared runDogfoodCmd helper continues to use
CombinedOutput; --help and similar callers still want both streams
in their output. Surface stderr in the error path via *exec.ExitError
so a real agent-context failure tells the user what broke instead
of "exit status 1".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(cli): extract runStdoutOnly helper, fix duplicate call site
The /simplify reviewer found a second caller of the same broken
pattern: pickGeneratedCommandFeatures in live_check.go also runs
agent-context via runDogfoodCmd (CombinedOutput) and parses the
result as JSON, with the same stderr-leak vulnerability. The bug
hadn't fired in a session yet but was exactly the same shape.
Extract runStdoutOnly as a sibling to runDogfoodCmd in dogfood.go.
Both agent-context callers now go through it; both wrap the returned
error with "agent-context failed: %w" for symmetric user-visible
messages. The helper keeps the timeout-then-stderr-then-bare-error
precedence the inline code had.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
internal/pipeline/dogfood.go | 32 ++++++++++++++++++--
internal/pipeline/dogfood_test.go | 62 +++++++++++++++++++++++++++++++++++++++
internal/pipeline/live_check.go | 6 ++--
3 files changed, 94 insertions(+), 6 deletions(-)
diff --git a/internal/pipeline/dogfood.go b/internal/pipeline/dogfood.go
index b48181c0..66d4fe7f 100644
--- a/internal/pipeline/dogfood.go
+++ b/internal/pipeline/dogfood.go
@@ -3,6 +3,7 @@ package pipeline
import (
"context"
"encoding/json"
+ "errors"
"fmt"
"os"
"os/exec"
@@ -1557,11 +1558,11 @@ func checkExamples(dir string) ExampleCheckResult {
}
func discoverExampleCheckCommands(binaryPath string) ([][]string, error) {
- out, err := runDogfoodCmd(binaryPath, 15*time.Second, "agent-context")
+ out, err := runStdoutOnly(binaryPath, 15*time.Second, "agent-context")
if err != nil {
- return nil, err
+ return nil, fmt.Errorf("agent-context failed: %w", err)
}
- paths, err := dogfoodExampleCommandPathsFromAgentContext([]byte(out))
+ paths, err := dogfoodExampleCommandPathsFromAgentContext(out)
if err != nil {
return nil, err
}
@@ -1571,6 +1572,31 @@ func discoverExampleCheckCommands(binaryPath string) ([][]string, error) {
return paths, nil
}
+// runStdoutOnly runs binaryPath with args and returns stdout. Unlike
+// runDogfoodCmd, stderr is captured separately so a leaky printed CLI
+// (deprecation warnings, config-load notices, panics that race with
+// stdout writes) doesn't prefix the JSON consumers expect to parse.
+// On non-zero exit, the trimmed stderr surfaces in the error so
+// callers can surface a meaningful "what broke" instead of "exit
+// status 1".
+func runStdoutOnly(binaryPath string, timeout time.Duration, args ...string) ([]byte, error) {
+ ctx, cancel := context.WithTimeout(context.Background(), timeout)
+ defer cancel()
+ cmd := exec.CommandContext(ctx, binaryPath, args...)
+ out, err := cmd.Output()
+ if ctx.Err() == context.DeadlineExceeded {
+ return nil, fmt.Errorf("timed out after %s", timeout)
+ }
+ if err != nil {
+ var exitErr *exec.ExitError
+ if errors.As(err, &exitErr) && len(exitErr.Stderr) > 0 {
+ return nil, fmt.Errorf("%s", strings.TrimSpace(string(exitErr.Stderr)))
+ }
+ return nil, err
+ }
+ return out, nil
+}
+
func dogfoodExampleCommandPathsFromAgentContext(data []byte) ([][]string, error) {
var ctx dogfoodAgentContext
if err := json.Unmarshal(data, &ctx); err != nil {
diff --git a/internal/pipeline/dogfood_test.go b/internal/pipeline/dogfood_test.go
index bd93efd2..5b0a2787 100644
--- a/internal/pipeline/dogfood_test.go
+++ b/internal/pipeline/dogfood_test.go
@@ -3,6 +3,7 @@ package pipeline
import (
"os"
"path/filepath"
+ "runtime"
"strings"
"testing"
@@ -1617,3 +1618,64 @@ func TestCollectDogfoodIssues_ThinTestsNotHardIssue(t *testing.T) {
assert.NotContains(t, i, "thin")
}
}
+
+// TestDiscoverExampleCheckCommandsIgnoresStderrNoise — printed CLIs
+// commonly write to stderr before agent-context succeeds (deprecation
+// warnings, config-load notices, panics that race with the JSON
+// write). Previously runDogfoodCmd merged stderr into stdout via
+// CombinedOutput, prefixing the JSON with text and breaking
+// json.Unmarshal with "invalid character 'E' looking for beginning of
+// value". discoverExampleCheckCommands must read stdout only so the
+// dogfood "Examples" check survives stderr leaks.
+func TestDiscoverExampleCheckCommandsIgnoresStderrNoise(t *testing.T) {
+ if runtime.GOOS == "windows" {
+ t.Skip("test uses a shell script as the fake binary; skip on Windows")
+ }
+ t.Parallel()
+
+ script := `#!/bin/sh
+# Deliberately write to stderr before the JSON write — simulates a
+# printed CLI emitting a deprecation warning or config-load notice.
+echo "WARN: legacy config path detected" >&2
+cat <<EOF
+{
+ "commands": [
+ {"name": "posts", "subcommands": [{"name": "list"}]},
+ {"name": "auth", "subcommands": [{"name": "login"}]}
+ ]
+}
+EOF
+`
+ binPath := filepath.Join(t.TempDir(), "fakebin")
+ require.NoError(t, os.WriteFile(binPath, []byte(script), 0o755))
+
+ paths, err := discoverExampleCheckCommands(binPath)
+ require.NoError(t, err, "stderr noise must not break JSON parsing")
+ // auth/login is filtered out because "auth" is in the framework
+ // skip set; only the posts subtree should survive.
+ assert.Equal(t, [][]string{{"posts", "list"}}, paths)
+}
+
+// TestDiscoverExampleCheckCommandsSurfacesStderrOnFailure — when the
+// binary actually fails (non-zero exit), stderr carries the reason
+// and the caller deserves to see it. Previously a CombinedOutput
+// failure left the user with "exit status 1"; the new path uses
+// exec.ExitError.Stderr to surface the underlying message.
+func TestDiscoverExampleCheckCommandsSurfacesStderrOnFailure(t *testing.T) {
+ if runtime.GOOS == "windows" {
+ t.Skip("test uses a shell script as the fake binary; skip on Windows")
+ }
+ t.Parallel()
+
+ script := `#!/bin/sh
+echo "config file ~/.fakebin/config.toml: permission denied" >&2
+exit 1
+`
+ binPath := filepath.Join(t.TempDir(), "fakebin")
+ require.NoError(t, os.WriteFile(binPath, []byte(script), 0o755))
+
+ _, err := discoverExampleCheckCommands(binPath)
+ require.Error(t, err)
+ assert.Contains(t, err.Error(), "permission denied",
+ "stderr from the failed agent-context call must surface in the error")
+}
diff --git a/internal/pipeline/live_check.go b/internal/pipeline/live_check.go
index 4c044ad2..10b82db8 100644
--- a/internal/pipeline/live_check.go
+++ b/internal/pipeline/live_check.go
@@ -267,11 +267,11 @@ func pickFeatures(r *ResearchResult) []NovelFeature {
}
func pickGeneratedCommandFeatures(binaryPath string) ([]NovelFeature, error) {
- out, err := runDogfoodCmd(binaryPath, 15*time.Second, "agent-context")
+ out, err := runStdoutOnly(binaryPath, 15*time.Second, "agent-context")
if err != nil {
- return nil, err
+ return nil, fmt.Errorf("agent-context failed: %w", err)
}
- paths, err := dogfoodExampleCommandPathsFromAgentContext([]byte(out))
+ paths, err := dogfoodExampleCommandPathsFromAgentContext(out)
if err != nil {
return nil, err
}
← 0954bcaf fix(cli): scorecard human output shows N/A for opt-out dimen
·
back to Cli Printing Press
·
fix(skills): drop context: fork from polish so AskUserQuesti 0683ef1f →