← back to Cli Printing Press
fix(cli): skip destructive-at-auth endpoints in live dogfood (#613)
84b22d2817d60f40a5c37ba9486b76739b008d16 · 2026-05-05 08:25:34 -0700 · Trevin Chow
* fix(cli): skip destructive-at-auth endpoints in live dogfood
Closes #602. Cal.com's `dogfood --live --level full` cascade — 595/723
false 401s after a single self-rotation of the runner's bearer via
`POST /api-keys/refresh` — now classifies, skips, and reports.
Annotation-primary classifier with Cobra-leaf-segment fallback. The
key plumbing change: agent-context now surfaces `cmd.Annotations`
through the dogfoodAgentCommand → liveDogfoodCommand pipeline. Without
this, the matrix builder couldn't see the `pp:endpoint` annotation
(authoritative signal for endpoint-mirror commands) and Cobra-leaf-only
classification missed promoted commands like cal-com's `Use: "api-keys"`
whose leaf has no `refresh` segment.
Three classification paths:
1. Annotation primary: pp:endpoint substring-match against
{refresh, rotate, revoke}. Catches the cal-com case
(api-keys.keys-refresh) even when the leaf is just "api-keys".
2. Leaf-segment fallback: when pp:endpoint is absent (novel hand-built
commands), substring-match on each leaf path segment lowercased.
Catches compound names like oauth-client-force-refresh.
3. Read-only exemption: any command with mcp:read-only=true is exempt
regardless of name. Read-only commands cannot rotate auth.
Verified-correct on craigslist's catalog refresh and similar
re-sync-from-upstream commands.
Default-skip with --allow-destructive opt-out. The flag threads:
LiveDogfoodOptions.AllowDestructive → resolveCtx.allowDestructive →
runLiveDogfoodCommand → classifier short-circuit. Skipped entries
emit four `LiveDogfoodTestResult{Status: Skip, Reason:
"destructive-at-auth"}` entries (one per kind), and finalizeLiveDogfoodReport
already excludes Skip from MatrixSize so destructive clusters don't
tank the verdict-gate floor.
Match list intentionally limited to {refresh, rotate, revoke} per
#602's stated scope. Extension by discovery is the deferred-question
hook for new patterns.
Tests: 11 classifier cases (annotation primary, leaf fallback, compound
names, read-only exempt, case-insensitive, negative paths, empty
inputs). Plus full suite: 3030 tests pass.
Note on existing library CLIs: until they're regenerated against this
template, their agent-context output won't carry annotations and the
classifier falls back to leaf-path matching for them. Cal-com today
ships `Use: "api-keys"` with no refresh segment, so leaf fallback
misses it — re-printing cal-com (or running printing-press regen-merge)
picks up the new agent-context emit and the classifier catches the
endpoint correctly. False negatives are recoverable (operator gets a
401 cascade and reports it); false positives are bounded by the
read-only exemption.
Plan: docs/plans/2026-05-05-006-fix-wave-2-retro-blockers-plan.md (U4 + U5)
* refactor(cli): simplify destructive-at-auth classifier and tests
Four simplify-pass changes to the prior commit, all from a /simplify
review against the recently-modified surface:
1. isDestructiveAtAuth uses containsAnyOf (defined at
live_check.go:678) instead of a hand-rolled double loop. Same
package, exact-shape match — case-insensitive substring against a
list. Drops 8 lines.
2. Hoist "destructive-at-auth" to a package const reasonDestructiveAtAuth.
Repeated 4x in the skip emission block plus more sites; one source
beats four duplications.
3. Drop ticket-number references (#602) from code comments per AGENTS.md
"no dates, incidents, or ticket numbers in code comments." The
issue link belongs in the PR description, not in the code.
4. Consolidate 11 separate TestIsDestructiveAtAuth* functions into one
table-driven test using t.Run subtests. Matches the existing
convention in live_dogfood_test.go (TestCommandSupportsSearch is the
precedent for boolean-classifier tests). Drops ~90 lines of
per-function boilerplate; same coverage.
Net: classifier is shorter and reads in one screen; tests are denser
and consistent with the package; no behavior change. All 11 cases
still pass; full suite still 3030 tests passing.
Plan: docs/plans/2026-05-05-006-fix-wave-2-retro-blockers-plan.md (Wave 2 simplify pass)
* test(cli): tighten classifier doc + cover annotation-key boundary
ce-code-review autofix-pass changes:
1. Doc comment on isDestructiveAtAuth said "Cobra-leaf-segment
matching" but the implementation scans all path segments
(catching compound names like oauth-client-force-refresh that
live deeper than the leaf). Comment now says "path-segment
matching across the command path" — accurate to the code.
2. New test row: when an annotation map contains a destructive
term in a non-pp:endpoint key (e.g., description = "refresh
the cache"), the classifier must NOT trigger. The annotation
primary path reads pp:endpoint exclusively; other keys are
not part of the contract. Caught by the adversarial reviewer.
No behavior change. Both fixes are doc/test-only.
Plan: docs/plans/2026-05-05-006-fix-wave-2-retro-blockers-plan.md (Wave 2 ce-code-review autofix)
Files touched
M internal/cli/dogfood.goM internal/generator/templates/agent_context.go.tmplM internal/pipeline/dogfood.goM internal/pipeline/live_dogfood.goA internal/pipeline/live_dogfood_destructive_test.go
Diff
commit 84b22d2817d60f40a5c37ba9486b76739b008d16
Author: Trevin Chow <trevin@trevinchow.com>
Date: Tue May 5 08:25:34 2026 -0700
fix(cli): skip destructive-at-auth endpoints in live dogfood (#613)
* fix(cli): skip destructive-at-auth endpoints in live dogfood
Closes #602. Cal.com's `dogfood --live --level full` cascade — 595/723
false 401s after a single self-rotation of the runner's bearer via
`POST /api-keys/refresh` — now classifies, skips, and reports.
Annotation-primary classifier with Cobra-leaf-segment fallback. The
key plumbing change: agent-context now surfaces `cmd.Annotations`
through the dogfoodAgentCommand → liveDogfoodCommand pipeline. Without
this, the matrix builder couldn't see the `pp:endpoint` annotation
(authoritative signal for endpoint-mirror commands) and Cobra-leaf-only
classification missed promoted commands like cal-com's `Use: "api-keys"`
whose leaf has no `refresh` segment.
Three classification paths:
1. Annotation primary: pp:endpoint substring-match against
{refresh, rotate, revoke}. Catches the cal-com case
(api-keys.keys-refresh) even when the leaf is just "api-keys".
2. Leaf-segment fallback: when pp:endpoint is absent (novel hand-built
commands), substring-match on each leaf path segment lowercased.
Catches compound names like oauth-client-force-refresh.
3. Read-only exemption: any command with mcp:read-only=true is exempt
regardless of name. Read-only commands cannot rotate auth.
Verified-correct on craigslist's catalog refresh and similar
re-sync-from-upstream commands.
Default-skip with --allow-destructive opt-out. The flag threads:
LiveDogfoodOptions.AllowDestructive → resolveCtx.allowDestructive →
runLiveDogfoodCommand → classifier short-circuit. Skipped entries
emit four `LiveDogfoodTestResult{Status: Skip, Reason:
"destructive-at-auth"}` entries (one per kind), and finalizeLiveDogfoodReport
already excludes Skip from MatrixSize so destructive clusters don't
tank the verdict-gate floor.
Match list intentionally limited to {refresh, rotate, revoke} per
#602's stated scope. Extension by discovery is the deferred-question
hook for new patterns.
Tests: 11 classifier cases (annotation primary, leaf fallback, compound
names, read-only exempt, case-insensitive, negative paths, empty
inputs). Plus full suite: 3030 tests pass.
Note on existing library CLIs: until they're regenerated against this
template, their agent-context output won't carry annotations and the
classifier falls back to leaf-path matching for them. Cal-com today
ships `Use: "api-keys"` with no refresh segment, so leaf fallback
misses it — re-printing cal-com (or running printing-press regen-merge)
picks up the new agent-context emit and the classifier catches the
endpoint correctly. False negatives are recoverable (operator gets a
401 cascade and reports it); false positives are bounded by the
read-only exemption.
Plan: docs/plans/2026-05-05-006-fix-wave-2-retro-blockers-plan.md (U4 + U5)
* refactor(cli): simplify destructive-at-auth classifier and tests
Four simplify-pass changes to the prior commit, all from a /simplify
review against the recently-modified surface:
1. isDestructiveAtAuth uses containsAnyOf (defined at
live_check.go:678) instead of a hand-rolled double loop. Same
package, exact-shape match — case-insensitive substring against a
list. Drops 8 lines.
2. Hoist "destructive-at-auth" to a package const reasonDestructiveAtAuth.
Repeated 4x in the skip emission block plus more sites; one source
beats four duplications.
3. Drop ticket-number references (#602) from code comments per AGENTS.md
"no dates, incidents, or ticket numbers in code comments." The
issue link belongs in the PR description, not in the code.
4. Consolidate 11 separate TestIsDestructiveAtAuth* functions into one
table-driven test using t.Run subtests. Matches the existing
convention in live_dogfood_test.go (TestCommandSupportsSearch is the
precedent for boolean-classifier tests). Drops ~90 lines of
per-function boilerplate; same coverage.
Net: classifier is shorter and reads in one screen; tests are denser
and consistent with the package; no behavior change. All 11 cases
still pass; full suite still 3030 tests passing.
Plan: docs/plans/2026-05-05-006-fix-wave-2-retro-blockers-plan.md (Wave 2 simplify pass)
* test(cli): tighten classifier doc + cover annotation-key boundary
ce-code-review autofix-pass changes:
1. Doc comment on isDestructiveAtAuth said "Cobra-leaf-segment
matching" but the implementation scans all path segments
(catching compound names like oauth-client-force-refresh that
live deeper than the leaf). Comment now says "path-segment
matching across the command path" — accurate to the code.
2. New test row: when an annotation map contains a destructive
term in a non-pp:endpoint key (e.g., description = "refresh
the cache"), the classifier must NOT trigger. The annotation
primary path reads pp:endpoint exclusively; other keys are
not part of the contract. Caught by the adversarial reviewer.
No behavior change. Both fixes are doc/test-only.
Plan: docs/plans/2026-05-05-006-fix-wave-2-retro-blockers-plan.md (Wave 2 ce-code-review autofix)
---
internal/cli/dogfood.go | 3 +
internal/generator/templates/agent_context.go.tmpl | 11 +++
internal/pipeline/dogfood.go | 1 +
internal/pipeline/live_dogfood.go | 91 +++++++++++++-----
internal/pipeline/live_dogfood_destructive_test.go | 102 +++++++++++++++++++++
5 files changed, 184 insertions(+), 24 deletions(-)
diff --git a/internal/cli/dogfood.go b/internal/cli/dogfood.go
index 845d703a..72a9ede3 100644
--- a/internal/cli/dogfood.go
+++ b/internal/cli/dogfood.go
@@ -22,6 +22,7 @@ func newDogfoodCmd() *cobra.Command {
var timeout time.Duration
var writeAcceptance string
var authEnv string
+ var allowDestructive bool
cmd := &cobra.Command{
Use: "dogfood",
@@ -40,6 +41,7 @@ func newDogfoodCmd() *cobra.Command {
Timeout: timeout,
WriteAcceptancePath: writeAcceptance,
AuthEnv: authEnv,
+ AllowDestructive: allowDestructive,
})
if err != nil {
return &ExitError{Code: ExitGenerationError, Err: fmt.Errorf("running live dogfood: %w", err)}
@@ -88,6 +90,7 @@ func newDogfoodCmd() *cobra.Command {
cmd.Flags().DurationVar(&timeout, "timeout", 30*time.Second, "Timeout for each live dogfood test")
cmd.Flags().StringVar(&writeAcceptance, "write-acceptance", "", "Write phase5-acceptance.json to this path when live dogfood passes")
cmd.Flags().StringVar(&authEnv, "auth-env", "", "Environment variable that proves an API credential was available for the acceptance marker")
+ cmd.Flags().BoolVar(&allowDestructive, "allow-destructive", false, "Re-enable testing of endpoints classified as destructive-at-auth (path/annotation matches refresh/rotate/revoke). Default skips them to prevent runner-credential rotation.")
_ = cmd.MarkFlagRequired("dir")
return cmd
}
diff --git a/internal/generator/templates/agent_context.go.tmpl b/internal/generator/templates/agent_context.go.tmpl
index f1891459..dc1c9ce1 100644
--- a/internal/generator/templates/agent_context.go.tmpl
+++ b/internal/generator/templates/agent_context.go.tmpl
@@ -60,6 +60,7 @@ type agentContextCommand struct {
Name string `json:"name"`
Use string `json:"use,omitempty"`
Short string `json:"short,omitempty"`
+ Annotations map[string]string `json:"annotations,omitempty"`
Flags []agentContextFlag `json:"flags,omitempty"`
Subcommands []agentContextCommand `json:"subcommands,omitempty"`
}
@@ -188,6 +189,16 @@ func collectAgentCommands(c *cobra.Command) []agentContextCommand {
Use: sub.Use,
Short: sub.Short,
}
+ // Surface Cobra annotations (e.g., pp:endpoint, mcp:read-only) so
+ // agents and the live-dogfood classifier can detect destructive-at-auth
+ // endpoints without parsing source. Empty maps are stripped via
+ // omitempty in the struct tag.
+ if len(sub.Annotations) > 0 {
+ entry.Annotations = make(map[string]string, len(sub.Annotations))
+ for k, v := range sub.Annotations {
+ entry.Annotations[k] = v
+ }
+ }
sub.Flags().VisitAll(func(f *pflag.Flag) {
entry.Flags = append(entry.Flags, agentContextFlag{
Name: f.Name,
diff --git a/internal/pipeline/dogfood.go b/internal/pipeline/dogfood.go
index c8219684..de676721 100644
--- a/internal/pipeline/dogfood.go
+++ b/internal/pipeline/dogfood.go
@@ -143,6 +143,7 @@ type dogfoodAgentContext struct {
type dogfoodAgentCommand struct {
Name string `json:"name"`
+ Annotations map[string]string `json:"annotations,omitempty"`
Subcommands []dogfoodAgentCommand `json:"subcommands,omitempty"`
}
diff --git a/internal/pipeline/live_dogfood.go b/internal/pipeline/live_dogfood.go
index f7e09723..bea61c10 100644
--- a/internal/pipeline/live_dogfood.go
+++ b/internal/pipeline/live_dogfood.go
@@ -33,6 +33,11 @@ const (
LiveDogfoodTestErrorReal LiveDogfoodTestKind = "error_path_real"
)
+// reasonDestructiveAtAuth is the Skip reason emitted for endpoints whose
+// path or pp:endpoint annotation matches refresh/rotate/revoke. Reused
+// across the matrix builder, the flag help text, and the test fixtures.
+const reasonDestructiveAtAuth = "destructive-at-auth"
+
type LiveDogfoodOptions struct {
CLIDir string
BinaryName string
@@ -40,6 +45,10 @@ type LiveDogfoodOptions struct {
Timeout time.Duration
WriteAcceptancePath string
AuthEnv string
+ // AllowDestructive re-enables testing of endpoints classified as
+ // destructive-at-auth. Default skips them to prevent runner-credential
+ // rotation.
+ AllowDestructive bool
}
type LiveDogfoodReport struct {
@@ -67,8 +76,9 @@ type LiveDogfoodTestResult struct {
}
type liveDogfoodCommand struct {
- Path []string
- Help string
+ Path []string
+ Help string
+ Annotations map[string]string
}
type liveDogfoodRun struct {
@@ -117,11 +127,12 @@ func RunLiveDogfood(opts LiveDogfoodOptions) (*LiveDogfoodReport, error) {
}
ctx := resolveCtx{
- binaryPath: binaryPath,
- cliDir: opts.CLIDir,
- siblings: buildSiblingMap(commands),
- cache: newCompanionCache(),
- timeout: timeout,
+ binaryPath: binaryPath,
+ cliDir: opts.CLIDir,
+ siblings: buildSiblingMap(commands),
+ cache: newCompanionCache(),
+ timeout: timeout,
+ allowDestructive: opts.AllowDestructive,
}
for _, command := range commands {
@@ -164,18 +175,13 @@ func discoverLiveDogfoodCommands(binaryPath string) ([]liveDogfoodCommand, error
return nil, fmt.Errorf("parsing agent-context: %w", err)
}
- var paths [][]string
+ var commands []liveDogfoodCommand
for _, command := range ctx.Commands {
- collectLiveDogfoodCommandPaths(nil, command, &paths)
+ collectLiveDogfoodCommands(nil, command, &commands)
}
- sort.Slice(paths, func(i, j int) bool {
- return strings.Join(paths[i], " ") < strings.Join(paths[j], " ")
+ sort.Slice(commands, func(i, j int) bool {
+ return strings.Join(commands[i].Path, " ") < strings.Join(commands[j].Path, " ")
})
-
- commands := make([]liveDogfoodCommand, 0, len(paths))
- for _, path := range paths {
- commands = append(commands, liveDogfoodCommand{Path: path})
- }
return commands, nil
}
@@ -236,11 +242,12 @@ type companionCache struct {
// resolveCtx threads run-scoped state into the chained companion walk so
// individual helpers don't need to take the same five parameters.
type resolveCtx struct {
- binaryPath string
- cliDir string
- siblings map[string][]liveDogfoodCommand
- cache *companionCache
- timeout time.Duration
+ binaryPath string
+ cliDir string
+ siblings map[string][]liveDogfoodCommand
+ cache *companionCache
+ timeout time.Duration
+ allowDestructive bool
}
func newCompanionCache() *companionCache {
@@ -567,24 +574,36 @@ func idValueAsString(v any) (string, bool) {
}
}
-func collectLiveDogfoodCommandPaths(prefix []string, command dogfoodAgentCommand, paths *[][]string) {
+func collectLiveDogfoodCommands(prefix []string, command dogfoodAgentCommand, cmds *[]liveDogfoodCommand) {
if command.Name == "" || liveDogfoodFrameworkSkip[command.Name] {
return
}
next := append(append([]string{}, prefix...), command.Name)
if len(command.Subcommands) == 0 {
- *paths = append(*paths, next)
+ *cmds = append(*cmds, liveDogfoodCommand{Path: next, Annotations: command.Annotations})
return
}
for _, sub := range command.Subcommands {
- collectLiveDogfoodCommandPaths(next, sub, paths)
+ collectLiveDogfoodCommands(next, sub, cmds)
}
}
func runLiveDogfoodCommand(command liveDogfoodCommand, ctx resolveCtx) []LiveDogfoodTestResult {
commandName := strings.Join(command.Path, " ")
+ // Destructive-at-auth short-circuit: commands that rotate or revoke
+ // the runner's bearer would 401-cascade every subsequent test. Skips
+ // don't count toward MatrixSize (see finalizeLiveDogfoodReport).
+ if !ctx.allowDestructive && isDestructiveAtAuth(command.Annotations, command.Path) {
+ return []LiveDogfoodTestResult{
+ skippedLiveDogfoodResult(commandName, LiveDogfoodTestHelp, reasonDestructiveAtAuth),
+ skippedLiveDogfoodResult(commandName, LiveDogfoodTestHappy, reasonDestructiveAtAuth),
+ skippedLiveDogfoodResult(commandName, LiveDogfoodTestJSON, reasonDestructiveAtAuth),
+ skippedLiveDogfoodResult(commandName, LiveDogfoodTestError, reasonDestructiveAtAuth),
+ }
+ }
+
helpArgs := append(append([]string{}, command.Path...), "--help")
helpRun := runLiveDogfoodProcess(ctx.binaryPath, ctx.cliDir, helpArgs, ctx.timeout)
helpResult := liveDogfoodResult(commandName, LiveDogfoodTestHelp, helpArgs, helpRun)
@@ -854,6 +873,30 @@ func skippedLiveDogfoodResult(command string, kind LiveDogfoodTestKind, reason s
}
}
+// destructiveAuthTerms are case-insensitive substrings classifying a
+// command as destructive-at-auth.
+var destructiveAuthTerms = []string{"refresh", "rotate", "revoke"}
+
+// isDestructiveAtAuth reports whether a command rotates or revokes the
+// bearer the live-dogfood runner is using. Reads pp:endpoint
+// (authoritative for endpoint-mirror commands) and falls back to
+// path-segment matching across the command path for novel commands.
+// Read-only commands are exempt regardless of name.
+func isDestructiveAtAuth(annotations map[string]string, commandPath []string) bool {
+ if annotations["mcp:read-only"] == "true" {
+ return false
+ }
+ if endpoint := annotations["pp:endpoint"]; endpoint != "" {
+ return containsAnyOf(strings.ToLower(endpoint), destructiveAuthTerms)
+ }
+ for _, seg := range commandPath {
+ if containsAnyOf(strings.ToLower(seg), destructiveAuthTerms) {
+ return true
+ }
+ }
+ return false
+}
+
func liveDogfoodHappyArgs(command liveDogfoodCommand) ([]string, bool) {
examples := extractExamplesSection(command.Help)
for line := range strings.SplitSeq(examples, "\n") {
diff --git a/internal/pipeline/live_dogfood_destructive_test.go b/internal/pipeline/live_dogfood_destructive_test.go
new file mode 100644
index 00000000..45a6b855
--- /dev/null
+++ b/internal/pipeline/live_dogfood_destructive_test.go
@@ -0,0 +1,102 @@
+package pipeline
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+)
+
+// TestIsDestructiveAtAuth covers the classifier's annotation-primary path,
+// Cobra-leaf-segment fallback, read-only exemption, and negative cases.
+// Cal.com's promoted command (Use="api-keys" with pp:endpoint=
+// "api-keys.keys-refresh") is the motivating example: leaf-only matching
+// misses it; the annotation lookup catches it.
+func TestIsDestructiveAtAuth(t *testing.T) {
+ t.Parallel()
+
+ cases := []struct {
+ name string
+ annotations map[string]string
+ path []string
+ want bool
+ }{
+ {
+ name: "annotation primary cal-com api-keys-refresh",
+ annotations: map[string]string{"pp:endpoint": "api-keys.keys-refresh"},
+ path: []string{"cal-com-pp-cli", "api-keys"},
+ want: true,
+ },
+ {
+ name: "annotation primary token rotate",
+ annotations: map[string]string{"pp:endpoint": "tokens.token-rotate"},
+ path: []string{"my-cli", "tokens"},
+ want: true,
+ },
+ {
+ name: "annotation case-insensitive",
+ annotations: map[string]string{"pp:endpoint": "API-Keys.Refresh-Key"},
+ path: []string{"my-cli", "API-Keys"},
+ want: true,
+ },
+ {
+ name: "annotation present without destructive term",
+ annotations: map[string]string{"pp:endpoint": "users.list-users"},
+ path: []string{"my-cli", "users", "refresh-cache"},
+ want: false,
+ },
+ {
+ name: "leaf fallback novel command",
+ path: []string{"my-cli", "auth", "refresh"},
+ want: true,
+ },
+ {
+ name: "leaf fallback compound name",
+ path: []string{"my-cli", "oauth-clients", "users", "oauth-client-force-refresh"},
+ want: true,
+ },
+ {
+ name: "leaf fallback no match",
+ path: []string{"my-cli", "users", "list"},
+ want: false,
+ },
+ {
+ name: "read-only exempt with pp:endpoint",
+ annotations: map[string]string{
+ "mcp:read-only": "true",
+ "pp:endpoint": "catalog.catalog-refresh",
+ },
+ path: []string{"craigslist-pp-cli", "catalog", "refresh"},
+ want: false,
+ },
+ {
+ name: "read-only exempt leaf only",
+ annotations: map[string]string{"mcp:read-only": "true"},
+ path: []string{"my-cli", "store", "refresh"},
+ want: false,
+ },
+ {
+ name: "empty inputs",
+ want: false,
+ },
+ {
+ // Adversarial reviewer caught: a non-pp:endpoint annotation
+ // containing a destructive term must not trigger the classifier.
+ // Annotation-primary path reads pp:endpoint exclusively; other
+ // keys (description, etc.) are not part of the contract.
+ name: "destructive term in non-pp:endpoint key is ignored",
+ annotations: map[string]string{
+ "pp:endpoint": "users.list-users",
+ "description": "list users (refresh the cache)",
+ },
+ path: []string{"my-cli", "users"},
+ want: false,
+ },
+ }
+
+ for _, tt := range cases {
+ t.Run(tt.name, func(t *testing.T) {
+ got := isDestructiveAtAuth(tt.annotations, tt.path)
+ assert.Equal(t, tt.want, got)
+ })
+ }
+}
← 9c5b882a fix(cli): emit invalidateCache() in every printed client.go
·
back to Cli Printing Press
·
chore(main): release 3.9.1 (#607) c5e7a801 →