← back to Cli Printing Press
fix(cli): live-dogfood matrix accuracy (WU-2, F4+F5) (#577)
7572b1e09ba4ae8ef4e29fe02e73fa799993e14e · 2026-05-04 15:30:18 -0700 · Trevin Chow
* docs(cli): add plan for live dogfood matrix accuracy (WU-2)
Captures the implementation plan for sub-issue #573 (parent retro
#571). Covers the four absorbed findings: camelCase ID recognition
with type fence (U1), chained companion walk for multi-positional
gets with per-companion cache (U2), search-aware error_path
dispatch (U3), and quick-level verdict gate restated to keep
skip-with-reason from flipping PASS to FAIL (U4).
* fix(cli): U1 — camelCase ID recognition in exampleValue with type fence
The exampleValue function (internal/generator/generator.go) only
recognised `*_id` (snake) and bare `id` as ID-shaped positionals.
camelCase names like `movieId`, `seriesId`, `personId` lower to
`movieid`/`seriesid`/`personid` and missed the existing
HasSuffix("_id") check, falling through to the literal
`example-value` placeholder. Every printed CLI generated from an
OpenAPI spec with camelCase ID positionals (TMDb, Notion, Linear,
Pagliacci, ESPN) shipped with broken Examples that live dogfood
probes treated as positional values, producing 404s.
Adds a third clause to the ID branch:
HasSuffix(nameLower, "id") && len(nameLower) > 2 &&
(p.Type == "string" || p.Type == "")
The string-type fence prevents boolean/numeric positionals like
`paid`, `valid`, `creditValid` from getting UUIDs — they flow
into their existing type-specific branches (boolean → "true",
integer → "42" or count-branch "50"). Empty Type passes through
to handle legacy YAML specs without explicit type info.
Pure-string false positives (`void`, `acid`, `arid` as required
URL path params) are accepted: a UUID at the wrong path produces
a clear 404 at verify time, strictly better than today's literal
`example-value`. Add a denylist if a real-spec FP surfaces.
Golden fixtures updated for projectId/taskId positionals in the
golden-api spec (intentional byte-level diff of two Example
lines).
* fix(cli): U4 — restate quick-level live-dogfood verdict gate
The previous gate required exactly MatrixSize == 6 with Passed >= 5
for a quick-level PASS. With U2's coming skip-with-reason paths
(get-shape commands without a sibling companion), MatrixSize can
drop below 6 even when nothing actually failed — flipping the
verdict from PASS to FAIL purely from increased Skip count.
Restate the switch:
Failed > 0 || MatrixSize == 0 → FAIL (failed-or-empty wins)
quick + (Passed + Skipped) >= 5 && MatrixSize >= 4 → PASS
quick fallthrough → FAIL
Skipped now counts toward the 5-entry quorum. The MatrixSize floor
of 4 guards against pathological all-skip outcomes — too few real
test runs to trust. Failed-dominance is preserved at every level.
Tests cover: classic 6-pass; 5-pass + 1-skip; 4-pass + 2-skip;
3-pass + 3-skip floor breach; pass + fail dominance; all-skip;
full-level pass and fail unaffected.
* fix(cli): U3 — search-aware error_path dispatch
The error_path probe ran every command with __printing_press_invalid__
and required a non-zero exit. Search-shape commands canonically return
exit 0 with empty (or fallback-recency) results on no-match — Movie
Goat shipped 6 false-fail entries on movies/people/tv/multi/search/
auth-set-token because of this uniform strategy.
Add commandSupportsSearch (--query flag OR <query> placeholder) and
branch the error_path strategy:
Search-shape:
exit non-zero → PASS (API rejected token)
exit 0 + valid JSON when --json → PASS (any result shape)
exit 0 + invalid JSON when --json → FAIL
exit 0 + non-JSON without --json → PASS
Mutation/plain-get (today's behavior preserved):
non-zero exit required.
The 'exit 0 + non-empty results = FAIL' branch from earlier drafts
is dropped: real-world content/feed APIs return recent items as a
fallback for unmatched queries, and treating non-empty as a failure
would block legitimate ships. Detecting broken filter logic is a
unit-test concern, not a live-dogfood signal.
Tests cover the heuristic across positive (--query flag, <query>
positional) and negative (list with --limit, dispatch with --queue,
delete) cases. The exact-match check on --queue confirms the flag-
name extractor doesn't substring-match.
* fix(cli): U2 — chained companion walk for live-dogfood id positionals
The live-dogfood matrix probed get-shape commands with the literal
`example-value` placeholder, producing 5 false-fail entries on
Movie Goat (movies get, people get, tv get, tv seasons get, export).
Even with U1's UUID-shape placeholder, real APIs 404 on synthetic
ids — the only way to actually test the get path live is to source
a real id from the API.
Add a chained companion walk that, for each id-shape positional in
the get-shape command's Usage line, sources a real id by running a
sibling list-shape companion under --json. The walk threads earlier-
resolved ids into later list calls, so nested resources work
end-to-end (e.g., projects/tasks/update <pid> <tid> sources pid
from `projects list` then tid from `projects tasks list <pid>`).
New mechanism in internal/pipeline/live_dogfood.go:
- buildSiblingMap groups commands by joined parent path for O(1)
sibling lookup.
- crossAPIListVerbs (list, all, index, query, find, search, discover,
browse, recent, feed) cover modern API conventions; cinemaListVerbs
retain TMDb-specific verbs (popular, trending, top_rated, etc.) so
the original motivating CLI keeps coverage.
- resolveCommandPositionals walks the chain: extract Usage placeholders,
reject non-id-shape mid-chain, find a list companion at each depth,
invoke it (with parent ids as positional context, --json, and
--limit 1 when supported), extract first id, accumulate.
- extractFirstIDFromJSON tries seven canonical shapes (results, top-
level array, items, data, list, GraphQL .data.<any>.nodes,
GraphQL .data.<any>.edges with depth-2 walk for Relay viewer.repos
shapes). Numeric ids decoded via UseNumber to preserve large
values (snowflake > 2^53) without scientific-notation rounding.
- companionCache (run-scoped) memoizes (companion-argv → id) so
sibling get-shape commands share lookups; --help is also cached
per companion path.
- substitutePositionals walks happyArgs left-to-right after
command.Path, replacing the first N non-flag args with the
resolved ids, preserving interleaved flags.
Wired into runLiveDogfoodCommand: after liveDogfoodHappyArgs, call
the resolver. On resolveSkip, happy_path + json_fidelity become
Status=Skip with a structured reason naming the failed link
('no list companion at depth N for <name>' / 'list companion
failed at depth N: exit X' / 'no id parseable from companion at
depth N' / 'non-id positional <name> at depth N'). error_path runs
independently (its own positional gate, U3-aware strategy).
Tests cover: TMDb/REST/Stripe/long-tail/GraphQL-Shopify/GraphQL-
Relay id extraction shapes; large numeric ids; sibling map
construction; companion-leaf detection; positional substitution
(single, two, before-flag, no-positionals); resolver skip paths
(no-positional pass-through, non-id mid-chain, no-companion,
path-segment shortage). All 935 pipeline package tests pass.
* refactor(cli): simplify per /ce-simplify-code review
Apply 3-reviewer (reuse / quality / efficiency) findings:
Quality (9 fixes):
- Extract resolveCtx struct — resolveCommandPositionals goes from 7
params to 3, companionSupportsLimit from 5 to 2.
- Cache commandSupportsSearch(command.Help) in a local bool — was
called twice in the error_path block, both calls re-walked the
full help text.
- Trim 15-line extractFirstIDFromJSON doc-comment listing the path
order; inline 'Path N' markers below already cover it.
- Trim 4-line index-math comment in resolveCommandPositionals to a
one-liner; the variable name (siblingKey) carries the rest.
- Drop multi-line 'chained companion walk' narration above the
resolver call site — the function's own doc comment says it.
- Drop test prose comment naming sibling tests in
TestResolveCommandPositionalsSkipPaths (will rot).
- Move companionCache field-level rationale onto the field doc-
comments (results, helps) instead of the type-level block.
- Drop 'U2's chained companion walk' ticket reference from
crossAPIListVerbs comment per AGENTS.md (no task IDs in code).
- Drop 'resolveStatus is the outcome enum' comment that restated
the type name.
- Trim camelCase exampleValue comment from 5 lines to 2.
- Fold cacheKey NUL-separator rationale into the trailing-comment
on the assignment.
Efficiency:
- isSearch + suppliedJSON cached locally in error_path; avoids
walking command.Help twice per command.
Skipped findings: liveDogfoodUsageSuffix vs runtime_commands.go
regex (different return shapes; risk to verify path); isIDShape
predicate spread across two packages (would require new shared
package for 4 lines — premature abstraction at 2 sites).
Behaviour preserved: 2949 tests pass, golden verify 11/11, vet
clean, golangci-lint clean.
* fix(cli): apply ce-code-review findings (F1-F3, F5-F9)
Code-review pass on the WU-2 PR via /ce-code-review (9 reviewers).
Applied 4 safe_auto fixes silently then auto-resolved 8 of 9
remaining findings with best judgment. F4 (test coverage gaps for
resolve-success / cache-hit / U3 integration) deferred to follow-up
since it requires extending the fake-binary fixture.
Safe-auto (already applied):
- Removed 'TMDb' brand name from cinemaListVerbs comment per
AGENTS.md 'no API names in reusable artifacts'.
- Hoisted extractFlagNames regex to package-level extractFlagNameRe
to stop recompiling 4x per search-shape command.
- Cached flagNames/hasQueryFlag/suppliedJSON locals in error_path
block to remove redundant scans of the same help text.
- Added cross-file comment on isIDShape predicate explaining
divergence from generator.go's exampleValue (no spec type info
available from CLI help text).
Best-judgment fixes:
- F1 (P1, adversarial): commandSupportsSearch over-matched via
extractFlagNames regexing the entire help text. A delete <id>
command whose help mentioned 'see widgets list --query=foo' would
be misclassified as search-shape, then its error_path probe would
silently PASS regardless of behavior. Added extractFlagsSection
to scope flag-name detection to the actual Flags: block. Tests
cover Examples and Long sections with --query references.
- F2 (P1, reliability): quick-level verdict gate
'Passed+Skipped >= 5' was structurally unreachable when
liveDogfoodQuickCommands returned 1 command (4 entries max).
Relaxed to 'Passed+Skipped >= min(5, MatrixSize)' while keeping
the MatrixSize >= 4 floor that blocks pathological all-skip
matrices. Added 1-command-all-pass test case.
- F3 (P1, reliability): companion timeout (exitCode=-1) returned
resolveSkip but did not write a negative-cache sentinel. Sibling
get-shape commands sharing the failed companion re-spawned the
30s subprocess. Now writes empty-string sentinel; cache-hit check
detects the sentinel and skips immediately.
- F5 (P2, adversarial): exampleValue type fence was '== string ||
== empty', which rejected typed-uuid/guid string-shaped types and
fell through to literal example-value. Inverted to 'not numeric
or boolean', which keeps the bool/int exclusion while accepting
any string-shaped type. Tests cover uuid and guid type values.
- F6 (P2, adversarial): a future API shipping --query as a filter
on a mutating verb (e.g., delete --query=...) would be
misclassified as search-shape. The error_path probe would run
'<verb> --query=__printing_press_invalid__' against the LIVE API.
Added isMutatingLeaf deny-list overlay covering delete/destroy/
remove/create/add/new/update/patch/edit/set/modify/replace.
- F8 (P2, maintainability): resolveStatus int enum had 2 values
with one if-status==resolveSkip check. Replaced with bool 'skip'
return value. Call sites and tests updated.
- F9 (P2, maintainability): runLiveDogfoodCommand took 6 params
then immediately constructed a resolveCtx from 5 of them.
Refactored to take resolveCtx directly; the struct that
resolveCommandPositionals already used now threads through the
outer call too. RunLiveDogfood builds the ctx once and reuses.
Skipped: F7 (merge crossAPI/cinema verb maps) contradicts the
deliberate split chosen during the prior /simplify walkthrough.
F4 (test fixture extension for resolve-success / cache-hit /
U3 integration) requires writing new fake-binary scripts; deferred
to follow-up.
Run artifact: /tmp/compound-engineering/ce-code-review/20260504-141500-907bca25/
Behaviour preserved: 2954 tests pass, golden verify 11/11, vet
clean, golangci-lint clean.
Files touched
A docs/plans/2026-05-04-003-fix-live-dogfood-matrix-accuracy-plan.mdA internal/generator/example_value_test.goM internal/generator/generator.goM internal/pipeline/dogfood.goM internal/pipeline/live_dogfood.goM internal/pipeline/live_dogfood_test.goM testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_list-project.goM testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_update-project.go
Diff
commit 7572b1e09ba4ae8ef4e29fe02e73fa799993e14e
Author: Trevin Chow <trevin@trevinchow.com>
Date: Mon May 4 15:30:18 2026 -0700
fix(cli): live-dogfood matrix accuracy (WU-2, F4+F5) (#577)
* docs(cli): add plan for live dogfood matrix accuracy (WU-2)
Captures the implementation plan for sub-issue #573 (parent retro
#571). Covers the four absorbed findings: camelCase ID recognition
with type fence (U1), chained companion walk for multi-positional
gets with per-companion cache (U2), search-aware error_path
dispatch (U3), and quick-level verdict gate restated to keep
skip-with-reason from flipping PASS to FAIL (U4).
* fix(cli): U1 — camelCase ID recognition in exampleValue with type fence
The exampleValue function (internal/generator/generator.go) only
recognised `*_id` (snake) and bare `id` as ID-shaped positionals.
camelCase names like `movieId`, `seriesId`, `personId` lower to
`movieid`/`seriesid`/`personid` and missed the existing
HasSuffix("_id") check, falling through to the literal
`example-value` placeholder. Every printed CLI generated from an
OpenAPI spec with camelCase ID positionals (TMDb, Notion, Linear,
Pagliacci, ESPN) shipped with broken Examples that live dogfood
probes treated as positional values, producing 404s.
Adds a third clause to the ID branch:
HasSuffix(nameLower, "id") && len(nameLower) > 2 &&
(p.Type == "string" || p.Type == "")
The string-type fence prevents boolean/numeric positionals like
`paid`, `valid`, `creditValid` from getting UUIDs — they flow
into their existing type-specific branches (boolean → "true",
integer → "42" or count-branch "50"). Empty Type passes through
to handle legacy YAML specs without explicit type info.
Pure-string false positives (`void`, `acid`, `arid` as required
URL path params) are accepted: a UUID at the wrong path produces
a clear 404 at verify time, strictly better than today's literal
`example-value`. Add a denylist if a real-spec FP surfaces.
Golden fixtures updated for projectId/taskId positionals in the
golden-api spec (intentional byte-level diff of two Example
lines).
* fix(cli): U4 — restate quick-level live-dogfood verdict gate
The previous gate required exactly MatrixSize == 6 with Passed >= 5
for a quick-level PASS. With U2's coming skip-with-reason paths
(get-shape commands without a sibling companion), MatrixSize can
drop below 6 even when nothing actually failed — flipping the
verdict from PASS to FAIL purely from increased Skip count.
Restate the switch:
Failed > 0 || MatrixSize == 0 → FAIL (failed-or-empty wins)
quick + (Passed + Skipped) >= 5 && MatrixSize >= 4 → PASS
quick fallthrough → FAIL
Skipped now counts toward the 5-entry quorum. The MatrixSize floor
of 4 guards against pathological all-skip outcomes — too few real
test runs to trust. Failed-dominance is preserved at every level.
Tests cover: classic 6-pass; 5-pass + 1-skip; 4-pass + 2-skip;
3-pass + 3-skip floor breach; pass + fail dominance; all-skip;
full-level pass and fail unaffected.
* fix(cli): U3 — search-aware error_path dispatch
The error_path probe ran every command with __printing_press_invalid__
and required a non-zero exit. Search-shape commands canonically return
exit 0 with empty (or fallback-recency) results on no-match — Movie
Goat shipped 6 false-fail entries on movies/people/tv/multi/search/
auth-set-token because of this uniform strategy.
Add commandSupportsSearch (--query flag OR <query> placeholder) and
branch the error_path strategy:
Search-shape:
exit non-zero → PASS (API rejected token)
exit 0 + valid JSON when --json → PASS (any result shape)
exit 0 + invalid JSON when --json → FAIL
exit 0 + non-JSON without --json → PASS
Mutation/plain-get (today's behavior preserved):
non-zero exit required.
The 'exit 0 + non-empty results = FAIL' branch from earlier drafts
is dropped: real-world content/feed APIs return recent items as a
fallback for unmatched queries, and treating non-empty as a failure
would block legitimate ships. Detecting broken filter logic is a
unit-test concern, not a live-dogfood signal.
Tests cover the heuristic across positive (--query flag, <query>
positional) and negative (list with --limit, dispatch with --queue,
delete) cases. The exact-match check on --queue confirms the flag-
name extractor doesn't substring-match.
* fix(cli): U2 — chained companion walk for live-dogfood id positionals
The live-dogfood matrix probed get-shape commands with the literal
`example-value` placeholder, producing 5 false-fail entries on
Movie Goat (movies get, people get, tv get, tv seasons get, export).
Even with U1's UUID-shape placeholder, real APIs 404 on synthetic
ids — the only way to actually test the get path live is to source
a real id from the API.
Add a chained companion walk that, for each id-shape positional in
the get-shape command's Usage line, sources a real id by running a
sibling list-shape companion under --json. The walk threads earlier-
resolved ids into later list calls, so nested resources work
end-to-end (e.g., projects/tasks/update <pid> <tid> sources pid
from `projects list` then tid from `projects tasks list <pid>`).
New mechanism in internal/pipeline/live_dogfood.go:
- buildSiblingMap groups commands by joined parent path for O(1)
sibling lookup.
- crossAPIListVerbs (list, all, index, query, find, search, discover,
browse, recent, feed) cover modern API conventions; cinemaListVerbs
retain TMDb-specific verbs (popular, trending, top_rated, etc.) so
the original motivating CLI keeps coverage.
- resolveCommandPositionals walks the chain: extract Usage placeholders,
reject non-id-shape mid-chain, find a list companion at each depth,
invoke it (with parent ids as positional context, --json, and
--limit 1 when supported), extract first id, accumulate.
- extractFirstIDFromJSON tries seven canonical shapes (results, top-
level array, items, data, list, GraphQL .data.<any>.nodes,
GraphQL .data.<any>.edges with depth-2 walk for Relay viewer.repos
shapes). Numeric ids decoded via UseNumber to preserve large
values (snowflake > 2^53) without scientific-notation rounding.
- companionCache (run-scoped) memoizes (companion-argv → id) so
sibling get-shape commands share lookups; --help is also cached
per companion path.
- substitutePositionals walks happyArgs left-to-right after
command.Path, replacing the first N non-flag args with the
resolved ids, preserving interleaved flags.
Wired into runLiveDogfoodCommand: after liveDogfoodHappyArgs, call
the resolver. On resolveSkip, happy_path + json_fidelity become
Status=Skip with a structured reason naming the failed link
('no list companion at depth N for <name>' / 'list companion
failed at depth N: exit X' / 'no id parseable from companion at
depth N' / 'non-id positional <name> at depth N'). error_path runs
independently (its own positional gate, U3-aware strategy).
Tests cover: TMDb/REST/Stripe/long-tail/GraphQL-Shopify/GraphQL-
Relay id extraction shapes; large numeric ids; sibling map
construction; companion-leaf detection; positional substitution
(single, two, before-flag, no-positionals); resolver skip paths
(no-positional pass-through, non-id mid-chain, no-companion,
path-segment shortage). All 935 pipeline package tests pass.
* refactor(cli): simplify per /ce-simplify-code review
Apply 3-reviewer (reuse / quality / efficiency) findings:
Quality (9 fixes):
- Extract resolveCtx struct — resolveCommandPositionals goes from 7
params to 3, companionSupportsLimit from 5 to 2.
- Cache commandSupportsSearch(command.Help) in a local bool — was
called twice in the error_path block, both calls re-walked the
full help text.
- Trim 15-line extractFirstIDFromJSON doc-comment listing the path
order; inline 'Path N' markers below already cover it.
- Trim 4-line index-math comment in resolveCommandPositionals to a
one-liner; the variable name (siblingKey) carries the rest.
- Drop multi-line 'chained companion walk' narration above the
resolver call site — the function's own doc comment says it.
- Drop test prose comment naming sibling tests in
TestResolveCommandPositionalsSkipPaths (will rot).
- Move companionCache field-level rationale onto the field doc-
comments (results, helps) instead of the type-level block.
- Drop 'U2's chained companion walk' ticket reference from
crossAPIListVerbs comment per AGENTS.md (no task IDs in code).
- Drop 'resolveStatus is the outcome enum' comment that restated
the type name.
- Trim camelCase exampleValue comment from 5 lines to 2.
- Fold cacheKey NUL-separator rationale into the trailing-comment
on the assignment.
Efficiency:
- isSearch + suppliedJSON cached locally in error_path; avoids
walking command.Help twice per command.
Skipped findings: liveDogfoodUsageSuffix vs runtime_commands.go
regex (different return shapes; risk to verify path); isIDShape
predicate spread across two packages (would require new shared
package for 4 lines — premature abstraction at 2 sites).
Behaviour preserved: 2949 tests pass, golden verify 11/11, vet
clean, golangci-lint clean.
* fix(cli): apply ce-code-review findings (F1-F3, F5-F9)
Code-review pass on the WU-2 PR via /ce-code-review (9 reviewers).
Applied 4 safe_auto fixes silently then auto-resolved 8 of 9
remaining findings with best judgment. F4 (test coverage gaps for
resolve-success / cache-hit / U3 integration) deferred to follow-up
since it requires extending the fake-binary fixture.
Safe-auto (already applied):
- Removed 'TMDb' brand name from cinemaListVerbs comment per
AGENTS.md 'no API names in reusable artifacts'.
- Hoisted extractFlagNames regex to package-level extractFlagNameRe
to stop recompiling 4x per search-shape command.
- Cached flagNames/hasQueryFlag/suppliedJSON locals in error_path
block to remove redundant scans of the same help text.
- Added cross-file comment on isIDShape predicate explaining
divergence from generator.go's exampleValue (no spec type info
available from CLI help text).
Best-judgment fixes:
- F1 (P1, adversarial): commandSupportsSearch over-matched via
extractFlagNames regexing the entire help text. A delete <id>
command whose help mentioned 'see widgets list --query=foo' would
be misclassified as search-shape, then its error_path probe would
silently PASS regardless of behavior. Added extractFlagsSection
to scope flag-name detection to the actual Flags: block. Tests
cover Examples and Long sections with --query references.
- F2 (P1, reliability): quick-level verdict gate
'Passed+Skipped >= 5' was structurally unreachable when
liveDogfoodQuickCommands returned 1 command (4 entries max).
Relaxed to 'Passed+Skipped >= min(5, MatrixSize)' while keeping
the MatrixSize >= 4 floor that blocks pathological all-skip
matrices. Added 1-command-all-pass test case.
- F3 (P1, reliability): companion timeout (exitCode=-1) returned
resolveSkip but did not write a negative-cache sentinel. Sibling
get-shape commands sharing the failed companion re-spawned the
30s subprocess. Now writes empty-string sentinel; cache-hit check
detects the sentinel and skips immediately.
- F5 (P2, adversarial): exampleValue type fence was '== string ||
== empty', which rejected typed-uuid/guid string-shaped types and
fell through to literal example-value. Inverted to 'not numeric
or boolean', which keeps the bool/int exclusion while accepting
any string-shaped type. Tests cover uuid and guid type values.
- F6 (P2, adversarial): a future API shipping --query as a filter
on a mutating verb (e.g., delete --query=...) would be
misclassified as search-shape. The error_path probe would run
'<verb> --query=__printing_press_invalid__' against the LIVE API.
Added isMutatingLeaf deny-list overlay covering delete/destroy/
remove/create/add/new/update/patch/edit/set/modify/replace.
- F8 (P2, maintainability): resolveStatus int enum had 2 values
with one if-status==resolveSkip check. Replaced with bool 'skip'
return value. Call sites and tests updated.
- F9 (P2, maintainability): runLiveDogfoodCommand took 6 params
then immediately constructed a resolveCtx from 5 of them.
Refactored to take resolveCtx directly; the struct that
resolveCommandPositionals already used now threads through the
outer call too. RunLiveDogfood builds the ctx once and reuses.
Skipped: F7 (merge crossAPI/cinema verb maps) contradicts the
deliberate split chosen during the prior /simplify walkthrough.
F4 (test fixture extension for resolve-success / cache-hit /
U3 integration) requires writing new fake-binary scripts; deferred
to follow-up.
Run artifact: /tmp/compound-engineering/ce-code-review/20260504-141500-907bca25/
Behaviour preserved: 2954 tests pass, golden verify 11/11, vet
clean, golangci-lint clean.
---
...04-003-fix-live-dogfood-matrix-accuracy-plan.md | 450 +++++++++++++++++
internal/generator/example_value_test.go | 72 +++
internal/generator/generator.go | 11 +-
internal/pipeline/dogfood.go | 5 +-
internal/pipeline/live_dogfood.go | 556 +++++++++++++++++++--
internal/pipeline/live_dogfood_test.go | 433 ++++++++++++++++
.../internal/cli/projects_tasks_list-project.go | 2 +-
.../internal/cli/projects_tasks_update-project.go | 2 +-
8 files changed, 1492 insertions(+), 39 deletions(-)
diff --git a/docs/plans/2026-05-04-003-fix-live-dogfood-matrix-accuracy-plan.md b/docs/plans/2026-05-04-003-fix-live-dogfood-matrix-accuracy-plan.md
new file mode 100644
index 00000000..dbccb32d
--- /dev/null
+++ b/docs/plans/2026-05-04-003-fix-live-dogfood-matrix-accuracy-plan.md
@@ -0,0 +1,450 @@
+---
+title: Live dogfood matrix accuracy — camelCase ID examples + kind-aware error_path
+type: fix
+status: active
+date: 2026-05-04
+deepened: 2026-05-04
+origin: https://github.com/mvanhorn/cli-printing-press/issues/573
+---
+
+# Live dogfood matrix accuracy — camelCase ID examples + kind-aware error_path
+
+## Summary
+
+Cut false-fail noise from the live dogfood matrix on every CLI with camelCase ID positionals or search-family commands. Three coordinated fixes: (1) `exampleValue` recognises camelCase ID suffixes (with a string-type fence) so emitted Examples carry a UUID-shape placeholder instead of the literal `example-value`; (2) the matrix walks the sibling list-shape chain to source a real id for each id-shape positional in a `get`-shape command, with a per-companion cache so siblings share lookups, and skip-with-reason when any link in the chain is unreachable; (3) the error_path strategy branches on command shape — search commands accept exit 0 (any result shape) OR non-zero exit, mutating commands keep today's "non-zero exit" expectation. Also restates the quick-level verdict gate so skip-with-reason no longer flips PASS to FAIL.
+
+---
+
+## Problem Frame
+
+The Movie Goat run produced 11 false-fail entries in dogfood: 5 from happy_path/json_fidelity probing get-shape commands with the placeholder string `example-value` (TMDb 404'd), and 6 from error_path probing search commands with `__printing_press_invalid__` (those commands correctly returned exit 0 with empty results — canonical Unix UX). Both classes are scorer accuracy bugs that recur on every CLI matching the same shapes (notion, linear, espn, pagliacci all carry camelCase IDs and search commands). The polish/promote gates trust the dogfood verdict, so the noise weakens an already load-bearing signal.
+
+---
+
+## Requirements
+
+- R1. Specs declaring a camelCase ID positional (`movieId`, `seriesId`, `personId`) emit Examples that use the UUID-shape placeholder, not the literal `example-value`. The recognition rule includes a string-type fence so non-string positionals (booleans like `paid`/`valid`, numerics) flow into their own type branches instead of getting UUIDs.
+- R2. Specs that already use snake_case (`movie_id`) or the bare `id` continue to emit the UUID-shape placeholder unchanged.
+- R3. Live dogfood for a get-shape command sources a real id for each id-shape positional by walking the sibling list-shape chain — a single-positional get queries one list companion; a nested-resource get (e.g., `projects tasks update <project-id> <task-id>`) walks the chain, threading earlier-resolved ids into later list calls. A per-companion cache shares lookups across sibling get-shape commands. happy_path and json_fidelity then exercise the get path against real data.
+- R4. When any link in the chain is unreachable (no sibling at the appropriate depth, companion errors, or no parseable id in the response), happy_path and json_fidelity for that get-shape command skip-with-reason, rather than failing with the placeholder.
+- R5. Live dogfood error_path for a search-shaped command (heuristic: `--query` flag is present, or Usage suffix contains a `<query>` placeholder) accepts exit 0 (regardless of result shape) OR non-zero exit as a pass. Only malformed JSON when `--json` was supplied counts as a fail.
+- R6. Live dogfood error_path for non-search commands (mutating writes/deletes, plain `get` with bogus id) preserves today's strategy: non-zero exit is required.
+- R7. The quick-level verdict gate is restated as `Failed == 0 && (Passed + Skipped) >= 5 && MatrixSize >= 4` so the new skip-with-reason paths cannot flip a PASS run to FAIL purely from increased Skip count.
+
+---
+
+## Scope Boundaries
+
+- Does NOT remove error_path coverage entirely — it adapts strategy by command shape.
+- Does NOT fix the `<resource>` placeholder used by `export` (separate concern, lower frequency, called out as out-of-scope in #573).
+- Does NOT extend `agent-context`'s emitted schema with a `kind` field. Search detection uses a help-text heuristic local to the live_dogfood subprocess. Schema extension would touch `agent_context.go.tmpl` and every printed CLI's contract for marginal benefit; revisit if a future scorer also wants kind-aware behavior.
+- Does NOT migrate live_dogfood to consume `Param.Default` from the spec (adjacent food52-retro work — `liveDogfoodHappyArgs` still parses `--help` examples, not the spec). The substitution introduced in U2 is a runtime override on top of whatever placeholder the example carries.
+
+---
+
+## Context & Research
+
+### Relevant Code and Patterns
+
+- `internal/generator/generator.go:2944` — `exampleValue(p spec.Param) string`. The first branch checks `strings.HasSuffix(nameLower, "_id") || nameLower == "id"`. CamelCase names lower to `movieid`/`seriesid`/`personid` and miss this branch.
+- `internal/generator/generator.go:2982` — `exampleLine` consumes `exampleValue` for every positional. Used for both human Examples and the SKILL template's example block.
+- `internal/pipeline/live_dogfood.go:195` — `runLiveDogfoodCommand`. Today's flow: extract Examples section from `--help`, parse the first runnable line via `liveDogfoodHappyArgs`, then run happy/json/error probes uniformly.
+- `internal/pipeline/live_dogfood.go:350` — `liveDogfoodHappyArgs(command) ([]string, bool)`. Returns the parsed example args; no positional substitution today.
+- `internal/pipeline/live_dogfood.go:260` — error_path branch. Appends `__printing_press_invalid__` and asserts non-zero exit uniformly across kinds.
+- `internal/pipeline/runtime_commands.go:189` — `extractPositionalPlaceholders` already strips `[--flag=<val>]` descriptors and yields `<id>`/`<query>` placeholder names from a Usage suffix. Reusable for both U2 and U3.
+- `internal/pipeline/dogfood.go:1779` — `extractFlagNames(text)` returns flag names from `--help`. Reusable for the `--query`-flag side of the search heuristic.
+- `internal/pipeline/dogfood.go:1747` — `extractExamplesSection(helpOutput)` returns the Examples section. Already used by live_dogfood.
+
+### Institutional Learnings
+
+- **food52 retro #20260427-014521** — proposed canonical mock values for verify positionals; `canonicalargs` registry plus `Param.Default` lookup chain (see `internal/pipeline/runtime_commands.go:resolvePositionalValue`). This plan extends that direction into live dogfood by sourcing real ids from sibling list calls. The two are complementary: verify-mock uses static fixtures; live dogfood uses runtime-discovered fixtures.
+- **AGENTS.md anti-pattern** — "Never change the machine for one CLI's edge case." The companion-leaf allowlist must be the cross-API set, not a TMDb-specific list.
+- **AGENTS.md golden harness** — "Run `scripts/golden.sh verify` whenever a change may affect CLI command output." U1 changes Example-line bytes for the golden fixture's `projectId`/`taskId` positionals; the fixture update is intentional and documented in U1's verification.
+
+### External References
+
+- TMDb v3 API (`api.themoviedb.org/3`) — primary live test target for U2/U3. Movie list endpoints (`/movie/popular`, `/movie/top_rated`, `/movie/now_playing`) return `{"results":[{"id":N, ...}]}`. Search endpoints (`/search/movie`) return exit 0 with `{"results":[]}` on no-match.
+
+---
+
+## Key Technical Decisions
+
+- **camelCase recognition rule, with type fence.** Extend the ID branch in `exampleValue` to: `nameLower == "id"` (existing) OR `strings.HasSuffix(nameLower, "_id")` (existing) OR `strings.HasSuffix(nameLower, "id") && len(nameLower) > 2 && (p.Type == "string" || p.Type == "")` (new). Rationale: matches `movieid`/`seriesid`/`personid`/`userid`/`groupid`/`pageid`/`issueid`/`eventid`/`teamid` — the cross-API set. The `string`-or-empty type fence prevents boolean positionals like `paid`/`valid` and numeric positionals from getting UUIDs (they fall through to the existing `boolean` / `integer` branches). Pure-string false positives (`acid`, `arid`, `void` as required URL path params) are accepted as a documented risk — they would produce one Example with a UUID and a clear 404 signal at verify time, strictly better than today's literal `example-value`. Add a denylist only when a real-spec FP surfaces.
+- **Search detection is help-text heuristic, not schema.** Reuse `extractFlagNames` to detect `--query`, and `extractPositionalPlaceholders` to detect `<query>` in the Usage suffix. Either signal flips the command into search-shaped. Local to the live_dogfood subprocess, no contract change. Defers schema-level kind dispatch until a second scorer needs it.
+- **Companion-leaf allowlist — two named sets.** Detection: the appropriate ancestor's parent path plus any leaf below.
+ - `crossAPIListVerbs = {list, all, index, query, find, search, discover, browse, recent, feed}` — generic verbs that appear across modern API conventions (Notion's `query`, Stripe's `list`, content-feed APIs' `recent`/`feed`/`browse`, Algolia/Elastic-style `search`).
+ - `cinemaListVerbs = {popular, trending, top_rated, latest, now_playing, upcoming, airing_today, on_the_air}` — TMDb / cinema-API specific. Kept because TMDb's printed CLI does not expose a plain `list` leaf.
+ - The union is checked at companion-resolve time. Future additions go to the right bucket (generic vs domain-specific) so the cinema bucket doesn't leak coverage promises across non-cinema CLIs.
+- **ID extraction try-list.** In order: `.results[0].id`, `.[0].id`, `.items[0].id`, `.data[0].id`, `.list[0].id`, `.data.<any>.nodes[0].id`, `.data.<any>.edges[0].node.id`. First match wins; coerced to string. Rationale: covers TMDb (`results`), top-level array (`[0]`), GitHub REST (`items`), Stripe (`data`), the long-tail (`list`), and GraphQL connection shapes (`.data.<resource>.nodes` for Shopify/Linear/Notion-database, `.data.<resource>.edges[].node` for GitHub GraphQL/Relay-style). The `<any>` wildcard means: walk any single-key field under `.data` and try the child path. When all seven miss, fall through to skip-with-reason — better than substituting a guessed value.
+- **Multi-positional resolution via chained companion walk.** For each id-shape positional in the get-shape command, walk up to the appropriate ancestor list-shape sibling, run it with already-resolved parent ids threaded into its argv as positional context, and extract the next id from the response. Bounded by command-tree depth (typically 1-3 levels). Example for `projects tasks update <project-id> <task-id>`: source `<project-id>` from `projects list --json`, source `<task-id>` from `projects tasks list <project-id> --json`, then run `projects tasks update <project-id> <task-id>`. Non-id-shape positionals in the chain (e.g., `<query>` somewhere in the path) abort the chain to skip-with-reason. Rationale: nested resources are common (subresources, comments, items, members); without the chain, a meaningful slice of get-shape commands across the catalog stay at FAIL. The chained walk converts those to live-tested PASS.
+- **Per-companion cache.** A run-scoped `map[string]string` keyed by the companion's full argv (path joined with parent ids) caches the extracted id. Sibling get-shape commands sharing a parent reuse one companion subprocess. Combined with the chained walk, this caps total companion subprocess count at O(unique-parent-paths × depth), not O(get-commands × depth).
+- **Skip-with-reason on chain failure.** When any link in the chain fails (no sibling at depth, companion errors, no id parseable), happy_path and json_fidelity become `Status=Skip` with a structured reason naming the failed link (`"no list companion at depth N for <name>"`, `"list companion failed at depth N: exit <X>"`, `"no id parseable from companion at depth N"`). Today's behavior is FAIL with the placeholder; skip-with-reason is strictly better signal because the fail was always a scorer artifact, never a real defect.
+- **Quick-level verdict gate restated.** Today's gate `MatrixSize == 6 && Passed >= 5` flips to FAIL whenever a Skip drops MatrixSize below 6 — a regression directly caused by this PR's new skip paths. Restate to `Failed == 0 && (Passed + Skipped) >= 5 && MatrixSize >= 4`: any non-failure pattern with at least 4 entries (allowing for one or two skips) and zero Failed counts as PASS. Lower MatrixSize floor (4) accommodates the case where both quick-selected commands skip happy_path + json_fidelity due to no companion. The `Failed == 0` clause keeps real failures fatal.
+- **Error_path skip-on-no-positional preserved.** Today's branch already skips error_path when the command has no positional. That's correct and unchanged. The new search dispatch only branches the strategy when a positional IS present.
+- **Search-shape error_path: exit 0 = PASS regardless of result shape.** When the command is search-shaped, accept exit 0 with any result shape (empty or non-empty results) OR non-zero exit as PASS. Only fail on exit 0 with malformed JSON when `--json` was supplied. Drops the "exit 0 + non-empty results = FAIL" branch from earlier drafts because real-world content/feed APIs intentionally return recent items as a fallback for unmatched queries (canonical UX). Detecting broken filter logic is a unit-test concern, not a live-dogfood signal — a search-error_path test that fires on production data shape variance would block legitimate ships.
+
+---
+
+## Open Questions
+
+### Resolved During Planning
+
+- **Should `exampleValue` also handle `Id` exactly (PascalCase single token)?** Resolved: `nameLower == "id"` already covers `Id` after lowercasing. No additional clause needed.
+- **Should U2 substitution run even when the placeholder isn't ID-shaped?** Resolved: only run substitution when the positional name (extracted from Usage) ends in `id` (the same check U1 wires into the generator). Rationale: substituting a real id into a `<query>` slot would corrupt the test; the heuristic is "only run companion lookup for id-shaped positionals." Non-id-shape positionals encountered mid-chain abort the walk to skip-with-reason.
+- **Multi-positional commands.** Resolved: chained companion walk (see Key Technical Decisions) — each id-shape positional is sourced from the appropriate ancestor list-shape sibling, with earlier-resolved ids threaded into later list calls. Skip-with-reason if any link fails.
+- **Companion subprocess fan-out.** Resolved: per-companion cache keyed by full argv (path + parent ids) so sibling get-shape commands share one subprocess. Cap is O(unique-parent-paths × depth), not O(get-commands × depth).
+- **Companion `--limit` flag detection.** Resolved (in U2 Approach): when the companion supports `--limit` (detected via `extractFlagNames(companion.Help)`), run with `--limit 1` to minimize the API call. If absent, run without and accept the full response.
+- **Quick-level verdict gate.** Resolved (R7): restate as `Failed == 0 && (Passed + Skipped) >= 5 && MatrixSize >= 4` so skip-with-reason cannot flip PASS to FAIL purely from increased Skip count.
+- **Search error_path strategy on non-empty results.** Resolved: exit 0 = PASS regardless of result shape. Detecting broken filter logic is a unit-test concern, not a live-dogfood signal. Real-world content/feed APIs return fallback items on unmatched queries.
+- **Empty-results predicate wording.** Resolved: the predicate looks for "any of {results, items, data, list} key whose value is an empty array" — not strict object equality. TMDb's `{"results":[],"total_results":0,"page":1}` matches because `results` exists with empty array; multi-key responses are not rejected.
+- **Does the heuristic-based search detection misclassify any existing commands?** Resolved: scan the catalog for commands carrying both `--query` flag and a mutating verb. None found in current catalog (search/list/find use `--query`; create/delete/update use bodies). Risk is low; reassessable from future retros.
+
+### Deferred to Implementation
+
+- **JSON id type coercion for non-string ids.** TMDb returns `id` as a number; some APIs return strings. The extracted value is converted via `fmt.Sprint` so both shapes serialize correctly into the substituted positional. Confirm at impl time that `json.Decoder.UseNumber()` is wired (not the default float64 decode) so very large ids (e.g., Twitter snowflake > 2^53) survive coercion without scientific-notation rounding.
+- **Whether to retain the placeholder result on companion-success but get-failure.** When the chain succeeds and substituted ids are sourced cleanly but the get probe still fails (genuine API/auth issue), happy_path stays FAIL with the standard exit-code reason. No special "chain succeeded but get failed" status — the existing failure signal is correct.
+- **Cache-population ordering.** The cache is built lazily as commands run. The matrix iterates alphabetically; sibling get-shape commands sharing a parent will populate the cache on the first call and reuse on subsequent calls. Confirm at impl time that no parallel iteration breaks the cache contract (today's iteration is sequential).
+
+---
+
+## Implementation Units
+
+- U1. **camelCase ID recognition in `exampleValue`**
+
+**Goal:** Recognise camelCase ID suffixes so generated Examples carry the UUID-shape placeholder instead of `example-value`.
+
+**Requirements:** R1, R2
+
+**Dependencies:** None.
+
+**Files:**
+- Modify: `internal/generator/generator.go` (`exampleValue` at line 2944)
+- Test: `internal/generator/generator_test.go` (add or extend `TestExampleValue`)
+- Update (golden): `testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_update-project.go`
+- Update (golden): `testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_list-project.go`
+- Update (golden, if any other files reference `example-value` for these positionals): inspect via `grep -r "example-value" testdata/golden/expected/`
+
+**Approach:**
+- Extend the first `if` in `exampleValue` to a three-clause check with type fence:
+ - `nameLower == "id"` (existing)
+ - OR `strings.HasSuffix(nameLower, "_id")` (existing)
+ - OR `strings.HasSuffix(nameLower, "id") && len(nameLower) > 2 && (p.Type == "string" || p.Type == "")` (new — camelCase + string-type fence)
+- The three clauses are flat OR — preserve order so the existing two paths stay first and the new one is the catch-all for camelCase shapes.
+- The `(p.Type == "string" || p.Type == "")` fence prevents boolean positionals (`paid`, `valid`) and numeric positionals from getting UUIDs — they fall through to the existing `p.Type == "boolean"` / `p.Type == "integer"` branches later in the function. Spec authors that don't set `p.Type` (legacy internal YAML or browser-sniffed specs) get the UUID by default; this is the safer fallback for unknown-type positionals.
+- After the change, run `scripts/golden.sh verify` — expect a deterministic byte-level diff in the two project Example lines (`example-value` → `550e8400-e29b-41d4-a716-446655440000` for `projectId` and `taskId`). Run `scripts/golden.sh update` to refresh, inspect, and explain the diff in the PR body.
+
+**Patterns to follow:**
+- Existing `exampleValue` style (flat string-suffix checks, no helper extraction).
+- Golden-update workflow per AGENTS.md "Golden Output Harness" section.
+
+**Test scenarios:**
+- Happy path — `exampleValue({Name: "movieId", Type: "string"})` returns `550e8400-e29b-41d4-a716-446655440000`.
+- Happy path — `exampleValue({Name: "seriesId", Type: "string"})` returns the UUID.
+- Happy path — `exampleValue({Name: "personId", Type: "string"})` returns the UUID.
+- Edge — `exampleValue({Name: "movieId"})` (no Type) returns the UUID — empty Type passes the fence.
+- Edge — `exampleValue({Name: "id", Type: "string"})` returns the UUID (existing equality clause unchanged).
+- Edge — `exampleValue({Name: "movie_id", Type: "string"})` returns the UUID (existing snake_case clause unchanged).
+- Edge — `exampleValue({Name: "ID", Type: "string"})` returns the UUID (case-insensitive via `nameLower`).
+- Edge — `exampleValue({Name: "userIdentifier", Type: "string"})` does NOT return UUID — name does not end in `id`. Falls through to default `"example-value"`.
+- Edge — `exampleValue({Name: "", Type: "string"})` does NOT return UUID — empty string fails `len > 2`.
+- Edge — `exampleValue({Name: "ix", Type: "string"})` does NOT return UUID — does not end in `id`.
+- Type fence — `exampleValue({Name: "paid", Type: "boolean"})` does NOT return UUID; falls through to the boolean branch and returns `"true"`. Same for `valid`/`unpaid`.
+- Type fence — `exampleValue({Name: "amountId", Type: "integer"})` does NOT return UUID; falls through to the integer branch and returns `"42"`.
+- Type fence — `exampleValue({Name: "creditValid", Type: "boolean"})` does NOT return UUID; ends in `id` but type is boolean — returns `"true"`.
+- Accepted FP — `exampleValue({Name: "void", Type: "string"})` DOES return UUID. Documented in Risks; if a real-spec FP surfaces, add a denylist entry.
+
+**Verification:**
+- `go test ./internal/generator/...` passes including new cases.
+- `scripts/golden.sh verify` produces diffs only in the two `projectId`/`taskId` example lines; running `scripts/golden.sh update` produces a clean updated fixture; manual inspection confirms the only changed bytes are the placeholder values.
+
+---
+
+- U2. **Chained companion walk — multi-positional ID resolution in live dogfood**
+
+**Goal:** For each id-shape positional in a get-shape command, source a real id by walking the sibling list-shape chain — single-positional gets query one list companion; nested-resource gets walk the chain, threading earlier-resolved ids into later list calls. A per-companion cache shares lookups across siblings. Skip-with-reason if any link fails. happy_path and json_fidelity exercise the get path against real data instead of failing on a placeholder.
+
+**Requirements:** R3, R4
+
+**Dependencies:** None — independent of U1 in code (U1 changes the placeholder bytes; U2 substitutes them at runtime regardless of which placeholder shipped).
+
+**Files:**
+- Modify: `internal/pipeline/live_dogfood.go` (extend `RunLiveDogfood` to build a sibling index and companion cache; extend `runLiveDogfoodCommand` signature to accept them; add `resolveCommandPositionals` helper, `findListCompanionAtDepth` helper, and `extractFirstIDFromJSON` helper)
+- Test: `internal/pipeline/live_dogfood_test.go` (extend the fake-binary fixture with sibling list companions at multiple depths; add tests for single-positional resolve, multi-positional chain resolve, GraphQL-shape resolve, sibling cache hit, resolve-skip at each chain depth)
+
+**Approach:**
+
+*Setup (in `RunLiveDogfood`):* After `discoverLiveDogfoodCommands`, build `siblings map[string][]liveDogfoodCommand` keyed by joined parent path (`strings.Join(path[:len(path)-1], " ")`); root-level commands key on `""`. Build an empty `companionCache := &companionCache{results: map[string]string{}}` shared across the iteration. Thread both into `runLiveDogfoodCommand`.
+
+*Companion-leaf check:* `isCompanionLeaf(name string) bool` returns true if `name` is in the union of `crossAPIListVerbs` and `cinemaListVerbs` (see Key Technical Decisions for the two named sets).
+
+*Chained walk (in `resolveCommandPositionals`):*
+
+```
+resolveCommandPositionals(command, happyArgs, siblings, cache, binaryPath, cliDir, timeout):
+ placeholders := extractPositionalPlaceholders(usageSuffix(command.Help))
+ if len(placeholders) == 0:
+ return happyArgs, resolveOK, ""
+
+ resolved := []string{} // accumulates ids for chain context
+ for i, name := range placeholders:
+ nameLower := strings.ToLower(name)
+ if !strings.HasSuffix(nameLower, "id") || len(nameLower) <= 2:
+ // Non-id-shape positional in a chain: abort.
+ return nil, resolveSkip, fmt.Sprintf(
+ "non-id positional %q at depth %d", name, i)
+
+ // Walk up: positional at depth i sources from a list-shape
+ // sibling at the appropriate ancestor level. For a path like
+ // [movies, seasons, episodes, get], depth-0 (<series-id>)
+ // sources from siblings of [movies], depth-1 (<season-id>)
+ // sources from siblings of [movies, <id>, seasons], depth-2
+ // (<episode-id>) sources from siblings of [movies, <id>,
+ // seasons, <id>, episodes].
+ //
+ // Practically: the ancestor's parent path is
+ // command.Path[: len(command.Path) - len(placeholders) + i]
+ // Excluding the trailing get verb, the ancestor's siblings live
+ // at that parent-path key.
+ ancestorParentPath := strings.Join(
+ command.Path[:len(command.Path) - len(placeholders) + i], " ")
+ listCmd := findListCompanionAtDepth(siblings, ancestorParentPath)
+ if listCmd == nil:
+ return nil, resolveSkip, fmt.Sprintf(
+ "no list companion at depth %d for %q", i, name)
+
+ // Build companion args: list path + already-resolved parent ids
+ // as positional context + --json, plus --limit 1 if supported.
+ listArgs := append([]string{}, listCmd.Path...)
+ listArgs = append(listArgs, resolved...)
+ listArgs = append(listArgs, "--json")
+ if companionSupportsLimit(listCmd, binaryPath, cliDir, timeout):
+ listArgs = append(listArgs, "--limit", "1")
+
+ cacheKey := strings.Join(listArgs, "\x00")
+ if id, ok := cache.results[cacheKey]; ok:
+ resolved = append(resolved, id)
+ continue
+
+ run := runLiveDogfoodProcess(binaryPath, cliDir, listArgs, timeout)
+ if run.exitCode != 0:
+ return nil, resolveSkip, fmt.Sprintf(
+ "list companion failed at depth %d: exit %d", i, run.exitCode)
+
+ id, ok := extractFirstIDFromJSON(run.stdout)
+ if !ok:
+ return nil, resolveSkip, fmt.Sprintf(
+ "no id parseable from companion at depth %d", i)
+
+ cache.results[cacheKey] = id
+ resolved = append(resolved, id)
+
+ // Substitute resolved ids into happyArgs. Walk happyArgs left-to-right
+ // after command.Path; each non-flag arg corresponds to the next
+ // positional in `placeholders`. Replace those slots with `resolved`.
+ return substitutePositionals(happyArgs, command.Path, resolved), resolveOK, ""
+```
+
+*Cache structure:*
+
+```
+type companionCache struct {
+ // key = listArgs joined with NUL (\x00) so paths and ids can't collide
+ // value = extracted id from that companion's response
+ results map[string]string
+}
+```
+
+*ID extraction (`extractFirstIDFromJSON`):* Try paths in order and return the first match coerced via `fmt.Sprint`:
+- `.results[0].id`
+- `.[0].id` (top-level array)
+- `.items[0].id`
+- `.data[0].id`
+- `.list[0].id`
+- `.data.<any>.nodes[0].id` (GraphQL connection — Shopify, Linear, Notion)
+- `.data.<any>.edges[0].node.id` (Relay-style — GitHub GraphQL)
+
+The `<any>` walks any single-key field under `.data` and tries the child path. Use `json.Decoder.UseNumber()` so large numeric ids (e.g., Twitter snowflake > 2^53) coerce cleanly via `fmt.Sprint(json.Number(...))` without scientific notation. If all seven paths miss, return `("", false)`.
+
+*Companion `--help` for `--limit` detection:* `companionSupportsLimit` runs `companion.Path + ["--help"]` once per companion (cached on the `liveDogfoodCommand` entry by mutating its `.Help` field, or via a parallel map). The companion's Help is needed to call `extractFlagNames` to check for `--limit`. Lazy-load is acceptable because companion lookup runs only for id-shape commands and the cache prevents repeat probes.
+
+*Wire into `runLiveDogfoodCommand`:* After `liveDogfoodHappyArgs` returns `(happyArgs, true)`, call `resolveCommandPositionals`. On `resolveSkip`, append `skippedLiveDogfoodResult(commandName, LiveDogfoodTestHappy, reason)` and `skippedLiveDogfoodResult(commandName, LiveDogfoodTestJSON, reason)`, then continue to the error_path branch (which has its own positional gate and is U3's domain). On `resolveOK` with rewritten args, proceed with the existing happy_path / json_fidelity flow using the new args.
+
+**Patterns to follow:**
+- `runLiveDogfoodProcess` for subprocess invocation.
+- `extractPositionalPlaceholders` (`runtime_commands.go:196`) for placeholder parsing.
+- `extractFlagNames` (`dogfood.go:1779`) for companion `--limit` detection.
+- `skippedLiveDogfoodResult` for skip status.
+
+**Test scenarios:**
+- Covers R3 (single-positional happy) — fixture exposes `widgets list` returning `{"results":[{"id":"42"}]}` and `widgets get` with `<id>` positional. Matrix probes `widgets get`; chain resolves to `["42"]`; the get probe runs against the fake binary's id=42 path and passes happy_path + json_fidelity.
+- Covers R3 (chained multi-positional happy) — fixture exposes `widgets list` returning `{"results":[{"id":"P1"}]}`, `widgets sublist` taking `<widget-id>` positional and returning `{"results":[{"id":"S7"}]}`, and `widgets subwidgets get` taking `<widget-id> <subwidget-id>`. Matrix probes the get command; chain resolves to `["P1", "S7"]` (the sublist call ran with `widgets sublist P1 --json`). get probe runs with `widgets subwidgets get P1 S7` and passes.
+- Edge (cinema verb) — companion is `widgets popular` (in `cinemaListVerbs`). Resolution succeeds.
+- Edge (top-level array) — companion shape `[{"id":"42"}]`. Resolution succeeds via `.[0].id`.
+- Edge (items shape) — companion shape `{"items":[{"id":"42"}]}`. Resolution succeeds via `.items[0].id`.
+- Edge (numeric id) — companion shape `{"data":[{"id":42}]}`. Resolution succeeds and serializes to `"42"` via `json.Decoder.UseNumber` + `fmt.Sprint`.
+- Edge (large numeric id) — companion returns `{"results":[{"id":1234567890123456789}]}` (snowflake size). Resolution serializes correctly without scientific notation.
+- Edge (Shopify GraphQL) — companion shape `{"data":{"products":{"nodes":[{"id":"gid://shopify/Product/42"}]}}}`. Resolution succeeds via `.data.<any>.nodes[0].id` returning `"gid://shopify/Product/42"`.
+- Edge (Relay-style GraphQL) — companion shape `{"data":{"viewer":{"repos":{"edges":[{"node":{"id":"R_kgABC123"}}]}}}}`. Resolution succeeds via `.data.<any>.edges[0].node.id` (the wildcard tolerates any single-key under `.data`, here `viewer`).
+- Cache hit — fixture has two get-shape commands `widgets get` and `widgets describe` sharing parent `widgets`. The `widgets list` companion is invoked once on the first get; the second get hits the cache. Verify by counting subprocess invocations recorded by the fake binary.
+- Covers R4 (no companion) — fixture removes the companion. happy_path = SKIP with reason `"no list companion at depth 0 for id"`. json_fidelity = SKIP with same reason.
+- Covers R4 (companion errors) — companion exists but exits non-zero. happy_path = SKIP with reason `"list companion failed at depth 0: exit 2"`.
+- Covers R4 (no id in response) — companion returns `{"results":[]}`. happy_path = SKIP with reason `"no id parseable from companion at depth 0"`.
+- Covers R4 (chain breaks at depth 1) — first list resolves; second list missing or errors. happy_path = SKIP with reason naming depth 1.
+- Covers R4 (non-id-shape mid-chain) — multi-positional command where one positional is `<query>`. happy_path = SKIP with reason `"non-id positional \"query\" at depth N"`. Deliberately fail closed — chain resolution requires every positional in the chain to be id-shape.
+- Edge (no positional) — command has no positional. resolveCommandPositionals returns happyArgs unchanged; existing flow exercised.
+- Edge (companion supports `--limit`) — resolve runs with `--limit 1`. Verifiable by recording the args passed to the fake binary.
+- Edge (root-level get) — top-level get like `accounts get <id>` (no parent path). Sibling map keys on `""`; resolution still works.
+
+**Verification:**
+- `go test ./internal/pipeline/...` passes including the new cases.
+- Test fixture demonstrates happy_path moving from FAIL→PASS when a companion is wired and from FAIL→SKIP when no companion is reachable.
+- Manual probe against a printed Movie Goat CLI (post-U1 regen) shows `movies get`, `tv get`, `people get` happy_path passing live with TMDb when authed, and skip-with-reason when run without a companion fixture.
+
+---
+
+- U3. **Search-aware error_path dispatch in live dogfood**
+
+**Goal:** Recognise search-shaped commands and accept either non-zero exit OR exit 0 with empty results under `--json` as a pass; preserve today's "non-zero exit required" behavior for non-search commands.
+
+**Requirements:** R5, R6
+
+**Dependencies:** None — independent of U1 and U2.
+
+**Files:**
+- Modify: `internal/pipeline/live_dogfood.go` (extend the error_path branch in `runLiveDogfoodCommand`; add `commandSupportsSearch` helper; add `errorPathEmptyResults` JSON-shape predicate)
+- Test: `internal/pipeline/live_dogfood_test.go` (extend the fixture with a `widgets search` command supporting `--query`; add tests for search-pass-on-empty-results, search-pass-on-non-zero, mutation-still-fails-on-zero-exit)
+
+**Approach:**
+- Add `commandSupportsSearch(help string) bool`:
+ - Returns true if `extractFlagNames(help)` contains `query`, OR
+ - The Usage suffix (via `liveDogfoodUsageSuffix`) yields a placeholder named `query` from `extractPositionalPlaceholders`.
+- Extend the error_path branch (`runLiveDogfoodCommand` line 260):
+ - If `liveDogfoodCommandTakesArg(command.Help)` is false, current skip path is unchanged.
+ - If `commandSupportsSearch(command.Help)` is true (search-shaped strategy):
+ - Build args: prefer `--query __printing_press_invalid__ --json` when `--query` flag is supported; otherwise positional `__printing_press_invalid__ --json`. If `--json` is not supported, drop it.
+ - Run the probe.
+ - **Pass criteria** (broad): exit 0 with valid JSON when `--json` was supplied; OR exit 0 with any output when `--json` was not supplied; OR exit non-zero (consistent with mutation behavior — non-zero is also a valid "no match" signal for some APIs).
+ - **Fail criteria** (narrow): exit 0 with `--json` supplied but stdout is not valid JSON. That's the only fail condition.
+ - Note: this strategy does NOT require an empty results array. Real-world content/feed APIs return recent items as a fallback for unmatched queries; treating non-empty results as FAIL would block legitimate ships. Detecting broken filter logic is a unit-test concern, not a live-dogfood signal.
+ - Else (mutation-shaped strategy, today's behavior preserved): non-zero exit required.
+- The mutation-side branch (write/delete with bogus body) is unchanged: `commandSupportsSearch` returns false for those (no `--query`, no `<query>` placeholder), so they fall through to the existing strategy.
+- The empty-results predicate from earlier drafts is dropped — search-shape PASS no longer depends on result emptiness, only on exit code and JSON validity.
+
+**Patterns to follow:**
+- `extractFlagNames`, `extractPositionalPlaceholders`, `liveDogfoodUsageSuffix` — already used in this file.
+- `commandSupportsJSON` for the existing helper shape (one-line predicate over `extractFlagNames`).
+
+**Test scenarios:**
+- Covers R5 (happy, --json + empty results) — fixture exposes `widgets search` with `--query` flag and `--json`. Probe runs `widgets search --query __printing_press_invalid__ --json`; fixture returns exit 0 + `{"results":[]}`. error_path = PASS.
+- Covers R5 (happy, --json + non-empty results from fallback API) — same fixture but returns exit 0 + `{"results":[{"id":"recent-1"},{"id":"recent-2"}]}` (simulating a content-feed API's recency fallback). error_path = PASS — non-empty under exit 0 is no longer a fail.
+- Covers R5 (alt happy, no --json) — search command without `--json` support. Probe runs `widgets search --query __printing_press_invalid__` (no --json); fixture returns exit 0 + `0 results found.` to stdout. error_path = PASS — exit 0 alone is sufficient when --json wasn't supplied.
+- Covers R5 (alt happy, positional <query>) — search command via positional `<query>` (no `--query` flag). Probe runs `widgets search __printing_press_invalid__ --json`; fixture returns exit 0 + `{"results":[]}`. error_path = PASS.
+- Covers R5 (non-zero exit) — search command returns exit non-zero (e.g., 4xx). error_path = PASS (consistent with non-search behavior).
+- Edge (only fail mode) — search command claims `--json` support but emits non-JSON when `--json` was supplied. error_path = FAIL with reason `"invalid JSON"`. This is the sole fail condition for search-shape commands.
+- Covers R6 (mutation) — fixture exposes `widgets delete` (no `--query`, no `<query>` placeholder). Probe runs `widgets delete __printing_press_invalid__`; fixture returns exit 2. error_path = PASS (existing behavior preserved).
+- Covers R6 (plain get) — fixture exposes `widgets get` (read-shape, no `--query`, no `<query>` — id positional). Probe runs `widgets get __printing_press_invalid__`; fixture returns exit 2. error_path = PASS (existing behavior preserved).
+- Edge — `commandSupportsSearch` returns true ONLY when query signal is present. Verify with a fixture that has another flag (e.g., `--queue`) — extractFlagNames must return `queue`, not match `query` partially.
+
+**Verification:**
+- `go test ./internal/pipeline/...` passes including new cases.
+- Test fixture demonstrates error_path moving from FAIL→PASS for the search command, while remaining PASS for the mutation/get cases.
+- Run live dogfood against a printed Movie Goat CLI (post-U1+U2 regen): the 6 false-fail error_path entries from the original retro (`movies search`, `people search`, `tv search`, `multi`, `search`, `auth set-token`) should now PASS.
+
+---
+
+- U4. **Quick-level verdict gate restated**
+
+**Goal:** Update `finalizeLiveDogfoodReport` so the new skip-with-reason paths (introduced by U2) cannot flip a quick-level run from PASS to FAIL purely by reducing MatrixSize.
+
+**Requirements:** R7
+
+**Dependencies:** Conceptually paired with U2 (both touch the `Skip` outcome). Implementation order doesn't matter — U4 can land first as a no-op for today's data, then U2 lights up the new skip paths.
+
+**Files:**
+- Modify: `internal/pipeline/live_dogfood.go` (`finalizeLiveDogfoodReport` switch around line 410)
+- Test: `internal/pipeline/live_dogfood_test.go` (add tests covering the new gate semantics)
+
+**Approach:**
+- Change the verdict switch from:
+ ```go
+ case report.Level == "quick" && report.MatrixSize == 6 && report.Passed >= 5:
+ report.Verdict = "PASS"
+ case report.Failed > 0 || report.MatrixSize == 0:
+ report.Verdict = "FAIL"
+ case report.Level == "quick" && report.MatrixSize != 6:
+ report.Verdict = "FAIL"
+ ```
+ to:
+ ```go
+ case report.Failed > 0 || report.MatrixSize == 0:
+ report.Verdict = "FAIL"
+ case report.Level == "quick" && (report.Passed + report.Skipped) >= 5 && report.MatrixSize >= 4:
+ report.Verdict = "PASS"
+ case report.Level == "quick":
+ report.Verdict = "FAIL"
+ ```
+- Semantics: PASS at quick-level requires zero Failed entries, at least 5 entries that aren't Failed (Passed + Skipped), and a minimum MatrixSize floor of 4 to guard against pathological cases (every test skipping). FAIL still fires for any real failure or empty matrix.
+- The switch keeps the `case report.Failed > 0` arm first so any real failure dominates over the quick-PASS arm.
+
+**Patterns to follow:**
+- Existing `finalizeLiveDogfoodReport` switch shape.
+- Existing `LiveDogfoodStatus` enum (Pass/Fail/Skip already accounted for in the count loop above the switch).
+
+**Test scenarios:**
+- Covers R7 (Pass + 1 Skip) — quick-level run with 4 Pass + 1 Skip + 0 Fail = MatrixSize 4 (Skip excluded). Total non-failed = 5. Verdict = PASS (today's gate would FAIL because MatrixSize != 6).
+- Covers R7 (Pass + 2 Skips) — 3 Pass + 2 Skip + 0 Fail = MatrixSize 3, non-failed = 5. MatrixSize floor = 4 not met. Verdict = FAIL (the matrix collapsed too far to trust).
+- Edge (zero failures, full pass) — 6 Pass + 0 Skip + 0 Fail = MatrixSize 6, non-failed = 6. Verdict = PASS (existing behavior preserved).
+- Edge (one failure) — 4 Pass + 1 Skip + 1 Fail = MatrixSize 5, non-failed = 5. Verdict = FAIL (Failed > 0 dominates).
+- Edge (all skip) — 0 Pass + 6 Skip + 0 Fail = MatrixSize 0. Verdict = FAIL (matrix size 0 dominates).
+- Full-level unchanged — full-level runs continue to use the `Failed > 0 || MatrixSize == 0` arm; the quick-specific arm doesn't affect them.
+
+**Verification:**
+- `go test ./internal/pipeline/...` passes including new cases.
+- A regen of Movie Goat at quick-level with companion-resolved gets shows quick-PASS even when one or two get-shape commands skip happy_path/json_fidelity due to no companion.
+
+---
+
+## System-Wide Impact
+
+- **Interaction graph.** Changes are contained to two files (`internal/generator/generator.go`, `internal/pipeline/live_dogfood.go`) plus their tests. No call into the printed CLI runtime, no template change, no `agent-context` schema change.
+- **Error propagation.** U2 introduces a new "skip-with-reason" status path for happy_path/json_fidelity, and U4 restates the quick-level verdict gate so this skip path no longer flips PASS to FAIL. The `finalizeLiveDogfoodReport` accumulator's count loop (line 397) already routes `Status=Skip` correctly into the `Skipped` counter; the gate change consumes that counter explicitly via the new `(Passed + Skipped) >= 5` arm.
+- **State lifecycle risks.** Within request-scoped subprocess execution. The new `companionCache` is run-scoped (built and dropped per `RunLiveDogfood` invocation); no cross-run persistence, no concurrent-mutation risk because the live_dogfood iteration is sequential.
+- **API surface parity.** Generator and scorer are both internal to the printing-press binary; no public Go package contract widens. The agent-context schema is unchanged (Decision: defer kind dispatch).
+- **Integration coverage.** U2's chained-walk and cache paths cross the subprocess boundary inside the live_dogfood test fixture. Unit-level fake-binary tests are sufficient; the existing `live_dogfood_test.go` shows the pattern for multi-command fixtures. The fake binary's args-recording capability is needed for cache-hit verification — extend the fixture to write per-call argv to a tempfile that the test asserts against.
+- **Subprocess-count budget.** Today: 4 subprocesses per command (help, happy, json, error). After U2 with no caching: up to 6 per id-shape get (adds companion --help + companion --json). With the U2 cache: O(unique-parent-paths × depth) total companion calls across the run, not O(get-commands × depth). For chained gets (multi-positional), each chain link adds one cached companion call. Worst-case wall-clock is bounded; expect 30-50% increase for full-mode runs on get-heavy CLIs, mostly amortized by the cache for sibling shares.
+- **Unchanged invariants.** The verify pipeline (`internal/pipeline/runtime_commands.go:resolvePositionalValue`) is untouched. The `canonicalargs` registry is untouched. The mock-mode dispatch path is untouched. The MCP runtime walker is untouched. Only the live_dogfood subprocess path and the generator's example-line emitter change.
+
+---
+
+## Risks & Dependencies
+
+| Risk | Mitigation |
+|------|------------|
+| U1's type-fence rule still matches pure-string positionals like `void`/`acid`/`arid` ending in `id` (false positive). | Such names as required URL path params are vanishingly rare in real API design. Cost of an FP is one Example line with a UUID — verify will produce a clear 404 signal at the wrong path, strictly better than today's literal `example-value`. Add a denylist entry only when a real-spec FP surfaces in a retro. |
+| U2's companion-leaf allowlist misses a CLI's preferred list verb. | Fall-through goes to skip-with-reason, not back to placeholder-fail. Two named sets (`crossAPIListVerbs`, `cinemaListVerbs`) make extension targeted: future generic verbs go in the cross-API set, future cinema-class verbs stay isolated. Add new entries on retro evidence. |
+| U2's ID extraction try-list misses an API shape (e.g., `.records[0].id`, deeply nested non-GraphQL shapes). | Falls through to skip-with-reason. Strictly better than today. The seven-path try-list (including two GraphQL shapes) covers TMDb, REST `data`/`items`/`list`, Shopify/Linear/Notion GraphQL, GitHub Relay-style. Add new paths on retro evidence. |
+| U2's chained walk fails when a non-id-shape positional appears mid-chain (e.g., `<query>` between two ids). | Chain fails closed to skip-with-reason. Documented test scenario. Acceptable — the chain semantics are "thread ids forward"; a non-id mid-chain breaks that contract. |
+| U3 misclassifies a non-search command that happens to ship `--query` (e.g., a list command with `--query` as a filter). | Such commands should also accept exit 0 — that's the correct behavior for a "no matches found" filter. The new strategy is strictly looser than the old one, so misclassification doesn't harm: search-strategy is a superset of acceptable outcomes. |
+| Future API ships `--query` on a mutating verb (Elasticsearch `_delete_by_query`, MongoDB `deleteMany --query=...`). The heuristic flags it as search-shape, then probes against the live API with `__printing_press_invalid__`. | Today's risk is theoretical (no such command in current catalog). If tokenization broadens the match, `__printing_press_invalid__` could match real production data and trigger the mutation. Add a verb-name guard if/when such an API enters the catalog: only enter search-strategy when the leaf name itself contains a search-coloured verb (search, find, query, lookup, browse) AND `--query` is present — exclude leaves containing delete/update/create/remove/destroy. Not in scope for this PR. |
+| Companion subprocess timing inflates wall-clock for full-mode runs on get-heavy CLIs. | Per-companion cache caps total companion calls at O(unique-parent-paths × depth). Empirical worst-case 30-50% wall-clock increase, mostly amortized for sibling shares. Tighten by lowering `--limit` if companion supports it. If a real run blows the per-process timeout, the existing process timeout fires per-call, not per-CLI; a single slow companion does not stall other commands. |
+| Golden update in U1 noisy on git blame. | Two-line bytes update in two files. Clearly explain in PR body that the diff is the intended consequence of the camelCase ID fix. |
+| Quick-level verdict regression risk (was: skip-flips-FAIL). | Resolved by U4 — the gate is restated as `Failed == 0 && (Passed + Skipped) >= 5 && MatrixSize >= 4`. Skip outcomes can no longer flip PASS to FAIL. |
+
+---
+
+## Documentation / Operational Notes
+
+- Mention the U1 golden diff in the PR body.
+- No SKILL change needed — the new behavior is mechanical and lives entirely in the scorer/generator. SKILL prose still says "live dogfood probes happy_path / json_fidelity / error_path"; the per-shape strategy is an internal scorer detail.
+- No change to `docs/PIPELINE.md` (the pipeline phase contracts are unchanged — only one phase's verdict logic gets sharper).
+- After landing, regenerate Movie Goat to confirm the 11 false-fail entries clear. That regen is operational follow-up, not part of this plan's scope.
+
+---
+
+## Sources & References
+
+- **Origin issue:** [cli-printing-press#573](https://github.com/mvanhorn/cli-printing-press/issues/573)
+- **Parent retro:** [cli-printing-press#571](https://github.com/mvanhorn/cli-printing-press/issues/571)
+- **Sibling WU (already shipped):** [cli-printing-press#572](https://github.com/mvanhorn/cli-printing-press/issues/572) — generator template polish, PR #576
+- Related code: `internal/generator/generator.go:exampleValue`, `internal/pipeline/live_dogfood.go`, `internal/pipeline/runtime_commands.go`
+- Related prior retros: food52 retro #20260427-014521 (canonical mock values for verify positionals; complementary direction)
diff --git a/internal/generator/example_value_test.go b/internal/generator/example_value_test.go
new file mode 100644
index 00000000..b6192eee
--- /dev/null
+++ b/internal/generator/example_value_test.go
@@ -0,0 +1,72 @@
+package generator
+
+import (
+ "testing"
+
+ "github.com/mvanhorn/cli-printing-press/v3/internal/spec"
+ "github.com/stretchr/testify/assert"
+)
+
+// TestExampleValueIDRecognition covers the three id-shape clauses: bare `id`,
+// snake_case `*_id`, and camelCase `*Id` (added with a string-type fence so
+// boolean/numeric positionals like `paid` / `valid` don't get UUIDs).
+func TestExampleValueIDRecognition(t *testing.T) {
+ const uuid = "550e8400-e29b-41d4-a716-446655440000"
+
+ tests := []struct {
+ name string
+ param spec.Param
+ want string
+ }{
+ // Bare id (existing behavior preserved).
+ {"bare id string", spec.Param{Name: "id", Type: "string"}, uuid},
+ {"bare ID uppercase", spec.Param{Name: "ID", Type: "string"}, uuid},
+ {"bare Id pascal", spec.Param{Name: "Id", Type: "string"}, uuid},
+
+ // snake_case (existing behavior preserved).
+ {"snake movie_id", spec.Param{Name: "movie_id", Type: "string"}, uuid},
+ {"snake user_id", spec.Param{Name: "user_id", Type: "string"}, uuid},
+
+ // camelCase recognition (the new clause).
+ {"camel movieId", spec.Param{Name: "movieId", Type: "string"}, uuid},
+ {"camel seriesId", spec.Param{Name: "seriesId", Type: "string"}, uuid},
+ {"camel personId", spec.Param{Name: "personId", Type: "string"}, uuid},
+ {"camel pageId", spec.Param{Name: "pageId", Type: "string"}, uuid},
+ {"camel issueId", spec.Param{Name: "issueId", Type: "string"}, uuid},
+
+ // Empty Type passes the fence (legacy specs without type info).
+ {"camel movieId no type", spec.Param{Name: "movieId"}, uuid},
+
+ // Type fence: boolean and numeric positionals named *id flow to
+ // their type branches, not UUID.
+ {"bool paid", spec.Param{Name: "paid", Type: "boolean"}, "true"},
+ {"bool valid", spec.Param{Name: "valid", Type: "boolean"}, "true"},
+ {"bool unpaid", spec.Param{Name: "unpaid", Type: "boolean"}, "true"},
+ {"bool creditValid", spec.Param{Name: "creditValid", Type: "boolean"}, "true"},
+ {"int amountId", spec.Param{Name: "amountId", Type: "integer"}, "42"},
+ // `countId` is integer-typed AND contains the substring "count",
+ // so it lands in the count/limit/size branch (returns "50") rather
+ // than the generic integer branch. Confirms the fence routes it
+ // away from the UUID branch and into the existing type logic.
+ {"int countId routes to count branch", spec.Param{Name: "countId", Type: "integer"}, "50"},
+
+ // String-shaped alternative types (uuid, guid) ARE matched by the
+ // not-numeric-or-bool fence. Spec authors emitting non-canonical
+ // type strings get UUID examples just like canonical "string".
+ {"camel movieId uuid type", spec.Param{Name: "movieId", Type: "uuid"}, uuid},
+ {"camel personId guid type", spec.Param{Name: "personId", Type: "guid"}, uuid},
+
+ // Negative — does not end in `id`.
+ {"userIdentifier", spec.Param{Name: "userIdentifier", Type: "string"}, "example-value"},
+ {"empty name", spec.Param{Name: "", Type: "string"}, "example-value"},
+ {"too short ix", spec.Param{Name: "ix", Type: "string"}, "example-value"},
+ {"too short id-but-len2 cd", spec.Param{Name: "cd", Type: "string"}, "example-value"},
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ got := exampleValue(tt.param)
+ assert.Equal(t, tt.want, got)
+ })
+ }
+}
diff --git a/internal/generator/generator.go b/internal/generator/generator.go
index 1053016f..d8ae7beb 100644
--- a/internal/generator/generator.go
+++ b/internal/generator/generator.go
@@ -2944,7 +2944,16 @@ func (g *Generator) mcpParamDescription(p spec.Param) string {
func exampleValue(p spec.Param) string {
nameLower := strings.ToLower(p.Name)
- if strings.HasSuffix(nameLower, "_id") || nameLower == "id" {
+ // camelCase `*Id` carries an exclusion fence so bool/numeric params
+ // ending in "id" (e.g. paid, valid) get their own branches. The fence
+ // is expressed as "not numeric/boolean" rather than "is string" so
+ // alternative string-shaped types (e.g., `uuid`, `guid`) still match.
+ isNumericOrBool := p.Type == "boolean" || p.Type == "bool" ||
+ p.Type == "integer" || p.Type == "int" ||
+ p.Type == "number" || p.Type == "float"
+ if nameLower == "id" ||
+ strings.HasSuffix(nameLower, "_id") ||
+ (strings.HasSuffix(nameLower, "id") && len(nameLower) > 2 && !isNumericOrBool) {
return "550e8400-e29b-41d4-a716-446655440000"
}
if strings.Contains(nameLower, "email") {
diff --git a/internal/pipeline/dogfood.go b/internal/pipeline/dogfood.go
index 805f9a3f..c8219684 100644
--- a/internal/pipeline/dogfood.go
+++ b/internal/pipeline/dogfood.go
@@ -1776,9 +1776,10 @@ func extractExamplesSection(helpOutput string) string {
return strings.TrimSpace(strings.Join(examples, "\n"))
}
+var extractFlagNameRe = regexp.MustCompile(`--([a-z][-a-z0-9]*)`)
+
func extractFlagNames(text string) []string {
- re := regexp.MustCompile(`--([a-z][-a-z0-9]*)`)
- matches := re.FindAllStringSubmatch(text, -1)
+ matches := extractFlagNameRe.FindAllStringSubmatch(text, -1)
seen := make(map[string]struct{})
var flags []string
for _, match := range matches {
diff --git a/internal/pipeline/live_dogfood.go b/internal/pipeline/live_dogfood.go
index 9d231b45..21eb03d6 100644
--- a/internal/pipeline/live_dogfood.go
+++ b/internal/pipeline/live_dogfood.go
@@ -115,10 +115,18 @@ func RunLiveDogfood(opts LiveDogfoodOptions) (*LiveDogfoodReport, error) {
RanAt: time.Now().UTC(),
}
+ ctx := resolveCtx{
+ binaryPath: binaryPath,
+ cliDir: opts.CLIDir,
+ siblings: buildSiblingMap(commands),
+ cache: newCompanionCache(),
+ timeout: timeout,
+ }
+
for _, command := range commands {
commandName := strings.Join(command.Path, " ")
report.Commands = append(report.Commands, commandName)
- report.Tests = append(report.Tests, runLiveDogfoodCommand(binaryPath, opts.CLIDir, command, timeout)...)
+ report.Tests = append(report.Tests, runLiveDogfoodCommand(command, ctx)...)
}
finalizeLiveDogfoodReport(report)
@@ -177,6 +185,387 @@ var liveDogfoodFrameworkSkip = map[string]bool{
"version": true,
}
+// crossAPIListVerbs are leaf names a modern API CLI may expose as a
+// list-shape companion to a get-shape command.
+var crossAPIListVerbs = map[string]bool{
+ "list": true, "all": true, "index": true,
+ "query": true, "find": true, "search": true,
+ "discover": true, "browse": true, "recent": true, "feed": true,
+}
+
+// cinemaListVerbs are domain-specific list verbs for media/cinema-class APIs
+// that expose `popular`/`trending`/etc. as the canonical list shape rather
+// than a plain `list` leaf. Keep cross-API generic verbs in
+// crossAPIListVerbs; route new media-class verbs here.
+var cinemaListVerbs = map[string]bool{
+ "popular": true, "trending": true, "top_rated": true,
+ "latest": true, "now_playing": true, "upcoming": true,
+ "airing_today": true, "on_the_air": true,
+}
+
+func isCompanionLeaf(name string) bool {
+ return crossAPIListVerbs[name] || cinemaListVerbs[name]
+}
+
+// mutatingVerbs name leaves whose semantics include writes/deletes against
+// the API. Used as a deny-list overlay on the search-shape heuristic so a
+// command like `delete --query=...` (mass delete by filter) is not probed
+// with __printing_press_invalid__ against the live API.
+var mutatingVerbs = map[string]bool{
+ "delete": true, "destroy": true, "remove": true,
+ "create": true, "add": true, "new": true,
+ "update": true, "patch": true, "edit": true,
+ "set": true, "modify": true, "replace": true,
+}
+
+func isMutatingLeaf(name string) bool {
+ return mutatingVerbs[name]
+}
+
+// companionCache is run-scoped: per-RunLiveDogfood maps keyed by the full
+// companion argv (NUL-joined to avoid path/id collisions).
+type companionCache struct {
+ // results: NUL-joined argv → extracted id.
+ results map[string]string
+ // helps: companion path → cached --help output, so `--limit` detection
+ // runs at most once per companion.
+ helps map[string]string
+}
+
+// resolveCtx threads run-scoped state into the chained companion walk so
+// individual helpers don't need to take the same five parameters.
+type resolveCtx struct {
+ binaryPath string
+ cliDir string
+ siblings map[string][]liveDogfoodCommand
+ cache *companionCache
+ timeout time.Duration
+}
+
+func newCompanionCache() *companionCache {
+ return &companionCache{
+ results: map[string]string{},
+ helps: map[string]string{},
+ }
+}
+
+// buildSiblingMap groups commands by their joined parent path so the chain
+// walker can look up sibling list-shape companions in O(1).
+func buildSiblingMap(commands []liveDogfoodCommand) map[string][]liveDogfoodCommand {
+ siblings := map[string][]liveDogfoodCommand{}
+ for _, c := range commands {
+ if len(c.Path) == 0 {
+ continue
+ }
+ key := strings.Join(c.Path[:len(c.Path)-1], " ")
+ siblings[key] = append(siblings[key], c)
+ }
+ return siblings
+}
+
+// findListCompanion picks the first sibling whose leaf name is in the
+// companion-leaf allowlist (cross-API or cinema). Returns nil when no
+// allowlisted sibling is present.
+func findListCompanion(candidates []liveDogfoodCommand) *liveDogfoodCommand {
+ for i := range candidates {
+ path := candidates[i].Path
+ if len(path) == 0 {
+ continue
+ }
+ if isCompanionLeaf(path[len(path)-1]) {
+ return &candidates[i]
+ }
+ }
+ return nil
+}
+
+// resolveCommandPositionals walks the sibling list-shape chain to source a
+// real id for each id-shape positional in command.Help's Usage line. Earlier-
+// resolved ids are threaded into later list calls as positional context, so
+// nested resources (projects/tasks/update <pid> <tid>) work end-to-end.
+//
+// Returns:
+// - (newArgs, false, "") — placeholders substituted; run happy_path with newArgs
+// - (nil, true, reason) — chain broke; caller must skip happy_path + json_fidelity
+// - (happyArgs, false, "") — no positionals at all; pass-through unchanged
+func resolveCommandPositionals(command liveDogfoodCommand, happyArgs []string, ctx resolveCtx) ([]string, bool, string) {
+ placeholders := extractPositionalPlaceholders(liveDogfoodUsageSuffix(command.Help))
+ if len(placeholders) == 0 {
+ return happyArgs, false, ""
+ }
+
+ pathLen := len(command.Path)
+ nPlaceholders := len(placeholders)
+ if pathLen < nPlaceholders+1 {
+ // More placeholders than path segments before the verb. Unusual
+ // shape (top-level command with multiple positionals); skip.
+ return nil, true, fmt.Sprintf(
+ "command path %v has fewer segments than placeholders (%d)", command.Path, nPlaceholders)
+ }
+
+ resolved := make([]string, 0, nPlaceholders)
+ for i, name := range placeholders {
+ nameLower := strings.ToLower(name)
+ // id-shape covers: bare "id", snake_case "*_id", or camelCase "*id"
+ // where the prefix has at least one character (len > 2). Broader than
+ // generator.go exampleValue's predicate — no spec type info is available
+ // from CLI help text, so the string-type fence applied there is omitted.
+ isIDShape := nameLower == "id" ||
+ (strings.HasSuffix(nameLower, "id") && len(nameLower) > 2)
+ if !isIDShape {
+ return nil, true, fmt.Sprintf("non-id positional %q at depth %d", name, i)
+ }
+
+ // parent path of the verb that expects this placeholder.
+ siblingKey := strings.Join(command.Path[:pathLen-nPlaceholders+i], " ")
+ listCmd := findListCompanion(ctx.siblings[siblingKey])
+ if listCmd == nil {
+ return nil, true, fmt.Sprintf("no list companion at depth %d for %q", i, name)
+ }
+
+ listArgs := append([]string{}, listCmd.Path...)
+ listArgs = append(listArgs, resolved...)
+ listArgs = append(listArgs, "--json")
+ if companionSupportsLimit(*listCmd, ctx) {
+ listArgs = append(listArgs, "--limit", "1")
+ }
+
+ cacheKey := strings.Join(listArgs, "\x00") // NUL avoids path/id collisions.
+ if id, ok := ctx.cache.results[cacheKey]; ok {
+ if id == "" {
+ // Negative-cache sentinel: this companion already failed in this
+ // run. Skip immediately so sibling get-shape commands sharing
+ // the same companion don't each block on the same 30s timeout.
+ return nil, true, fmt.Sprintf(
+ "list companion previously failed at depth %d for %q", i, name)
+ }
+ resolved = append(resolved, id)
+ continue
+ }
+
+ run := runLiveDogfoodProcess(ctx.binaryPath, ctx.cliDir, listArgs, ctx.timeout)
+ if run.exitCode != 0 {
+ ctx.cache.results[cacheKey] = "" // negative-cache sentinel
+ return nil, true, fmt.Sprintf(
+ "list companion failed at depth %d: exit %d", i, run.exitCode)
+ }
+
+ id, ok := extractFirstIDFromJSON(run.stdout)
+ if !ok {
+ ctx.cache.results[cacheKey] = "" // negative-cache sentinel
+ return nil, true, fmt.Sprintf(
+ "no id parseable from companion at depth %d", i)
+ }
+
+ ctx.cache.results[cacheKey] = id
+ resolved = append(resolved, id)
+ }
+
+ return substitutePositionals(happyArgs, command.Path, resolved), false, ""
+}
+
+// substitutePositionals replaces the first len(resolved) non-flag args in
+// happyArgs (after command.Path) with the resolved ids. The walk preserves
+// flags interleaved with positionals so an example like
+// `--limit 5 widgets get <id>` stays intact when the placeholder is
+// substituted in. Args before command.Path are preserved untouched.
+func substitutePositionals(happyArgs, commandPath []string, resolved []string) []string {
+ out := make([]string, 0, len(happyArgs))
+ out = append(out, happyArgs[:min(len(commandPath), len(happyArgs))]...)
+ idx := 0
+ for j := len(commandPath); j < len(happyArgs); j++ {
+ arg := happyArgs[j]
+ if !strings.HasPrefix(arg, "-") && idx < len(resolved) {
+ out = append(out, resolved[idx])
+ idx++
+ } else {
+ out = append(out, arg)
+ }
+ }
+ return out
+}
+
+// companionSupportsLimit checks the companion's --help for a --limit flag,
+// caching the result. Lazy: only invoked once per companion path because
+// the chain walker calls findListCompanion before each invocation and we
+// only consult --help when a companion was actually selected.
+func companionSupportsLimit(companion liveDogfoodCommand, ctx resolveCtx) bool {
+ pathKey := strings.Join(companion.Path, " ")
+ help, cached := ctx.cache.helps[pathKey]
+ if !cached {
+ helpArgs := append(append([]string{}, companion.Path...), "--help")
+ run := runLiveDogfoodProcess(ctx.binaryPath, ctx.cliDir, helpArgs, ctx.timeout)
+ if run.exitCode != 0 {
+ ctx.cache.helps[pathKey] = ""
+ return false
+ }
+ help = run.stdout + run.stderr
+ ctx.cache.helps[pathKey] = help
+ }
+ return slices.Contains(extractFlagNames(help), "limit")
+}
+
+// extractFirstIDFromJSON tries canonical REST and GraphQL response shapes
+// in order; see inline `// Path N:` comments for the priority list.
+// UseNumber() preserves large numeric ids (e.g., snowflake > 2^53) through
+// fmt.Sprint without scientific notation.
+func extractFirstIDFromJSON(stdout string) (string, bool) {
+ dec := json.NewDecoder(strings.NewReader(stdout))
+ dec.UseNumber()
+ var root any
+ if err := dec.Decode(&root); err != nil {
+ return "", false
+ }
+
+ // Path 1: .results[0].id
+ if id, ok := pickIDFromArrayKey(root, "results"); ok {
+ return id, true
+ }
+ // Path 2: top-level array .[0].id
+ if id, ok := pickIDFromTopArray(root); ok {
+ return id, true
+ }
+ // Path 3: .items[0].id
+ if id, ok := pickIDFromArrayKey(root, "items"); ok {
+ return id, true
+ }
+ // Path 4: .data[0].id (only when .data is an ARRAY — GraphQL data is an object)
+ if obj, ok := root.(map[string]any); ok {
+ if dataArr, ok := obj["data"].([]any); ok {
+ if id, ok := firstIDFromArray(dataArr); ok {
+ return id, true
+ }
+ }
+ }
+ // Path 5: .list[0].id
+ if id, ok := pickIDFromArrayKey(root, "list"); ok {
+ return id, true
+ }
+ // Path 6: .data.<any>.nodes[0].id
+ if id, ok := pickIDFromGraphQLConnection(root, "nodes", false); ok {
+ return id, true
+ }
+ // Path 7: .data.<any>.edges[0].node.id
+ if id, ok := pickIDFromGraphQLConnection(root, "edges", true); ok {
+ return id, true
+ }
+ return "", false
+}
+
+func pickIDFromArrayKey(root any, key string) (string, bool) {
+ obj, ok := root.(map[string]any)
+ if !ok {
+ return "", false
+ }
+ arr, ok := obj[key].([]any)
+ if !ok {
+ return "", false
+ }
+ return firstIDFromArray(arr)
+}
+
+func pickIDFromTopArray(root any) (string, bool) {
+ arr, ok := root.([]any)
+ if !ok {
+ return "", false
+ }
+ return firstIDFromArray(arr)
+}
+
+func firstIDFromArray(arr []any) (string, bool) {
+ if len(arr) == 0 {
+ return "", false
+ }
+ first, ok := arr[0].(map[string]any)
+ if !ok {
+ return "", false
+ }
+ return idValueAsString(first["id"])
+}
+
+// pickIDFromGraphQLConnection walks .data... looking for a `connectionKey`
+// (`nodes` or `edges`) array within a bounded subtree. Handles two shapes:
+//
+// Shape A — depth 1 under .data (Shopify, Linear, Notion):
+// .data.<resource>.<connectionKey>[0]...
+//
+// Shape B — depth 2 under .data (GitHub Relay viewer.repos.edges):
+// .data.<wrapper>.<resource>.<connectionKey>[0]...
+//
+// edgeShape=true reads id from .node.id under each entry (Relay edges);
+// edgeShape=false reads id directly from each entry (nodes). The walk is
+// bounded to depth 2 to avoid pathological recursion on deeply nested
+// responses that don't carry an id-shaped first element.
+func pickIDFromGraphQLConnection(root any, connectionKey string, edgeShape bool) (string, bool) {
+ obj, ok := root.(map[string]any)
+ if !ok {
+ return "", false
+ }
+ data, ok := obj["data"].(map[string]any)
+ if !ok {
+ return "", false
+ }
+ // Try depth 1 then depth 2.
+ for depth := 1; depth <= 2; depth++ {
+ if id, ok := walkForConnection(data, connectionKey, edgeShape, depth); ok {
+ return id, true
+ }
+ }
+ return "", false
+}
+
+// walkForConnection descends `depth` levels into nested map[string]any
+// values, returning the first matching connection's id.
+func walkForConnection(node map[string]any, connectionKey string, edgeShape bool, depth int) (string, bool) {
+ if depth == 0 {
+ arr, ok := node[connectionKey].([]any)
+ if !ok || len(arr) == 0 {
+ return "", false
+ }
+ first, ok := arr[0].(map[string]any)
+ if !ok {
+ return "", false
+ }
+ if edgeShape {
+ n, ok := first["node"].(map[string]any)
+ if !ok {
+ return "", false
+ }
+ return idValueAsString(n["id"])
+ }
+ return idValueAsString(first["id"])
+ }
+ for _, child := range node {
+ childObj, ok := child.(map[string]any)
+ if !ok {
+ continue
+ }
+ if id, ok := walkForConnection(childObj, connectionKey, edgeShape, depth-1); ok {
+ return id, true
+ }
+ }
+ return "", false
+}
+
+func idValueAsString(v any) (string, bool) {
+ if v == nil {
+ return "", false
+ }
+ switch t := v.(type) {
+ case string:
+ if t == "" {
+ return "", false
+ }
+ return t, true
+ case json.Number:
+ return t.String(), true
+ case bool:
+ return "", false
+ default:
+ return fmt.Sprint(v), true
+ }
+}
+
func collectLiveDogfoodCommandPaths(prefix []string, command dogfoodAgentCommand, paths *[][]string) {
if command.Name == "" || liveDogfoodFrameworkSkip[command.Name] {
return
@@ -192,11 +581,11 @@ func collectLiveDogfoodCommandPaths(prefix []string, command dogfoodAgentCommand
}
}
-func runLiveDogfoodCommand(binaryPath, cliDir string, command liveDogfoodCommand, timeout time.Duration) []LiveDogfoodTestResult {
+func runLiveDogfoodCommand(command liveDogfoodCommand, ctx resolveCtx) []LiveDogfoodTestResult {
commandName := strings.Join(command.Path, " ")
helpArgs := append(append([]string{}, command.Path...), "--help")
- helpRun := runLiveDogfoodProcess(binaryPath, cliDir, helpArgs, timeout)
+ helpRun := runLiveDogfoodProcess(ctx.binaryPath, ctx.cliDir, helpArgs, ctx.timeout)
helpResult := liveDogfoodResult(commandName, LiveDogfoodTestHelp, helpArgs, helpRun)
helpPassed := helpRun.exitCode == 0
help := helpRun.stdout + helpRun.stderr
@@ -231,42 +620,95 @@ func runLiveDogfoodCommand(binaryPath, cliDir string, command liveDogfoodCommand
return results
}
- happyRun := runLiveDogfoodProcess(binaryPath, cliDir, happyArgs, timeout)
- happyResult := liveDogfoodResult(commandName, LiveDogfoodTestHappy, happyArgs, happyRun)
- if happyRun.exitCode == 0 {
- happyResult.Status = LiveDogfoodStatusPass
- happyResult.Reason = ""
- }
- results = append(results, happyResult)
+ // error_path gate runs independently below; its own branch.
+ resolvedArgs, resolveSkipped, resolveReason := resolveCommandPositionals(command, happyArgs, ctx)
+ if resolveSkipped {
+ results = append(results,
+ skippedLiveDogfoodResult(commandName, LiveDogfoodTestHappy, resolveReason),
+ skippedLiveDogfoodResult(commandName, LiveDogfoodTestJSON, resolveReason),
+ )
+ } else {
+ happyArgs = resolvedArgs
- if commandSupportsJSON(command.Help) {
- jsonArgs := appendJSONArg(happyArgs)
- jsonRun := runLiveDogfoodProcess(binaryPath, cliDir, jsonArgs, timeout)
- jsonResult := liveDogfoodResult(commandName, LiveDogfoodTestJSON, jsonArgs, jsonRun)
- if jsonRun.exitCode == 0 {
- if !json.Valid([]byte(jsonRun.stdout)) {
- jsonResult.Status = LiveDogfoodStatusFail
- jsonResult.Reason = "invalid JSON"
- } else {
- jsonResult.Status = LiveDogfoodStatusPass
- jsonResult.Reason = ""
+ happyRun := runLiveDogfoodProcess(ctx.binaryPath, ctx.cliDir, happyArgs, ctx.timeout)
+ happyResult := liveDogfoodResult(commandName, LiveDogfoodTestHappy, happyArgs, happyRun)
+ if happyRun.exitCode == 0 {
+ happyResult.Status = LiveDogfoodStatusPass
+ happyResult.Reason = ""
+ }
+ results = append(results, happyResult)
+
+ if commandSupportsJSON(command.Help) {
+ jsonArgs := appendJSONArg(happyArgs)
+ jsonRun := runLiveDogfoodProcess(ctx.binaryPath, ctx.cliDir, jsonArgs, ctx.timeout)
+ jsonResult := liveDogfoodResult(commandName, LiveDogfoodTestJSON, jsonArgs, jsonRun)
+ if jsonRun.exitCode == 0 {
+ if !json.Valid([]byte(jsonRun.stdout)) {
+ jsonResult.Status = LiveDogfoodStatusFail
+ jsonResult.Reason = "invalid JSON"
+ } else {
+ jsonResult.Status = LiveDogfoodStatusPass
+ jsonResult.Reason = ""
+ }
}
+ results = append(results, jsonResult)
+ } else {
+ results = append(results, skippedLiveDogfoodResult(commandName, LiveDogfoodTestJSON, "--json not supported"))
}
- results = append(results, jsonResult)
- } else {
- results = append(results, skippedLiveDogfoodResult(commandName, LiveDogfoodTestJSON, "--json not supported"))
}
if liveDogfoodCommandTakesArg(command.Help) {
- errorArgs := append(append([]string{}, command.Path...), "__printing_press_invalid__")
- errorRun := runLiveDogfoodProcess(binaryPath, cliDir, errorArgs, timeout)
+ flagNames := extractFlagNames(command.Help)
+ hasQueryFlag := slices.Contains(flagNames, "query")
+ // Search-shape strategy is suppressed for mutating leaves so a
+ // `delete --query=...` mass-delete is never probed with the
+ // invalid-token sentinel against the live API.
+ leaf := command.Path[len(command.Path)-1]
+ isSearch := commandSupportsSearch(command.Help) && !isMutatingLeaf(leaf)
+ suppliedJSON := slices.Contains(flagNames, "json")
+
+ var errorArgs []string
+ if isSearch {
+ errorArgs = append([]string{}, command.Path...)
+ if hasQueryFlag {
+ errorArgs = append(errorArgs, "--query", "__printing_press_invalid__")
+ } else {
+ errorArgs = append(errorArgs, "__printing_press_invalid__")
+ }
+ if suppliedJSON {
+ errorArgs = appendJSONArg(errorArgs)
+ }
+ } else {
+ errorArgs = append(append([]string{}, command.Path...), "__printing_press_invalid__")
+ }
+
+ errorRun := runLiveDogfoodProcess(ctx.binaryPath, ctx.cliDir, errorArgs, ctx.timeout)
errorResult := liveDogfoodResult(commandName, LiveDogfoodTestError, errorArgs, errorRun)
- if errorRun.exitCode != 0 {
- errorResult.Status = LiveDogfoodStatusPass
- errorResult.Reason = ""
+
+ if isSearch {
+ // Real-world feed/content APIs return recent items as a fallback
+ // for unmatched queries, so non-empty results under exit 0 are
+ // not a failure signal. The only fail mode is invalid JSON when
+ // the caller asked for --json.
+ switch {
+ case errorRun.exitCode != 0:
+ errorResult.Status = LiveDogfoodStatusPass
+ errorResult.Reason = ""
+ case suppliedJSON && !json.Valid([]byte(errorRun.stdout)):
+ errorResult.Status = LiveDogfoodStatusFail
+ errorResult.Reason = "invalid JSON under --json"
+ default:
+ errorResult.Status = LiveDogfoodStatusPass
+ errorResult.Reason = ""
+ }
} else {
- errorResult.Status = LiveDogfoodStatusFail
- errorResult.Reason = "expected non-zero exit for invalid argument"
+ if errorRun.exitCode != 0 {
+ errorResult.Status = LiveDogfoodStatusPass
+ errorResult.Reason = ""
+ } else {
+ errorResult.Status = LiveDogfoodStatusFail
+ errorResult.Reason = "expected non-zero exit for invalid argument"
+ }
}
results = append(results, errorResult)
} else {
@@ -276,6 +718,47 @@ func runLiveDogfoodCommand(binaryPath, cliDir string, command liveDogfoodCommand
return results
}
+// commandSupportsSearch reports whether a command behaves like a search:
+// either it ships a --query flag, or its Usage suffix carries a <query>
+// positional placeholder. Search-shape commands canonically return exit 0
+// with empty (or fallback) results on no-match, so error_path treats them
+// differently from mutating writes.
+//
+// Flag detection is scoped to the Flags: section so cross-references in
+// Examples or Long descriptions (e.g., "see widgets list --query=foo") do
+// not contaminate the heuristic.
+func commandSupportsSearch(help string) bool {
+ if slices.Contains(extractFlagNames(extractFlagsSection(help)), "query") {
+ return true
+ }
+ return slices.Contains(extractPositionalPlaceholders(liveDogfoodUsageSuffix(help)), "query")
+}
+
+// extractFlagsSection returns the body of a Cobra `--help` "Flags:" or
+// "Global Flags:" block — everything from the section header through the
+// next blank line. Used to scope flag-name extraction so cross-reference
+// strings outside the actual flag section can't trigger false positives.
+func extractFlagsSection(help string) string {
+ lines := strings.Split(help, "\n")
+ var out []string
+ inFlags := false
+ for _, line := range lines {
+ trimmed := strings.TrimSpace(line)
+ if trimmed == "Flags:" || trimmed == "Global Flags:" {
+ inFlags = true
+ continue
+ }
+ if inFlags {
+ if trimmed == "" {
+ inFlags = false
+ continue
+ }
+ out = append(out, line)
+ }
+ }
+ return strings.Join(out, "\n")
+}
+
func runLiveDogfoodProcess(binaryPath, cliDir string, args []string, timeout time.Duration) liveDogfoodRun {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
@@ -407,12 +890,17 @@ func finalizeLiveDogfoodReport(report *LiveDogfoodReport) {
report.Skipped++
}
}
+ // Failed-or-empty wins: any real failure or a fully empty matrix is FAIL.
+ // The quick-level PASS arm uses min(5, MatrixSize) for the threshold so
+ // single-command quick runs (~4 entries when all probes succeed) can
+ // PASS, while keeping a MatrixSize >= 4 floor that blocks pathological
+ // matrices where most entries skipped.
switch {
- case report.Level == "quick" && report.MatrixSize == 6 && report.Passed >= 5:
- report.Verdict = "PASS"
case report.Failed > 0 || report.MatrixSize == 0:
report.Verdict = "FAIL"
- case report.Level == "quick" && report.MatrixSize != 6:
+ case report.Level == "quick" && report.MatrixSize >= 4 && report.Passed+report.Skipped >= min(5, report.MatrixSize):
+ report.Verdict = "PASS"
+ case report.Level == "quick":
report.Verdict = "FAIL"
}
}
diff --git a/internal/pipeline/live_dogfood_test.go b/internal/pipeline/live_dogfood_test.go
index 398e5cec..f48dfc1f 100644
--- a/internal/pipeline/live_dogfood_test.go
+++ b/internal/pipeline/live_dogfood_test.go
@@ -129,6 +129,439 @@ func TestRunLiveDogfoodAcceptanceRequiresManifestIdentity(t *testing.T) {
assert.Contains(t, err.Error(), "CLI manifest")
}
+// TestFinalizeLiveDogfoodReportVerdictGate exercises the quick-level verdict
+// switch directly against synthesized reports. The new gate must accept
+// skip-with-reason as a non-failure (Passed + Skipped >= 5) with a MatrixSize
+// floor of 4, while preserving Failed-dominance and full-level semantics.
+func TestFinalizeLiveDogfoodReportVerdictGate(t *testing.T) {
+ mkResult := func(status LiveDogfoodStatus) LiveDogfoodTestResult {
+ return LiveDogfoodTestResult{Status: status}
+ }
+
+ tests := []struct {
+ name string
+ level string
+ results []LiveDogfoodTestResult
+ want string
+ }{
+ {
+ name: "quick all pass classic",
+ level: "quick",
+ results: []LiveDogfoodTestResult{
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ },
+ want: "PASS",
+ },
+ {
+ name: "quick 5 pass + 1 skip — companion missing",
+ level: "quick",
+ results: []LiveDogfoodTestResult{
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusSkip),
+ },
+ want: "PASS",
+ },
+ {
+ name: "quick 4 pass + 2 skip — multi-positional skip + no-companion skip",
+ level: "quick",
+ results: []LiveDogfoodTestResult{
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusSkip), mkResult(LiveDogfoodStatusSkip),
+ },
+ want: "PASS",
+ },
+ {
+ name: "quick 3 pass + 3 skip — MatrixSize floor (3) below 4",
+ level: "quick",
+ results: []LiveDogfoodTestResult{
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusSkip),
+ mkResult(LiveDogfoodStatusSkip), mkResult(LiveDogfoodStatusSkip),
+ },
+ want: "FAIL",
+ },
+ {
+ name: "quick 1-command all pass — 4 entries should PASS via min(5, M) threshold",
+ level: "quick",
+ results: []LiveDogfoodTestResult{
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ },
+ want: "PASS",
+ },
+ {
+ name: "quick 4 pass + 1 fail — Failed dominates",
+ level: "quick",
+ results: []LiveDogfoodTestResult{
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusFail),
+ },
+ want: "FAIL",
+ },
+ {
+ name: "quick all skip — MatrixSize 0",
+ level: "quick",
+ results: []LiveDogfoodTestResult{mkResult(LiveDogfoodStatusSkip), mkResult(LiveDogfoodStatusSkip)},
+ want: "FAIL",
+ },
+ {
+ name: "full all pass — full-level PASS preserved (verdict default)",
+ level: "full",
+ results: []LiveDogfoodTestResult{
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusPass),
+ },
+ want: "PASS",
+ },
+ {
+ name: "full one fail — Failed dominates at full level",
+ level: "full",
+ results: []LiveDogfoodTestResult{
+ mkResult(LiveDogfoodStatusPass), mkResult(LiveDogfoodStatusPass),
+ mkResult(LiveDogfoodStatusFail),
+ },
+ want: "FAIL",
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ report := &LiveDogfoodReport{
+ Level: tt.level,
+ Verdict: "PASS",
+ Tests: tt.results,
+ }
+ finalizeLiveDogfoodReport(report)
+ assert.Equal(t, tt.want, report.Verdict, "Passed=%d Failed=%d Skipped=%d MatrixSize=%d",
+ report.Passed, report.Failed, report.Skipped, report.MatrixSize)
+ })
+ }
+}
+
+func TestExtractFirstIDFromJSON(t *testing.T) {
+ tests := []struct {
+ name string
+ stdout string
+ want string
+ ok bool
+ }{
+ {
+ name: "TMDb results shape",
+ stdout: `{"results":[{"id":"42"}],"total_results":1}`,
+ want: "42", ok: true,
+ },
+ {
+ name: "top-level array",
+ stdout: `[{"id":"first"},{"id":"second"}]`,
+ want: "first", ok: true,
+ },
+ {
+ name: "items shape (GitHub REST)",
+ stdout: `{"items":[{"id":"abc"}],"total_count":1}`,
+ want: "abc", ok: true,
+ },
+ {
+ name: "data array (Stripe)",
+ stdout: `{"object":"list","data":[{"id":"cus_xyz"}],"has_more":false}`,
+ want: "cus_xyz", ok: true,
+ },
+ {
+ name: "list shape (long-tail)",
+ stdout: `{"list":[{"id":"L1"}]}`,
+ want: "L1", ok: true,
+ },
+ {
+ name: "GraphQL nodes (Shopify)",
+ stdout: `{"data":{"products":{"nodes":[{"id":"gid://shopify/Product/42"}]}}}`,
+ want: "gid://shopify/Product/42", ok: true,
+ },
+ {
+ name: "GraphQL edges (Relay-style)",
+ stdout: `{"data":{"viewer":{"repos":{"edges":[{"node":{"id":"R_kgABC123"}}]}}}}`,
+ want: "R_kgABC123", ok: true,
+ },
+ {
+ name: "numeric id preserved as string",
+ stdout: `{"results":[{"id":12345}]}`,
+ want: "12345", ok: true,
+ },
+ {
+ name: "snowflake-size numeric id (no scientific notation)",
+ stdout: `{"results":[{"id":1234567890123456789}]}`,
+ want: "1234567890123456789", ok: true,
+ },
+ {
+ name: "empty results — no id",
+ stdout: `{"results":[]}`,
+ want: "", ok: false,
+ },
+ {
+ name: "results without id field",
+ stdout: `{"results":[{"name":"thing"}]}`,
+ want: "", ok: false,
+ },
+ {
+ name: "invalid JSON",
+ stdout: `not json at all`,
+ want: "", ok: false,
+ },
+ {
+ name: "matches REST results before GraphQL — REST wins",
+ stdout: `{"results":[{"id":"REST"}],"data":{"x":{"nodes":[{"id":"GQL"}]}}}`,
+ want: "REST", ok: true,
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ got, ok := extractFirstIDFromJSON(tt.stdout)
+ assert.Equal(t, tt.ok, ok)
+ assert.Equal(t, tt.want, got)
+ })
+ }
+}
+
+func TestBuildSiblingMap(t *testing.T) {
+ commands := []liveDogfoodCommand{
+ {Path: []string{"projects", "list"}},
+ {Path: []string{"projects", "get"}},
+ {Path: []string{"projects", "tasks", "list"}},
+ {Path: []string{"projects", "tasks", "update"}},
+ {Path: []string{"users", "get"}},
+ }
+ siblings := buildSiblingMap(commands)
+
+ // Top-level commands keyed by "" (parent path).
+ assert.Len(t, siblings["projects"], 2, "projects subcommands")
+ assert.Len(t, siblings["projects tasks"], 2, "projects tasks subcommands")
+ assert.Len(t, siblings["users"], 1, "users subcommands")
+}
+
+func TestFindListCompanion(t *testing.T) {
+ candidates := []liveDogfoodCommand{
+ {Path: []string{"widgets", "get"}},
+ {Path: []string{"widgets", "list"}},
+ {Path: []string{"widgets", "delete"}},
+ }
+ got := findListCompanion(candidates)
+ if assert.NotNil(t, got) {
+ assert.Equal(t, []string{"widgets", "list"}, got.Path)
+ }
+
+ // Cinema verb fallback.
+ cinema := []liveDogfoodCommand{
+ {Path: []string{"movies", "get"}},
+ {Path: []string{"movies", "popular"}},
+ }
+ got = findListCompanion(cinema)
+ if assert.NotNil(t, got) {
+ assert.Equal(t, []string{"movies", "popular"}, got.Path)
+ }
+
+ // No allowlisted leaf.
+ none := []liveDogfoodCommand{
+ {Path: []string{"x", "delete"}},
+ {Path: []string{"x", "update"}},
+ }
+ assert.Nil(t, findListCompanion(none))
+}
+
+func TestSubstitutePositionals(t *testing.T) {
+ tests := []struct {
+ name string
+ happyArgs []string
+ commandPath []string
+ resolved []string
+ want []string
+ }{
+ {
+ name: "single positional",
+ happyArgs: []string{"widgets", "get", "example-value"},
+ commandPath: []string{"widgets", "get"},
+ resolved: []string{"42"},
+ want: []string{"widgets", "get", "42"},
+ },
+ {
+ name: "two positionals",
+ happyArgs: []string{"projects", "tasks", "update", "ph1", "ph2"},
+ commandPath: []string{"projects", "tasks", "update"},
+ resolved: []string{"P1", "T1"},
+ want: []string{"projects", "tasks", "update", "P1", "T1"},
+ },
+ {
+ name: "positional before flag",
+ happyArgs: []string{"widgets", "update", "ph1", "--name", "thing"},
+ commandPath: []string{"widgets", "update"},
+ resolved: []string{"abc"},
+ want: []string{"widgets", "update", "abc", "--name", "thing"},
+ },
+ {
+ name: "no positionals (resolved empty)",
+ happyArgs: []string{"widgets", "list", "--limit", "5"},
+ commandPath: []string{"widgets", "list"},
+ resolved: nil,
+ want: []string{"widgets", "list", "--limit", "5"},
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ got := substitutePositionals(tt.happyArgs, tt.commandPath, tt.resolved)
+ assert.Equal(t, tt.want, got)
+ })
+ }
+}
+
+func TestResolveCommandPositionalsSkipPaths(t *testing.T) {
+ ctx := resolveCtx{
+ siblings: map[string][]liveDogfoodCommand{},
+ cache: newCompanionCache(),
+ timeout: time.Second,
+ }
+
+ // No positionals → not-skipped, happyArgs unchanged.
+ cmd := liveDogfoodCommand{
+ Path: []string{"widgets", "list"},
+ Help: "Usage:\n cli widgets list [flags]\n",
+ }
+ args, skipped, _ := resolveCommandPositionals(cmd, []string{"widgets", "list"}, ctx)
+ assert.False(t, skipped)
+ assert.Equal(t, []string{"widgets", "list"}, args)
+
+ // Non-id-shape positional (<query>) at depth 0 → skip.
+ cmd = liveDogfoodCommand{
+ Path: []string{"widgets", "search"},
+ Help: "Usage:\n cli widgets search <query> [flags]\n",
+ }
+ _, skipped, reason := resolveCommandPositionals(cmd, []string{"widgets", "search", "x"}, ctx)
+ assert.True(t, skipped)
+ assert.Contains(t, reason, "non-id positional")
+
+ // id-shape positional (bare `id`) but no companion → skip.
+ cmd = liveDogfoodCommand{
+ Path: []string{"widgets", "get"},
+ Help: "Usage:\n cli widgets get <id> [flags]\n",
+ }
+ _, skipped, reason = resolveCommandPositionals(cmd, []string{"widgets", "get", "x"}, ctx)
+ assert.True(t, skipped)
+ assert.Contains(t, reason, "no list companion")
+
+ // camelCase id-shape positional (movieId) but no companion → skip.
+ cmd = liveDogfoodCommand{
+ Path: []string{"movies", "get"},
+ Help: "Usage:\n cli movies get <movieId> [flags]\n",
+ }
+ _, skipped, reason = resolveCommandPositionals(cmd, []string{"movies", "get", "x"}, ctx)
+ assert.True(t, skipped)
+ assert.Contains(t, reason, "no list companion")
+
+ // Path shorter than placeholders + 1 → skip.
+ cmd = liveDogfoodCommand{
+ Path: []string{"get"},
+ Help: "Usage:\n cli get <id> <name> [flags]\n",
+ }
+ _, skipped, _ = resolveCommandPositionals(cmd, []string{"get", "x", "y"}, ctx)
+ assert.True(t, skipped)
+}
+
+func TestCommandSupportsSearch(t *testing.T) {
+ tests := []struct {
+ name string
+ help string
+ want bool
+ }{
+ {
+ name: "search via --query flag",
+ help: `Usage:
+ fixture-pp-cli widgets search [flags]
+
+Flags:
+ --query string Search query
+ --json Output JSON
+`,
+ want: true,
+ },
+ {
+ name: "search via positional <query>",
+ help: `Usage:
+ fixture-pp-cli widgets search <query> [flags]
+
+Flags:
+ --json Output JSON
+`,
+ want: true,
+ },
+ {
+ name: "non-search list command — no query signal",
+ help: `Usage:
+ fixture-pp-cli widgets list [flags]
+
+Flags:
+ --limit int Max items
+ --json Output JSON
+`,
+ want: false,
+ },
+ {
+ name: "exact-match flag — --queue must not match --query",
+ help: `Usage:
+ fixture-pp-cli widgets dispatch [flags]
+
+Flags:
+ --queue string Job queue name
+`,
+ want: false,
+ },
+ {
+ name: "Examples block mentioning --query does NOT trigger search-shape (Flags-section scoping)",
+ help: `Usage:
+ fixture-pp-cli widgets delete <id> [flags]
+
+Examples:
+ fixture-pp-cli widgets delete 42
+ # related: fixture-pp-cli widgets list --query=foo
+
+Flags:
+ --yes Confirm
+`,
+ want: false,
+ },
+ {
+ name: "Long block mentioning --query does NOT trigger search-shape",
+ help: `Long: To delete by filter, see the related --query syntax in widgets list.
+
+Usage:
+ fixture-pp-cli widgets purge <id> [flags]
+
+Flags:
+ --force Skip confirmation
+`,
+ want: false,
+ },
+ {
+ name: "mutation command — no query signal",
+ help: `Usage:
+ fixture-pp-cli widgets delete <id> [flags]
+
+Flags:
+ --yes Confirm
+`,
+ want: false,
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ assert.Equal(t, tt.want, commandSupportsSearch(tt.help))
+ })
+ }
+}
+
func TestRunLiveDogfoodJSONFlagDetectionIsExact(t *testing.T) {
help := `Usage:
fixture-pp-cli widgets list [flags]
diff --git a/testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_list-project.go b/testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_list-project.go
index 656112f6..57dc171c 100644
--- a/testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_list-project.go
+++ b/testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_list-project.go
@@ -21,7 +21,7 @@ func newProjectsTasksListProjectCmd(flags *rootFlags) *cobra.Command {
Use: "list-project <projectId>",
Aliases: []string{"get"},
Short: "List project tasks",
- Example: " printing-press-golden-pp-cli projects tasks list-project example-value",
+ Example: " printing-press-golden-pp-cli projects tasks list-project 550e8400-e29b-41d4-a716-446655440000",
Annotations: map[string]string{"pp:endpoint": "tasks.list-project", "mcp:read-only": "true"},
RunE: func(cmd *cobra.Command, args []string) error {
if len(args) == 0 {
diff --git a/testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_update-project.go b/testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_update-project.go
index 829f1909..313acb9d 100644
--- a/testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_update-project.go
+++ b/testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/projects_tasks_update-project.go
@@ -22,7 +22,7 @@ func newProjectsTasksUpdateProjectCmd(flags *rootFlags) *cobra.Command {
Use: "update-project <projectId> <taskId>",
Aliases: []string{"update"},
Short: "Update project task",
- Example: " printing-press-golden-pp-cli projects tasks update-project example-value example-value",
+ Example: " printing-press-golden-pp-cli projects tasks update-project 550e8400-e29b-41d4-a716-446655440000 550e8400-e29b-41d4-a716-446655440000",
Annotations: map[string]string{"pp:endpoint": "tasks.update-project"},
RunE: func(cmd *cobra.Command, args []string) error {
if len(args) == 0 {
← 542835d3 fix(cli): generator template polish (WU-1, F1+F2+F3+F6) (#57
·
back to Cli Printing Press
·
fix(cli): preserve query params in generated MCP handlers (# e9696c9b →