← back to Cli Printing Press
feat(cli): add validate-narrative subcommand to replace bash recipe (#433) (#456)
8a548d098ccca4474367c5cf8268759c412e766b · 2026-04-30 22:07:56 -0700 · Trevin Chow
* feat(cli): add validate-narrative subcommand to replace bash recipe (#433)
The SKILL.md bash recipe that walked research.json's narrative commands
against a built CLI was tedious to test, hard to reuse from other
pipeline stages, and easy to skip silently. Lift it into the binary so
the same wordlist rules + --help walk are scriptable from dogfood,
scorecard, or any future stage that wants narrative validation.
The new package mirrors the bash recipe's awk wordlist semantics
exactly so Go and bash classifiers stay byte-identical. Strict mode
exits non-zero on missing commands or empty narrative; JSON mode emits
the report for downstream tooling.
Closes #433
* refactor(narrativecheck): typed Status, ExitError, cached stub
Simplify pass on the validate-narrative subcommand:
- Add typed Status const (StatusOK/StatusMissing/StatusEmptyWords) so
the producer/consumer/printer trio stops sharing magic strings.
- Wrap required-flag and strict-failure errors in ExitError with
ExitInputError, matching the verify-skill/shipcheck convention so
scripts can switch on exit codes consistently.
- Set SilenceUsage/SilenceErrors so flag errors don't dump the help
text on top of the structured failure.
- Have extractSubcommandWords return []string instead of joining and
re-splitting in classify; Words on the Result is the joined display.
- Cache the stub binary build in a sync.Once so the package's two
end-to-end tests pay one go-build cost instead of two.
- Strip trivial doc comments that just restated type names.
* refactor(cli): tighten strict-mode error + cover --strict exit code
Two polish items from the post-simplify review:
- Drop the count-replaying message from the strict-mode error wrap;
the human report already prints DONE: N ok, M missing, K empty-words
on the line above. Callers reading stderr no longer see two slightly
different count summaries.
- Add a cobra-level test that --strict wraps a ResearchEmpty result in
ExitError{ExitInputError}, covering the path where strict mode
promotes a non-failure report to an exit-code failure.
Files touched
M internal/cli/root.goA internal/cli/validate_narrative.goA internal/cli/validate_narrative_test.goA internal/narrativecheck/narrativecheck.goA internal/narrativecheck/narrativecheck_test.goM skills/printing-press/SKILL.md
Diff
commit 8a548d098ccca4474367c5cf8268759c412e766b
Author: Trevin Chow <trevin@trevinchow.com>
Date: Thu Apr 30 22:07:56 2026 -0700
feat(cli): add validate-narrative subcommand to replace bash recipe (#433) (#456)
* feat(cli): add validate-narrative subcommand to replace bash recipe (#433)
The SKILL.md bash recipe that walked research.json's narrative commands
against a built CLI was tedious to test, hard to reuse from other
pipeline stages, and easy to skip silently. Lift it into the binary so
the same wordlist rules + --help walk are scriptable from dogfood,
scorecard, or any future stage that wants narrative validation.
The new package mirrors the bash recipe's awk wordlist semantics
exactly so Go and bash classifiers stay byte-identical. Strict mode
exits non-zero on missing commands or empty narrative; JSON mode emits
the report for downstream tooling.
Closes #433
* refactor(narrativecheck): typed Status, ExitError, cached stub
Simplify pass on the validate-narrative subcommand:
- Add typed Status const (StatusOK/StatusMissing/StatusEmptyWords) so
the producer/consumer/printer trio stops sharing magic strings.
- Wrap required-flag and strict-failure errors in ExitError with
ExitInputError, matching the verify-skill/shipcheck convention so
scripts can switch on exit codes consistently.
- Set SilenceUsage/SilenceErrors so flag errors don't dump the help
text on top of the structured failure.
- Have extractSubcommandWords return []string instead of joining and
re-splitting in classify; Words on the Result is the joined display.
- Cache the stub binary build in a sync.Once so the package's two
end-to-end tests pay one go-build cost instead of two.
- Strip trivial doc comments that just restated type names.
* refactor(cli): tighten strict-mode error + cover --strict exit code
Two polish items from the post-simplify review:
- Drop the count-replaying message from the strict-mode error wrap;
the human report already prints DONE: N ok, M missing, K empty-words
on the line above. Callers reading stderr no longer see two slightly
different count summaries.
- Add a cobra-level test that --strict wraps a ResearchEmpty result in
ExitError{ExitInputError}, covering the path where strict mode
promotes a non-failure report to an exit-code failure.
---
internal/cli/root.go | 1 +
internal/cli/validate_narrative.go | 107 ++++++++++
internal/cli/validate_narrative_test.go | 70 +++++++
internal/narrativecheck/narrativecheck.go | 214 +++++++++++++++++++
internal/narrativecheck/narrativecheck_test.go | 279 +++++++++++++++++++++++++
skills/printing-press/SKILL.md | 66 +-----
6 files changed, 682 insertions(+), 55 deletions(-)
diff --git a/internal/cli/root.go b/internal/cli/root.go
index f54717c7..84ba2b0e 100644
--- a/internal/cli/root.go
+++ b/internal/cli/root.go
@@ -43,6 +43,7 @@ func Execute() error {
rootCmd.AddCommand(newGenerateCmd())
rootCmd.AddCommand(newScorecardCmd())
rootCmd.AddCommand(newDogfoodCmd())
+ rootCmd.AddCommand(newValidateNarrativeCmd())
rootCmd.AddCommand(newVerifyCmd())
rootCmd.AddCommand(newVerifySkillCmd())
rootCmd.AddCommand(newEmbossCmd())
diff --git a/internal/cli/validate_narrative.go b/internal/cli/validate_narrative.go
new file mode 100644
index 00000000..26026ddd
--- /dev/null
+++ b/internal/cli/validate_narrative.go
@@ -0,0 +1,107 @@
+package cli
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "os"
+ "os/signal"
+
+ "github.com/mvanhorn/cli-printing-press/v3/internal/narrativecheck"
+ "github.com/spf13/cobra"
+)
+
+func newValidateNarrativeCmd() *cobra.Command {
+ var (
+ researchPath string
+ binaryPath string
+ strict bool
+ asJSON bool
+ )
+
+ cmd := &cobra.Command{
+ Use: "validate-narrative",
+ Short: "Verify research.json narrative commands resolve in a built CLI's Cobra tree",
+ SilenceUsage: true,
+ SilenceErrors: true,
+ Long: `Walks every narrative.quickstart[].command and narrative.recipes[].command
+in research.json and runs '<binary> <words> --help' to confirm each command
+path exists. Replaces the bash recipe in skills/printing-press/SKILL.md so
+the same check is testable, scriptable, and reusable from dogfood/scorecard.
+
+Without this check, broken commands ship to the README's Quick Start and
+the SKILL's recipes; users hit "unknown command" on copy-paste.`,
+ Example: ` # Default: warn-only, exits 0 even when commands are missing
+ printing-press validate-narrative \
+ --research $API_RUN_DIR/research.json \
+ --binary $CLI_WORK_DIR/myapi-pp-cli
+
+ # Strict: exits non-zero on missing commands or empty narrative
+ printing-press validate-narrative --strict \
+ --research $API_RUN_DIR/research.json \
+ --binary $CLI_WORK_DIR/myapi-pp-cli
+
+ # JSON output for downstream tooling
+ printing-press validate-narrative --json \
+ --research $API_RUN_DIR/research.json \
+ --binary $CLI_WORK_DIR/myapi-pp-cli`,
+ RunE: func(cmd *cobra.Command, args []string) error {
+ if researchPath == "" {
+ return &ExitError{Code: ExitInputError, Err: fmt.Errorf("--research is required")}
+ }
+ if binaryPath == "" {
+ return &ExitError{Code: ExitInputError, Err: fmt.Errorf("--binary is required")}
+ }
+
+ // Honor SIGINT so a stuck `<binary> --help` (e.g., a CLI
+ // that itself spawns a child) doesn't block forever.
+ ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
+ defer cancel()
+
+ report, err := narrativecheck.Validate(ctx, researchPath, binaryPath)
+ if err != nil {
+ return &ExitError{Code: ExitInputError, Err: err}
+ }
+
+ // Human report goes to stderr so --json on stdout pipes cleanly.
+ if asJSON {
+ if err := json.NewEncoder(cmd.OutOrStdout()).Encode(report); err != nil {
+ return err
+ }
+ } else {
+ printHumanReport(cmd.OutOrStderr(), report)
+ }
+
+ if strict && (report.HasFailures() || report.ResearchEmpty) {
+ return &ExitError{Code: ExitInputError, Err: errors.New("narrative validation failed")}
+ }
+ return nil
+ },
+ }
+ cmd.Flags().StringVar(&researchPath, "research", "", "Path to research.json (required)")
+ cmd.Flags().StringVar(&binaryPath, "binary", "", "Path to the built CLI binary to walk (required)")
+ cmd.Flags().BoolVar(&strict, "strict", false, "Exit non-zero on any missing command or empty narrative (default: warn-only)")
+ cmd.Flags().BoolVar(&asJSON, "json", false, "Emit machine-readable JSON instead of the human report")
+ return cmd
+}
+
+func printHumanReport(w io.Writer, report *narrativecheck.Report) {
+ if report.ResearchEmpty {
+ fmt.Fprintln(w, "WARNING: research.json has no narrative.quickstart or narrative.recipes entries")
+ }
+ for _, r := range report.Results {
+ switch r.Status {
+ case narrativecheck.StatusMissing:
+ fmt.Fprintf(w, "MISSING [%s]: %s → %s\n", r.Section, r.Command, r.Words)
+ case narrativecheck.StatusEmptyWords:
+ fmt.Fprintf(w, "EMPTY [%s]: %s has no subcommand words to verify\n", r.Section, r.Command)
+ }
+ }
+ if report.Missing+report.Empty == 0 && !report.ResearchEmpty {
+ fmt.Fprintf(w, "OK: %d narrative commands resolved against the CLI tree\n", report.Walked)
+ return
+ }
+ fmt.Fprintf(w, "DONE: %d ok, %d missing, %d empty-words\n", report.Walked, report.Missing, report.Empty)
+}
diff --git a/internal/cli/validate_narrative_test.go b/internal/cli/validate_narrative_test.go
new file mode 100644
index 00000000..2c75bd69
--- /dev/null
+++ b/internal/cli/validate_narrative_test.go
@@ -0,0 +1,70 @@
+package cli
+
+import (
+ "bytes"
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+func TestValidateNarrativeCmd_RequiresFlags(t *testing.T) {
+ t.Parallel()
+
+ cases := []struct {
+ name string
+ args []string
+ want string
+ }{
+ {"no flags", nil, "--research is required"},
+ {"only research", []string{"--research", "/dev/null"}, "--binary is required"},
+ }
+
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ t.Parallel()
+ cmd := newValidateNarrativeCmd()
+ cmd.SetArgs(tc.args)
+ cmd.SetOut(new(bytes.Buffer))
+ cmd.SetErr(new(bytes.Buffer))
+
+ err := cmd.Execute()
+ if err == nil {
+ t.Fatalf("expected error mentioning %q, got nil", tc.want)
+ }
+ if !strings.Contains(err.Error(), tc.want) {
+ t.Errorf("error %q should contain %q", err.Error(), tc.want)
+ }
+ })
+ }
+}
+
+// TestValidateNarrativeCmd_StrictExitCode confirms --strict wraps a
+// ResearchEmpty (or HasFailures) result in ExitError so callers can
+// switch on shell exit codes consistently with verify-skill/shipcheck.
+func TestValidateNarrativeCmd_StrictExitCode(t *testing.T) {
+ t.Parallel()
+
+ research := filepath.Join(t.TempDir(), "research.json")
+ if err := os.WriteFile(research, []byte(`{"narrative":{}}`), 0o644); err != nil {
+ t.Fatal(err)
+ }
+
+ cmd := newValidateNarrativeCmd()
+ cmd.SetArgs([]string{"--strict", "--research", research, "--binary", "/nonexistent-but-not-invoked"})
+ cmd.SetOut(new(bytes.Buffer))
+ cmd.SetErr(new(bytes.Buffer))
+
+ err := cmd.Execute()
+ if err == nil {
+ t.Fatal("expected --strict to fail on empty narrative, got nil")
+ }
+ var exitErr *ExitError
+ if !errors.As(err, &exitErr) {
+ t.Fatalf("expected *ExitError, got %T: %v", err, err)
+ }
+ if exitErr.Code != ExitInputError {
+ t.Errorf("Code = %d, want ExitInputError (%d)", exitErr.Code, ExitInputError)
+ }
+}
diff --git a/internal/narrativecheck/narrativecheck.go b/internal/narrativecheck/narrativecheck.go
new file mode 100644
index 00000000..dc860fd2
--- /dev/null
+++ b/internal/narrativecheck/narrativecheck.go
@@ -0,0 +1,214 @@
+// Package narrativecheck validates that command strings in
+// research.json's narrative.quickstart and narrative.recipes resolve
+// against a built printed-CLI binary's Cobra tree.
+//
+// The narrative is LLM-authored (or hand-edited) and easily drifts from
+// the actual CLI surface — e.g., research.json names `<cli> stats` but
+// the real shape is `<cli> reports stats`, or a command was dropped
+// because its endpoint had a complex body. Without this check, broken
+// commands ship to the README's Quick Start and the SKILL's recipes;
+// users hit "unknown command" on their very first copy-paste.
+package narrativecheck
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "os/exec"
+ "strings"
+)
+
+// Section names the narrative section a command lives in. Matches the
+// JSON path used by the bash recipe this package replaces, so log
+// output is consistent across the two implementations.
+type Section string
+
+const (
+ SectionQuickstart Section = "quickstart"
+ SectionRecipes Section = "recipes"
+)
+
+// Status is a command's classification after the --help walk.
+type Status string
+
+const (
+ StatusOK Status = "ok"
+ StatusMissing Status = "missing"
+ StatusEmptyWords Status = "empty-words"
+)
+
+type Result struct {
+ Section Section `json:"section"`
+ Command string `json:"command"`
+ // Words is the extracted subcommand path (e.g., `reports stats`)
+ // after stripping the binary name and the first --flag/positional.
+ // Empty when the command was a bare binary or pure-flag invocation.
+ Words string `json:"words,omitempty"`
+ Status Status `json:"status"`
+ Error string `json:"error,omitempty"`
+}
+
+type Report struct {
+ Walked int `json:"walked"`
+ Missing int `json:"missing"`
+ Empty int `json:"empty"`
+ Results []Result `json:"results"`
+ // ResearchEmpty is true when neither narrative.quickstart nor
+ // narrative.recipes contained any entries. The LLM may have
+ // omitted both sections by mistake; the caller's --strict flag
+ // can decide whether that's an error.
+ ResearchEmpty bool `json:"research_empty,omitempty"`
+}
+
+// Validate parses researchPath, walks every narrative.quickstart and
+// narrative.recipes command, and resolves it against the binary's
+// Cobra tree by running `<binary> <words> --help`. ctx scopes every
+// subprocess so callers can interrupt cleanly.
+func Validate(ctx context.Context, researchPath, binaryPath string) (*Report, error) {
+ commands, err := loadCommands(researchPath)
+ if err != nil {
+ return nil, err
+ }
+
+ report := &Report{
+ Results: make([]Result, 0, len(commands)),
+ ResearchEmpty: len(commands) == 0,
+ }
+ for _, sc := range commands {
+ r := classify(ctx, binaryPath, sc.Section, sc.Command)
+ switch r.Status {
+ case StatusOK:
+ report.Walked++
+ case StatusMissing:
+ report.Missing++
+ case StatusEmptyWords:
+ report.Empty++
+ }
+ report.Results = append(report.Results, r)
+ }
+ return report, nil
+}
+
+// HasFailures reports whether the run found any missing or empty-words
+// entries. Callers gate --strict exit codes on this.
+func (r *Report) HasFailures() bool {
+ return r.Missing > 0 || r.Empty > 0
+}
+
+type sectionCommand struct {
+ Section Section
+ Command string
+}
+
+func loadCommands(researchPath string) ([]sectionCommand, error) {
+ data, err := os.ReadFile(researchPath)
+ if err != nil {
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil, fmt.Errorf("research file %s not found; cannot validate narrative commands", researchPath)
+ }
+ return nil, fmt.Errorf("reading %s: %w", researchPath, err)
+ }
+
+ // Decode just the narrative subtree we care about. Tolerates extra
+ // fields in research.json (the schema is wider than narrative).
+ var doc struct {
+ Narrative struct {
+ Quickstart []struct {
+ Command string `json:"command"`
+ } `json:"quickstart"`
+ Recipes []struct {
+ Command string `json:"command"`
+ } `json:"recipes"`
+ } `json:"narrative"`
+ }
+ if err := json.Unmarshal(data, &doc); err != nil {
+ return nil, fmt.Errorf("%s is not valid JSON: %w", researchPath, err)
+ }
+
+ var out []sectionCommand
+ for _, q := range doc.Narrative.Quickstart {
+ if cmd := strings.TrimSpace(q.Command); cmd != "" {
+ out = append(out, sectionCommand{Section: SectionQuickstart, Command: cmd})
+ }
+ }
+ for _, r := range doc.Narrative.Recipes {
+ if cmd := strings.TrimSpace(r.Command); cmd != "" {
+ out = append(out, sectionCommand{Section: SectionRecipes, Command: cmd})
+ }
+ }
+ return out, nil
+}
+
+// classify mirrors the bash recipe's wordlist rule: drop the leading
+// binary name, keep words until the first flag (starts with `-`) or
+// non-identifier character. Hyphens stay because Cobra subcommands use
+// them (`list-projects`).
+func classify(ctx context.Context, binaryPath string, section Section, command string) Result {
+ words := extractSubcommandWords(command)
+ r := Result{Section: section, Command: command, Words: strings.Join(words, " ")}
+
+ if len(words) == 0 {
+ r.Status = StatusEmptyWords
+ r.Error = "command has no subcommand words to verify (bare binary or pure-flag invocation)"
+ return r
+ }
+
+ args := append(words, "--help")
+ if err := exec.CommandContext(ctx, binaryPath, args...).Run(); err != nil {
+ r.Status = StatusMissing
+ r.Error = fmt.Sprintf("%s %s --help failed: %v", binaryPath, r.Words, err)
+ return r
+ }
+
+ r.Status = StatusOK
+ return r
+}
+
+// extractSubcommandWords replicates the bash recipe's awk wordlist
+// extraction so the Go and bash implementations classify identically:
+//
+// for (i=2; i<=NF; i++) {
+// if ($i ~ /^-/ || $i ~ /[^a-zA-Z0-9_-]/) break
+// print $i
+// }
+//
+// Strip the first token (binary name), then keep tokens until the first
+// flag or any token containing a character outside [A-Za-z0-9_-].
+func extractSubcommandWords(command string) []string {
+ tokens := strings.Fields(command)
+ if len(tokens) <= 1 {
+ return nil
+ }
+ var words []string
+ for _, tok := range tokens[1:] {
+ if strings.HasPrefix(tok, "-") || !isIdentifierToken(tok) {
+ break
+ }
+ words = append(words, tok)
+ }
+ return words
+}
+
+// isIdentifierToken reports whether s contains only ASCII alphanumerics,
+// underscores, and hyphens. Anything else (=, :, /, quotes, JSON-string
+// arguments, etc.) signals the start of a non-subcommand token and ends
+// the wordlist scan.
+func isIdentifierToken(s string) bool {
+ if s == "" {
+ return false
+ }
+ for _, r := range s {
+ switch {
+ case r >= 'a' && r <= 'z':
+ case r >= 'A' && r <= 'Z':
+ case r >= '0' && r <= '9':
+ case r == '_' || r == '-':
+ default:
+ return false
+ }
+ }
+ return true
+}
diff --git a/internal/narrativecheck/narrativecheck_test.go b/internal/narrativecheck/narrativecheck_test.go
new file mode 100644
index 00000000..8d874598
--- /dev/null
+++ b/internal/narrativecheck/narrativecheck_test.go
@@ -0,0 +1,279 @@
+package narrativecheck
+
+import (
+ "context"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "sync"
+ "testing"
+)
+
+// TestExtractSubcommandWords pins the wordlist rule against the bash
+// recipe it replaces. Each case is a research.json `command` string;
+// the want is what the bash recipe's awk pipeline would produce.
+func TestExtractSubcommandWords(t *testing.T) {
+ t.Parallel()
+
+ cases := []struct {
+ name string
+ in string
+ want string
+ }{
+ {"single subcommand", "mycli widgets", "widgets"},
+ {"nested subcommands", "mycli reports stats", "reports stats"},
+ {"hyphenated subcommand", "mycli list-projects", "list-projects"},
+ {"deep nesting", "mycli a b c d", "a b c d"},
+ {"trailing flag", "mycli widgets list --json", "widgets list"},
+ {"trailing flag with value", "mycli widgets list --since 7d", "widgets list"},
+ {"flag mid-tokens", "mycli widgets --since 7d list", "widgets"},
+ {"positional value with equals", "mycli widgets q=hello", "widgets"},
+ // awk matches the whole token against the non-identifier regex,
+ // so "ns:resource" emits nothing (not "ns").
+ {"positional value with colon", "mycli ns:resource list", ""},
+ {"bare binary", "mycli", ""},
+ {"binary plus flag only", "mycli --version", ""},
+ {"empty string", "", ""},
+ {"single token with leading dash", "--help", ""},
+ }
+
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ t.Parallel()
+ got := strings.Join(extractSubcommandWords(tc.in), " ")
+ if got != tc.want {
+ t.Errorf("extractSubcommandWords(%q) = %q, want %q", tc.in, got, tc.want)
+ }
+ })
+ }
+}
+
+// TestLoadCommands_Shapes covers the JSON-parsing contract: missing
+// file, malformed JSON, empty narrative (both sections empty), partial
+// narrative (one section populated).
+func TestLoadCommands_Shapes(t *testing.T) {
+ t.Parallel()
+
+ t.Run("missing file", func(t *testing.T) {
+ t.Parallel()
+ _, err := loadCommands(filepath.Join(t.TempDir(), "nope.json"))
+ if err == nil {
+ t.Fatal("expected error for missing file")
+ }
+ if !strings.Contains(err.Error(), "not found") {
+ t.Errorf("error %q should mention 'not found'", err)
+ }
+ })
+
+ t.Run("malformed JSON", func(t *testing.T) {
+ t.Parallel()
+ path := writeFile(t, "{ not json")
+ _, err := loadCommands(path)
+ if err == nil || !strings.Contains(err.Error(), "not valid JSON") {
+ t.Errorf("error %v should mention 'not valid JSON'", err)
+ }
+ })
+
+ t.Run("no narrative section at all", func(t *testing.T) {
+ t.Parallel()
+ path := writeFile(t, `{"other_field": "ignored"}`)
+ got, err := loadCommands(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(got) != 0 {
+ t.Errorf("expected 0 commands, got %d", len(got))
+ }
+ })
+
+ t.Run("only quickstart populated", func(t *testing.T) {
+ t.Parallel()
+ path := writeFile(t, `{"narrative":{"quickstart":[{"command":"mycli a"},{"command":"mycli b"}]}}`)
+ got, err := loadCommands(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(got) != 2 || got[0].Section != SectionQuickstart {
+ t.Errorf("expected 2 quickstart entries, got %+v", got)
+ }
+ })
+
+ t.Run("both sections populated, order preserved", func(t *testing.T) {
+ t.Parallel()
+ path := writeFile(t, `{"narrative":{
+ "quickstart":[{"command":"mycli q1"}],
+ "recipes":[{"command":"mycli r1"},{"command":"mycli r2"}]
+ }}`)
+ got, err := loadCommands(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(got) != 3 {
+ t.Fatalf("expected 3 commands, got %d", len(got))
+ }
+ if got[0].Section != SectionQuickstart || got[1].Section != SectionRecipes {
+ t.Errorf("expected quickstart before recipes, got %+v", got)
+ }
+ })
+
+ t.Run("empty command strings are dropped", func(t *testing.T) {
+ t.Parallel()
+ path := writeFile(t, `{"narrative":{"quickstart":[{"command":""},{"command":" "},{"command":"mycli x"}]}}`)
+ got, err := loadCommands(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(got) != 1 || got[0].Command != "mycli x" {
+ t.Errorf("expected single non-empty command, got %+v", got)
+ }
+ })
+}
+
+// TestValidate_EndToEnd builds a tiny stub binary that responds OK to
+// some commands and "unknown command" to others, then runs Validate
+// across a fixture research.json. Confirms the resolution pipeline
+// (parse → words → exec → classify) end-to-end.
+func TestValidate_EndToEnd(t *testing.T) {
+ t.Parallel()
+
+ binary := buildStubBinary(t)
+ research := writeFile(t, `{"narrative":{
+ "quickstart":[
+ {"command":"stub widgets list"},
+ {"command":"stub typo-here"},
+ {"command":"stub --version"}
+ ],
+ "recipes":[
+ {"command":"stub widgets show 42"}
+ ]
+ }}`)
+
+ report, err := Validate(context.Background(), research, binary)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ if report.Walked != 2 {
+ t.Errorf("Walked = %d, want 2 (widgets-list, widgets-show)", report.Walked)
+ }
+ if report.Missing != 1 {
+ t.Errorf("Missing = %d, want 1 (typo-here)", report.Missing)
+ }
+ if report.Empty != 1 {
+ t.Errorf("Empty = %d, want 1 (--version is bare-flag)", report.Empty)
+ }
+ if !report.HasFailures() {
+ t.Error("HasFailures should be true with missing+empty entries")
+ }
+
+ // Verify per-result classification + section attribution
+ bySection := map[Section]int{}
+ for _, r := range report.Results {
+ bySection[r.Section]++
+ }
+ if bySection[SectionQuickstart] != 3 || bySection[SectionRecipes] != 1 {
+ t.Errorf("section counts wrong: %+v", bySection)
+ }
+}
+
+// TestValidate_EmptyResearchFlagsResearchEmpty covers the LLM-omitted-
+// both-sections case.
+func TestValidate_EmptyResearchFlagsResearchEmpty(t *testing.T) {
+ t.Parallel()
+
+ research := writeFile(t, `{"narrative":{}}`)
+ binary := buildStubBinary(t)
+
+ report, err := Validate(context.Background(), research, binary)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !report.ResearchEmpty {
+ t.Error("ResearchEmpty should be true when both sections are empty")
+ }
+ if report.Walked != 0 || report.Missing != 0 {
+ t.Errorf("expected no walked or missing entries, got walked=%d missing=%d", report.Walked, report.Missing)
+ }
+}
+
+// writeFile writes content to a temp file and returns the path.
+func writeFile(t *testing.T, content string) string {
+ t.Helper()
+ path := filepath.Join(t.TempDir(), "research.json")
+ if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ return path
+}
+
+// buildStubBinary compiles a small Go program that simulates a printed
+// CLI: it accepts `widgets list --help`, `widgets show <id> --help`,
+// and exits non-zero for anything else. The stub is the most direct
+// way to test the exec path without depending on a fully generated CLI.
+//
+// The build is cached across tests via sync.Once — go build is the
+// slowest step in the package's test runtime.
+var (
+ stubOnce sync.Once
+ stubPath string
+ stubErr error
+)
+
+func buildStubBinary(t *testing.T) string {
+ t.Helper()
+ stubOnce.Do(func() {
+ src := `package main
+
+import (
+ "fmt"
+ "os"
+ "strings"
+)
+
+var validPathPrefixes = []string{
+ "widgets list",
+ "widgets show",
+}
+
+func main() {
+ args := os.Args[1:]
+ var path []string
+ for _, a := range args {
+ if strings.HasPrefix(a, "-") {
+ break
+ }
+ path = append(path, a)
+ }
+ joined := strings.Join(path, " ")
+ for _, prefix := range validPathPrefixes {
+ if joined == prefix || strings.HasPrefix(joined, prefix+" ") {
+ fmt.Println("usage stub:", prefix)
+ return
+ }
+ }
+ fmt.Fprintln(os.Stderr, "unknown command:", joined)
+ os.Exit(1)
+}
+`
+ dir, err := os.MkdirTemp("", "narrativecheck-stub-")
+ if err != nil {
+ stubErr = err
+ return
+ }
+ srcPath := filepath.Join(dir, "stub.go")
+ if err := os.WriteFile(srcPath, []byte(src), 0o644); err != nil {
+ stubErr = err
+ return
+ }
+ stubPath = filepath.Join(dir, "stub")
+ if out, err := exec.Command("go", "build", "-o", stubPath, srcPath).CombinedOutput(); err != nil {
+ stubErr = fmt.Errorf("building stub: %v\n%s", err, out)
+ }
+ })
+ if stubErr != nil {
+ t.Fatal(stubErr)
+ }
+ return stubPath
+}
diff --git a/skills/printing-press/SKILL.md b/skills/printing-press/SKILL.md
index 26f9592e..1441f3d2 100644
--- a/skills/printing-press/SKILL.md
+++ b/skills/printing-press/SKILL.md
@@ -1671,68 +1671,24 @@ the broken commands ship to the README's Quick Start (`narrative.quickstart`) an
SKILL's recipes (`narrative.recipes`); users copy-paste them and hit `unknown command`
on the very first invocation.
-Build the CLI binary first (the CLI does not need to run against a live API for this
-check; `--help` is offline). Then for each `narrative.quickstart[].command` and
-`narrative.recipes[].command`, strip the binary name and trailing arguments to get the
-command path, and confirm it walks the Cobra tree:
+Build the CLI binary, then run `printing-press validate-narrative` against it. The
+subcommand walks every `narrative.quickstart[].command` and `narrative.recipes[].command`,
+strips the binary name and trailing arguments, and runs `<binary> <words> --help` for
+each. The check is offline — no live API access needed.
```bash
QUICKSTART_BINARY="$CLI_WORK_DIR/<api>-pp-cli"
go build -o "$QUICKSTART_BINARY" "$CLI_WORK_DIR/cmd/<api>-pp-cli"
-# Fail loudly if research.json is missing or malformed — silent jq output
-# would otherwise pass an empty pipeline through the loop and falsely
-# report "everything looks fine" when nothing was actually checked.
-if [ ! -f "$API_RUN_DIR/research.json" ]; then
- echo "ERROR: $API_RUN_DIR/research.json not found; cannot validate narrative commands" >&2
- exit 1
-fi
-if ! jq empty "$API_RUN_DIR/research.json" >/dev/null 2>&1; then
- echo "ERROR: $API_RUN_DIR/research.json is not valid JSON" >&2
- exit 1
-fi
-
-# Track whether any narrative command was actually walked. An empty quickstart
-# AND empty recipes list is itself worth flagging — the LLM authoring the
-# research.json may have omitted both sections by mistake.
-walked=0
-missing=0
-
-while IFS=$'\t' read -r section cmd; do
- # Drop the leading binary name and any --flag/positional arg suffix.
- # Keep only the literal subcommand words (everything before the first
- # word that starts with `-` or that contains `=`/`:`/non-alphanumerics).
- words=$(printf '%s\n' "$cmd" \
- | awk '{ for (i=2; i<=NF; i++) { if ($i ~ /^-/ || $i ~ /[^a-zA-Z0-9_-]/) break; printf "%s ", $i } }')
- # Strip trailing whitespace; awk's "%s " emits a trailing space.
- words=$(printf '%s' "$words" | sed 's/[[:space:]]*$//')
- if [ -z "$words" ]; then
- # Bare-binary or pure-flag commands ("<cli>", "<cli> --version") have
- # nothing for `--help` to validate. Flag instead of silently passing.
- echo "EMPTY [$section]: $cmd has no subcommand words to verify" >&2
- missing=$((missing + 1))
- continue
- fi
- walked=$((walked + 1))
- # shellcheck disable=SC2086 # words is a deliberate splat into argv
- if ! "$QUICKSTART_BINARY" $words --help >/dev/null 2>&1; then
- echo "MISSING [$section]: $cmd → $words" >&2
- missing=$((missing + 1))
- fi
-done < <(jq -r '
- ((.narrative.quickstart // []) | .[] | "quickstart\t" + .command),
- ((.narrative.recipes // []) | .[] | "recipes\t" + .command)
-' "$API_RUN_DIR/research.json")
-
-if [ "$walked" -eq 0 ] && [ "$missing" -eq 0 ]; then
- echo "WARNING: research.json has no narrative.quickstart or narrative.recipes entries" >&2
-fi
-if [ "$missing" -gt 0 ]; then
- echo "ERROR: $missing narrative command(s) failed validation; fix research.json before continuing" >&2
- exit 1
-fi
+printing-press validate-narrative --strict \
+ --research "$API_RUN_DIR/research.json" \
+ --binary "$QUICKSTART_BINARY"
```
+`--strict` exits non-zero on any missing command, empty subcommand-words entry, or
+empty narrative (both sections omitted). Drop `--strict` to get a warn-only report,
+or add `--json` for machine-readable output.
+
If any commands are reported missing, fix them in `research.json` before continuing.
Common causes:
← d909d6cf Fix docs-only test filtering (#455)
·
back to Cli Printing Press
·
refactor(cli): extract body-map block to a shared helper (#4 2c59d1c6 →