← back to Cli Printing Press
feat(cli): add unknown-command check + sync test for verify-skill (#339)
907c6986b51119eb767a0d82f8d33361b62e1499 · 2026-04-27 02:35:46 -0700 · Trevin Chow
The bundled verify-skill script (internal/cli/verify_skill_bundled.py)
that ships in the printing-press binary was running 3 checks: flag-names,
flag-commands, positional-args. The library repo's CI runs a fork that
adds an unknown-command check — every command path referenced in SKILL.md
must map to a real cobra Use: declaration. That fork rejected the dub
PR (mvanhorn/printing-press-library#135) for phantom paths like `qr
get-qrcode` while local Phase 4 reported PASS. Refs issue #336 (U2).
The two scripts in this repo (scripts/verify-skill/verify_skill.py and
internal/cli/verify_skill_bundled.py) were already byte-identical at HEAD
— both 785 lines, neither implementing unknown-command. The retro's "200
lines drift" was a snapshot at a past commit; whatever drift existed had
been manually reconciled. So this PR has two parts:
1. Port the unknown-command check.
The upstream library-repo script and this repo's canonical have
diverged ~1,400 lines — they're forks with different helper graphs.
A verbatim lift would not work. The new check is re-derived using
this repo's existing find_command_source helper (which walks the
rootCmd.AddCommand graph and resolves multi-level paths cleanly,
richer than upstream's). It surfaces command paths from two surfaces:
- Bash recipes (already extracted by extract_recipes for the other
checks; previously dropped silently when the command was missing)
- Inline backtick references inside the `## Command Reference`
section (scoped to that section to avoid false positives in
narrative prose)
Each unique path is reported at most once. Cobra's auto-registered
builtins (help, completion, version) are whitelisted.
Updates to scripts/verify-skill/verify_skill.py:
- Add COMMAND_REFERENCE_SECTION_RE and BUILTIN_COMMANDS constants
- Add _extract_inline_commands helper
- Add check_unknown_commands function
- Wire into run_checks dispatcher and --only argparse choices
- Update header docstring (4 checks now)
2. Drift prevention.
internal/cli/verify_skill_sync_test.go adds TestVerifySkillScriptInSync
which sha256-hashes both files and fails if they differ, with a clear
error pointing the developer at the canonical as source-of-truth and
the lefthook command to regenerate.
lefthook.yml gains a pre-commit `sync-verify-skill` hook that copies
canonical → bundled and re-stages the bundled file. Mirrors the
existing fmt block's `stage_fixed: true` shape. The hook prevents
drift on the happy path; the Go sync test catches `--no-verify`.
Three new integration tests in
internal/cli/verify_skill_unknown_command_test.go:
- TestVerifySkill_DetectsUnknownCommand — phantom path → exit 1 +
"[unknown-command]" + "closest existing prefix is `<cli> qr`"
- TestVerifySkill_UnknownCommandPassesWhenAllPathsResolve — clean
SKILL passes
- TestVerifySkill_UnknownCommandSkipsBuiltins — help/completion/version
not flagged
Validation against existing library CLIs found real findings on
allrecipes (4 phantom paths: category, cuisine, ingredient, occasion)
and cal-com (3 phantom paths: api-keys/keys-refresh, auth/oauth2-*).
These are pre-existing bugs that the new check will surface on next
regeneration. The dub library copy passes (was hand-fixed in PR #135).
Refs #336, #333
Files touched
M internal/cli/verify_skill_bundled.pyA internal/cli/verify_skill_sync_test.goA internal/cli/verify_skill_unknown_command_test.goM lefthook.ymlM scripts/verify-skill/verify_skill.py
Diff
commit 907c6986b51119eb767a0d82f8d33361b62e1499
Author: Trevin Chow <trevin@trevinchow.com>
Date: Mon Apr 27 02:35:46 2026 -0700
feat(cli): add unknown-command check + sync test for verify-skill (#339)
The bundled verify-skill script (internal/cli/verify_skill_bundled.py)
that ships in the printing-press binary was running 3 checks: flag-names,
flag-commands, positional-args. The library repo's CI runs a fork that
adds an unknown-command check — every command path referenced in SKILL.md
must map to a real cobra Use: declaration. That fork rejected the dub
PR (mvanhorn/printing-press-library#135) for phantom paths like `qr
get-qrcode` while local Phase 4 reported PASS. Refs issue #336 (U2).
The two scripts in this repo (scripts/verify-skill/verify_skill.py and
internal/cli/verify_skill_bundled.py) were already byte-identical at HEAD
— both 785 lines, neither implementing unknown-command. The retro's "200
lines drift" was a snapshot at a past commit; whatever drift existed had
been manually reconciled. So this PR has two parts:
1. Port the unknown-command check.
The upstream library-repo script and this repo's canonical have
diverged ~1,400 lines — they're forks with different helper graphs.
A verbatim lift would not work. The new check is re-derived using
this repo's existing find_command_source helper (which walks the
rootCmd.AddCommand graph and resolves multi-level paths cleanly,
richer than upstream's). It surfaces command paths from two surfaces:
- Bash recipes (already extracted by extract_recipes for the other
checks; previously dropped silently when the command was missing)
- Inline backtick references inside the `## Command Reference`
section (scoped to that section to avoid false positives in
narrative prose)
Each unique path is reported at most once. Cobra's auto-registered
builtins (help, completion, version) are whitelisted.
Updates to scripts/verify-skill/verify_skill.py:
- Add COMMAND_REFERENCE_SECTION_RE and BUILTIN_COMMANDS constants
- Add _extract_inline_commands helper
- Add check_unknown_commands function
- Wire into run_checks dispatcher and --only argparse choices
- Update header docstring (4 checks now)
2. Drift prevention.
internal/cli/verify_skill_sync_test.go adds TestVerifySkillScriptInSync
which sha256-hashes both files and fails if they differ, with a clear
error pointing the developer at the canonical as source-of-truth and
the lefthook command to regenerate.
lefthook.yml gains a pre-commit `sync-verify-skill` hook that copies
canonical → bundled and re-stages the bundled file. Mirrors the
existing fmt block's `stage_fixed: true` shape. The hook prevents
drift on the happy path; the Go sync test catches `--no-verify`.
Three new integration tests in
internal/cli/verify_skill_unknown_command_test.go:
- TestVerifySkill_DetectsUnknownCommand — phantom path → exit 1 +
"[unknown-command]" + "closest existing prefix is `<cli> qr`"
- TestVerifySkill_UnknownCommandPassesWhenAllPathsResolve — clean
SKILL passes
- TestVerifySkill_UnknownCommandSkipsBuiltins — help/completion/version
not flagged
Validation against existing library CLIs found real findings on
allrecipes (4 phantom paths: category, cuisine, ingredient, occasion)
and cal-com (3 phantom paths: api-keys/keys-refresh, auth/oauth2-*).
These are pre-existing bugs that the new check will surface on next
regeneration. The dub library copy passes (was hand-fixed in PR #135).
Refs #336, #333
---
internal/cli/verify_skill_bundled.py | 127 ++++++++++++++++++-
internal/cli/verify_skill_sync_test.go | 87 +++++++++++++
internal/cli/verify_skill_unknown_command_test.go | 146 ++++++++++++++++++++++
lefthook.yml | 11 ++
scripts/verify-skill/verify_skill.py | 127 ++++++++++++++++++-
5 files changed, 492 insertions(+), 6 deletions(-)
diff --git a/internal/cli/verify_skill_bundled.py b/internal/cli/verify_skill_bundled.py
index 7f3a41b0..915ee4c7 100755
--- a/internal/cli/verify_skill_bundled.py
+++ b/internal/cli/verify_skill_bundled.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""verify_skill.py — validate that SKILL.md matches the shipped CLI source.
-Three checks run in sequence:
+Four checks run in sequence:
1. flag-names — every `--flag` in SKILL.md is declared as a cobra flag
somewhere in internal/cli/*.go.
@@ -9,6 +9,11 @@ Three checks run in sequence:
on that command (or as a persistent/root flag).
3. positional-args — positional args in bash recipes match the command's
`Use:` field signature (required + optional + variadic).
+ 4. unknown-command — every command path referenced in SKILL.md (in bash
+ recipes and inline backticks under `## Command Reference`) maps to a
+ real cobra `Use:` declaration in internal/cli/*.go. Catches docs that
+ promise commands the binary does not implement (e.g. SKILL.md lists
+ `qr get-qrcode` but the CLI only registers a leaf `qr` after promotion).
The checks are pattern-matching heuristics against Go AST-adjacent text.
False positives are possible for edge cases:
@@ -25,6 +30,7 @@ USAGE
python3 verify_skill.py --dir <cli-dir>
python3 verify_skill.py --dir <cli-dir> --json
python3 verify_skill.py --dir <cli-dir> --only flag-names
+ python3 verify_skill.py --dir <cli-dir> --only unknown-command
python3 verify_skill.py --dir <cli-dir> --strict # treat known-FPs as failures
Exit codes:
@@ -57,6 +63,15 @@ COMMON_FLAGS = {
}
CODEBLOCK_BASH = re.compile(r"```bash\n(.*?)\n```", re.DOTALL)
+COMMAND_REFERENCE_SECTION_RE = re.compile(
+ r"^##\s+Command\s+Reference\s*$(.*?)(?=^##\s+|\Z)",
+ re.DOTALL | re.MULTILINE | re.IGNORECASE,
+)
+# Cobra registers help/completion automatically; treat as always-present.
+# Other CLIs may surface version as a real cobra command, but it is also a
+# common --version flag pattern; we conservatively whitelist it too so a
+# `<binary> version` reference never fires this check.
+BUILTIN_COMMANDS = {"help", "completion", "version"}
USE_RE = re.compile(r'Use:\s*"([^"]+)"')
ARGS_RE = re.compile(
r'Args:\s*cobra\.(ExactArgs|MinimumNArgs|MaximumNArgs|RangeArgs|NoArgs|OnlyValidArgs|ExactValidArgs)\s*\(([^)]*)\)'
@@ -666,6 +681,109 @@ def check_positional_args(cli_dir: Path, skill: Path, cli_binary: str, report: R
)
+def _extract_inline_commands(skill_text: str, cli_binary: str) -> list[list[str]]:
+ """Pull `<binary> <cmd> [more]` snippets from inline backticks under the
+ `## Command Reference` section. Returns command paths only, no flags or
+ positional args (those are surfaced through the bash-recipe checks).
+
+ Why scoped to ## Command Reference: SKILL.md narrative prose mentions
+ binary names in flowing text where false positives would be high. The
+ Command Reference section is the canonical promise to the reader.
+ """
+ sec = COMMAND_REFERENCE_SECTION_RE.search(skill_text)
+ if not sec:
+ return []
+ section_body = sec.group(1)
+ binary_token = re.escape(cli_binary)
+ inline_re = re.compile(rf"`({binary_token}(?:\s+[^`]+)?)`")
+ paths: list[list[str]] = []
+ for m in inline_re.finditer(section_body):
+ snippet = m.group(1).strip()
+ after = snippet[len(cli_binary):].strip()
+ if not after:
+ continue
+ tokens = after.split()
+ cmd_path: list[str] = []
+ for t in tokens:
+ if t.startswith("-") or t.startswith("<") or t.startswith("[") \
+ or t.startswith("$") or t.startswith("\"") or t.startswith("'") \
+ or t.startswith("`") or "/" in t or "=" in t:
+ break
+ if not re.match(r"^[a-z][a-z0-9-]*$", t):
+ break
+ cmd_path.append(t)
+ if len(cmd_path) >= 3:
+ break
+ if cmd_path:
+ paths.append(cmd_path)
+ return paths
+
+
+def check_unknown_commands(cli_dir: Path, skill: Path, cli_binary: str, report: Report) -> None:
+ """Report command paths in SKILL.md that have no matching cobra Use:
+ declaration in internal/cli/*.go. Source paths come from two surfaces:
+
+ - Bash recipes (extract_recipes), which the other checks already walk
+ but skip silently when the command is missing
+ - Inline backtick references inside the `## Command Reference` section
+
+ Each unique cmd_path is reported at most once per SKILL.md.
+
+ Uses the in-repo find_command_source which walks the rootCmd.AddCommand
+ graph and resolves multi-level command paths (e.g., `links stale` vs
+ `profile save`) without false-positive collisions on shared leaf names.
+ """
+ skill_text = skill.read_text()
+ seen: set[tuple[str, ...]] = set()
+ sources: list[tuple[list[str], str]] = []
+
+ for cmd_path, _pos, _flags in extract_recipes(skill, cli_binary, cli_dir):
+ if cmd_path:
+ sources.append((cmd_path, "bash recipe"))
+ for cmd_path in _extract_inline_commands(skill_text, cli_binary):
+ sources.append((cmd_path, "Command Reference inline"))
+
+ for cmd_path, surface in sources:
+ if not cmd_path:
+ continue
+ head = cmd_path[0]
+ # Skip non-command tokens that the recipe parser may have promoted
+ # into cmd_path[0]: flags, placeholders, env vars, etc. These belong
+ # to other checks or are documentation conventions, not commands.
+ if head in BUILTIN_COMMANDS:
+ continue
+ if head.startswith(("-", "<", "[", "$")) or "=" in head:
+ continue
+ if not re.match(r"^[a-z][a-z0-9-]*$", head):
+ continue
+ key = tuple(cmd_path)
+ if key in seen:
+ continue
+ seen.add(key)
+ files, _use, _args = find_command_source(cli_dir, cmd_path)
+ if files:
+ continue
+ # Walk back to the longest existing prefix for a clearer error.
+ detail = "command path not found in internal/cli/*.go (no matching Use: declaration)"
+ for k in range(len(cmd_path) - 1, 0, -1):
+ prefix_files, _, _ = find_command_source(cli_dir, cmd_path[:k])
+ if prefix_files:
+ detail = (
+ f"command path not found in internal/cli/*.go; "
+ f"closest existing prefix is `{cli_binary} {' '.join(cmd_path[:k])}`"
+ )
+ break
+ report.findings.append(
+ Finding(
+ check="unknown-command",
+ severity="error",
+ command=f"{cli_binary} {' '.join(cmd_path)}",
+ detail=detail,
+ evidence=surface,
+ )
+ )
+
+
# ---------------------------------------------------------------------------
# Runner
# ---------------------------------------------------------------------------
@@ -697,7 +815,7 @@ def run_checks(cli_dir: Path, only: set[str] | None) -> Report:
cli_binary = derive_cli_binary(cli_dir)
report = Report(cli_dir=str(cli_dir), skill_path=str(skill))
- checks = only or {"flag-names", "flag-commands", "positional-args"}
+ checks = only or {"flag-names", "flag-commands", "positional-args", "unknown-command"}
if "flag-names" in checks:
report.checks_run.append("flag-names")
check_flag_names(cli_dir, skill, report)
@@ -707,6 +825,9 @@ def run_checks(cli_dir: Path, only: set[str] | None) -> Report:
if "positional-args" in checks:
report.checks_run.append("positional-args")
check_positional_args(cli_dir, skill, cli_binary, report)
+ if "unknown-command" in checks:
+ report.checks_run.append("unknown-command")
+ check_unknown_commands(cli_dir, skill, cli_binary, report)
return report
@@ -757,7 +878,7 @@ def main():
p.add_argument("--dir", required=True, help="CLI directory (contains SKILL.md + internal/cli/)")
p.add_argument(
"--only",
- choices=["flag-names", "flag-commands", "positional-args"],
+ choices=["flag-names", "flag-commands", "positional-args", "unknown-command"],
action="append",
help="Run only the named check(s). Pass multiple times to include multiple.",
)
diff --git a/internal/cli/verify_skill_sync_test.go b/internal/cli/verify_skill_sync_test.go
new file mode 100644
index 00000000..e7bfb658
--- /dev/null
+++ b/internal/cli/verify_skill_sync_test.go
@@ -0,0 +1,87 @@
+// Copyright 2026 trevin-chow. Licensed under Apache-2.0. See LICENSE.
+
+package cli
+
+import (
+ "crypto/sha256"
+ "encoding/hex"
+ "os"
+ "path/filepath"
+ "runtime"
+ "testing"
+)
+
+// TestVerifySkillScriptInSync ensures the script embedded into the binary
+// (internal/cli/verify_skill_bundled.py) matches the canonical script that
+// the library repo's CI runs (scripts/verify-skill/verify_skill.py).
+//
+// The two used to drift: a bundled vendor copy, hand-maintained, eventually
+// missed checks the canonical added (most recently the unknown-command
+// check, ported in U2). The fix is mechanical: a lefthook pre-commit hook
+// copies canonical → bundled on every commit that touches the canonical,
+// and this test catches any commit that bypasses the hook (--no-verify).
+//
+// To regenerate the bundled copy after editing the canonical:
+//
+// cp scripts/verify-skill/verify_skill.py internal/cli/verify_skill_bundled.py
+//
+// Or just run lefthook:
+//
+// lefthook run pre-commit
+func TestVerifySkillScriptInSync(t *testing.T) {
+ t.Parallel()
+
+ repoRoot := findRepoRoot(t)
+ canonical := filepath.Join(repoRoot, "scripts", "verify-skill", "verify_skill.py")
+ bundled := filepath.Join(repoRoot, "internal", "cli", "verify_skill_bundled.py")
+
+ canonicalBytes, err := os.ReadFile(canonical)
+ if err != nil {
+ t.Fatalf("read canonical script %s: %v", canonical, err)
+ }
+ bundledBytes, err := os.ReadFile(bundled)
+ if err != nil {
+ t.Fatalf("read bundled script %s: %v", bundled, err)
+ }
+
+ canonicalHash := sha256.Sum256(canonicalBytes)
+ bundledHash := sha256.Sum256(bundledBytes)
+
+ if canonicalHash != bundledHash {
+ t.Fatalf(
+ "verify-skill scripts have diverged. "+
+ "\n scripts/verify-skill/verify_skill.py sha256=%s (%d bytes)"+
+ "\n internal/cli/verify_skill_bundled.py sha256=%s (%d bytes)"+
+ "\n\n"+
+ "The canonical script (scripts/verify-skill/verify_skill.py) is the source of truth. "+
+ "To resync, run:\n\n"+
+ " cp scripts/verify-skill/verify_skill.py internal/cli/verify_skill_bundled.py\n\n"+
+ "Or run lefthook:\n\n"+
+ " lefthook run pre-commit",
+ hex.EncodeToString(canonicalHash[:]), len(canonicalBytes),
+ hex.EncodeToString(bundledHash[:]), len(bundledBytes),
+ )
+ }
+}
+
+// findRepoRoot walks up from the test file's location until it finds go.mod.
+// This is more robust than relying on PWD or runtime.Caller alone, because
+// `go test ./...` runs each package from its own directory.
+func findRepoRoot(t *testing.T) string {
+ t.Helper()
+ _, thisFile, _, ok := runtime.Caller(0)
+ if !ok {
+ t.Fatal("could not determine test file location")
+ }
+ dir := filepath.Dir(thisFile)
+ for {
+ if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil {
+ return dir
+ }
+ parent := filepath.Dir(dir)
+ if parent == dir {
+ t.Fatalf("could not find repo root (no go.mod) starting from %s", filepath.Dir(thisFile))
+ }
+ dir = parent
+ }
+}
diff --git a/internal/cli/verify_skill_unknown_command_test.go b/internal/cli/verify_skill_unknown_command_test.go
new file mode 100644
index 00000000..77607c5f
--- /dev/null
+++ b/internal/cli/verify_skill_unknown_command_test.go
@@ -0,0 +1,146 @@
+// Copyright 2026 trevin-chow. Licensed under Apache-2.0. See LICENSE.
+
+package cli_test
+
+import (
+ "os"
+ "os/exec"
+ "path/filepath"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+)
+
+// TestVerifySkill_DetectsUnknownCommand integration-tests the new
+// unknown-command check from U2: a SKILL that references an op-id-shaped
+// path (`<cli> qr get-qrcode`) for a resource the cobra source actually
+// registers as a leaf (`<cli> qr`) is rejected.
+func TestVerifySkill_DetectsUnknownCommand(t *testing.T) {
+ bin := buildPrintingPressBinary(t)
+ dir := t.TempDir()
+
+ require.NoError(t, os.MkdirAll(filepath.Join(dir, "internal", "cli"), 0o755))
+
+ // Minimal cobra source: only `qr` exists as a leaf. SKILL claims `qr get-qrcode`.
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "internal", "cli", "root.go"), []byte(`package cli
+import "github.com/spf13/cobra"
+func newRootCmd() *cobra.Command {
+ rootCmd := &cobra.Command{Use: "fixture-pp-cli"}
+ rootCmd.AddCommand(newQrCmd())
+ return rootCmd
+}
+`), 0o644))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "internal", "cli", "qr.go"), []byte(`package cli
+import "github.com/spf13/cobra"
+func newQrCmd() *cobra.Command {
+ return &cobra.Command{Use: "qr <url>"}
+}
+`), 0o644))
+
+ skill := `---
+name: pp-fixture
+description: "fixture"
+---
+
+# Fixture
+
+## Command Reference
+
+- ` + "`fixture-pp-cli qr get-qrcode <url>`" + ` — phantom op-id form
+`
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "SKILL.md"), []byte(skill), 0o644))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, ".printing-press.json"), []byte(`{"cli_name":"fixture-pp-cli"}`), 0o644))
+
+ out, err := exec.Command(bin, "verify-skill", "--dir", dir).CombinedOutput()
+ require.Error(t, err, "verifier must exit non-zero when SKILL references an unknown command path")
+ exitErr, ok := err.(*exec.ExitError)
+ require.True(t, ok)
+ require.Equal(t, 1, exitErr.ExitCode(), "exit 1 signals findings (not usage error)")
+ require.Contains(t, string(out), "[unknown-command]",
+ "output must label the finding as unknown-command")
+ require.Contains(t, string(out), "qr get-qrcode",
+ "diagnostic must name the phantom path so the SKILL author knows what to fix")
+ require.Contains(t, string(out), "closest existing prefix is `fixture-pp-cli qr`",
+ "diagnostic must name the closest valid prefix to guide the fix")
+}
+
+// TestVerifySkill_UnknownCommandPassesWhenAllPathsResolve confirms the
+// negative case: a SKILL whose command-reference paths all map to real
+// cobra Use: declarations passes the unknown-command check.
+func TestVerifySkill_UnknownCommandPassesWhenAllPathsResolve(t *testing.T) {
+ bin := buildPrintingPressBinary(t)
+ dir := t.TempDir()
+
+ require.NoError(t, os.MkdirAll(filepath.Join(dir, "internal", "cli"), 0o755))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "internal", "cli", "root.go"), []byte(`package cli
+import "github.com/spf13/cobra"
+func newRootCmd() *cobra.Command {
+ rootCmd := &cobra.Command{Use: "fixture-pp-cli"}
+ rootCmd.AddCommand(newQrCmd())
+ return rootCmd
+}
+`), 0o644))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "internal", "cli", "qr.go"), []byte(`package cli
+import "github.com/spf13/cobra"
+func newQrCmd() *cobra.Command {
+ return &cobra.Command{Use: "qr <url>"}
+}
+`), 0o644))
+
+ // SKILL uses the leaf form — the real, registered path.
+ skill := `---
+name: pp-fixture
+description: "fixture"
+---
+
+# Fixture
+
+## Command Reference
+
+- ` + "`fixture-pp-cli qr <url>`" + ` — leaf form, resolves correctly
+`
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "SKILL.md"), []byte(skill), 0o644))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, ".printing-press.json"), []byte(`{"cli_name":"fixture-pp-cli"}`), 0o644))
+
+ out, err := exec.Command(bin, "verify-skill", "--dir", dir, "--only", "unknown-command").CombinedOutput()
+ require.NoError(t, err, "unknown-command must NOT fire when every path resolves: %s", string(out))
+ require.Contains(t, string(out), "All checks passed",
+ "output must indicate clean pass on the unknown-command check")
+}
+
+// TestVerifySkill_UnknownCommandSkipsBuiltins confirms cobra's auto-registered
+// built-in commands (help, completion, version) are whitelisted — references
+// to `<cli> help` in SKILL.md must NOT fire unknown-command.
+func TestVerifySkill_UnknownCommandSkipsBuiltins(t *testing.T) {
+ bin := buildPrintingPressBinary(t)
+ dir := t.TempDir()
+
+ require.NoError(t, os.MkdirAll(filepath.Join(dir, "internal", "cli"), 0o755))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "internal", "cli", "root.go"), []byte(`package cli
+import "github.com/spf13/cobra"
+func newRootCmd() *cobra.Command {
+ return &cobra.Command{Use: "fixture-pp-cli"}
+}
+`), 0o644))
+
+ skill := `---
+name: pp-fixture
+description: "fixture"
+---
+
+# Fixture
+
+## Command Reference
+
+- ` + "`fixture-pp-cli help`" + ` — cobra auto-registered, must not flag
+- ` + "`fixture-pp-cli completion`" + ` — cobra auto-registered, must not flag
+- ` + "`fixture-pp-cli version`" + ` — common pattern, must not flag
+`
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "SKILL.md"), []byte(skill), 0o644))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, ".printing-press.json"), []byte(`{"cli_name":"fixture-pp-cli"}`), 0o644))
+
+ out, err := exec.Command(bin, "verify-skill", "--dir", dir, "--only", "unknown-command").CombinedOutput()
+ require.NoError(t, err, "unknown-command must NOT fire on cobra builtins: %s", string(out))
+ require.NotContains(t, string(out), "[unknown-command]",
+ "no findings expected — help/completion/version are whitelisted")
+}
diff --git a/lefthook.yml b/lefthook.yml
index 3881c3c1..52ab241a 100644
--- a/lefthook.yml
+++ b/lefthook.yml
@@ -26,6 +26,17 @@ pre-commit:
files="$(printf '%s\n' {staged_files} | grep -v '^testdata/golden/expected/' || true)"
test -z "$files" || { printf '%s\n' "$files" | xargs gofmt -w; printf '%s\n' "$files" | xargs git add; }
stage_fixed: true
+ sync-verify-skill:
+ # Keep internal/cli/verify_skill_bundled.py byte-identical to the
+ # canonical scripts/verify-skill/verify_skill.py so the binary's
+ # `verify-skill` runs the same checks as the library repo's CI. The
+ # Go test TestVerifySkillScriptInSync is the safety net for commits
+ # that bypass lefthook (--no-verify); this hook is the auto-fix path.
+ glob: "scripts/verify-skill/verify_skill.py"
+ run: |
+ cp scripts/verify-skill/verify_skill.py internal/cli/verify_skill_bundled.py
+ git add internal/cli/verify_skill_bundled.py
+ stage_fixed: true
pre-push:
commands:
diff --git a/scripts/verify-skill/verify_skill.py b/scripts/verify-skill/verify_skill.py
index 7f3a41b0..915ee4c7 100755
--- a/scripts/verify-skill/verify_skill.py
+++ b/scripts/verify-skill/verify_skill.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""verify_skill.py — validate that SKILL.md matches the shipped CLI source.
-Three checks run in sequence:
+Four checks run in sequence:
1. flag-names — every `--flag` in SKILL.md is declared as a cobra flag
somewhere in internal/cli/*.go.
@@ -9,6 +9,11 @@ Three checks run in sequence:
on that command (or as a persistent/root flag).
3. positional-args — positional args in bash recipes match the command's
`Use:` field signature (required + optional + variadic).
+ 4. unknown-command — every command path referenced in SKILL.md (in bash
+ recipes and inline backticks under `## Command Reference`) maps to a
+ real cobra `Use:` declaration in internal/cli/*.go. Catches docs that
+ promise commands the binary does not implement (e.g. SKILL.md lists
+ `qr get-qrcode` but the CLI only registers a leaf `qr` after promotion).
The checks are pattern-matching heuristics against Go AST-adjacent text.
False positives are possible for edge cases:
@@ -25,6 +30,7 @@ USAGE
python3 verify_skill.py --dir <cli-dir>
python3 verify_skill.py --dir <cli-dir> --json
python3 verify_skill.py --dir <cli-dir> --only flag-names
+ python3 verify_skill.py --dir <cli-dir> --only unknown-command
python3 verify_skill.py --dir <cli-dir> --strict # treat known-FPs as failures
Exit codes:
@@ -57,6 +63,15 @@ COMMON_FLAGS = {
}
CODEBLOCK_BASH = re.compile(r"```bash\n(.*?)\n```", re.DOTALL)
+COMMAND_REFERENCE_SECTION_RE = re.compile(
+ r"^##\s+Command\s+Reference\s*$(.*?)(?=^##\s+|\Z)",
+ re.DOTALL | re.MULTILINE | re.IGNORECASE,
+)
+# Cobra registers help/completion automatically; treat as always-present.
+# Other CLIs may surface version as a real cobra command, but it is also a
+# common --version flag pattern; we conservatively whitelist it too so a
+# `<binary> version` reference never fires this check.
+BUILTIN_COMMANDS = {"help", "completion", "version"}
USE_RE = re.compile(r'Use:\s*"([^"]+)"')
ARGS_RE = re.compile(
r'Args:\s*cobra\.(ExactArgs|MinimumNArgs|MaximumNArgs|RangeArgs|NoArgs|OnlyValidArgs|ExactValidArgs)\s*\(([^)]*)\)'
@@ -666,6 +681,109 @@ def check_positional_args(cli_dir: Path, skill: Path, cli_binary: str, report: R
)
+def _extract_inline_commands(skill_text: str, cli_binary: str) -> list[list[str]]:
+ """Pull `<binary> <cmd> [more]` snippets from inline backticks under the
+ `## Command Reference` section. Returns command paths only, no flags or
+ positional args (those are surfaced through the bash-recipe checks).
+
+ Why scoped to ## Command Reference: SKILL.md narrative prose mentions
+ binary names in flowing text where false positives would be high. The
+ Command Reference section is the canonical promise to the reader.
+ """
+ sec = COMMAND_REFERENCE_SECTION_RE.search(skill_text)
+ if not sec:
+ return []
+ section_body = sec.group(1)
+ binary_token = re.escape(cli_binary)
+ inline_re = re.compile(rf"`({binary_token}(?:\s+[^`]+)?)`")
+ paths: list[list[str]] = []
+ for m in inline_re.finditer(section_body):
+ snippet = m.group(1).strip()
+ after = snippet[len(cli_binary):].strip()
+ if not after:
+ continue
+ tokens = after.split()
+ cmd_path: list[str] = []
+ for t in tokens:
+ if t.startswith("-") or t.startswith("<") or t.startswith("[") \
+ or t.startswith("$") or t.startswith("\"") or t.startswith("'") \
+ or t.startswith("`") or "/" in t or "=" in t:
+ break
+ if not re.match(r"^[a-z][a-z0-9-]*$", t):
+ break
+ cmd_path.append(t)
+ if len(cmd_path) >= 3:
+ break
+ if cmd_path:
+ paths.append(cmd_path)
+ return paths
+
+
+def check_unknown_commands(cli_dir: Path, skill: Path, cli_binary: str, report: Report) -> None:
+ """Report command paths in SKILL.md that have no matching cobra Use:
+ declaration in internal/cli/*.go. Source paths come from two surfaces:
+
+ - Bash recipes (extract_recipes), which the other checks already walk
+ but skip silently when the command is missing
+ - Inline backtick references inside the `## Command Reference` section
+
+ Each unique cmd_path is reported at most once per SKILL.md.
+
+ Uses the in-repo find_command_source which walks the rootCmd.AddCommand
+ graph and resolves multi-level command paths (e.g., `links stale` vs
+ `profile save`) without false-positive collisions on shared leaf names.
+ """
+ skill_text = skill.read_text()
+ seen: set[tuple[str, ...]] = set()
+ sources: list[tuple[list[str], str]] = []
+
+ for cmd_path, _pos, _flags in extract_recipes(skill, cli_binary, cli_dir):
+ if cmd_path:
+ sources.append((cmd_path, "bash recipe"))
+ for cmd_path in _extract_inline_commands(skill_text, cli_binary):
+ sources.append((cmd_path, "Command Reference inline"))
+
+ for cmd_path, surface in sources:
+ if not cmd_path:
+ continue
+ head = cmd_path[0]
+ # Skip non-command tokens that the recipe parser may have promoted
+ # into cmd_path[0]: flags, placeholders, env vars, etc. These belong
+ # to other checks or are documentation conventions, not commands.
+ if head in BUILTIN_COMMANDS:
+ continue
+ if head.startswith(("-", "<", "[", "$")) or "=" in head:
+ continue
+ if not re.match(r"^[a-z][a-z0-9-]*$", head):
+ continue
+ key = tuple(cmd_path)
+ if key in seen:
+ continue
+ seen.add(key)
+ files, _use, _args = find_command_source(cli_dir, cmd_path)
+ if files:
+ continue
+ # Walk back to the longest existing prefix for a clearer error.
+ detail = "command path not found in internal/cli/*.go (no matching Use: declaration)"
+ for k in range(len(cmd_path) - 1, 0, -1):
+ prefix_files, _, _ = find_command_source(cli_dir, cmd_path[:k])
+ if prefix_files:
+ detail = (
+ f"command path not found in internal/cli/*.go; "
+ f"closest existing prefix is `{cli_binary} {' '.join(cmd_path[:k])}`"
+ )
+ break
+ report.findings.append(
+ Finding(
+ check="unknown-command",
+ severity="error",
+ command=f"{cli_binary} {' '.join(cmd_path)}",
+ detail=detail,
+ evidence=surface,
+ )
+ )
+
+
# ---------------------------------------------------------------------------
# Runner
# ---------------------------------------------------------------------------
@@ -697,7 +815,7 @@ def run_checks(cli_dir: Path, only: set[str] | None) -> Report:
cli_binary = derive_cli_binary(cli_dir)
report = Report(cli_dir=str(cli_dir), skill_path=str(skill))
- checks = only or {"flag-names", "flag-commands", "positional-args"}
+ checks = only or {"flag-names", "flag-commands", "positional-args", "unknown-command"}
if "flag-names" in checks:
report.checks_run.append("flag-names")
check_flag_names(cli_dir, skill, report)
@@ -707,6 +825,9 @@ def run_checks(cli_dir: Path, only: set[str] | None) -> Report:
if "positional-args" in checks:
report.checks_run.append("positional-args")
check_positional_args(cli_dir, skill, cli_binary, report)
+ if "unknown-command" in checks:
+ report.checks_run.append("unknown-command")
+ check_unknown_commands(cli_dir, skill, cli_binary, report)
return report
@@ -757,7 +878,7 @@ def main():
p.add_argument("--dir", required=True, help="CLI directory (contains SKILL.md + internal/cli/)")
p.add_argument(
"--only",
- choices=["flag-names", "flag-commands", "positional-args"],
+ choices=["flag-names", "flag-commands", "positional-args", "unknown-command"],
action="append",
help="Run only the named check(s). Pass multiple times to include multiple.",
)
← 4219e712 feat(cli): emit promoted-leaf paths in SKILL.md (#338)
·
back to Cli Printing Press
·
fix(cli): recover RunID in NewMinimalState so lock promote e b25c2b90 →