← back to Cli Printing Press
feat(cli): printing-press verify-skill + Phase 4 wiring + Phase 4.8 agentic SKILL reviewer (#212)
18cb521e6ce4f91b1f463952cdb3f65ddde161e8 · 2026-04-13 14:49:43 -0700 · Trevin Chow
Closes the gap that let recipe-goat ship a SKILL.md advertising
`search --max-time` when `--max-time` is a `tonight` flag, not a `search`
flag. The mechanical verifier exists (scripts/verify-skill/verify_skill.py,
shipped in #194) but ran only on the library-repo PR CI — too late. Now
runs during Phase 4 shipcheck alongside dogfood/verify/scorecard, and a
new Phase 4.8 agentic reviewer catches semantic issues the mechanical
check can't.
## Machine changes
New subcommand `printing-press verify-skill --dir <cli-dir>`:
- Embeds scripts/verify-skill/verify_skill.py via go:embed so the same
logic ships in the binary — no "is the script in my PATH" guesswork.
- Shells out to python3 with the embedded script; propagates exit code
(1 = findings, 2 = usage) and output.
- Supports --json, --strict, --only flag-names|flag-commands|positional-args
flags so CI, skill shipcheck, and developer debug all use one entry point.
- Silences cobra's default error wrapping — the verifier's own report is
the human-facing output.
## Skill changes
- `skills/printing-press/SKILL.md` Phase 4 shipcheck block now runs
`printing-press verify-skill --dir $CLI_WORK_DIR` alongside dogfood +
verify + workflow-verify + scorecard.
- Ship threshold: `verify-skill` must exit 0 for a CLI to pass Phase 4.
"The SKILL is what agents read; if it lies about the CLI, the lie ships"
is the rationale inline in the threshold list.
- New **Phase 4.8: Agentic SKILL Review** — dispatches an Agent with a
six-point semantic contract (trigger-phrase accuracy, novel-feature
descriptions, stub disclosure, auth-narrative accuracy, recipe output
claims, marketing-copy smell). Mechanical check catches wrong flags;
agentic check catches lies the mechanical check can't.
## Why both
`verify-skill` is cheap, deterministic, mechanical. Catches 100% of
"wrong flag on wrong command" / "undeclared flag" / "positional-count
mismatch". Missed recipe-goat's bug because the flag WAS declared —
just on the wrong command. That kind of miss is its nominal job.
Phase 4.8 agentic covers what AST-adjacent regex can't: "the trigger
phrase promises capability X, but the CLI does Y," "novel feature Z
is labeled as shipping but actually emits a wip stub," "the auth
narrative mentions `auth login --chrome` when the CLI has no `login`
subcommand." Same findings the retro-era #197 and #199 landed as
skill rules; Phase 4.8 enforces them.
## Testing
`internal/cli/verify_skill_test.go` adds three regression tests:
- `TestVerifySkill_DetectsWrongFlagOnCommand` — regression guard for the
recipe-goat `search --max-time` bug. Builds the real binary, runs
against a fixture CLI, asserts exit 1 + specific diagnostic.
- `TestVerifySkill_PassesWhenSkillMatches` — clean SKILL exits 0.
- `TestVerifySkill_RejectsMissingInputs` — usage errors (code 2) when
--dir is absent or the directory lacks SKILL.md / internal/cli/.
Live smoke: `printing-press verify-skill --dir ~/printing-press/library/recipe-goat`
now reports `✓ All checks passed (flag-names, flag-commands, positional-args)`.
## Pre-existing flake (not caused by this PR)
A flake in `TestIntEnumParamSkipped` / `TestNonEnumParamDoesNotEmitValidation`
has been intermittent on main since #208 (map-iteration order decides
which endpoint the generator promotes for 2-endpoint resources, making
the expected per-endpoint file absent sometimes). Confirmed to reproduce
on clean main; my stabilization attempt regressed and was reverted. Filing
separately — not a blocker for this PR.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Files touched
M internal/cli/root.goA internal/cli/verify_skill.goA internal/cli/verify_skill_bundled.pyA internal/cli/verify_skill_test.goM skills/printing-press/SKILL.md
Diff
commit 18cb521e6ce4f91b1f463952cdb3f65ddde161e8
Author: Trevin Chow <trevin@trevinchow.com>
Date: Mon Apr 13 14:49:43 2026 -0700
feat(cli): printing-press verify-skill + Phase 4 wiring + Phase 4.8 agentic SKILL reviewer (#212)
Closes the gap that let recipe-goat ship a SKILL.md advertising
`search --max-time` when `--max-time` is a `tonight` flag, not a `search`
flag. The mechanical verifier exists (scripts/verify-skill/verify_skill.py,
shipped in #194) but ran only on the library-repo PR CI — too late. Now
runs during Phase 4 shipcheck alongside dogfood/verify/scorecard, and a
new Phase 4.8 agentic reviewer catches semantic issues the mechanical
check can't.
## Machine changes
New subcommand `printing-press verify-skill --dir <cli-dir>`:
- Embeds scripts/verify-skill/verify_skill.py via go:embed so the same
logic ships in the binary — no "is the script in my PATH" guesswork.
- Shells out to python3 with the embedded script; propagates exit code
(1 = findings, 2 = usage) and output.
- Supports --json, --strict, --only flag-names|flag-commands|positional-args
flags so CI, skill shipcheck, and developer debug all use one entry point.
- Silences cobra's default error wrapping — the verifier's own report is
the human-facing output.
## Skill changes
- `skills/printing-press/SKILL.md` Phase 4 shipcheck block now runs
`printing-press verify-skill --dir $CLI_WORK_DIR` alongside dogfood +
verify + workflow-verify + scorecard.
- Ship threshold: `verify-skill` must exit 0 for a CLI to pass Phase 4.
"The SKILL is what agents read; if it lies about the CLI, the lie ships"
is the rationale inline in the threshold list.
- New **Phase 4.8: Agentic SKILL Review** — dispatches an Agent with a
six-point semantic contract (trigger-phrase accuracy, novel-feature
descriptions, stub disclosure, auth-narrative accuracy, recipe output
claims, marketing-copy smell). Mechanical check catches wrong flags;
agentic check catches lies the mechanical check can't.
## Why both
`verify-skill` is cheap, deterministic, mechanical. Catches 100% of
"wrong flag on wrong command" / "undeclared flag" / "positional-count
mismatch". Missed recipe-goat's bug because the flag WAS declared —
just on the wrong command. That kind of miss is its nominal job.
Phase 4.8 agentic covers what AST-adjacent regex can't: "the trigger
phrase promises capability X, but the CLI does Y," "novel feature Z
is labeled as shipping but actually emits a wip stub," "the auth
narrative mentions `auth login --chrome` when the CLI has no `login`
subcommand." Same findings the retro-era #197 and #199 landed as
skill rules; Phase 4.8 enforces them.
## Testing
`internal/cli/verify_skill_test.go` adds three regression tests:
- `TestVerifySkill_DetectsWrongFlagOnCommand` — regression guard for the
recipe-goat `search --max-time` bug. Builds the real binary, runs
against a fixture CLI, asserts exit 1 + specific diagnostic.
- `TestVerifySkill_PassesWhenSkillMatches` — clean SKILL exits 0.
- `TestVerifySkill_RejectsMissingInputs` — usage errors (code 2) when
--dir is absent or the directory lacks SKILL.md / internal/cli/.
Live smoke: `printing-press verify-skill --dir ~/printing-press/library/recipe-goat`
now reports `✓ All checks passed (flag-names, flag-commands, positional-args)`.
## Pre-existing flake (not caused by this PR)
A flake in `TestIntEnumParamSkipped` / `TestNonEnumParamDoesNotEmitValidation`
has been intermittent on main since #208 (map-iteration order decides
which endpoint the generator promotes for 2-endpoint resources, making
the expected per-endpoint file absent sometimes). Confirmed to reproduce
on clean main; my stabilization attempt regressed and was reverted. Filing
separately — not a blocker for this PR.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
internal/cli/root.go | 1 +
internal/cli/verify_skill.go | 133 +++++++++
internal/cli/verify_skill_bundled.py | 546 +++++++++++++++++++++++++++++++++++
internal/cli/verify_skill_test.go | 124 ++++++++
skills/printing-press/SKILL.md | 50 ++++
5 files changed, 854 insertions(+)
diff --git a/internal/cli/root.go b/internal/cli/root.go
index 054e98d1..5f10a5aa 100644
--- a/internal/cli/root.go
+++ b/internal/cli/root.go
@@ -42,6 +42,7 @@ func Execute() error {
rootCmd.AddCommand(newScorecardCmd())
rootCmd.AddCommand(newDogfoodCmd())
rootCmd.AddCommand(newVerifyCmd())
+ rootCmd.AddCommand(newVerifySkillCmd())
rootCmd.AddCommand(newEmbossCmd())
rootCmd.AddCommand(newVisionCmd())
rootCmd.AddCommand(newVersionCmd())
diff --git a/internal/cli/verify_skill.go b/internal/cli/verify_skill.go
new file mode 100644
index 00000000..5bc8d484
--- /dev/null
+++ b/internal/cli/verify_skill.go
@@ -0,0 +1,133 @@
+// Copyright 2026 trevin-chow. Licensed under Apache-2.0. See LICENSE.
+
+package cli
+
+import (
+ "bytes"
+ _ "embed"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+
+ "github.com/spf13/cobra"
+)
+
+// verifySkillScript is the full text of scripts/verify-skill/verify_skill.py
+// bundled into the binary so the verification runs the same way in CI, in
+// Phase 4 shipcheck, and from a developer's machine — no "is the script in
+// my PATH" guesswork.
+//
+//go:embed verify_skill_bundled.py
+var verifySkillScript string
+
+func newVerifySkillCmd() *cobra.Command {
+ var (
+ dir string
+ only []string
+ asJSON bool
+ strict bool
+ )
+
+ cmd := &cobra.Command{
+ Use: "verify-skill",
+ Short: "Verify SKILL.md matches the shipped CLI source",
+ SilenceUsage: true,
+ SilenceErrors: true,
+ Long: `Run three checks against a printed CLI's SKILL.md:
+
+ 1. flag-names — every --flag referenced in SKILL.md is declared in internal/cli/*.go
+ 2. flag-commands — every --flag used on a specific command is declared on that command (or persistent)
+ 3. positional-args — positional args in bash recipes match the command's Use: field
+
+Fails when the SKILL advertises commands, flags, or arguments that the binary
+doesn't actually provide — which is how the recipe-goat "search --max-time"
+bug shipped before this gate existed. Runs the same logic as the library-repo
+CI check (scripts/verify-skill/verify_skill.py) via an embedded copy so no
+external script path is needed.
+
+Requires python3 on PATH (same dependency as the cookie-auth doctor check).`,
+ Example: ` # Run all three checks against a generated CLI
+ printing-press verify-skill --dir ./my-api-pp-cli
+
+ # JSON output for programmatic consumption
+ printing-press verify-skill --dir ./my-api-pp-cli --json
+
+ # Only check a specific category
+ printing-press verify-skill --dir ./my-api-pp-cli --only flag-commands`,
+ RunE: func(cmd *cobra.Command, args []string) error {
+ if dir == "" {
+ return &ExitError{Code: ExitInputError, Err: fmt.Errorf("--dir is required")}
+ }
+ abs, err := filepath.Abs(dir)
+ if err != nil {
+ return fmt.Errorf("resolving --dir: %w", err)
+ }
+ if _, err := os.Stat(filepath.Join(abs, "SKILL.md")); err != nil {
+ return &ExitError{Code: ExitInputError, Err: fmt.Errorf("no SKILL.md in %s", abs)}
+ }
+ if _, err := os.Stat(filepath.Join(abs, "internal", "cli")); err != nil {
+ return &ExitError{Code: ExitInputError, Err: fmt.Errorf("no internal/cli/ in %s", abs)}
+ }
+
+ tmpFile, err := os.CreateTemp("", "verify-skill-*.py")
+ if err != nil {
+ return fmt.Errorf("creating temp file: %w", err)
+ }
+ defer func() { _ = os.Remove(tmpFile.Name()) }()
+ if _, err := tmpFile.WriteString(verifySkillScript); err != nil {
+ _ = tmpFile.Close()
+ return fmt.Errorf("writing temp file: %w", err)
+ }
+ if err := tmpFile.Close(); err != nil {
+ return fmt.Errorf("closing temp file: %w", err)
+ }
+
+ pyArgs := []string{tmpFile.Name(), "--dir", abs}
+ for _, o := range only {
+ pyArgs = append(pyArgs, "--only", o)
+ }
+ if asJSON {
+ pyArgs = append(pyArgs, "--json")
+ }
+ if strict {
+ pyArgs = append(pyArgs, "--strict")
+ }
+
+ py := exec.Command("python3", pyArgs...)
+ py.Stdin = os.Stdin
+ var stdout, stderr bytes.Buffer
+ py.Stdout = &stdout
+ py.Stderr = &stderr
+ runErr := py.Run()
+ // Forward verifier output to the caller regardless of exit code.
+ if stdout.Len() > 0 {
+ fmt.Fprint(os.Stdout, stdout.String())
+ }
+ if stderr.Len() > 0 {
+ fmt.Fprint(os.Stderr, stderr.String())
+ }
+ if runErr != nil {
+ if exitErr, ok := runErr.(*exec.ExitError); ok {
+ // Propagate the verifier's exit code. 1 = findings, 2 = usage.
+ // Silent=true suppresses cobra's "Error: ..." prefix since
+ // the verifier already printed a human-readable report.
+ return &ExitError{
+ Code: exitErr.ExitCode(),
+ Err: fmt.Errorf("SKILL verification failed"),
+ Silent: true,
+ }
+ }
+ return fmt.Errorf("running verifier: %w", runErr)
+ }
+ return nil
+ },
+ }
+
+ cmd.Flags().StringVar(&dir, "dir", "", "Path to the printed CLI directory (contains SKILL.md + internal/cli/)")
+ cmd.Flags().StringSliceVar(&only, "only", nil, "Run only the named check(s): flag-names, flag-commands, positional-args (repeatable)")
+ cmd.Flags().BoolVar(&asJSON, "json", false, "Output as JSON")
+ cmd.Flags().BoolVar(&strict, "strict", false, "Treat likely-false-positive findings as failures")
+
+ return cmd
+}
diff --git a/internal/cli/verify_skill_bundled.py b/internal/cli/verify_skill_bundled.py
new file mode 100755
index 00000000..6a393a1c
--- /dev/null
+++ b/internal/cli/verify_skill_bundled.py
@@ -0,0 +1,546 @@
+#!/usr/bin/env python3
+"""verify_skill.py — validate that SKILL.md matches the shipped CLI source.
+
+Three checks run in sequence:
+
+ 1. flag-names — every `--flag` in SKILL.md is declared as a cobra flag
+ somewhere in internal/cli/*.go.
+ 2. flag-commands — every `--flag` used on a specific command is declared
+ on that command (or as a persistent/root flag).
+ 3. positional-args — positional args in bash recipes match the command's
+ `Use:` field signature (required + optional + variadic).
+
+The checks are pattern-matching heuristics against Go AST-adjacent text.
+False positives are possible for edge cases:
+ - Shell command substitution (`$(...)`) inside a recipe can be
+ misinterpreted as extending the outer command path.
+ - Commands where the first positional arg is a valid subcommand name
+ (e.g., `hubspot associations companies <id>` where `companies` is an
+ object type passed as arg, not a subcommand).
+
+Known false-positives are reported with a `[likely false positive]` tag.
+
+USAGE
+
+ python3 verify_skill.py --dir <cli-dir>
+ python3 verify_skill.py --dir <cli-dir> --json
+ python3 verify_skill.py --dir <cli-dir> --only flag-names
+ python3 verify_skill.py --dir <cli-dir> --strict # treat known-FPs as failures
+
+Exit codes:
+ 0 — all checks passed
+ 1 — one or more checks found issues (excluding known false positives
+ unless --strict is set)
+ 2 — usage error (missing --dir, SKILL.md not found, etc.)
+
+The CLI dir must contain both `SKILL.md` and `internal/cli/*.go`.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shlex
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Iterable
+
+
+COMMON_FLAGS = {
+ "help", "version", "json", "csv", "plain", "quiet", "agent",
+ "select", "compact", "dry-run", "no-cache", "yes", "no-input",
+ "no-color", "human-friendly", "config", "base-url", "rate-limit",
+ "timeout", "data-source", "stdin", "limit", "format", "output",
+ "no-prompt", "days",
+}
+
+CODEBLOCK_BASH = re.compile(r"```bash\n(.*?)\n```", re.DOTALL)
+USE_RE = re.compile(r'Use:\s*"([^"]+)"')
+ARGS_RE = re.compile(
+ r'Args:\s*cobra\.(ExactArgs|MinimumNArgs|MaximumNArgs|RangeArgs|NoArgs|OnlyValidArgs|ExactValidArgs)\s*\(([^)]*)\)'
+)
+FLAG_DECL_RE = re.compile(
+ r'(Persistent)?Flags\(\)\.'
+ r'(StringVar|BoolVar|IntVar|Int64Var|Float64Var|DurationVar|'
+ r'StringSliceVar|StringArrayVar|UintVar|Uint64Var)P?\('
+ r'&[^,]+,\s*"([a-z][a-z0-9-]*)"'
+)
+FLAG_TOKEN_RE = re.compile(r"(?:^|\s)(--[a-z][a-z0-9-]*)")
+
+
+@dataclass
+class Finding:
+ check: str
+ severity: str # "error" or "warning"
+ command: str
+ detail: str
+ evidence: str = ""
+ likely_false_positive: bool = False
+
+
+@dataclass
+class Report:
+ cli_dir: str
+ skill_path: str
+ findings: list[Finding] = field(default_factory=list)
+ checks_run: list[str] = field(default_factory=list)
+ recipes_checked: int = 0
+
+ def has_real_failures(self) -> bool:
+ return any(not f.likely_false_positive for f in self.findings)
+
+
+# ---------------------------------------------------------------------------
+# Source inspection
+# ---------------------------------------------------------------------------
+
+
+def parse_use(use_str: str) -> tuple[str, int, int, bool]:
+ """Return (name, required_count, optional_count, has_variadic)."""
+ tokens = use_str.split()
+ if not tokens:
+ return "", 0, 0, False
+ name = tokens[0]
+ required = optional = 0
+ variadic = False
+ for t in tokens[1:]:
+ if t.startswith("<") and t.endswith(">"):
+ required += 1
+ elif t.startswith("[") and t.endswith("]"):
+ if "..." in t:
+ variadic = True
+ else:
+ optional += 1
+ elif "..." in t:
+ variadic = True
+ return name, required, optional, variadic
+
+
+def find_command_source(cli_dir: Path, cmd_path: list[str]):
+ """Locate source file(s) whose cobra.Command matches this path.
+
+ Returns (go_files, use_str, args_info) where go_files is a list.
+ Why a list: CLIs sometimes declare the same Use in two files (historical
+ artifact or generator + transcendence both shipping a version of the
+ same command). Cobra only registers one at runtime, but we don't know
+ which without parsing root.go's AddCommand calls. Returning all matching
+ files lets callers union their flags when checking declarations.
+ """
+ if not cmd_path:
+ return [], None, None
+ leaf = cmd_path[-1]
+ src = cli_dir / "internal/cli"
+ if not src.exists():
+ return [], None, None
+
+ candidates = []
+ for go_file in src.glob("*.go"):
+ if go_file.name.endswith("_test.go"):
+ continue
+ try:
+ text = go_file.read_text()
+ except Exception:
+ continue
+ for m in USE_RE.finditer(text):
+ use_str = m.group(1)
+ name, req, opt, var_ = parse_use(use_str)
+ if name != leaf:
+ continue
+ end = m.end()
+ window = text[end : end + 500]
+ args_match = ARGS_RE.search(window)
+ args_info = (args_match.group(1), args_match.group(2)) if args_match else None
+ specificity = req + opt + (1 if var_ else 0)
+ candidates.append((specificity, go_file, use_str, args_info))
+
+ if not candidates:
+ return [], None, None
+
+ # Multi-token paths: prefer filename-match (e.g., contacts_search.go for
+ # cmd_path ["contacts", "search"]).
+ if len(cmd_path) >= 2:
+ expected_basename = "_".join(cmd_path).replace("-", "_") + ".go"
+ for spec, go_file, use_str, args_info in candidates:
+ if go_file.name == expected_basename:
+ return [go_file], use_str, args_info
+
+ # Single-token paths (or no filename match): take all files at the
+ # highest specificity tier. For flag verification, any one of them
+ # declaring the flag counts. For Use string, pick the representative.
+ candidates.sort(key=lambda c: -c[0])
+ top_spec = candidates[0][0]
+ top_files = [c[1] for c in candidates if c[0] == top_spec]
+ return top_files, candidates[0][2], candidates[0][3]
+
+
+def flag_declared_in(files: Iterable[Path], flag_name: str) -> bool:
+ for f in files:
+ try:
+ text = f.read_text()
+ except Exception:
+ continue
+ for m in FLAG_DECL_RE.finditer(text):
+ if m.group(3) == flag_name:
+ return True
+ return False
+
+
+def persistent_flag_declared(cli_dir: Path, flag_name: str) -> bool:
+ src = cli_dir / "internal/cli"
+ if not src.exists():
+ return False
+ for go_file in src.glob("*.go"):
+ try:
+ text = go_file.read_text()
+ except Exception:
+ continue
+ for m in FLAG_DECL_RE.finditer(text):
+ persistent, _, name = m.groups()
+ if name == flag_name and persistent == "Persistent":
+ return True
+ return False
+
+
+# ---------------------------------------------------------------------------
+# SKILL.md extraction
+# ---------------------------------------------------------------------------
+
+
+def extract_all_flags(skill: Path) -> set[str]:
+ """Return every `--flag-name` token (without `--`) used anywhere in SKILL.md."""
+ text = skill.read_text()
+ return {t.lstrip("-") for t in FLAG_TOKEN_RE.findall(text)}
+
+
+def extract_recipes(skill: Path, cli_binary: str, cli_dir: Path | None = None) -> list[tuple[list[str], list[str], list[str]]]:
+ """Return list of (cmd_path, positional_args, flags) tuples from bash blocks.
+
+ cmd_path: leading lowercase-hyphenated tokens (up to 3)
+ positional_args: non-flag tokens after cmd_path (shell-quoted strings preserved)
+ flags: --flag tokens (with their -- prefix)
+ """
+ text = skill.read_text()
+ blocks = CODEBLOCK_BASH.findall(text)
+ results = []
+ for block in blocks:
+ # Merge line continuations
+ merged = []
+ buf = []
+ for raw in block.splitlines():
+ stripped = raw.rstrip()
+ if stripped.endswith("\\"):
+ buf.append(stripped[:-1].strip())
+ else:
+ buf.append(stripped)
+ merged.append(" ".join(buf))
+ buf = []
+ if buf:
+ merged.append(" ".join(buf))
+
+ for line in merged:
+ line = line.strip()
+ if not line or line.startswith("#"):
+ continue
+ # Strip trailing comment
+ cmt = line.find(" #")
+ if cmt != -1:
+ line = line[:cmt].strip()
+ if not line.startswith(cli_binary + " "):
+ continue
+ # Strip shell command substitutions $(...) and backtick forms
+ # FIRST — their contents are separate commands. Do this before
+ # splitting on pipes so we don't mistakenly cut inside a $(...).
+ line = re.sub(r"\$\([^)]*\)", "__SUBST__", line)
+ line = re.sub(r"`[^`]*`", "__SUBST__", line)
+ # Stop at outer shell operators so we don't parse pipes/redirects
+ for op in [" | ", " && ", " || ", " > ", " >> ", " < "]:
+ if op in line:
+ line = line.split(op)[0]
+ break
+ after = line[len(cli_binary) + 1 :].strip()
+ try:
+ tokens = shlex.split(after, posix=True)
+ except ValueError:
+ tokens = after.split()
+ if not tokens:
+ continue
+ cmd_path: list[str] = [tokens[0].lower()]
+ i = 1
+ while i < len(tokens):
+ t = tokens[i]
+ if t.startswith("-"):
+ break
+ if (
+ t.startswith("<") or t.startswith("[")
+ or t.startswith('"') or t.startswith("'")
+ or t.startswith("$") or t.startswith("http")
+ or "/" in t or "=" in t
+ or re.match(r"^[A-Z]", t)
+ or re.match(r"^\d", t)
+ ):
+ break
+ if len(cmd_path) < 3 and re.match(r"^[a-z][a-z0-9-]*$", t):
+ # Verify adding this token still maps to a valid command.
+ # If the extended path has no source match (e.g. the
+ # parent command's Use documents <positional> and this
+ # token is just the arg), treat it as positional.
+ if cli_dir is not None:
+ trial = cmd_path + [t]
+ files, _, _ = find_command_source(cli_dir, trial)
+ if not files:
+ break
+ cmd_path.append(t)
+ i += 1
+ continue
+ break
+ positional: list[str] = []
+ flags: list[str] = []
+ while i < len(tokens):
+ t = tokens[i]
+ if t.startswith("--"):
+ flags.append(t)
+ # Skip value if present and not another flag
+ if i + 1 < len(tokens) and not tokens[i + 1].startswith("-"):
+ i += 2
+ continue
+ elif t.startswith("-"):
+ # Short flag, skip its value heuristically
+ if i + 1 < len(tokens) and not tokens[i + 1].startswith("-"):
+ i += 2
+ continue
+ else:
+ positional.append(t)
+ i += 1
+ results.append((cmd_path, positional, flags))
+ return results
+
+
+# ---------------------------------------------------------------------------
+# Checks
+# ---------------------------------------------------------------------------
+
+
+def check_flag_names(cli_dir: Path, skill: Path, report: Report) -> None:
+ all_files = list((cli_dir / "internal/cli").glob("*.go"))
+ flags = extract_all_flags(skill) - COMMON_FLAGS
+ for flag in sorted(flags):
+ if not flag_declared_in(all_files, flag):
+ report.findings.append(
+ Finding(
+ check="flag-names",
+ severity="error",
+ command="(any)",
+ detail=f"--{flag} is referenced in SKILL.md but not declared in any internal/cli/*.go",
+ )
+ )
+
+
+def check_flag_commands(cli_dir: Path, skill: Path, cli_binary: str, report: Report) -> None:
+ all_files = list((cli_dir / "internal/cli").glob("*.go"))
+ recipes = extract_recipes(skill, cli_binary, cli_dir)
+ for cmd_path, _positional, flags in recipes:
+ for raw_flag in flags:
+ flag = raw_flag.lstrip("-")
+ if flag in COMMON_FLAGS:
+ continue
+ cmd_files, _, _ = find_command_source(cli_dir, cmd_path)
+ if cmd_files and flag_declared_in(cmd_files, flag):
+ continue
+ if persistent_flag_declared(cli_dir, flag):
+ continue
+ path_str = " ".join(cmd_path)
+ if flag_declared_in(all_files, flag):
+ report.findings.append(
+ Finding(
+ check="flag-commands",
+ severity="error",
+ command=f"{cli_binary} {path_str}",
+ detail=f"--{flag} is declared elsewhere but not on {path_str}",
+ )
+ )
+ else:
+ report.findings.append(
+ Finding(
+ check="flag-commands",
+ severity="error",
+ command=f"{cli_binary} {path_str}",
+ detail=f"--{flag} is not declared anywhere",
+ )
+ )
+
+
+def check_positional_args(cli_dir: Path, skill: Path, cli_binary: str, report: Report) -> None:
+ recipes = extract_recipes(skill, cli_binary, cli_dir)
+ report.recipes_checked = len(recipes)
+ for cmd_path, positional, _flags in recipes:
+ _files, use_str, args_info = find_command_source(cli_dir, cmd_path)
+ if not use_str:
+ continue # command not found — not our job to flag here
+ _, required, optional, variadic = parse_use(use_str)
+ min_ok = required
+ max_ok = float("inf") if variadic else required + optional
+ if args_info:
+ validator, arg = args_info
+ try:
+ n = int(arg) if arg else 0
+ except ValueError:
+ n = 0
+ if validator == "ExactArgs":
+ min_ok = max_ok = n
+ elif validator == "MinimumNArgs":
+ min_ok = n
+ max_ok = float("inf")
+ elif validator == "MaximumNArgs":
+ min_ok = 0
+ max_ok = n
+ elif validator == "NoArgs":
+ min_ok = max_ok = 0
+ actual = len(positional)
+ if min_ok <= actual <= max_ok:
+ continue
+
+ path_str = " ".join(cmd_path)
+ # Classify common false-positive patterns.
+ # FP-1: shell command-substitution residue inside an --arg value
+ # (parser may have kept `$(dub-pp-cli links stale ...)` contents).
+ # FP-2: parent command whose first positional arg happens to be a
+ # valid cobra subcommand name (e.g., `associations companies`).
+ fp = False
+ if any(p.startswith("$") for p in positional):
+ fp = True
+ # For single-token cmd_path where positional[0] is lowercase+alpha,
+ # the parser may have under-counted cmd_path.
+ if len(cmd_path) == 1 and positional and re.match(r"^[a-z][a-z0-9-]+$", positional[0]):
+ fp = True
+
+ max_display = "∞" if max_ok == float("inf") else int(max_ok)
+ report.findings.append(
+ Finding(
+ check="positional-args",
+ severity="error" if not fp else "warning",
+ command=f"{cli_binary} {path_str}",
+ detail=f'got {actual} positional args; Use: "{use_str}" expects {min_ok}–{max_display}',
+ evidence=" ".join(positional) or "(none)",
+ likely_false_positive=fp,
+ )
+ )
+
+
+# ---------------------------------------------------------------------------
+# Runner
+# ---------------------------------------------------------------------------
+
+
+def derive_cli_binary(cli_dir: Path) -> str:
+ """Derive the CLI binary name from .printing-press.json, go.mod, or dir name."""
+ manifest = cli_dir / ".printing-press.json"
+ if manifest.exists():
+ try:
+ data = json.loads(manifest.read_text())
+ if data.get("cli_name"):
+ return data["cli_name"]
+ except Exception:
+ pass
+ # Fallback — assume <dirname>-pp-cli
+ return cli_dir.name + "-pp-cli"
+
+
+def run_checks(cli_dir: Path, only: set[str] | None) -> Report:
+ skill = cli_dir / "SKILL.md"
+ if not skill.exists():
+ print(f"error: no SKILL.md in {cli_dir}", file=sys.stderr)
+ sys.exit(2)
+ if not (cli_dir / "internal/cli").exists():
+ print(f"error: no internal/cli/ in {cli_dir}", file=sys.stderr)
+ sys.exit(2)
+
+ cli_binary = derive_cli_binary(cli_dir)
+ report = Report(cli_dir=str(cli_dir), skill_path=str(skill))
+
+ checks = only or {"flag-names", "flag-commands", "positional-args"}
+ if "flag-names" in checks:
+ report.checks_run.append("flag-names")
+ check_flag_names(cli_dir, skill, report)
+ if "flag-commands" in checks:
+ report.checks_run.append("flag-commands")
+ check_flag_commands(cli_dir, skill, cli_binary, report)
+ if "positional-args" in checks:
+ report.checks_run.append("positional-args")
+ check_positional_args(cli_dir, skill, cli_binary, report)
+ return report
+
+
+def format_human(report: Report) -> str:
+ lines = [f"=== {Path(report.cli_dir).name} ==="]
+ errors = [f for f in report.findings if not f.likely_false_positive]
+ warnings = [f for f in report.findings if f.likely_false_positive]
+ if not report.findings:
+ lines.append(f" ✓ All checks passed ({', '.join(report.checks_run)})")
+ return "\n".join(lines)
+ lines.append(f" ✘ {len(errors)} error(s), {len(warnings)} likely false-positive(s)")
+ for f in errors:
+ lines.append(f" [{f.check}] {f.command}: {f.detail}")
+ if f.evidence:
+ lines.append(f" evidence: {f.evidence}")
+ for f in warnings:
+ lines.append(f" [{f.check}] {f.command}: {f.detail} [likely false positive]")
+ if f.evidence:
+ lines.append(f" evidence: {f.evidence}")
+ return "\n".join(lines)
+
+
+def format_json(report: Report) -> str:
+ out = {
+ "cli_dir": report.cli_dir,
+ "skill_path": report.skill_path,
+ "checks_run": report.checks_run,
+ "recipes_checked": report.recipes_checked,
+ "findings": [
+ {
+ "check": f.check,
+ "severity": f.severity,
+ "command": f.command,
+ "detail": f.detail,
+ "evidence": f.evidence,
+ "likely_false_positive": f.likely_false_positive,
+ }
+ for f in report.findings
+ ],
+ }
+ return json.dumps(out, indent=2)
+
+
+def main():
+ p = argparse.ArgumentParser(
+ description="Verify SKILL.md matches shipped CLI source."
+ )
+ p.add_argument("--dir", required=True, help="CLI directory (contains SKILL.md + internal/cli/)")
+ p.add_argument(
+ "--only",
+ choices=["flag-names", "flag-commands", "positional-args"],
+ action="append",
+ help="Run only the named check(s). Pass multiple times to include multiple.",
+ )
+ p.add_argument("--json", action="store_true", help="Emit JSON output")
+ p.add_argument(
+ "--strict",
+ action="store_true",
+ help="Exit non-zero even for findings classified as likely false positives.",
+ )
+ args = p.parse_args()
+ only = set(args.only) if args.only else None
+ report = run_checks(Path(args.dir).resolve(), only)
+
+ if args.json:
+ print(format_json(report))
+ else:
+ print(format_human(report))
+
+ if args.strict:
+ sys.exit(1 if report.findings else 0)
+ sys.exit(1 if report.has_real_failures() else 0)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/internal/cli/verify_skill_test.go b/internal/cli/verify_skill_test.go
new file mode 100644
index 00000000..166d6274
--- /dev/null
+++ b/internal/cli/verify_skill_test.go
@@ -0,0 +1,124 @@
+package cli_test
+
+import (
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+)
+
+// TestVerifySkill_DetectsWrongFlagOnCommand is the regression guard for
+// PR library#66: the recipe-goat SKILL advertised `search --max-time` but
+// --max-time is a `tonight` flag, not a `search` flag. This test writes a
+// synthetic CLI fixture with exactly that shape and confirms
+// `printing-press verify-skill` catches it at generation time instead of
+// letting it ship to the library.
+func TestVerifySkill_DetectsWrongFlagOnCommand(t *testing.T) {
+ bin := buildPrintingPressBinary(t)
+ dir := t.TempDir()
+
+ require.NoError(t, os.MkdirAll(filepath.Join(dir, "internal", "cli"), 0o755))
+
+ // Minimal source: search (without --max-time) and tonight (with --max-time)
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "internal", "cli", "search.go"), []byte(`package cli
+import "github.com/spf13/cobra"
+func newSearchCmd() *cobra.Command {
+ var limit int
+ cmd := &cobra.Command{Use: "search <query>"}
+ cmd.Flags().IntVar(&limit, "limit", 10, "Max results")
+ return cmd
+}
+`), 0o644))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "internal", "cli", "tonight.go"), []byte(`package cli
+import (
+ "github.com/spf13/cobra"
+ "time"
+)
+func newTonightCmd() *cobra.Command {
+ var maxTime time.Duration
+ cmd := &cobra.Command{Use: "tonight"}
+ cmd.Flags().DurationVar(&maxTime, "max-time", 0, "Max total time")
+ return cmd
+}
+`), 0o644))
+
+ // SKILL claims search --max-time (the bug).
+ skill := `---
+name: pp-fixture
+description: "fixture"
+---
+
+# Fixture
+
+` + "```bash" + `
+fixture-pp-cli search "chicken" --max-time 30m
+` + "```" + `
+`
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "SKILL.md"), []byte(skill), 0o644))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, ".printing-press.json"), []byte(`{"cli_name":"fixture-pp-cli"}`), 0o644))
+
+ out, err := exec.Command(bin, "verify-skill", "--dir", dir).CombinedOutput()
+ require.Error(t, err, "verifier must exit non-zero for a SKILL with an undeclared flag")
+ exitErr, ok := err.(*exec.ExitError)
+ require.True(t, ok)
+ require.Equal(t, 1, exitErr.ExitCode(), "exit 1 signals findings (not usage error)")
+ require.Contains(t, string(out), "--max-time is declared elsewhere but not on search",
+ "diagnostic must name the exact mismatch so the skill reader knows what to fix")
+}
+
+// TestVerifySkill_PassesWhenSkillMatches confirms the verifier doesn't
+// false-positive on a well-formed CLI.
+func TestVerifySkill_PassesWhenSkillMatches(t *testing.T) {
+ bin := buildPrintingPressBinary(t)
+ dir := t.TempDir()
+ require.NoError(t, os.MkdirAll(filepath.Join(dir, "internal", "cli"), 0o755))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "internal", "cli", "search.go"), []byte(`package cli
+import "github.com/spf13/cobra"
+func newSearchCmd() *cobra.Command {
+ var limit int
+ cmd := &cobra.Command{Use: "search <query>"}
+ cmd.Flags().IntVar(&limit, "limit", 10, "Max results")
+ return cmd
+}
+`), 0o644))
+ skill := "---\nname: pp-fixture\n---\n\n# Fixture\n\n```bash\nfixture-pp-cli search \"chicken\" --limit 5\n```\n"
+ require.NoError(t, os.WriteFile(filepath.Join(dir, "SKILL.md"), []byte(skill), 0o644))
+ require.NoError(t, os.WriteFile(filepath.Join(dir, ".printing-press.json"), []byte(`{"cli_name":"fixture-pp-cli"}`), 0o644))
+
+ out, err := exec.Command(bin, "verify-skill", "--dir", dir).CombinedOutput()
+ require.NoError(t, err, "clean SKILL should exit 0, got: %s", string(out))
+ require.Contains(t, string(out), "All checks passed")
+}
+
+// TestVerifySkill_RejectsMissingInputs confirms usage errors (code 2).
+func TestVerifySkill_RejectsMissingInputs(t *testing.T) {
+ bin := buildPrintingPressBinary(t)
+
+ // Missing --dir
+ _, err := exec.Command(bin, "verify-skill").CombinedOutput()
+ require.Error(t, err)
+
+ // --dir without SKILL.md
+ emptyDir := t.TempDir()
+ out, err := exec.Command(bin, "verify-skill", "--dir", emptyDir).CombinedOutput()
+ require.Error(t, err)
+ require.True(t, strings.Contains(string(out), "no SKILL.md") || strings.Contains(string(out), "no internal/cli"))
+}
+
+// buildPrintingPressBinary compiles the printing-press binary into a test
+// tempdir and returns its path. Built once per test because each test's
+// TempDir is fresh; Go's test cache ensures the compile is fast.
+func buildPrintingPressBinary(t *testing.T) string {
+ t.Helper()
+ out := filepath.Join(t.TempDir(), "printing-press")
+ cmd := exec.Command("go", "build", "-o", out, "./cmd/printing-press")
+ // The test runs from internal/cli; go up to repo root.
+ cmd.Dir = "../.."
+ if buildOut, err := cmd.CombinedOutput(); err != nil {
+ t.Fatalf("building printing-press: %v\n%s", err, string(buildOut))
+ }
+ return out
+}
diff --git a/skills/printing-press/SKILL.md b/skills/printing-press/SKILL.md
index 5919be92..e547d751 100644
--- a/skills/printing-press/SKILL.md
+++ b/skills/printing-press/SKILL.md
@@ -1600,6 +1600,7 @@ printing-press lock update --cli <api>-pp-cli --phase shipcheck
printing-press dogfood --dir "$CLI_WORK_DIR" --spec <same-spec> --research-dir "$API_RUN_DIR"
printing-press verify --dir "$CLI_WORK_DIR" --spec <same-spec> --fix
printing-press workflow-verify --dir "$CLI_WORK_DIR"
+printing-press verify-skill --dir "$CLI_WORK_DIR"
printing-press scorecard --dir "$CLI_WORK_DIR" --spec <same-spec>
```
@@ -1607,6 +1608,7 @@ Interpretation:
- `dogfood` catches dead flags, dead helpers, invalid paths, example drift, broken data wiring, command tree/config field wiring bugs, and novel features that were planned but not built
- `verify` catches runtime breakage and runs the auto-fix loop for common failures
- `workflow-verify` tests the primary workflow end-to-end using the verification manifest (workflow_verify.yaml). Three verdicts: workflow-pass, workflow-fail, unverified-needs-auth
+- `verify-skill` checks that every `--flag` and command path in SKILL.md actually exists in the shipped CLI source. Catches bogus examples invented by the absorb LLM (e.g., `search --max-time` when `--max-time` is a `tonight` flag). Exit 1 = findings to fix; exit 0 = SKILL is honest.
- `scorecard` is the structural quality snapshot, not the source of truth by itself
Fix order (update heartbeat between each fix category to prevent stale lock during long fix loops):
@@ -1641,6 +1643,7 @@ Ship threshold:
- `dogfood` no longer fails because of spec parsing, binary path, or skipped examples
- `dogfood` wiring checks pass (no unregistered commands, no config field mismatches)
- `workflow-verify` verdict is `workflow-pass` or `unverified-needs-auth` (not `workflow-fail`)
+- `verify-skill` exits 0 (no mechanical mismatches between SKILL.md and CLI source). Treat non-zero as a fix-before-ship blocker — the SKILL is what agents read; if it lies about the CLI, the lie ships.
- `scorecard` is at least 65 and **no flagship or approved-in-Phase-1.5 feature returns wrong/empty output**
**Behavioral correctness is part of the ship threshold, not just structural quality.** A Grade A scorecard with a broken flagship feature (e.g., `goat "brownies"` returning a chili recipe) does NOT pass the ship threshold. Run a sample invocation of every novel-feature command before declaring shipcheck complete.
@@ -1671,6 +1674,53 @@ printing-press lock release --cli <api>-pp-cli
```
The working copy remains in `$CLI_WORK_DIR` for potential future retry. Proceed to Phase 5.6 to archive manuscripts (archiving still happens on hold).
+## Phase 4.8: Agentic SKILL Review
+
+**Runs after shipcheck, before Phase 5.** `verify-skill` (Phase 4) is a mechanical check — it catches wrong flags on wrong commands, undeclared flags, and positional-arg count mismatches. It cannot catch **semantic** issues that only a reader notices:
+
+- A trigger phrase promises behavior the CLI doesn't have ("plan dinners for the week" when there's no `meal-plan suggest`, only manual `meal-plan set`)
+- A novel-feature description says the feature does X; the actual command does Y
+- The AuthNarrative mentions `auth login --chrome` when the CLI's auth subcommands are only `set-token`/`logout`/`status`
+- Novel features shipped as stubs aren't labeled as such in the SKILL (contradicts Phase 1.5 stub-marking rule)
+- Recipes/worked examples produce output that doesn't match their prose claims
+- Trigger phrases sound agent-natural or sound like marketing copy
+
+### Dispatch
+
+Use the Agent tool (general-purpose or a dedicated reviewer) with this prompt contract:
+
+> Review the SKILL.md at `$CLI_WORK_DIR/SKILL.md` against the shipped CLI. You have these ground-truth sources:
+>
+> - `<cli> --help` output — enumerate it recursively if needed.
+> - The absorb manifest in `$RESEARCH_DIR/<stamp>-feat-<api>-pp-cli-absorb-manifest.md`.
+> - The `research.json` `novel_features` (planned) and `novel_features_built` (verified) fields.
+> - The README at `$CLI_WORK_DIR/README.md`.
+>
+> For each of these semantic checks, report findings under 50 words each:
+>
+> 1. **Trigger phrases match capabilities.** Does every trigger phrase in the SKILL's description frontmatter correspond to something the CLI can actually do? Flag phrases that imply missing capabilities.
+> 2. **Novel-feature descriptions match commands.** For each feature in the "Unique Capabilities" section, run `<cli> <command> --help` and verify the description matches the actual behavior. Mismatches are findings.
+> 3. **Stub disclosure.** If `novel_features` has items that are absent from `novel_features_built`, they shipped as stubs. The SKILL must label them (see Phase 1.5 stub-marking rule). Unlabeled stubs are findings.
+> 4. **Auth narrative accuracy.** Read the auth section. Does every `auth login/set-token/status` invocation mentioned actually exist on the CLI? Does the narrative match the CLI's auth type (api_key vs cookie vs session_handshake)?
+> 5. **Recipe output claims.** For the worked examples, does the prose claim match what the command actually produces? (Not the exact output — the shape and intent.)
+> 6. **Marketing-copy smell.** Does the SKILL read like ad copy ("comprehensive", "seamless", "powerful") instead of concrete capability descriptions? Those phrases are findings.
+>
+> Return a list of findings. For each: check name, severity (error/warning), line number, one-sentence fix. If SKILL passes all six checks, return "PASS — no findings."
+
+### Gate
+
+- If the reviewer returns PASS, proceed to Phase 5.
+- If the reviewer returns findings of severity `error`, fix them before Phase 5. Same fix-now contract as other shipcheck findings.
+- If the reviewer returns only `warning` findings, surface them to the user and proceed if they approve.
+
+### Why agentic vs template-only
+
+A template-level check would require every possible semantic mismatch to be pattern-matchable against source. Many aren't — "does this trigger phrase correspond to what the CLI does" is an LLM-shaped question. Accept the token cost for the catch.
+
+### Known blind spots
+
+The agent can't verify runtime behavior without running commands; stick to help-text and source-based claims. For runtime-behavior claims (e.g., "returns 5 matching recipes"), Phase 5 dogfood is the right gate.
+
## Phase 5: Dogfood Testing
**MANDATORY when an API key is available. Do NOT skip or shortcut this phase.**
← 831040d5 feat(cli): auth.optional spec field — doctor INFO not FAIL,
·
back to Cli Printing Press
·
feat(cli): machine-output-verification Wave A — cliutil pack 65dacc25 →