selftest: cover the credential lens (it shipped untested)

7f0c63466b7f8e99b94cb8b796d0217a83a69d6a · 2026-09-12 06:40:19 -0700 · Steve Abrams

The credential lens landed in a23289a — flagging a visible username, API secret,
basic-auth blob or backend host in a submission clip — but selftest.sh was
written before it and never exercised it. Zero of the six cases touched it, so
a broken lens would have passed the suite silently. That is the exact thing
CLAUDE.md forbids: a check with no negative test proving it goes red.

Three cases added, one of which is the half that usually gets skipped:
  - D_cred.mov (Username / demo_reviewer / an agentabrams host) -> lens MUST fire
  - D_cred.mov -> verdict MUST stay TECH_PASS_PRIVACY_PENDING, because the lens
    warns by design (Apple 2.1 item #4 explicitly ASKS for demo credentials, so
    visible credentials must never hard-fail a clip)
  - C.mov (clean) -> lens MUST stay silent: the false-positive guard

Proven by fault injection, not just by passing: neutering all four CRED_*
patterns makes the suite report "credential lens did NOT fire" / SOME FAILED;
restoring them returns ALL PASS (9/9). recording-qa.sh verified byte-identical
to HEAD afterwards, so the injection left no residue.

Also re-verified the pre-existing gate end to end before touching it: ffprobe,
ffmpeg, tesseract and shasum all present, and the original 6 cases still pass —
including the two that matter most for App Review, a simulator clip rejected as
REJECTED_NOT_DEVICE and --approve refusing to bind a review to a PII clip.

Refs TK-11155.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Files touched

M selftest.sh

Diff

commit 7f0c63466b7f8e99b94cb8b796d0217a83a69d6a
Author: Steve Abrams <steve@designerwallcoverings.com>
Date:   Sat Sep 12 06:40:19 2026 -0700

    selftest: cover the credential lens (it shipped untested)
    
    The credential lens landed in a23289a — flagging a visible username, API secret,
    basic-auth blob or backend host in a submission clip — but selftest.sh was
    written before it and never exercised it. Zero of the six cases touched it, so
    a broken lens would have passed the suite silently. That is the exact thing
    CLAUDE.md forbids: a check with no negative test proving it goes red.
    
    Three cases added, one of which is the half that usually gets skipped:
      - D_cred.mov (Username / demo_reviewer / an agentabrams host) -> lens MUST fire
      - D_cred.mov -> verdict MUST stay TECH_PASS_PRIVACY_PENDING, because the lens
        warns by design (Apple 2.1 item #4 explicitly ASKS for demo credentials, so
        visible credentials must never hard-fail a clip)
      - C.mov (clean) -> lens MUST stay silent: the false-positive guard
    
    Proven by fault injection, not just by passing: neutering all four CRED_*
    patterns makes the suite report "credential lens did NOT fire" / SOME FAILED;
    restoring them returns ALL PASS (9/9). recording-qa.sh verified byte-identical
    to HEAD afterwards, so the injection left no residue.
    
    Also re-verified the pre-existing gate end to end before touching it: ffprobe,
    ffmpeg, tesseract and shasum all present, and the original 6 cases still pass —
    including the two that matter most for App Review, a simulator clip rejected as
    REJECTED_NOT_DEVICE and --approve refusing to bind a review to a PII clip.
    
    Refs TK-11155.
    
    Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 selftest.sh | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/selftest.sh b/selftest.sh
index 8ab2019..470e674 100755
--- a/selftest.sh
+++ b/selftest.sh
@@ -17,11 +17,13 @@ def frame(txt,out):
     d.multiline_text((90,880),txt,fill="black",font=f); im.save(out)
 frame("Contact test@evil.com", os.path.join(T,"pii.png"))
 frame("Opportunities\nListing Settings", os.path.join(T,"clean.png"))
+frame("Username\ndemo_reviewer\nhttps://api.agentabrams.com", os.path.join(T,"cred.png"))
 PY
 
 ffmpeg -y -loglevel error -f lavfi -i color=c=black:s=1080x1920:d=5 -pix_fmt yuv420p "$T/A.mov"
 ffmpeg -y -loglevel error -loop 1 -i "$T/pii.png"   -t 25 -r 5 -pix_fmt yuv420p "$T/B.mov"
 ffmpeg -y -loglevel error -loop 1 -i "$T/clean.png" -t 25 -r 5 -pix_fmt yuv420p "$T/C.mov"
+ffmpeg -y -loglevel error -loop 1 -i "$T/cred.png"  -t 25 -r 5 -pix_fmt yuv420p "$T/D_cred.mov"
 cp "$T/C.mov" "$T/E_simulator.mov"
 
 v(){ "$QA" --json "$1" 2>/dev/null | sed 's/.*"verdict":"//;s/".*//'; }
@@ -35,6 +37,27 @@ assert "$T/B.mov" PRIVACY_FLAGGED
 assert "$T/C.mov" TECH_PASS_PRIVACY_PENDING
 assert "$T/E_simulator.mov" REJECTED_NOT_DEVICE
 # --approve refuses a PII clip
+# credential lens (added with a507... a23289a, previously UNTESTED — TK-11155).
+# It WARNS by design (Apple 2.1 #4 asks for demo credentials) so the verdict must
+# stay TECH_PASS_PRIVACY_PENDING; what we prove is that the lens actually fires,
+# and — the half that matters — that it does NOT fire on a clean clip.
+credout=$("$QA" "$T/D_cred.mov" 2>&1)
+if printf '%s' "$credout" | grep -q '🔑 CRED'; then
+  echo "PASS  D_cred.mov -> credential lens FIRED ($(printf '%s' "$credout" | grep -c '🔑 CRED') hit(s))"
+else
+  echo "FAIL  D_cred.mov -> credential lens did NOT fire"; fail=1
+fi
+if printf '%s' "$credout" | grep -q 'TECH_PASS_PRIVACY_PENDING'; then
+  echo "PASS  D_cred.mov -> still WARN, not a hard fail (by design)"
+else
+  echo "FAIL  D_cred.mov -> credential visibility hard-failed; it must only WARN"; fail=1
+fi
+if "$QA" "$T/C.mov" 2>&1 | grep -q '🔑 CRED'; then
+  echo "FAIL  C.mov -> credential lens FALSE POSITIVE on a clean clip"; fail=1
+else
+  echo "PASS  C.mov -> credential lens silent on a clean clip (no false positive)"
+fi
+
 "$QA" --approve "$T/B.mov" >/dev/null 2>&1 && { echo "FAIL  --approve accepted a PII clip"; fail=1; } || echo "PASS  --approve refused the PII clip"
 # approve the clean clip -> READY
 "$QA" --approve "$T/C.mov" >/dev/null 2>&1

← a23289a recording-qa: add credential lens — flag visible username/AP · back to Ios Fleet Recording Qa · (newest)