Add a minimal RDMA CM selftest suite that captures observability baselines and runs trace, counter-delta, and fault-injection oriented checks, plus a review-loop helper for repeated validation rounds. Usage (client side): - export CM_WORKLOAD_CMD='ib_send_bw -d -i -R -g ' (User can customize CM_WORKLOAD_CMD) - sudo -E make -C tools/testing/selftests TARGETS=drivers/net/rdma run_tests Signed-off-by: Chenguang Zhao --- The first patch adds a focused RDMA CM selftest suite under kselftest to make CM behavior easier to observe and validate in routine regression runs. It introduces baseline collection, trace-sequence checks, counter-delta checks, and failslab-based recovery checks, plus a review-loop script for one-shot serial execution. It also registers drivers/net/rdma in the top-level selftests TARGETS, so the suite runs through standard kselftest flow (make ... TARGETS=drivers/net/rdma run_tests) instead of requiring manual script-by-script execution. --- tools/testing/selftests/Makefile | 1 + .../selftests/drivers/net/rdma/Makefile | 13 ++ .../selftests/drivers/net/rdma/README.md | 168 ++++++++++++++++++ .../drivers/net/rdma/rdma_cm_baseline.sh | 58 ++++++ .../drivers/net/rdma/rdma_cm_counter_delta.sh | 72 ++++++++ .../net/rdma/rdma_cm_fault_injection.sh | 95 ++++++++++ .../drivers/net/rdma/rdma_cm_review_loop.sh | 35 ++++ .../net/rdma/rdma_cm_trace_sequence.sh | 83 +++++++++ .../selftests/drivers/net/rdma/rdma_common.sh | 126 +++++++++++++ 9 files changed, 651 insertions(+) create mode 100644 tools/testing/selftests/drivers/net/rdma/Makefile create mode 100644 tools/testing/selftests/drivers/net/rdma/README.md create mode 100755 tools/testing/selftests/drivers/net/rdma/rdma_cm_baseline.sh create mode 100755 tools/testing/selftests/drivers/net/rdma/rdma_cm_counter_delta.sh create mode 100755 tools/testing/selftests/drivers/net/rdma/rdma_cm_fault_injection.sh create mode 100755 tools/testing/selftests/drivers/net/rdma/rdma_cm_review_loop.sh create mode 100755 tools/testing/selftests/drivers/net/rdma/rdma_cm_trace_sequence.sh create mode 100755 tools/testing/selftests/drivers/net/rdma/rdma_common.sh diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 984abb6d42ab..0df7034f46b2 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -22,6 +22,7 @@ TARGETS += drivers/ntsync TARGETS += drivers/s390x/uvdevice TARGETS += drivers/net TARGETS += drivers/net/bonding +TARGETS += drivers/net/rdma TARGETS += drivers/net/netconsole TARGETS += drivers/net/team TARGETS += drivers/net/virtio_net diff --git a/tools/testing/selftests/drivers/net/rdma/Makefile b/tools/testing/selftests/drivers/net/rdma/Makefile new file mode 100644 index 000000000000..42d042aac1f0 --- /dev/null +++ b/tools/testing/selftests/drivers/net/rdma/Makefile @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: GPL-2.0 + +TEST_PROGS := \ + rdma_cm_baseline.sh \ + rdma_cm_trace_sequence.sh \ + rdma_cm_counter_delta.sh \ + rdma_cm_fault_injection.sh + +TEST_FILES := \ + rdma_common.sh \ + rdma_cm_review_loop.sh + +include ../../../lib.mk diff --git a/tools/testing/selftests/drivers/net/rdma/README.md b/tools/testing/selftests/drivers/net/rdma/README.md new file mode 100644 index 000000000000..a9caca638b20 --- /dev/null +++ b/tools/testing/selftests/drivers/net/rdma/README.md @@ -0,0 +1,168 @@ +# RDMA CM Selftests Usage Guide + +These scripts provide baseline observability and regression checks for RDMA/CM +paths under the Linux `kselftest` framework. + +Files: + +- `rdma_cm_baseline.sh` +- `rdma_cm_trace_sequence.sh` +- `rdma_cm_counter_delta.sh` +- `rdma_cm_fault_injection.sh` +- `rdma_cm_review_loop.sh` +- `rdma_common.sh` + +The scripts use a fixed test flow and only require workload commands from +environment variables. + +## 1. Use Cases + +- CM main-flow observability checks (REQ/REP/RTU) +- CM counter delta validation +- Recovery validation after fault injection (`failslab`) +- One-shot serial regression run + +## 2. Requirements + +- root privileges +- Reachable client/server network path +- Perftest command available on the remote side (default: `ib_send_bw -R`) +- For fault injection: kernel support for `failslab` and access to + `/sys/kernel/debug/failslab` + +## 3. Recommended Execution Order + +```bash +./rdma_cm_baseline.sh +./rdma_cm_trace_sequence.sh +./rdma_cm_counter_delta.sh +./rdma_cm_fault_injection.sh +``` + +Or run all in sequence: + +```bash +./rdma_cm_review_loop.sh +``` + +## 4. Quick Start (Two Hosts) + +### 4.1 Server side (recommended: loop and keep listening) + +```bash +while true; do + ib_send_bw -d -i -R + sleep 1 +done +``` + +### 4.2 Client side (set workload command) + +```bash +export CM_WORKLOAD_CMD='ib_send_bw -d rocep1s0f0 -i 1 -R -g 3 192.168.1.22' +export CM_VALIDATE_RECOVERY_CMD="${CM_WORKLOAD_CMD}" + +./rdma_cm_review_loop.sh +``` + +### 4.3 Run through kselftest harness + +```bash +sudo -E make -C tools/testing/selftests TARGETS=drivers/net/rdma run_tests +``` + +`sudo -E` keeps exported workload variables for test scripts. + +### 4.4 Run a single script directly + +```bash +cd tools/testing/selftests/drivers/net/rdma +sudo -E ./rdma_cm_counter_delta.sh +``` + +## 5. Configuration Parameters + +Only workload command variables are supported: + +- `CM_WORKLOAD_CMD`: required; workload command used by trace/counter/fault tests +- `CM_VALIDATE_RECOVERY_CMD`: optional; command for recovery stage in fault + injection test (falls back to `CM_WORKLOAD_CMD`) + +Fixed internal settings: + +- Counter pre-wait: `2s` +- Recovery pre-wait: `2s` +- Failslab path: `/sys/kernel/debug/failslab` +- Failslab knobs: `task-filter=1`, `probability=1`, `interval=100`, `times=1` +- Counter limits: `cm_rx_duplicates.* <= 10`, `cm_tx_retries.* <= 10` +- Trace log path: `/tmp/rdma_cm_trace..log` + +## 6. Exit Codes + +- `0`: pass +- `4`: skip (environment not ready, e.g. missing tracefs/failslab/counters) +- other non-zero: fail + +## 7. Result Interpretation + +When running with kselftest (`make ... run_tests`), TAP output looks like: + +```text +ok 1 selftests: drivers/net/rdma: rdma_cm_baseline.sh +ok 2 selftests: drivers/net/rdma: rdma_cm_trace_sequence.sh +not ok 3 selftests: drivers/net/rdma: rdma_cm_counter_delta.sh # exit=1 +``` + +- `ok N ...`: that script passed +- `not ok N ... # exit=1`: that script failed +- `not ok N ... # exit=4`: that script was skipped by environment checks + +When running `rdma_cm_review_loop.sh` directly, check the final summary block: + +```text +==== summary ==== +baseline=0 +trace=0 +counters=1 +fault_injection=0 +``` + +Each value is the corresponding script return code. + +## 8. Common Issues + +### 8.1 `cm counters are unavailable under /sys/class/infiniband` + +The script did not find `cm_tx_msgs` (and related) counters. Check: + +- whether `cm_tx_msgs` exists under any available RDMA port path + +### 8.2 `missing CM send trace events (req/rep/rtu)` + +This usually means workload did not create a CM handshake. Verify +`CM_WORKLOAD_CMD` and remote server readiness. + +### 8.3 `Unexpected CM event ... 8` + +Usually means the server was not ready for the next connection. Try: + +- keep server in a listening loop +- ensure the remote server is still listening before the recovery stage + +### 8.4 `failslab is unavailable` + +Expected skip when failslab is not exposed by kernel/debugfs. Check: + +```bash +mount | grep debugfs +ls /sys/kernel/debug/failslab +``` + +## 9. Minimal Regression Profile + +```bash +export CM_WORKLOAD_CMD='ib_send_bw -d -i 1 -R -g ' +export CM_VALIDATE_RECOVERY_CMD="${CM_WORKLOAD_CMD}" + +./rdma_cm_review_loop.sh +``` diff --git a/tools/testing/selftests/drivers/net/rdma/rdma_cm_baseline.sh b/tools/testing/selftests/drivers/net/rdma/rdma_cm_baseline.sh new file mode 100755 index 000000000000..b0d8b3e46470 --- /dev/null +++ b/tools/testing/selftests/drivers/net/rdma/rdma_cm_baseline.sh @@ -0,0 +1,58 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +source "${SCRIPT_DIR}/rdma_common.sh" + +require_root +require_cmd date +require_cmd uname + +trace_dir="$(tracefs_dir || true)" +counter_root="$(find_cm_counter_root || true)" +out_dir="/tmp/rdma_cm_baseline.$(date +%s)" +dmesg_lines=400 +dmesg_pattern="ib_cm|infiniband|rdma|roce|mlx|hns_roce|irdma|siw|rxe" + +mkdir -p "${out_dir}" + +log_info "writing baseline to ${out_dir}" + +{ + echo "timestamp=$(date -u +%FT%TZ)" + echo "kernel=$(uname -r)" + echo "hostname=$(uname -n)" + echo "dmesg_lines=${dmesg_lines}" + echo "dmesg_pattern=${dmesg_pattern}" +} >"${out_dir}/env.txt" + +if [[ -n "${trace_dir}" && -d "${trace_dir}/events/ib_cma" ]]; then + find "${trace_dir}/events/ib_cma" -maxdepth 2 -name enable -print \ + >"${out_dir}/trace_events.list" 2>/dev/null || true +else + log_warn "tracefs or ib_cma trace events are unavailable" +fi + +if [[ -n "${counter_root}" ]]; then + { + echo "counter_root=${counter_root}" + for group in "${RDMA_COUNTER_GROUPS[@]}"; do + for attr in "${RDMA_COUNTER_ATTRS[@]}"; do + value="$(read_cm_counter "${counter_root}" "${group}" "${attr}")" + echo "${group}.${attr}=${value}" + done + done + } >"${out_dir}/cm_counters.before" +else + log_warn "cm counters are unavailable under /sys/class/infiniband" +fi + +if command -v dmesg >/dev/null 2>&1; then + dmesg | tail -n "${dmesg_lines}" | grep -E "${dmesg_pattern}" \ + >"${out_dir}/dmesg.rdma.tail" || true +fi + +log_info "baseline collection completed" +exit 0 diff --git a/tools/testing/selftests/drivers/net/rdma/rdma_cm_counter_delta.sh b/tools/testing/selftests/drivers/net/rdma/rdma_cm_counter_delta.sh new file mode 100755 index 000000000000..060adf9fe78a --- /dev/null +++ b/tools/testing/selftests/drivers/net/rdma/rdma_cm_counter_delta.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +source "${SCRIPT_DIR}/rdma_common.sh" + +require_root +counter_root="$(find_cm_counter_root || true)" +counter_wait_sec=2 + +if [[ -z "${counter_root}" ]]; then + log_warn "cm counters are unavailable under /sys/class/infiniband" + exit "${ksft_skip}" +fi + +declare -A before after + +for group in "${RDMA_COUNTER_GROUPS[@]}"; do + for attr in "${RDMA_COUNTER_ATTRS[@]}"; do + key="${group}.${attr}" + before["${key}"]="$(read_cm_counter "${counter_root}" "${group}" "${attr}")" + done +done + +if [[ "${counter_wait_sec}" != "0" ]]; then + log_info "waiting ${counter_wait_sec}s before workload" + sleep "${counter_wait_sec}" +fi + +workload_rc=0 +run_workload || workload_rc=$? +if [[ "${workload_rc}" -eq "${ksft_skip}" ]]; then + exit "${ksft_skip}" +fi +if [[ "${workload_rc}" -ne 0 ]]; then + log_err "workload failed with rc=${workload_rc}" + exit "${workload_rc}" +fi + +for group in "${RDMA_COUNTER_GROUPS[@]}"; do + for attr in "${RDMA_COUNTER_ATTRS[@]}"; do + key="${group}.${attr}" + after["${key}"]="$(read_cm_counter "${counter_root}" "${group}" "${attr}")" + delta=$((after["${key}"] - before["${key}"])) + echo "${key}.delta=${delta}" + if ((delta < 0)); then + log_err "counter regressed: ${key}" + exit 1 + fi + done +done + +dup_limit=10 +retry_limit=10 + +for attr in "${RDMA_COUNTER_ATTRS[@]}"; do + dup_delta=$((after["cm_rx_duplicates.${attr}"] - before["cm_rx_duplicates.${attr}"])) + retry_delta=$((after["cm_tx_retries.${attr}"] - before["cm_tx_retries.${attr}"])) + + if ((dup_delta > dup_limit)); then + log_err "duplicate counter exceeds limit: ${attr}=${dup_delta}" + exit 1 + fi + if ((retry_delta > retry_limit)); then + log_err "retry counter exceeds limit: ${attr}=${retry_delta}" + exit 1 + fi +done + +exit 0 diff --git a/tools/testing/selftests/drivers/net/rdma/rdma_cm_fault_injection.sh b/tools/testing/selftests/drivers/net/rdma/rdma_cm_fault_injection.sh new file mode 100755 index 000000000000..0202ee901386 --- /dev/null +++ b/tools/testing/selftests/drivers/net/rdma/rdma_cm_fault_injection.sh @@ -0,0 +1,95 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +source "${SCRIPT_DIR}/rdma_common.sh" + +require_root + +debugfs_fail="/sys/kernel/debug/failslab" +recovery_wait_sec=2 +if [[ ! -d "${debugfs_fail}" ]]; then + log_warn "failslab is unavailable: ${debugfs_fail}" + exit "${ksft_skip}" +fi + +for knob in probability interval times task-filter; do + if [[ ! -f "${debugfs_fail}/${knob}" ]]; then + log_warn "failslab knob missing: ${knob}" + exit "${ksft_skip}" + fi +done + +orig_probability="$(cat "${debugfs_fail}/probability")" +orig_interval="$(cat "${debugfs_fail}/interval")" +orig_times="$(cat "${debugfs_fail}/times")" +orig_task_filter="$(cat "${debugfs_fail}/task-filter")" + +restore_knobs() +{ + echo "${orig_probability}" >"${debugfs_fail}/probability" || true + echo "${orig_interval}" >"${debugfs_fail}/interval" || true + echo "${orig_times}" >"${debugfs_fail}/times" || true + echo "${orig_task_filter}" >"${debugfs_fail}/task-filter" || true +} + +trap restore_knobs EXIT + +log_failslab_state() +{ + local state="$1" + local task_filter probability interval times + + task_filter="$(cat "${debugfs_fail}/task-filter")" + probability="$(cat "${debugfs_fail}/probability")" + interval="$(cat "${debugfs_fail}/interval")" + times="$(cat "${debugfs_fail}/times")" + + log_info "failslab ${state}: task-filter=${task_filter} probability=${probability}" + log_info "failslab ${state}: interval=${interval} times=${times}" +} + +echo 1 >"${debugfs_fail}/task-filter" +echo 1 >"${debugfs_fail}/probability" +echo 100 >"${debugfs_fail}/interval" +echo 1 >"${debugfs_fail}/times" +log_failslab_state "enabled" + +if [[ -z "${CM_WORKLOAD_CMD:-}" && -n "${CM_VALIDATE_RECOVERY_CMD:-}" ]]; then + CM_WORKLOAD_CMD="${CM_VALIDATE_RECOVERY_CMD}" + log_warn "CM_WORKLOAD_CMD is not set; fallback to CM_VALIDATE_RECOVERY_CMD" +fi + +injected_rc=0 +run_workload || injected_rc=$? +if [[ "${injected_rc}" -eq "${ksft_skip}" ]]; then + exit "${ksft_skip}" +fi +log_info "workload rc under injection=${injected_rc}" + +echo 0 >"${debugfs_fail}/probability" +echo 0 >"${debugfs_fail}/times" +echo 0 >"${debugfs_fail}/task-filter" +log_failslab_state "disabled" + +recovery_cmd="${CM_VALIDATE_RECOVERY_CMD:-${CM_WORKLOAD_CMD:-}}" +if [[ -z "${recovery_cmd}" ]]; then + log_warn "CM_VALIDATE_RECOVERY_CMD and CM_WORKLOAD_CMD are both unset" + exit "${ksft_skip}" +fi + +if [[ "${recovery_wait_sec}" != "0" ]]; then + log_info "waiting ${recovery_wait_sec}s before recovery workload" + sleep "${recovery_wait_sec}" +fi + +log_info "running recovery workload: ${recovery_cmd}" +if ! bash -c "${recovery_cmd}"; then + log_err "recovery workload failed after disabling fault injection" + log_err "hint: ensure remote server is restarted and listening for a second connection" + exit 1 +fi + +exit 0 diff --git a/tools/testing/selftests/drivers/net/rdma/rdma_cm_review_loop.sh b/tools/testing/selftests/drivers/net/rdma/rdma_cm_review_loop.sh new file mode 100755 index 000000000000..c156090b17e3 --- /dev/null +++ b/tools/testing/selftests/drivers/net/rdma/rdma_cm_review_loop.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +cd "${SCRIPT_DIR}" + +declare -A rc + +run_step() +{ + local name="$1" + local cmd="$2" + + echo "==== ${name} ====" + if bash -c "${cmd}"; then + rc["${name}"]=0 + else + rc["${name}"]=$? + fi + echo "==== ${name} rc=${rc["${name}"]} ====" +} + +run_step baseline "./rdma_cm_baseline.sh" +run_step trace "./rdma_cm_trace_sequence.sh" +run_step counters "./rdma_cm_counter_delta.sh" +run_step fault_injection "./rdma_cm_fault_injection.sh" + +echo "==== summary ====" +for name in baseline trace counters fault_injection; do + echo "${name}=${rc["${name}"]}" +done + +exit 0 diff --git a/tools/testing/selftests/drivers/net/rdma/rdma_cm_trace_sequence.sh b/tools/testing/selftests/drivers/net/rdma/rdma_cm_trace_sequence.sh new file mode 100755 index 000000000000..7e68289345e8 --- /dev/null +++ b/tools/testing/selftests/drivers/net/rdma/rdma_cm_trace_sequence.sh @@ -0,0 +1,83 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +source "${SCRIPT_DIR}/rdma_common.sh" + +require_root +require_cmd bash +require_cmd grep + +trace_dir="$(tracefs_dir || true)" +if [[ -z "${trace_dir}" ]]; then + log_warn "tracefs is unavailable" + exit "${ksft_skip}" +fi + +if [[ ! -d "${trace_dir}/events/ib_cma" ]]; then + log_warn "ib_cma trace events are unavailable" + exit "${ksft_skip}" +fi + +workload_rc=0 + +cleanup_trace() +{ + local event + + for event in icm_send_req icm_send_rep icm_send_rtu icm_recv_unknown_attr; do + [[ -f "${trace_dir}/events/ib_cma/${event}/enable" ]] && \ + echo 0 >"${trace_dir}/events/ib_cma/${event}/enable" + done + [[ -f "${trace_dir}/events/ib_cma/enable" ]] && echo 0 >"${trace_dir}/events/ib_cma/enable" + echo 0 >"${trace_dir}/tracing_on" +} + +trap cleanup_trace EXIT + +echo 0 >"${trace_dir}/tracing_on" +echo >"${trace_dir}/trace" +echo 1 >"${trace_dir}/events/ib_cma/enable" + +for event in icm_send_req icm_send_rep icm_send_rtu; do + if [[ -f "${trace_dir}/events/ib_cma/${event}/enable" ]]; then + echo 1 >"${trace_dir}/events/ib_cma/${event}/enable" + fi +done + +echo 1 >"${trace_dir}/tracing_on" +run_workload || workload_rc=$? +echo 0 >"${trace_dir}/tracing_on" + +if [[ "${workload_rc}" -eq "${ksft_skip}" ]]; then + exit "${ksft_skip}" +fi + +trace_log="/tmp/rdma_cm_trace.$(date +%s).log" +cat "${trace_dir}/trace" >"${trace_log}" +log_info "captured trace at ${trace_log}" + +if ! grep -Eq "icm_send_(req|rep|rtu)" "${trace_log}"; then + log_err "missing CM send trace events (req/rep/rtu)" + exit 1 +fi + +err_lines="$(grep "icm_.*_err" "${trace_log}" || true)" +if [[ -n "${err_lines}" ]]; then + # DREP send failure while already in TIMEWAIT is a common teardown + # race and is tolerated for this smoke-style validation script. + untolerated_err_lines="$( + printf '%s\n' "${err_lines}" | \ + grep -Ev "icm_send_drep_err: .*state=TIMEWAIT" || true + )" + if [[ -n "${untolerated_err_lines}" ]]; then + log_err "error trace event detected in ib_cma path" + printf '%s\n' "${untolerated_err_lines}" >&2 + exit 1 + fi + log_warn "only tolerated TIMEWAIT drep errors observed" +fi + +exit 0 diff --git a/tools/testing/selftests/drivers/net/rdma/rdma_common.sh b/tools/testing/selftests/drivers/net/rdma/rdma_common.sh new file mode 100755 index 000000000000..ee3d8b0d86b2 --- /dev/null +++ b/tools/testing/selftests/drivers/net/rdma/rdma_common.sh @@ -0,0 +1,126 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +ksft_skip=4 +RET=0 + +RDMA_COUNTER_GROUPS=( + cm_tx_msgs + cm_tx_retries + cm_rx_msgs + cm_rx_duplicates +) + +RDMA_COUNTER_ATTRS=( + req + mra + rej + rep + rtu + dreq + drep + sidr_req + sidr_rep + lap + apr +) + +log_info() +{ + echo "INFO: $*" +} + +log_warn() +{ + echo "WARN: $*" >&2 +} + +log_err() +{ + echo "ERROR: $*" >&2 +} + +require_root() +{ + if [[ "$(id -u)" -ne 0 ]]; then + log_warn "this test requires root privileges" + exit "${ksft_skip}" + fi +} + +require_cmd() +{ + local cmd="$1" + + command -v "${cmd}" >/dev/null 2>&1 || { + log_warn "missing required command: ${cmd}" + exit "${ksft_skip}" + } +} + +tracefs_dir() +{ + if [[ -d /sys/kernel/tracing ]]; then + echo /sys/kernel/tracing + elif [[ -d /sys/kernel/debug/tracing ]]; then + echo /sys/kernel/debug/tracing + else + return 1 + fi +} + +find_cm_counter_root() +{ + local base + local port + local candidate + + for base in /sys/class/infiniband/*; do + [[ -d "${base}" ]] || continue + + for port in "${base}"/ports/*; do + [[ -d "${port}" ]] || continue + # RoCE / newer sysfs: cm_* groups live directly under ports// + if [[ -d "${port}/cm_tx_msgs" ]]; then + echo "${port}" + return 0 + fi + # Legacy layout: under counters/ or hw_counters/ + for candidate in "${port}/counters" "${port}/hw_counters"; do + [[ -d "${candidate}/cm_tx_msgs" ]] || continue + echo "${candidate}" + return 0 + done + done + done + + return 1 +} + +read_cm_counter() +{ + local root="$1" + local group="$2" + local attr="$3" + local path="${root}/${group}/${attr}" + + if [[ -f "${path}" ]]; then + cat "${path}" 2>/dev/null + else + echo 0 + fi +} + +run_workload() +{ + local cmd="${CM_WORKLOAD_CMD:-}" + + if [[ -z "${cmd}" ]]; then + log_warn "CM_WORKLOAD_CMD is not set" + return "${ksft_skip}" + fi + + log_info "running workload: ${cmd}" + bash -c "${cmd}" +} + -- 2.25.1