The stress test module starts a worker on every online CPU, and each one holds its lock for 20ms with interrupts disabled before releasing it and queueing again at once. A waiter that is not at the head of the queue has no timeout, so the queue holds every worker but the holder, and the last one in line waits for all of them: the longest wait, spent with interrupts off, grows linearly with the number of CPUs. On a 4 CPU VM that is 40ms, on a 128 CPU VM it is 2.5s, and the NMI handlers of the test spend the same time in the queue: INFO: NMI handler (perf_event_nmi_handler) took too long to run: 2560.853 msecs perf: interrupt took too long (33533733 > 25008210), lowering kernel.perf_event_max_sample_rate to 1000 tsc: Marking TSC unstable due to clocksource watchdog Enabling the perf events alone takes over a minute, as each IPI waits for its CPU to reenable interrupts, and the test takes 145s instead of 10s. Once the host is oversubscribed, the waits stretch further, timers run late, and the soft lockup detector fires, which panics the VM with CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=1 from the selftests config: WARNING: kernel/time/hrtimer.c:1098 at hrtimer_forward+0x76/0xc0, CPU#6: rqsl_w/6/1185 watchdog: BUG: soft lockup - CPU#25 stuck for 21s! [kworker/R-mm_pe:13] Andrii reports that the test panics his 128 vCPU VM every time, while the same kernel and module pass with 8 vCPUs. Limit the test to the first 8 CPUs by default, which bounds the longest wait to 120ms with the default critical section length. A new max_cpus module parameter overrides the limit, and 0 restores the old behavior of using every online CPU. The worker threads and the perf events are both created for the capped set of CPUs, and teardown walks the threads by index instead of by online CPU so that it covers the same set. Fixes: 15cf39221e89 ("selftests/bpf: Add stress test for rqspinlock in NMI") Reported-by: Andrii Nakryiko Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/test_kmods/bpf_test_rqspinlock.c | 21 ++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_test_rqspinlock.c b/tools/testing/selftests/bpf/test_kmods/bpf_test_rqspinlock.c index 7b4ae5e81d32..a9cc5f496f46 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_test_rqspinlock.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_test_rqspinlock.c @@ -68,6 +68,11 @@ module_param(nmi_delay, int, 0644); MODULE_PARM_DESC(nmi_delay, "rqspinlock critical section length for NMI context (10ms default)"); +static int max_cpus = 8; +module_param(max_cpus, int, 0444); +MODULE_PARM_DESC(max_cpus, + "maximum number of CPUs taking part in the test, 0 for all online CPUs (8 default)"); + static struct perf_event **rqsl_evts; static int rqsl_nevts; @@ -216,7 +221,7 @@ static void free_rqsl_threads(void) int i; if (rqsl_threads) { - for_each_online_cpu(i) { + for (i = 0; i < rqsl_nthreads; i++) { if (rqsl_threads[i]) kthread_stop(rqsl_threads[i]); } @@ -247,7 +252,17 @@ static int bpf_test_rqspinlock_init(void) return -EINVAL; } - pr_err("Mode = %s\n", rqsl_mode_names[test_mode]); + /* + * Every worker holds its lock for normal_delay ms with interrupts + * disabled, and a queued waiter does not time out until it reaches the + * head of the queue, so the longest wait, spent with interrupts off, + * grows with the number of contending CPUs. Limit the test to the first + * max_cpus CPUs to keep it bounded on large machines. + */ + if (max_cpus > 0) + ncpus = min(ncpus, max_cpus); + + pr_err("Mode = %s, CPUs = %d\n", rqsl_mode_names[test_mode], ncpus); if (ncpus < test_mode + 2) return -ENOTSUPP; @@ -279,7 +294,7 @@ static int bpf_test_rqspinlock_init(void) } rqsl_nthreads = ncpus; - for_each_online_cpu(i) { + for (i = 0; i < ncpus; i++) { struct task_struct *t; t = kthread_create(rqspinlock_worker_fn, NULL, "rqsl_w/%d", i); base-commit: b116ac6087541d3d98031e786adbe9c733cd6df3 -- 2.53.0-Meta