Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker threads from executing block work on isolated CPUs. When all CPUs mapped to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx) evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with the intention of offloading execution. However, kblockd_workqueue is allocated as a per-CPU workqueue (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the workqueue core resolves the target CPU via raw_smp_processor_id(). Consequently, if a task executing on an isolated CPU submits I/O and triggers an asynchronous queue dispatch with zero delay (for instance, upon hardware queue congestion, tag/resource exhaustion returning BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto the local isolated CPU's worker pool. This schedules the local per-CPU high-priority kworker (such as kworker/6:1H-kblockd) on the isolated CPU. In real-time environments, an active SCHED_FIFO workload will starve this SCHED_NORMAL worker and stall hardware dispatch, whilst also introducing unwanted latency jitter on nohz_full cores, defeating that commit's original objective. Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU from housekeeping_cpumask(HK_TYPE_DOMAIN) via cpumask_any_and_distribute() when executing on an isolated CPU, and otherwise return WORK_CPU_UNBOUND. Gate the helper behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION so that common configurations without CPU isolation incur zero overhead across single-queue devices, hardware queues with empty cpumasks, and offline CPU fallback paths, whilst non-isolated CPUs preserve local execution via WORK_CPU_UNBOUND. Guard the slow-path check and cpumask selection with guard(preempt)() to pin raw_smp_processor_id(), satisfy RCU dereferencing of housekeeping_cpumask(HK_TYPE_DOMAIN), and protect the per-CPU distribution state. Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs") Cc: Ming Lei Cc: Jens Axboe Cc: Frederic Weisbecker Cc: Marco Crivellari Cc: stable@vger.kernel.org Signed-off-by: Aaron Tomlin --- Note: As Frederic Weisbecker pointed out on v4, workqueue-level support (WQ_PREFER_PERCPU) is currently in development to handle isolated CPUs automatically, at which point kblockd_workqueue will be a candidate for simplification; in the meantime, this provides a minimal fix suitable for backporting. Changes since v4: - Rebased against v7.4-rc1 - Hoisted #ifdef CONFIG_CPU_ISOLATION and static_branch_unlikely(&housekeeping_overridden) into blk_mq_hctx_unbound_cpu() so blk_mq_hctx_next_cpu() retains a single fast-path check and non-isolated configurations avoid any RCU or preemption guards across all unbound fallback paths - Moved guard(preempt)() ahead of cpu_is_isolated(raw_smp_processor_id()) to prevent migration between reading the local CPU ID and selecting a housekeeping CPU under CONFIG_PREEMPT_RCU, whilst satisfying rcu_dereference_all_check() without a separate guard(rcu)() - Consolidated the commit message summary into prose - Link to v4: https://lore.kernel.org/lkml/20261022034737.47820-1-atomlin@atomlin.com/ Changes since v3: - Create companion blktests test case (block/050) to reproduce this issue and verify the fix (Daniel Wagner) - Gate the single-queue (nr_hw_queues == 1) check behind the static key housekeeping_overridden under CONFIG_CPU_ISOLATION to ensure the common case without CPU isolation incurs zero overhead on the fast path (Jens Axboe) - Wrap the evaluation of cpu_is_isolated() inside guard(rcu)() in blk_mq_hctx_unbound_cpu() to prevent an RCU lockdep warning and ensure safe dereferencing of the housekeeping cpumask while concurrent cpuset updates occur - Disable preemption via guard(preempt)() around cpumask_any_and_distribute() to prevent a preemption warning under CONFIG_PREEMPT_RCU and avoid data races when accessing per-CPU distribution state - Link to v3: https://lore.kernel.org/all/20261016052203.489760-1-atomlin@atomlin.com/ Changes since v2: - Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER to align with cpu_is_isolated(), workqueue affinity rules, and cpuset isolated partitions - Query the housekeeping cpumask under guard(rcu)() to safely handle dynamic cpuset updates without lockdep warnings - Validate that cpumask_any_and_distribute() returns a valid cpu ID before returning, falling back cleanly to WORK_CPU_UNBOUND - Link to v2: https://lore.kernel.org/lkml/20261015020225.422107-1-atomlin@atomlin.com/ Changes since v1: - Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work to an online housekeeping CPU only when executing on an isolated core, preserving WORK_CPU_UNBOUND and local cache locality for housekeeping cores - Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline CPU fallback path. - Elaborate in the commit log on the triggers for asynchronous queue dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and plugging) and the SCHED_FIFO worker starvation hazard on isolated cores - Link to v1: https://lore.kernel.org/lkml/20261012012905.267920-1-atomlin@atomlin.com/ --- block/blk-mq.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index d9ab295be4d9..4657a3287f3a 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2252,6 +2252,25 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx) return hctx->next_cpu >= nr_cpu_ids; } +static inline int blk_mq_hctx_unbound_cpu(void) +{ +#ifdef CONFIG_CPU_ISOLATION + if (static_branch_unlikely(&housekeeping_overridden)) { + int cpu; + + guard(preempt)(); + if (!cpu_is_isolated(raw_smp_processor_id())) + return WORK_CPU_UNBOUND; + + cpu = cpumask_any_and_distribute(housekeeping_cpumask(HK_TYPE_DOMAIN), + cpu_online_mask); + if (cpu < nr_cpu_ids) + return cpu; + } +#endif + return WORK_CPU_UNBOUND; +} + /* * It'd be great if the workqueue API had a way to pass * in a mask and had some smarts for more clever placement. @@ -2265,7 +2284,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx) /* Switch to unbound if no allowable CPUs in this hctx */ if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx)) - return WORK_CPU_UNBOUND; + return blk_mq_hctx_unbound_cpu(); if (--hctx->next_cpu_batch <= 0) { select_cpu: @@ -2292,7 +2311,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx) */ hctx->next_cpu = next_cpu; hctx->next_cpu_batch = 1; - return WORK_CPU_UNBOUND; + return blk_mq_hctx_unbound_cpu(); } hctx->next_cpu = next_cpu; -- 2.55.0