From: Ionut Nechita Commit 679b1874eba7 ("block: fix ordering between checking QUEUE_FLAG_QUIESCED request adding") introduced queue_lock acquisition in blk_mq_run_hw_queue() to synchronize QUEUE_FLAG_QUIESCED checks. On RT kernels (CONFIG_PREEMPT_RT), regular spinlocks are converted to rt_mutex (sleeping locks). When multiple MSI-X IRQ threads process I/O completions concurrently, they contend on queue_lock in the hot path, causing all IRQ threads to enter D (uninterruptible sleep) state. This serializes interrupt processing completely. Test case (MegaRAID 12GSAS with 8 MSI-X vectors on RT kernel): - Good (v6.6.52-rt): 640 MB/s sequential read - Bad (v6.6.64-rt): 153 MB/s sequential read (-76% regression) - 6-8 out of 8 MSI-X IRQ threads stuck in D-state waiting on queue_lock The original commit message mentioned memory barriers as an alternative approach. Use full memory barriers (smp_mb) instead of queue_lock to provide the same ordering guarantees without sleeping in RT kernel. Memory barriers ensure proper synchronization: - CPU0 either sees QUEUE_FLAG_QUIESCED cleared, OR - CPU1 sees dispatch list/sw queue bitmap updates This maintains correctness while avoiding lock contention that causes RT kernel IRQ threads to sleep in the I/O completion path. Fixes: 679b1874eba7 ("block: fix ordering between checking QUEUE_FLAG_QUIESCED request adding") Cc: stable@vger.kernel.org Signed-off-by: Ionut Nechita --- block/blk-mq.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 5da948b07058..5fb8da4958d0 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2292,22 +2292,19 @@ void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async) might_sleep_if(!async && hctx->flags & BLK_MQ_F_BLOCKING); + /* + * First lockless check to avoid unnecessary overhead. + * Memory barrier below synchronizes with blk_mq_unquiesce_queue(). + */ need_run = blk_mq_hw_queue_need_run(hctx); if (!need_run) { - unsigned long flags; - - /* - * Synchronize with blk_mq_unquiesce_queue(), because we check - * if hw queue is quiesced locklessly above, we need the use - * ->queue_lock to make sure we see the up-to-date status to - * not miss rerunning the hw queue. - */ - spin_lock_irqsave(&hctx->queue->queue_lock, flags); + /* Synchronize with blk_mq_unquiesce_queue() */ + smp_mb(); need_run = blk_mq_hw_queue_need_run(hctx); - spin_unlock_irqrestore(&hctx->queue->queue_lock, flags); - if (!need_run) return; + /* Ensure dispatch list/sw queue updates visible before execution */ + smp_mb(); } if (async || !cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask)) { -- 2.52.0 From: Ionut Nechita Commit 58bf93580fec ("blk-mq: move cpuhp callback registering out of q->sysfs_lock") introduced a global mutex blk_mq_cpuhp_lock to avoid lockdep warnings between sysfs_lock and CPU hotplug lock. On RT kernels (CONFIG_PREEMPT_RT), regular mutexes are converted to rt_mutex (sleeping locks). When block layer operations need to acquire blk_mq_cpuhp_lock, IRQ threads processing I/O completions may sleep, causing additional contention on top of the queue_lock issue from commit 679b1874eba7 ("block: fix ordering between checking QUEUE_FLAG_QUIESCED request adding"). Test case (MegaRAID 12GSAS with 8 MSI-X vectors on RT kernel): - v6.6.68-rt with queue_lock fix: 640 MB/s (queue_lock fixed) - v6.6.69-rt: still exhibits contention due to cpuhp_lock mutex The functions protected by blk_mq_cpuhp_lock only perform fast, non-sleeping operations: - hlist_unhashed() checks - cpuhp_state_add_instance_nocalls() - just hlist manipulation - cpuhp_state_remove_instance_nocalls() - just hlist manipulation - INIT_HLIST_NODE() initialization The _nocalls variants do not invoke state callbacks and only manipulate data structures, making them safe to call under raw_spinlock. Convert blk_mq_cpuhp_lock from mutex to raw_spinlock to prevent it from becoming a sleeping lock in RT kernel. This eliminates the contention bottleneck while maintaining the lockdep fix's original intent. Fixes: 58bf93580fec ("blk-mq: move cpuhp callback registering out of q->sysfs_lock") Cc: stable@vger.kernel.org Signed-off-by: Ionut Nechita --- block/blk-mq.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 5fb8da4958d0..3982e24b1081 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -43,7 +43,7 @@ static DEFINE_PER_CPU(struct llist_head, blk_cpu_done); static DEFINE_PER_CPU(call_single_data_t, blk_cpu_csd); -static DEFINE_MUTEX(blk_mq_cpuhp_lock); +static DEFINE_RAW_SPINLOCK(blk_mq_cpuhp_lock); static void blk_mq_insert_request(struct request *rq, blk_insert_t flags); static void blk_mq_request_bypass_insert(struct request *rq, @@ -3641,9 +3641,9 @@ static void __blk_mq_remove_cpuhp(struct blk_mq_hw_ctx *hctx) static void blk_mq_remove_cpuhp(struct blk_mq_hw_ctx *hctx) { - mutex_lock(&blk_mq_cpuhp_lock); + raw_spin_lock(&blk_mq_cpuhp_lock); __blk_mq_remove_cpuhp(hctx); - mutex_unlock(&blk_mq_cpuhp_lock); + raw_spin_unlock(&blk_mq_cpuhp_lock); } static void __blk_mq_add_cpuhp(struct blk_mq_hw_ctx *hctx) @@ -3683,9 +3683,9 @@ static void blk_mq_remove_hw_queues_cpuhp(struct request_queue *q) list_splice_init(&q->unused_hctx_list, &hctx_list); spin_unlock(&q->unused_hctx_lock); - mutex_lock(&blk_mq_cpuhp_lock); + raw_spin_lock(&blk_mq_cpuhp_lock); __blk_mq_remove_cpuhp_list(&hctx_list); - mutex_unlock(&blk_mq_cpuhp_lock); + raw_spin_unlock(&blk_mq_cpuhp_lock); spin_lock(&q->unused_hctx_lock); list_splice(&hctx_list, &q->unused_hctx_list); @@ -3702,10 +3702,10 @@ static void blk_mq_add_hw_queues_cpuhp(struct request_queue *q) struct blk_mq_hw_ctx *hctx; unsigned long i; - mutex_lock(&blk_mq_cpuhp_lock); + raw_spin_lock(&blk_mq_cpuhp_lock); queue_for_each_hw_ctx(q, hctx, i) __blk_mq_add_cpuhp(hctx); - mutex_unlock(&blk_mq_cpuhp_lock); + raw_spin_unlock(&blk_mq_cpuhp_lock); } /* -- 2.52.0