Since commit cebf123c634a ("io_uring: implement our own schedule timeout handling"), a CQ wait with a timeout sleeps on io_uring's own hrtimer instead of an hrtimer sleeper. On PREEMPT_RT, an hrtimer that isn't marked for hard interrupt expiry expires in the per-CPU ktimers thread, which runs at SCHED_FIFO 1. A real-time task waiting in io_uring_enter() is therefore not woken at its deadline while any other real-time task is runnable on its CPU: the wakeup waits until every one of them has blocked. hrtimer sleepers, which schedule_hrtimeout_range_clock() used to give io_uring, don't have this problem: __hrtimer_setup_sleeper() marks the timer hard when the sleeping task is in a real-time or deadline class. Both of io_uring's callbacks are fine in hard interrupt context: io_cqring_timer_wakeup() sets a flag and calls wake_up_process(), and io_cqring_min_timer_wakeup() only reads ring state, under RCU, before doing the same or re-arming the timer. Apply the sleeper rule to them. Measured on a Jetson Orin Nano running 6.18.39-rt with this change backported: a SCHED_FIFO 40 task waits in io_uring_enter() for 40,000 deadlines 500us apart, while a SCHED_FIFO 20 task runs bursts of up to 1ms on the same CPU. Before, 6,647 wakeups were more than 100us late, the latest by 1,040us, and 2,758 deadlines passed without a wakeup at all. After, none were more than 100us late and the latest was 18us. This mirrors epoll: epoll_pwait2() waits on an hrtimer sleeper, so its timeout is already hard for real-time tasks, and on the same setup it was never more than 17us late. Fixes: cebf123c634a ("io_uring: implement our own schedule timeout handling") Cc: stable@vger.kernel.org Assisted-by: LLM Signed-off-by: Austin Schuh --- Notes: Based on for-7.4/io_uring (89f193dc3ef4). Built for arm64 with PREEMPT_RT and with PREEMPT, W=1, no new warnings. The numbers above are from the same change on 6.18.39-rt (in 6.18 the code is still in io_uring.c). I haven't booted for-7.4 on the board. The test program is a small C file, which I can post if it's useful. io_uring/wait.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/io_uring/wait.c b/io_uring/wait.c index b7a0292767c7..ef4b89d3c66e 100644 --- a/io_uring/wait.c +++ b/io_uring/wait.c @@ -3,6 +3,7 @@ * Waiting for completion events */ #include +#include #include #include #include @@ -120,23 +121,39 @@ static enum hrtimer_restart io_cqring_min_timer_wakeup(struct hrtimer *timer) return io_cqring_timer_wakeup(timer); } +/* + * On PREEMPT_RT, hrtimers not marked for hard interrupt expiry expire in the + * ktimers thread, at the lowest real-time priority, so a real-time waiter + * isn't woken until every runnable real-time task on its CPU has blocked. + * Like hrtimer sleepers (see __hrtimer_setup_sleeper()), expire the timer in + * hard interrupt context if the waiter is a real-time task. Both callbacks + * only look at the ring and wake the task, which is safe there. + */ +static enum hrtimer_mode io_cqring_timer_mode(void) +{ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && rt_or_dl_task_policy(current)) + return HRTIMER_MODE_ABS_HARD; + return HRTIMER_MODE_ABS; +} + static int io_cqring_schedule_timeout(struct io_wait_queue *iowq, clockid_t clock_id, ktime_t start_time) { + enum hrtimer_mode mode = io_cqring_timer_mode(); ktime_t timeout; if (iowq->min_timeout) { timeout = ktime_add_ns(iowq->min_timeout, start_time); hrtimer_setup_on_stack(&iowq->t, io_cqring_min_timer_wakeup, clock_id, - HRTIMER_MODE_ABS); + mode); } else { timeout = iowq->timeout; hrtimer_setup_on_stack(&iowq->t, io_cqring_timer_wakeup, clock_id, - HRTIMER_MODE_ABS); + mode); } hrtimer_set_expires_range_ns(&iowq->t, timeout, 0); - hrtimer_start_expires(&iowq->t, HRTIMER_MODE_ABS); + hrtimer_start_expires(&iowq->t, mode); if (!READ_ONCE(iowq->hit_timeout)) schedule(); base-commit: 89f193dc3ef462059387e9691bf6af4b1808379a -- 2.47.3