A worker that is the last one to go to sleep while work is pending asks for another worker. io_wq_dec_running() bumps ->nr_running on behalf of the new worker and queues create_worker_cb() as task_work for the owner of the io-wq. Until that has run, everybody else takes it that a worker is running which is going to look after the work list: io_wq_enqueue() doesn't create one, and other workers that go to sleep don't find themselves to be the last. If the worker that made the request exits, io_worker_exit() cancels the task_work, and io_worker_cancel_cb() drops ->nr_running without a look at the work list: worker A worker B io_wq_enqueue() blocks, work is pending nr_running++ queues create_worker_cb() wakes, handles the work idle timeout, decides to exit picks up work 1 queues work 2 B is on the free list still: wakes B work 1 blocks nr_running 2 -> 1 io_worker_exit() io_worker_cancel_cb() nr_running 1 -> 0 Work 2 has to wait for work 1, although there is room for more workers. It takes an owner that doesn't get around to run its task_work for the time of the idle timeout, less if the worker exits because its affinity was changed or the io-wq is marked exit-on-idle. Let the worker stay around until its request has been dealt with. Only the worker itself sets ->create_state, so it can't be set behind its back once it has decided to leave. Fixes: 1d5f5ea7cb7d ("io-wq: remove worker to owner tw dependency") Assisted-by: LLM Signed-off-by: Josef Bacik --- io_uring/io-wq.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 89b598b724c4..31c7d6fc426b 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -765,9 +765,13 @@ static int io_wq_worker(void *data) * idle-exit, drop the worker as well. This is used to avoid * keeping io-wq workers around for tasks that no longer have * any active io_uring instances. + * Not as long as a request of ours for another worker is + * pending, though. It is accounted as a running worker, which + * others rely on, and io_worker_exit() would cancel it. */ - if ((last_timeout && (exit_mask || acct->nr_workers > 1)) || - test_bit(IO_WQ_BIT_EXIT_ON_IDLE, &wq->state)) { + if (((last_timeout && (exit_mask || acct->nr_workers > 1)) || + test_bit(IO_WQ_BIT_EXIT_ON_IDLE, &wq->state)) && + !test_bit(0, &worker->create_state)) { acct->nr_workers--; if (test_and_clear_bit(IO_WORKER_F_FREE, &worker->flags)) hlist_nulls_del_init_rcu(&worker->nulls_node); -- 2.55.0