ep_poll_callback() takes ep->lock and only then checks whether the event matches epi->event.events. But epi->event.events is not protected by ep->lock, ep_modify() and the EPOLLONESHOT disarm update it under ep->mtx. So a wakeup for an event nobody asked for takes and releases ep->lock for nothing. E.g. a UDP socket registered with EPOLLIN only gets an EPOLLOUT wakeup for every freed tx skb. Check the mask first and return early if the event is not wanted. The comment in ep_modify() counts on ep_poll_callback() taking ep->lock, so do smp_mb() and read the mask again before dropping the event. Wanted events take the lock as before. Benchmark, based on stress-epoll.c [1]: N threads share one epoll, each thread registers its own eventfd with EPOLLPRI, which eventfd never raises, and loops on write() + read(). Both call ep_poll_callback(), with EPOLLIN and EPOLLOUT. 12 vCPU VM, ops/ms: threads before after 1 396 402 4 1514 1604 12 1264 4817 The original stress-epoll.c hits this path too: the poller read()s the eventfds, and eventfd_read() wakes with EPOLLOUT while they are registered with EPOLLIN. 32 vCPU VM, events/ms, average of 6 runs: threads before after 8 1293 1315 16 1407 1422 32 1304 1431 Real case, udp rr over veth with 1000 connected sockets registered with EPOLLIN: _raw_spin_lock_irqsave drops by ~1%. [1] https://github.com/rouming/test-tools/blob/master/stress-epoll.c Signed-off-by: Jiayuan Chen --- fs/eventpoll.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index e0c4bf88a838..29e034e3a295 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1495,8 +1495,22 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v struct eventpoll *ep = epi->ep; __poll_t pollflags = key_to_poll(key); unsigned long flags; + __poll_t events; int ewake = 0; + /* + * epi->event.events is updated under ep->mtx, not ep->lock, so an + * event nobody asked for can be dropped without taking ep->lock. + */ + events = READ_ONCE(epi->event.events); + if (pollflags && !(pollflags & (events | POLLFREE))) { + /* Pairs with smp_mb() in ep_modify(). */ + smp_mb(); + events = READ_ONCE(epi->event.events); + if (!(pollflags & events)) + return !(events & EPOLLEXCLUSIVE); + } + spin_lock_irqsave(&ep->lock, flags); ep_set_busy_poll_napi_id(epi); -- 2.43.0