Since commit 9d3684c24a52 ("veth: create by default nr_possible_cpus queues"), each veth device is allocated with one rx and one tx queue per possible CPU, and veth_newlink() reduces the real number of queues to one unless the user asked for a number. Both devices are already registered at that point, so netif_set_real_num_tx_queues() and netif_set_real_num_rx_queues() remove the sysfs objects of every extra queue that register_netdevice() has just created, each one emitting an add and a remove uevent, and the tx side also waits for an expedited RCU grace period. This happens with RTNL held, twice per pair, and grows with the number of possible CPUs: with 240 of them a veth pair emits about 1900 uevents, and udev takes minutes to work through those of a few hundred pairs. Set the number of queues before registering each device. For a device that is not registered yet this only updates the counters, so the extra queues never appear in sysfs, and the error paths that undid a registration are no longer needed. Creating and then deleting 500 veth pairs, one ip -batch each, in a 16 vCPU guest, median of 3 runs, per pair with udev stopped, and how long udev stays busy after the deletion with it running: before after 16 possible CPUs create 902 us 295 us delete 6180 us 6147 us udev busy > 300 s 4.5 s uevents 126 6 240 possible CPUs create 9354 us 505 us delete 7343 us 6916 us udev busy > 300 s 6.2 s uevents 1918 6 Signed-off-by: Qiliang Yuan Tested-by: Wang Zhan Reviewed-by: Wang Zhan --- V1 -> V2: - Rebase onto net-next, where the peer's queues are set from its own attributes (tbp), and keep setting them before registering the peer - Retest in another guest with udev stopped, where deletion is not faster, and add how long udev stays busy afterwards, as deletion times with udev running mostly depend on its backlog - Keep Wang Zhan's tags, as the code only changes by the rebase v1: https://lore.kernel.org/r/20261016-bug-veth-init-queues-before-register-v1-1-f48eb2bb25a0@gmail.com --- drivers/net/veth.c | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index a61842b2585bf..0539b3661793d 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -1875,6 +1875,10 @@ static int veth_newlink(struct net_device *dev, netif_inherit_tso_max(peer, dev); + err = veth_init_queues(peer, tbp); + if (err) + goto err_register_peer; + err = register_netdevice(peer); if (err < 0) goto err_register_peer; @@ -1904,6 +1908,10 @@ static int veth_newlink(struct net_device *dev, else snprintf(dev->name, IFNAMSIZ, DRV_NAME "%%d"); + err = veth_init_queues(dev, tb); + if (err) + goto err_register_dev; + err = register_netdevice(dev); if (err < 0) goto err_register_dev; @@ -1917,16 +1925,10 @@ static int veth_newlink(struct net_device *dev, priv = netdev_priv(dev); rcu_assign_pointer(priv->peer, peer); netdev_hold(peer, &priv->peer_tracker, GFP_KERNEL); - err = veth_init_queues(dev, tb); - if (err) - goto err_queues; priv = netdev_priv(peer); rcu_assign_pointer(priv->peer, dev); netdev_hold(dev, &priv->peer_tracker, GFP_KERNEL); - err = veth_init_queues(peer, tbp); - if (err) - goto err_peer_queues; veth_disable_gro(dev); /* update XDP supported features */ @@ -1935,12 +1937,6 @@ static int veth_newlink(struct net_device *dev, return 0; -err_peer_queues: - netdev_put(dev, &priv->peer_tracker); - priv = netdev_priv(dev); -err_queues: - netdev_put(peer, &priv->peer_tracker); - unregister_netdevice(dev); err_register_dev: /* nothing to do */ err_configure_peer: --- base-commit: fa5196392691051963726787944db872420dfa75 change-id: 20261016-bug-veth-init-queues-before-register-eebe763a01a7 Best regards, -- Qiliang Yuan