XDP needs XDP_PACKET_HEADROOM (256 bytes), but locally generated packets only get needed_headroom. XDP on the peer of a veth or netkit device then copies every packet. Generic XDP copies it in netif_skb_check_for_xdp(), and native XDP on veth in veth_convert_skb_to_xdp_buff(). While the peer has an XDP program, set tx_extra_headroom of the device to XDP_PACKET_HEADROOM minus needed_headroom. netdev_update_tx_extra_headroom() recalculates it when an XDP program is attached or detached, also when an upper device such as a bond propagates its program to the peer with netif_xdp_propagate(). It does not recalculate it: - for IFLA_NETKIT_HEADROOM, which cannot change after the netkit device is created, - in dev_xdp_uninstall(), because veth and netkit unregister both ends together, - when bridge or openvswitch change needed_headroom later. openvswitch changes needed_headroom under ovs_mutex, without RTNL. There are three ways to handle a later change of needed_headroom: - Recalculate tx_extra_headroom in netdev_set_rx_headroom(). This races with an XDP attach or detach under RTNL. The last write can leave tx_extra_headroom at 0 while XDP is attached. - Do not subtract needed_headroom. This always reserves needed_headroom bytes more than XDP needs. - Subtract needed_headroom only when tx_extra_headroom is recalculated. This patch uses the last one, because it does not race and reserves only what XDP needs. If needed_headroom shrinks later, XDP on the peer copies packets again until the next recalculation. tx_extra_headroom is used rather than needed_headroom, because raising needed_headroom can regress the performance of other paths. Signed-off-by: Tianyi Gao --- For native XDP on veth, this patch removes one of two copy conditions, the missing headroom. The other condition is a head that is not a page fragment. A follow-up veth series will fix that. v2: - New patch, replaces IFLA_TX_EXTRA_HEADROOM for XDP on the peer (Stephen). --- Documentation/networking/netdevices.rst | 11 +++++++ include/linux/netdevice.h | 1 + net/core/dev.c | 38 +++++++++++++++++++++++++ 3 files changed, 50 insertions(+) diff --git a/Documentation/networking/netdevices.rst b/Documentation/networking/netdevices.rst index f593b242da9c..53ca8439f73b 100644 --- a/Documentation/networking/netdevices.rst +++ b/Documentation/networking/netdevices.rst @@ -454,6 +454,17 @@ devices do not inherit it. It is not part of ``LL_RESERVED_SPACE()``, so a packet that reaches the device with less headroom, such as a forwarded packet, is not reallocated because of it. +The kernel sets ``tx_extra_headroom`` of a device to the sum of these +values: + +* ``XDP_PACKET_HEADROOM`` (256 bytes) minus ``needed_headroom``, when the + peer of a veth or netkit device has an XDP program + +When the peer has an XDP program, locally generated UDP, raw and +AF_PACKET packets and AF_XDP packets in copy mode have enough headroom +for it. ``tx_extra_headroom`` is not recalculated when a bridge or +openvswitch changes ``needed_headroom`` later. + NETDEV_INTERNAL symbol namespace ================================ diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 392e6fc81e34..786b63f21d4f 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -4484,6 +4484,7 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf); int dev_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf); u8 dev_xdp_sb_prog_count(struct net_device *dev); u32 dev_xdp_prog_id(struct net_device *dev, enum bpf_xdp_mode mode); +void netdev_update_tx_extra_headroom(struct net_device *dev); u32 dev_get_min_mp_channel_count(const struct net_device *dev); diff --git a/net/core/dev.c b/net/core/dev.c index 85b7c61015c7..6e053f612d72 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -10666,6 +10666,7 @@ int netif_xdp_propagate(struct net_device *dev, struct netdev_bpf *bpf) if (old_prog) bpf_prog_put(old_prog); + netdev_update_tx_extra_headroom(dev); return 0; } EXPORT_SYMBOL_GPL(netif_xdp_propagate); @@ -10677,6 +10678,41 @@ u32 dev_xdp_prog_id(struct net_device *dev, enum bpf_xdp_mode mode) return prog ? prog->aux->id : 0; } +static void __netdev_update_tx_extra_headroom(struct net_device *dev, + struct net_device *peer) +{ + unsigned int needed_headroom = READ_ONCE(dev->needed_headroom); + unsigned int tx_extra_headroom = 0; + + if (peer && dev_xdp_prog_count(peer) && + needed_headroom < XDP_PACKET_HEADROOM) + tx_extra_headroom = XDP_PACKET_HEADROOM - needed_headroom; + + WRITE_ONCE(dev->tx_extra_headroom, tx_extra_headroom); +} + +/** + * netdev_update_tx_extra_headroom - recalculate tx_extra_headroom + * @dev: the device to check + * + * Recalculate tx_extra_headroom of @dev and its peer. Should be called + * after conditions that influence tx_extra_headroom might have changed. + * The caller must hold the RTNL lock. + */ +void netdev_update_tx_extra_headroom(struct net_device *dev) +{ + struct net_device *peer = NULL; + + rcu_read_lock(); + if (dev->netdev_ops->ndo_get_peer_dev) + peer = dev->netdev_ops->ndo_get_peer_dev(dev); + __netdev_update_tx_extra_headroom(dev, peer); + if (peer) + __netdev_update_tx_extra_headroom(peer, dev); + rcu_read_unlock(); +} +EXPORT_SYMBOL_GPL(netdev_update_tx_extra_headroom); + static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode, bpf_op_t bpf_op, struct netlink_ext_ack *extack, u32 flags, struct bpf_prog *prog) @@ -10887,6 +10923,7 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack if (cur_prog) bpf_prog_put(cur_prog); + netdev_update_tx_extra_headroom(dev); return 0; } @@ -10913,6 +10950,7 @@ static int dev_xdp_detach_link(struct net_device *dev, bpf_op = dev_xdp_bpf_op(dev, mode); WARN_ON(dev_xdp_install(dev, mode, bpf_op, NULL, 0, NULL)); dev_xdp_set_link(dev, mode, NULL); + netdev_update_tx_extra_headroom(dev); return 0; } -- 2.56.0