From: David Keogh Programs attached at TC egress commonly push additional headers onto a packet, for example to encapsulate it. When the packet does not carry enough headroom the BPF helpers reallocate the head, which costs an allocation and a copy per packet. Locally generated UDP, raw and AF_PACKET traffic is allocated with exactly LL_RESERVED_SPACE() of headroom, which only covers the link-layer header and the driver's needed_headroom. Add a tx_extra_headroom field to struct net_device for that purpose, together with netdev_get_alloc_headroom() and netdev_get_alloc_headroom_hlen(). These add the extra reservation to the header length before the HH_DATA_MOD alignment, so the alignment of the link-layer header does not change. netdev_get_alloc_needed_headroom() returns needed_headroom plus the extra reservation, for callers that do their own alignment. The new functions are intended only for sites that allocate new packets. LL_RESERVED_SPACE() keeps its current meaning and remains the threshold used when checking whether an existing packet must be expanded, so the extra reservation never forces a reallocation on its own: packets that already have enough room for the headers being pushed, such as TCP segments built with MAX_TCP_HEADER, are left alone. Keep the extra reservation separate from needed_headroom so that driver updates of their own requirement do not clobber it. Place the field in the padding that follows needed_headroom, which keeps it in the TX read-mostly cacheline group. The field is not set anywhere yet. Later patches set it from XDP on a veth peer, from TC egress programs and from a veth attribute. Also remove LL_RESERVED_SPACE_EXTRA(). It has had no user since commit 6d5431555de0 ("caif: remove CAIF NETWORK LAYER"). It could add the extra headroom, but it reads dev->hard_header_len itself. AF_PACKET must use the value that it read once, see commit 03390aa32e66 ("packet: use consistent hard_header_len in non-ring send paths"). netdev_get_alloc_headroom_hlen() takes the header length as an argument instead. Signed-off-by: David Keogh Co-developed-by: Tianyi Gao Signed-off-by: Tianyi Gao --- v2: - Turn LL_ALLOC_SPACE() and LL_ALLOC_SPACE_EX() into the static inline functions netdev_get_alloc_headroom() and netdev_get_alloc_headroom_hlen(), and rename netdev_tx_alloc_headroom() to netdev_get_alloc_needed_headroom() (Willem). - Move the netdevices.rst section here from v1 3/8, without the IFLA part. --- .../networking/net_cachelines/net_device.rst | 1 + Documentation/networking/netdevices.rst | 13 +++++++ include/linux/netdevice.h | 34 +++++++++++++++++-- net/core/dev.c | 1 + 4 files changed, 46 insertions(+), 3 deletions(-) diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Documentation/networking/net_cachelines/net_device.rst index 7620d3fcb8dc..4341e9e443ba 100644 --- a/Documentation/networking/net_cachelines/net_device.rst +++ b/Documentation/networking/net_cachelines/net_device.rst @@ -35,6 +35,7 @@ unsigned_short gflags unsigned_short hard_header_len read_mostly read_mostly ip6_xmit(tx);gro_list_prepare(rx) unsigned_int mtu read_mostly ip_finish_output2 unsigned_short needed_headroom read_mostly LL_RESERVED_SPACE/ip_finish_output2 +unsigned_short tx_extra_headroom read_mostly netdev_get_alloc_headroom/__ip_append_data unsigned_short needed_tailroom netdev_features_t features read_mostly read_mostly HARD_TX_LOCK,netif_skb_features,sk_setup_caps(tx);netif_elide_gro(rx) netdev_features_t hw_features diff --git a/Documentation/networking/netdevices.rst b/Documentation/networking/netdevices.rst index 6a87f09e7015..f593b242da9c 100644 --- a/Documentation/networking/netdevices.rst +++ b/Documentation/networking/netdevices.rst @@ -441,6 +441,19 @@ from a notifier. Drivers **must not** generate nested notifications of the ops-locked types. +TX headroom reservation +======================= + +``tx_extra_headroom`` is extra headroom for locally generated packets. The +stack adds it when it allocates such a packet for the device, so that +headers pushed later, for example at TC egress, fit without a +reallocation. It is at most 512 bytes. + +Unlike ``needed_headroom``, it is not a driver requirement and stacked +devices do not inherit it. It is not part of ``LL_RESERVED_SPACE()``, so a +packet that reaches the device with less headroom, such as a forwarded +packet, is not reallocated because of it. + NETDEV_INTERNAL symbol namespace ================================ diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 0325c6f2a7d1..392e6fc81e34 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -311,9 +311,6 @@ struct hh_cache { & ~(HH_DATA_MOD - 1)) + HH_DATA_MOD) #define LL_RESERVED_SPACE(dev) \ LL_RESERVED_SPACE_EX(dev, (dev)->hard_header_len) -#define LL_RESERVED_SPACE_EXTRA(dev,extra) \ - ((((dev)->hard_header_len + READ_ONCE((dev)->needed_headroom) + (extra)) \ - & ~(HH_DATA_MOD - 1)) + HH_DATA_MOD) struct header_ops { int (*create) (struct sk_buff *skb, struct net_device *dev, @@ -1934,6 +1931,8 @@ enum netdev_reg_state { * * @needed_headroom: Extra headroom the hardware may need, but not in all * cases can this be guaranteed + * @tx_extra_headroom: Extra headroom for locally generated packets, for + * headers pushed after the allocation * @needed_tailroom: Extra tailroom the hardware may need, but not in all * cases can this be guaranteed. Some cases also use * LL_MAX_HEADER instead to allocate the skb @@ -2196,6 +2195,8 @@ struct net_device { */ unsigned int mtu; unsigned short needed_headroom; +#define NETDEV_TX_EXTRA_HEADROOM_MAX 512 + unsigned short tx_extra_headroom; struct netdev_tc_txq tc_to_txq[TC_MAX_QUEUE]; #ifdef CONFIG_XPS struct xps_dev_maps __rcu *xps_maps[XPS_MAPS_MAX]; @@ -2778,6 +2779,33 @@ struct netdev_queue *netdev_core_pick_tx(struct net_device *dev, struct sk_buff *skb, struct net_device *sb_dev); +/* Headroom to reserve when allocating a locally generated packet for dev. + * Code that checks the headroom of an existing packet keeps using + * LL_RESERVED_SPACE(), so tx_extra_headroom never forces a reallocation. + */ +static inline unsigned int +netdev_get_alloc_headroom_hlen(const struct net_device *dev, unsigned int hlen) +{ + return LL_RESERVED_SPACE_EX(dev, + hlen + READ_ONCE(dev->tx_extra_headroom)); +} + +static inline unsigned int +netdev_get_alloc_headroom(const struct net_device *dev) +{ + return netdev_get_alloc_headroom_hlen(dev, dev->hard_header_len); +} + +/* Same as netdev_get_alloc_headroom() without hard_header_len and alignment, + * for callers that size the headroom themselves. + */ +static inline unsigned int +netdev_get_alloc_needed_headroom(const struct net_device *dev) +{ + return READ_ONCE(dev->needed_headroom) + + READ_ONCE(dev->tx_extra_headroom); +} + /* returns the headroom that the master device needs to take in account * when forwarding to this dev */ diff --git a/net/core/dev.c b/net/core/dev.c index d88af07aea29..85b7c61015c7 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -13692,6 +13692,7 @@ static void __init net_dev_struct_check(void) CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, num_tc); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, mtu); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, needed_headroom); + CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, tx_extra_headroom); CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, tc_to_txq); #ifdef CONFIG_XPS CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_tx, xps_maps); -- 2.56.0