With MAX_SKB_FRAGS > TX_MAX_FRAGS, the xmit path linearizes skbs with more frags than a packet can have BDs. For GSO skbs, linearizing needs a large allocation, can produce a head which doesn't fit in a BD with BIG TCP, and always fails for unreadable skbs. Clear the GSO features of such skbs in ndo_features_check instead, so that the stack segments them. Non-GSO skbs are still linearized. Suggested-by: Eric Dumazet Link: https://lore.kernel.org/all/CANn89i+ps_CozfOyRnKav0_LUmSekJ9ExB5JkDbTAVf_ss_98g@mail.gmail.com/ Signed-off-by: Stanislav Fomichev Reviewed-by: Mina Almasry Reviewed-by: Dragos Tatulea Reviewed-by: Breno Leitao Reviewed-by: Michael Chan --- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 33c4cacc8ffe..f20b61180ab2 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -14320,6 +14320,11 @@ static netdev_features_t bnxt_features_check(struct sk_buff *skb, skb_shinfo(skb)->gso_size)) features &= ~NETIF_F_GSO_MASK; +#if (MAX_SKB_FRAGS > TX_MAX_FRAGS) + if (skb_shinfo(skb)->nr_frags > TX_MAX_FRAGS) + features &= ~NETIF_F_GSO_MASK; +#endif + switch (vlan_get_protocol(skb)) { case htons(ETH_P_IP): if (!skb->encapsulation) -- 2.53.0-Meta Devmem TX charges each frag the net_iov size, PAGE_SIZE today: zerocopy_fill_skb_from_devmem() adds PAGE_SIZE to skb->truesize per frag and __zerocopy_sg_from_iter() charges it to sk_wmem_queued. TCP uncharges in bytes as acked data is trimmed from the skb (tcp_trim_head()) and uncharges the rest when the skb is freed. A frag shorter than a page leaves the difference charged until then, which is small with PAGE_SIZE net_iovs. TX net_iovs are about to grow up to 2MB, while frags stay bounded by the skb and many frags share a net_iov. Charging the net_iov size would charge up to 2MB per frag: a 64KB skb with two frags would charge 4MB and fill the default 4MB send buffer on its own. Charge the frag length instead, which also matches what TCP uncharges. Signed-off-by: Stanislav Fomichev Reviewed-by: Mina Almasry Reviewed-by: Dragos Tatulea Reviewed-by: Breno Leitao --- net/core/datagram.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/datagram.c b/net/core/datagram.c index ecc30a6f8302..489d92fe6dae 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -737,7 +737,7 @@ zerocopy_fill_skb_from_devmem(struct sk_buff *skb, struct iov_iter *from, netmem_inc_ref(net_iov_to_netmem(niov)); skb_add_rx_frag_netmem(skb, i, net_iov_to_netmem(niov), off, - size, PAGE_SIZE); + size, size); iov_iter_advance(from, size); length -= size; i++; -- 2.53.0-Meta TX bindings use PAGE_SIZE net_iovs, and the TX lookup hard-codes it. Derive the index and offset from binding->niov_shift instead, so TX bindings can use larger net_iovs. No functional change, TX bindings are still created with PAGE_SHIFT. Signed-off-by: Stanislav Fomichev Reviewed-by: Mina Almasry Reviewed-by: Dragos Tatulea --- net/core/devmem.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/net/core/devmem.c b/net/core/devmem.c index 5c7d9640e9d1..c6868f39261a 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -581,13 +581,15 @@ struct net_iov * net_devmem_get_niov_at(struct net_devmem_dmabuf_binding *binding, size_t virt_addr, size_t *off, size_t *size) { + size_t niov_size = 1UL << binding->niov_shift; + if (virt_addr >= binding->dmabuf->size) return NULL; - *off = virt_addr % PAGE_SIZE; - *size = PAGE_SIZE - *off; + *off = virt_addr & (niov_size - 1); + *size = niov_size - *off; - return binding->tx_vec[virt_addr / PAGE_SIZE]; + return binding->tx_vec[virt_addr >> binding->niov_shift]; } /*** "Dmabuf devmem memory provider" ***/ -- 2.53.0-Meta PAGE_SIZE TX net_iovs cap devmem skbs at MAX_SKB_FRAGS pages (~68KB), which makes BIG TCP ineffective. TX net_iovs only need to be DMA contiguous. Use the largest power of 2, from PAGE_SIZE up to 2MB, which aligns every DMA segment of the dmabuf: - 2MB hugepage-backed udmabufs get 2MB net_iovs - no UAPI change, TX addresses are dmabuf offsets - skbs keep their size, only the frags get larger The net_iov size doesn't depend on the device. A frag is never larger than its skb, and the skb is never larger than the device's gso_max_size. TCP sizes skbs to fit, and validate_xmit_skb() segments bigger ones. gso_max_size is 64KB unless BIG TCP is enabled. With BIG TCP, page frags can already be 64KB or larger, for example with MSG_ZEROCOPY from huge pages or with 64KB pages. So drivers that enable BIG TCP already have to handle them. bnxt doesn't handle them yet. It has a 16-bit BD length and needs a separate fix. Signed-off-by: Stanislav Fomichev Reviewed-by: Mina Almasry Reviewed-by: Dragos Tatulea --- net/core/devmem.c | 28 ++++++++++++++++++++++++++-- net/core/netdev-genl.c | 2 +- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/net/core/devmem.c b/net/core/devmem.c index c6868f39261a..713528048897 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -303,6 +303,20 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, return err; } +static bool net_devmem_sgt_aligned(struct sg_table *sgt, size_t size) +{ + struct scatterlist *sg; + unsigned int i; + + for_each_sgtable_dma_sg(sgt, sg, i) { + if (!IS_ALIGNED(sg_dma_address(sg), size) || + !IS_ALIGNED(sg_dma_len(sg), size)) + return false; + } + return true; +} + +/* niov_shift == 0 picks the net_iov size from the dmabuf, TX only */ struct net_devmem_dmabuf_binding * net_devmem_bind_dmabuf(struct net_device *dev, void *vdev, struct device *dma_dev, @@ -313,11 +327,11 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev, struct netlink_ext_ack *extack) { struct net_devmem_dmabuf_binding *binding; - size_t niov_size = 1UL << niov_shift; static u32 id_alloc_next; struct scatterlist *sg; struct dma_buf *dmabuf; unsigned int sg_idx; + size_t niov_size; size_t niov_idx; size_t i; long err; @@ -342,7 +356,6 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev, binding->vdev = vdev; write_pnet(&binding->net, sock_net(nl_sk)); binding->uid = sk_uid(nl_sk); - binding->niov_shift = niov_shift; xa_init_flags(&binding->bound_rxqs, XA_FLAGS_ALLOC); err = percpu_ref_init(&binding->ref, @@ -387,6 +400,17 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev, goto err_unmap; } + if (!niov_shift) { + niov_size = SZ_2M; + while (niov_size > PAGE_SIZE && + (!IS_ALIGNED(dmabuf->size, niov_size) || + !net_devmem_sgt_aligned(binding->sgt, niov_size))) + niov_size /= 2; + niov_shift = ilog2(niov_size); + } + binding->niov_shift = niov_shift; + niov_size = 1UL << niov_shift; + if (!IS_ALIGNED(dmabuf->size, niov_size)) { err = -EINVAL; NL_SET_ERR_MSG_FMT(extack, diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index 9c5bf17b04e5..d2ccb1090496 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -1261,7 +1261,7 @@ int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct genl_info *info) binding = net_devmem_bind_dmabuf(bind_dev, bind_dev != netdev ? netdev : NULL, dma_dev, DMA_TO_DEVICE, dmabuf_fd, - PAGE_SHIFT, priv, NETLINK_CB(skb).sk, + 0, priv, NETLINK_CB(skb).sk, info->extack); if (IS_ERR(binding)) { err = PTR_ERR(binding); -- 2.53.0-Meta TX net_iovs are now sized from the dmabuf layout, but the TX tests use a regular memfd, which usually keeps them small. Add -H to ncdevmem to back the udmabuf with 2MB hugepages and a TX test which sends a 3MB buffer from it, so that frags end at and continue past the 2MB net_iov boundary. The payload has a 7 byte period to catch offset errors. Signed-off-by: Stanislav Fomichev Reviewed-by: Mina Almasry Reviewed-by: Dragos Tatulea --- .../selftests/drivers/net/hw/devmem.py | 10 ++++- .../selftests/drivers/net/hw/devmem_lib.py | 43 +++++++++++++++++-- .../selftests/drivers/net/hw/ncdevmem.c | 8 +++- .../selftests/drivers/net/hw/nk_devmem.py | 11 ++++- 4 files changed, 63 insertions(+), 9 deletions(-) diff --git a/tools/testing/selftests/drivers/net/hw/devmem.py b/tools/testing/selftests/drivers/net/hw/devmem.py index 82c11ffc4add..aa7141fb36f7 100755 --- a/tools/testing/selftests/drivers/net/hw/devmem.py +++ b/tools/testing/selftests/drivers/net/hw/devmem.py @@ -3,7 +3,7 @@ from os import path from devmem_lib import (setup_test, run_rx, run_tx, run_tx_chunks, run_rx_hds, - run_rx_large_niov) + run_rx_large_niov, run_tx_large_niov) from lib.py import ksft_run, ksft_exit, ksft_disruptive from lib.py import NetDrvEpEnv @@ -36,12 +36,18 @@ from lib.py import NetDrvEpEnv run_rx_large_niov(cfg) +@ksft_disruptive +def check_tx_large_niov(cfg) -> None: + """Run the devmem TX test from a hugepage-backed dmabuf.""" + run_tx_large_niov(cfg) + + def main() -> None: """Run the devmem test cases.""" with NetDrvEpEnv(__file__) as cfg: setup_test(cfg, path.abspath(path.dirname(__file__) + "/ncdevmem")) ksft_run([check_rx, check_tx, check_tx_chunks, check_rx_hds, - check_rx_large_niov], + check_rx_large_niov, check_tx_large_niov], args=(cfg,)) ksft_exit() diff --git a/tools/testing/selftests/drivers/net/hw/devmem_lib.py b/tools/testing/selftests/drivers/net/hw/devmem_lib.py index 71b916c16196..8058fd1ddd50 100644 --- a/tools/testing/selftests/drivers/net/hw/devmem_lib.py +++ b/tools/testing/selftests/drivers/net/hw/devmem_lib.py @@ -2,6 +2,7 @@ # pylint: disable=invalid-name,too-many-arguments """Shared helpers for devmem TCP selftests.""" +import hashlib import os import re @@ -42,15 +43,17 @@ NR_HUGEPAGES_FILE = "/proc/sys/vm/nr_hugepages" defer(_restore_nr_hugepages, nr_hugepages) -def _probe_devmem(cfg, rx_page_size): +def _probe_devmem(cfg, rx_page_size, hugepages=False): """Return True if ncdevmem can bind cfg.ifname at @rx_page_size.""" probe_command = f"{cfg.bin_local} -f {cfg.ifname}" if rx_page_size != RX_PAGE_SIZE_DEFAULT: probe_command += f" -b {rx_page_size}" + if hugepages: + probe_command += " -H" return cmd(probe_command, fail=False, shell=True).ret == 0 -def require_devmem(cfg, rx_page_size=RX_PAGE_SIZE_DEFAULT): +def require_devmem(cfg, rx_page_size=RX_PAGE_SIZE_DEFAULT, hugepages=False): """Probe ncdevmem on cfg.ifname and SKIP the test if devmem isn't supported.""" if rx_page_size not in PROBE_RX_PAGE_SIZES: raise RuntimeError( @@ -63,10 +66,15 @@ NR_HUGEPAGES_FILE = "/proc/sys/vm/nr_hugepages" # ncdevmem's queue range and cause the probe to fail. cfg.devmem_supported = {size: _probe_devmem(cfg, size) for size in PROBE_RX_PAGE_SIZES} + cfg.devmem_hugepages = _probe_devmem(cfg, RX_PAGE_SIZE_DEFAULT, + hugepages=True) if not cfg.devmem_supported[RX_PAGE_SIZE_DEFAULT]: raise KsftSkipEx("Test requires devmem support") + if hugepages and not cfg.devmem_hugepages: + raise KsftSkipEx("Test requires a 2MB hugepage-backed dmabuf") + if rx_page_size != RX_PAGE_SIZE_DEFAULT: page_size = os.sysconf("SC_PAGE_SIZE") if not _is_aligned(rx_page_size, page_size): @@ -169,7 +177,7 @@ def ncdevmem_rx(cfg, port, verify=True, fail_on_linear=False, flow_steer=False, return " ".join(parts) -def ncdevmem_tx(cfg, port, chunk_size=0): +def ncdevmem_tx(cfg, port, chunk_size=0, hugepages=False): """Build the ncdevmem TX send command.""" if hasattr(cfg, 'netns'): ifname = cfg.nk_guest_ifname @@ -182,6 +190,8 @@ def ncdevmem_rx(cfg, port, verify=True, fail_on_linear=False, flow_steer=False, if chunk_size: extras.append(f"-z {chunk_size}") + if hugepages: + extras.append("-H") parts = [cfg.bin_local, f"-f {ifname}", f"-s {addr}", f"-p {port}", *extras] @@ -269,6 +279,33 @@ def ncdevmem_rx(cfg, port, verify=True, fail_on_linear=False, flow_steer=False, ksft_eq(socat.stdout.strip(), "hello\nworld") +def run_tx_large_niov(cfg): + """Run the devmem TX test from a 2MB hugepage-backed dmabuf. + + The kernel sizes TX niovs from the dmabuf layout. Send a single buffer + larger than a hugepage so that frags end at and continue past the niov + boundary. + """ + require_devmem(cfg, hugepages=True) + _reserve_hugepages() + configure_nic(cfg) + netns = getattr(cfg, "netns", None) + port = rand_port() + size = 3 * 1024 * 1024 + pattern = "abcdefg" + expected = (pattern * (size // len(pattern) + 1))[:size] + tx_cmd = ncdevmem_tx(cfg, port, hugepages=True) + # Hash on the remote, bkg() only drains stdout once the command exits. + listen_cmd = f"{socat_listen(cfg, port)} | sha256sum" + + with bkg(listen_cmd, host=cfg.remote, exit_wait=True) as socat: + wait_port_listen(port, host=cfg.remote) + cmd(f"bash -c 'yes {pattern} | tr -d \"\\n\" | head -c {size} | {tx_cmd}'", + ns=netns, shell=True) + ksft_eq(socat.stdout.split()[0], + hashlib.sha256(expected.encode()).hexdigest()) + + def run_rx_large_niov(cfg): """Run the devmem RX test with a large niov (rx-page-size > PAGE_SIZE). diff --git a/tools/testing/selftests/drivers/net/hw/ncdevmem.c b/tools/testing/selftests/drivers/net/hw/ncdevmem.c index 918e3b51f3b8..e29ff27087e7 100644 --- a/tools/testing/selftests/drivers/net/hw/ncdevmem.c +++ b/tools/testing/selftests/drivers/net/hw/ncdevmem.c @@ -104,6 +104,7 @@ static uint32_t tx_dmabuf_id; static int waittime_ms = 500; static bool fail_on_linear; static uint32_t rx_page_size; +static bool use_hugepages; /* System state loaded by current_config_load() */ #define MAX_FLOWS 8 @@ -162,7 +163,7 @@ static struct memory_buffer *udmabuf_alloc(size_t size) } memfd_flags = MFD_ALLOW_SEALING; - if (rx_page_size > getpagesize()) + if (use_hugepages || rx_page_size > getpagesize()) memfd_flags |= MFD_HUGETLB | MFD_HUGE_2MB; ctx->memfd = memfd_create("udmabuf-test", memfd_flags); @@ -1428,11 +1429,14 @@ int main(int argc, char *argv[]) int is_server = 0, opt; int ret, err = 1; - while ((opt = getopt(argc, argv, "Lls:c:p:v:q:t:f:z:nb:")) != -1) { + while ((opt = getopt(argc, argv, "Lls:c:p:v:q:t:f:z:nb:H")) != -1) { switch (opt) { case 'L': fail_on_linear = true; break; + case 'H': + use_hugepages = true; + break; case 'l': is_server = 1; break; diff --git a/tools/testing/selftests/drivers/net/hw/nk_devmem.py b/tools/testing/selftests/drivers/net/hw/nk_devmem.py index 61c6f31f01e5..799030db7c12 100755 --- a/tools/testing/selftests/drivers/net/hw/nk_devmem.py +++ b/tools/testing/selftests/drivers/net/hw/nk_devmem.py @@ -4,7 +4,7 @@ import os from devmem_lib import (setup_test, run_rx, run_tx, run_tx_chunks, run_rx_hds, - run_rx_large_niov) + run_rx_large_niov, run_tx_large_niov) from lib.py import ksft_run, ksft_exit, ksft_disruptive from lib.py import NetDrvContEnv @@ -37,6 +37,12 @@ from lib.py import NetDrvContEnv run_rx_large_niov(cfg) +@ksft_disruptive +def check_nk_tx_large_niov(cfg) -> None: + """Run the devmem TX large-niov test through netkit.""" + run_tx_large_niov(cfg) + + def main() -> None: """Run the netkit devmem test cases.""" with NetDrvContEnv(__file__, rxqueues=2, primary_rx_redirect=True) as cfg: @@ -44,7 +50,8 @@ from lib.py import NetDrvContEnv os.path.join(os.path.dirname(os.path.abspath(__file__)), "ncdevmem")) ksft_run([check_nk_rx, check_nk_tx, check_nk_tx_chunks, - check_nk_rx_hds, check_nk_rx_large_niov], args=(cfg,)) + check_nk_rx_hds, check_nk_rx_large_niov, + check_nk_tx_large_niov], args=(cfg,)) ksft_exit() -- 2.53.0-Meta