From: Maciej Fijalkowski Currently, generic XDP hook uses xdp_rxq_info from netstack Rx queues which do not have its XDP memory model registered. There is a case when XDP program calls bpf_xdp_adjust_tail() BPF helper, which in turn releases underlying memory. This happens when it consumes enough amount of bytes and when XDP buffer has fragments. For this action the memory model knowledge passed to XDP program is crucial so that core can call suitable function for freeing/recycling the page. For netstack queues it defaults to MEM_TYPE_PAGE_SHARED (0) due to lack of mem model registration. The problem we're fixing here is when kernel copied the skb to new buffer backed by system's page_pool and XDP buffer is built around it. Then when bpf_xdp_adjust_tail() calls __xdp_return(), it acts incorrectly due to mem type not being set to MEM_TYPE_PAGE_POOL and causes a page leak. Pull out the existing code from bpf_prog_run_generic_xdp() that init/prepares xdp_buff onto new helper xdp_convert_skb_to_buff() and embed there rxq's mem_type initialization that is assigned to xdp_buff. Make it agnostic to current skb->data position. The mem type is only consulted when the program releases fragments; the head can not be freed from under the skb. All current users pass skbs with fragments through skb_pp_cow_data() first, and when the CoW fails the skb is linearized, so fragments seen here are always page_pool backed. Hence set MEM_TYPE_PAGE_POOL unconditionally and warn under CONFIG_DEBUG_NET if a non-linear skb shows up without pp_recycle. This problem was triggered by syzbot as well as AF_XDP test suite which is about to be integrated to BPF CI. Reported-by: syzbot+ff145014d6b0ce64a173@syzkaller.appspotmail.com Closes: https://lore.kernel.org/netdev/6756c37b.050a0220.a30f1.019a.GAE@google.com/ Fixes: e6d5dbdd20aa ("xdp: add multi-buff support for xdp running in generic mode") Suggested-by: Jakub Kicinski Co-developed-by: Octavian Purdila Signed-off-by: Octavian Purdila # whole analysis, testing, initiating a fix Signed-off-by: Maciej Fijalkowski # commit msg and proposed more robust fix [sst: rebased onto bpf/master; set MEM_TYPE_PAGE_POOL unconditionally, add DEBUG_NET_WARN_ON_ONCE() and kdoc as agreed in v4/v5 review; dropped Tested-by/Reviewed-by as the code changed] Signed-off-by: Slawomir Stepien --- include/net/xdp.h | 36 ++++++++++++++++++++++++++++++++++++ net/core/dev.c | 25 ++++--------------------- 2 files changed, 40 insertions(+), 21 deletions(-) diff --git a/include/net/xdp.h b/include/net/xdp.h index 07231adfb5f8..28bd78dbef20 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -389,6 +389,42 @@ struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, struct net_device *dev); struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf); +/** + * xdp_convert_skb_to_buff - build an xdp_buff on top of an skb + * @skb: skb to describe, packet data is taken from its MAC header onwards + * @xdp: xdp_buff to initialize + * @xdp_rxq: Rx queue info to attach to @xdp + * + * The memory type set on @xdp_rxq is only used to release fragments that + * the XDP program consumes; the head can not be freed from under the skb. + * A non-linear @skb is expected to have gone through skb_pp_cow_data(), so + * all of its fragments are page_pool backed. + */ +static inline +void xdp_convert_skb_to_buff(struct sk_buff *skb, struct xdp_buff *xdp, + struct xdp_rxq_info *xdp_rxq) +{ + u32 frame_sz, pkt_len; + + /* SKB "head" area always have tailroom for skb_shared_info */ + frame_sz = skb_end_pointer(skb) - skb->head; + frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + pkt_len = skb_tail_pointer(skb) - skb_mac_header(skb); + + xdp_init_buff(xdp, frame_sz, xdp_rxq); + xdp_prepare_buff(xdp, skb->head, skb->mac_header, pkt_len, true); + + if (skb_is_nonlinear(skb)) { + skb_shinfo(skb)->xdp_frags_size = skb->data_len; + xdp_buff_set_frags_flag(xdp); + } else { + xdp_buff_clear_frags_flag(xdp); + } + + DEBUG_NET_WARN_ON_ONCE(skb_is_nonlinear(skb) && !skb->pp_recycle); + xdp->rxq->mem.type = MEM_TYPE_PAGE_POOL; +} + static inline void xdp_convert_frame_to_buff(const struct xdp_frame *frame, struct xdp_buff *xdp) diff --git a/net/core/dev.c b/net/core/dev.c index 3ac700d37e59..4eeeb0182a6a 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -5667,35 +5667,18 @@ static struct netdev_rx_queue *netif_get_rxqueue(struct sk_buff *skb) u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp, const struct bpf_prog *xdp_prog) { - void *orig_data, *orig_data_end, *hard_start; + void *orig_data, *orig_data_end; struct netdev_rx_queue *rxqueue; bool orig_bcast, orig_host; - u32 mac_len, frame_sz; + u32 metalen, act, mac_len; __be16 orig_eth_type; struct ethhdr *eth; - u32 metalen, act; int off; - /* The XDP program wants to see the packet starting at the MAC - * header. - */ + rxqueue = netif_get_rxqueue(skb); mac_len = skb->data - skb_mac_header(skb); - hard_start = skb->data - skb_headroom(skb); - - /* SKB "head" area always have tailroom for skb_shared_info */ - frame_sz = (void *)skb_end_pointer(skb) - hard_start; - frame_sz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); - rxqueue = netif_get_rxqueue(skb); - xdp_init_buff(xdp, frame_sz, &rxqueue->xdp_rxq); - xdp_prepare_buff(xdp, hard_start, skb_headroom(skb) - mac_len, - skb_headlen(skb) + mac_len, true); - if (skb_is_nonlinear(skb)) { - skb_shinfo(skb)->xdp_frags_size = skb->data_len; - xdp_buff_set_frags_flag(xdp); - } else { - xdp_buff_clear_frags_flag(xdp); - } + xdp_convert_skb_to_buff(skb, xdp, &rxqueue->xdp_rxq); orig_data_end = xdp->data_end; orig_data = xdp->data; -- 2.56.0