Set SO_BUSY_POLL (busy_poll_us) on each paired socket via bpf_setsockopt() so the splice receiver busy-polls the ring instead of parking - without net.core.busy_read or an application setsockopt. The sock_ops prog runs for both the active and passive established callbacks, so each endpoint sets its own socket. This is done before the peer-not-found early return: pairing is asymmetric (only the second side to establish finds a peer and calls bpf_sock_splice_pair), so setting it only on the pairing side would leave the other end without busy-poll. bpf_setsockopt acts on skops->sk; the peer sets itself on its own callback. Busy polling is a receive-path optimization (splice_busy_loop() in tcp_bpf_splice_recvmsg()); TCP is full-duplex so both ends are receivers and both need it, which the per-endpoint setting provides. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Cong Wang --- .../selftests/bpf/progs/test_tcp_splice.c | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/test_tcp_splice.c b/tools/testing/selftests/bpf/progs/test_tcp_splice.c index 09c7f0f9e311..da43f00046c0 100644 --- a/tools/testing/selftests/bpf/progs/test_tcp_splice.c +++ b/tools/testing/selftests/bpf/progs/test_tcp_splice.c @@ -9,6 +9,13 @@ #include #include +#ifndef SOL_SOCKET +#define SOL_SOCKET 1 +#endif +#ifndef SO_BUSY_POLL +#define SO_BUSY_POLL 46 +#endif + struct flow_key { __u32 saddr; __u32 daddr; @@ -29,6 +36,8 @@ void *bpf_cast_to_kern_ctx(void *obj) __ksym; __u32 pair_ok; __u32 pair_other_err; +__u32 busy_poll_us; + /* IPv4 only: the verifier doesn't accept memcpy from sock_ops ctx * because it lowers to "ctx + reg" pointer arithmetic. IPv6 support * would need explicit field-by-field reads of local_ip6[i] / @@ -71,6 +80,21 @@ int sockops_splice(struct bpf_sock_ops *skops) if (skops->family != 2 /* AF_INET */) return 0; + /* Enable busy-poll on this socket. Both endpoints run this callback, + * so each sets its own socket; this must happen here, before the + * peer-not-found early return below, because pairing is asymmetric - + * only the second side to establish finds a peer and calls + * bpf_sock_splice_pair. Setting it only on the pairing side would + * leave the other side without busy-poll. bpf_setsockopt acts on + * skops->sk only - there is no variant to set the peer - but the peer + * sets itself when its own ESTABLISHED callback fires. + */ + if (busy_poll_us) { + int us = busy_poll_us; + + bpf_setsockopt(skops, SOL_SOCKET, SO_BUSY_POLL, &us, sizeof(us)); + } + mk_key(skops, &self_key, 0); mk_key(skops, &peer_key, 1); -- 2.43.0