AI Trajectory

Trajectory: (Expand/Collapse All)
Seq	Timestamp	Type	Name	Duration
0/0	2026/08/13 20:58	flow	patch-triage	0m Args: null Results: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT" ], "FocusSymbols": [ "bpf_kfunc_ret_reg_pair", "btf_type_is_scalar_struct", "bpf_jit_supports_kfunc_ret_reg_pair", "bpf_compute_live_registers" ], "KMSANReasoning": "The patch extends the BPF verifier and JIT compilers to support returning values up to 16 bytes (e.g., `__int128` or small structs) from BPF subprograms and kfuncs using the R0 and R2 register pair. The changes primarily involve updating the verifier's liveness tracking, backtracking, and return value checks to ensure that both R0 and R2 are properly initialized before a BPF program returns. The JIT changes simply move the native ABI's second return register into BPF's R2. \n\nThere are no new kernel data structures allocated and copied to user space, nor any C-level uninitialized memory risks introduced. The verifier statically enforces that BPF programs initialize the R2 register when returning a pair, preventing uninitialized register leaks. Furthermore, KMSAN does not instrument JIT-compiled BPF code, so it would not provide additional coverage for the BPF program execution itself. Standard KASAN and verifier fuzzing are sufficient to detect any logic bugs, out-of-bounds accesses, or memory corruptions introduced by these changes.", "NeedsKMSAN": false, "Reasoning": "The patch introduces support for kfuncs and BPF subprograms returning larger values (up to 16 bytes) in the R0:R2 register pair. This involves changes to the BPF verifier (liveness tracking, return value checking, BTF validation) and JIT compilers (x86, arm64, riscv). These are significant functional changes to the core BPF subsystem that should be fuzzed to ensure verifier safety and JIT correctness.", "WorthFuzzing": true }
1/1	2026/08/13 20:58	action	read-patch-diff	0m Args: null Results: { "PatchDiff": "commit bafb5dc76d94a94d43e7b1575db7d4920a549687\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Thu Aug 13 20:58:12 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst\nindex 10e725cbe64cd..5d840d3000338 100644\n--- a/Documentation/bpf/kfuncs.rst\n+++ b/Documentation/bpf/kfuncs.rst\n@@ -575,6 +575,69 @@ is also covered by this recovery. A kfunc handed an arena pointer may\n therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking\n against the arena. Larger accesses must verify the range explicitly.\n \n+2.9 kfunc Return Values\n+-----------------------\n+\n+A kfunc may return a scalar, a pointer, or a small struct or union by\n+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.\n+\n+A struct or union returned by value must be composed only of scalars\n+(recursively), where a scalar is an integer or an enum; arrays of scalars are\n+allowed as members. Its bytes are handed back to the program as the raw\n+contents of R0 (and R2), so a pointer field would be laundered into a scalar\n+and escape the verifier's pointer provenance and reference tracking. A struct\n+or union with a pointer member is therefore rejected at load time, and so is\n+one with a floating-point member, which the ABI may not return in R0:R2 at\n+all.\n+\n+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a\n+scalar-only struct or union, or an ``__int128``. Such a value is returned\n+in the register pair R0:R2, matching the convention LLVM uses for the BPF\n+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or\n+union of 8 bytes or less is returned in R0 alone.\n+\n+::\n+\n+ struct bpf_pair { __u64 a, b; }; /* 16 bytes /\n+\n+ __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)\n+ {\n+ struct bpf_pair p = { .a = 1, .b = 2 };\n+\n+ return p; / p.a in R0, p.b in R2 /\n+ }\n+\n+Returning a value in the R0:R2 pair requires the JIT to place the second\n+half of the return value into R2, which not every architecture supports\n+right now. A kfunc with a return value larger than 8 bytes is therefore\n+rejected at load time on a JIT that does not advertise this capability (see\n+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run\n+by the interpreter. A return value larger than 16 bytes is not supported.\n+\n+The same R0:R2 convention applies to a BPF subprogram, global or static,\n+that returns an ``__int128`` or a struct or union larger than 8 bytes. Such a\n+program also requires the JIT, since the interpreter propagates only R0 out\n+of a subprogram. A global subprogram is verified in isolation, so its\n+by-value struct or union return is restricted to scalars just like a kfunc's;\n+a static subprogram is verified inline and has no such restriction. The main\n+program is not covered: its return value is the program's exit code, read out\n+of R0 alone, so a declared upper half is never looked at.\n+\n+A global subprogram must leave a scalar in every* register of the pair, so\n+both halves of the returned value have to be assigned. Leaving the upper half\n+uninitialized is not merely untidy: the compiler is then free to leave R2\n+holding whatever it happened to hold, which for a subprogram taking a pointer\n+argument is typically that pointer. Handing the caller an unknown scalar built\n+from a pointer is a leak, so the verifier rejects it with::\n+\n+ At subprogram exit the register R2 is not a scalar value (...)\n+\n+Initialize the whole return value, for example ``struct pair p = {};``, to\n+avoid this. A static subprogram is exempt from the scalar-only rule: it is\n+verified inline, so an unassigned R2 is simply passed back to the caller as\n+uninitialized and only a caller that reads it fails. A stack pointer left in\n+R2 is still rejected there, just as one in R0 is.\n+\n .. _BPF_kfunc_lifecycle_expectations:\n \n 3. kfunc lifecycle expectations\ndiff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c\nindex 74b4083791da3..7a4c5968976a6 100644\n--- a/arch/arm64/net/bpf_jit_comp.c\n+++ b/arch/arm64/net/bpf_jit_comp.c\n@@ -2346,6 +2346,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_stack_args(void)\n {\n \treturn true;\ndiff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c\nindex 2504df1fa1118..49220765e96d4 100644\n--- a/arch/riscv/net/bpf_jit_comp64.c\n+++ b/arch/riscv/net/bpf_jit_comp64.c\n@@ -2120,6 +2120,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_ptr_xchg(void)\n {\n \treturn true;\ndiff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c\nindex d920772af7d5f..2ec09dd02b187 100644\n--- a/arch/x86/net/bpf_jit_comp.c\n+++ b/arch/x86/net/bpf_jit_comp.c\n@@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip,\n * arena NULL is offset 0. Return the number of emitted bytes.\n /\n static int emit_kfunc_arena_args(struct bpf_prog bpf_prog,\n-\t\t\t\t const struct bpf_insn insn, u8 pprog)\n+\t\t\t\t const struct btf_func_model fm, u8 *pprog)\n {\n-\tconst struct btf_func_model fm;\n \tu8 prog = pprog;\n \tu8 start = prog;\n \tint i;\n \n-\tfm = bpf_jit_find_kfunc_model(bpf_prog, insn);\n-\tif (!fm)\n-\t\treturn -EINVAL;\n-\n \tfor (i = 0; i \u003c min_t(int, fm-\u003enr_args, MAX_BPF_FUNC_REG_ARGS); i++) {\n \t\tu8 flags = fm-\u003earg_flags[i];\n \t\tu32 reg = BPF_REG_1 + i;\n@@ -2644,6 +2639,8 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \n \t\t\t/ call /\n \t\tcase BPF_JMP \| BPF_CALL: {\n+\t\t\tconst struct btf_func_model fm = NULL;\n+\n \t\t\tfunc = (u8 ) __bpf_call_base + imm32;\n \t\t\tif (src_reg == BPF_PSEUDO_CALL \u0026\u0026 tail_call_reachable) {\n \t\t\t\tLOAD_TAIL_CALL_CNT_PTR(stack_depth);\n@@ -2652,7 +2649,10 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tif (!imm32)\n \t\t\t\treturn -EINVAL;\n \t\t\tif (src_reg == BPF_PSEUDO_KFUNC_CALL) {\n-\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, insn, \u0026prog);\n+\t\t\t\tfm = bpf_jit_find_kfunc_model(bpf_prog, insn);\n+\t\t\t\tif (!fm)\n+\t\t\t\t\treturn -EINVAL;\n+\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, fm, \u0026prog);\n \t\t\t\tif (err \u003c 0)\n \t\t\t\t\treturn err;\n \t\t\t\tip += err;\n@@ -2666,6 +2666,14 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\treturn -EINVAL;\n \t\t\tif (priv_frame_ptr)\n \t\t\t\tpop_r9(\u0026prog);\n+\t\t\t/\n+\t\t\t * A kfunc returning more than 8 bytes hands the second\n+\t\t\t * half back in RDX (the native ABI's second return reg),\n+\t\t\t * but BPF expects it in R0:R2. BPF R0 is RAX (no move\n+\t\t\t * needed), while BPF R2 is RSI, so copy RDX into RSI.\n+\t\t\t /\n+\t\t\tif (fm \u0026\u0026 fm-\u003eret_size \u003e 8)\n+\t\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_2, BPF_REG_3);\n \t\t\tbreak;\n \t\t}\n \n@@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_stack_args(void)\n {\n \treturn true;\ndiff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h\nindex 27b43fda9b178..899e5ce986432 100644\n--- a/include/linux/bpf_verifier.h\n+++ b/include/linux/bpf_verifier.h\n@@ -814,6 +814,8 @@ struct bpf_subprog_info {\n \tbool is_async_cb: 1;\n \tbool is_exception_cb: 1;\n \tbool args_cached: 1;\n+\t/ true if the return value is passed in the R0:R2 register pair /\n+\tbool ret_reg_pair: 1;\n \t/ true if bpf_fastcall stack region is used by functions that can't be inlined /\n \tbool keep_fastcall_stack: 1;\n \tbool changes_pkt_data: 1;\n@@ -1048,6 +1050,13 @@ static inline struct bpf_subprog_info subprog_info(struct bpf_verifier_env env\n \treturn \u0026env-\u003esubprog_info[subprog];\n }\n \n+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env env, int subprog)\n+{\n+\treturn subprog_info(env, subprog)-\u003eret_reg_pair;\n+}\n+\n+bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env env, struct bpf_insn insn);\n+\n struct bpf_call_summary {\n \tu8 num_params;\n \tbool is_void;\n@@ -1439,6 +1448,8 @@ int bpf_jmp_offset(struct bpf_insn insn);\n struct bpf_iarray bpf_insn_successors(struct bpf_verifier_env env, u32 idx);\n void bpf_fmt_stack_mask(char buf, ssize_t buf_sz, u64 stack_mask);\n bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog);\n+bool btf_type_is_scalar_struct(struct bpf_verifier_env env, const struct btf btf,\n+\t\t\t const struct btf_type t, int rec);\n \n int bpf_find_subprog(struct bpf_verifier_env env, int off);\n bool bpf_is_throw_kfunc(struct bpf_insn insn);\ndiff --git a/include/linux/filter.h b/include/linux/filter.h\nindex 4a9bc6a848f2e..6e746b0a09306 100644\n--- a/include/linux/filter.h\n+++ b/include/linux/filter.h\n@@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);\n bool bpf_jit_supports_subprog_tailcalls(void);\n bool bpf_jit_supports_percpu_insn(void);\n bool bpf_jit_supports_kfunc_call(void);\n+bool bpf_jit_supports_kfunc_ret_reg_pair(void);\n bool bpf_jit_supports_stack_args(void);\n bool bpf_jit_supports_arena_args(void);\n bool bpf_jit_supports_far_kfunc_call(void);\ndiff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c\nindex 40bd04421a991..3ef8a3da726de 100644\n--- a/kernel/bpf/backtrack.c\n+++ b/kernel/bpf/backtrack.c\n@@ -425,6 +425,16 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx,\n \t\t\t\t /\n \t\t\t\tverifier_bug_if(idx + 1 != subseq_idx, env,\n \t\t\t\t\t\t\"extra insn from subprog\");\n+\t\t\t\t/* a global subprog returning more than 8 bytes\n+\t\t\t\t * sets R2 as well. R2 is part of the args mask\n+\t\t\t\t * checked just below, so clear it here rather\n+\t\t\t\t * than next to R0. Only a subprog that does\n+\t\t\t\t * return a pair defines R2, so leave the mask\n+\t\t\t\t * alone otherwise and let the check below catch\n+\t\t\t\t * an R2 that has no business being set.\n+\t\t\t\t /\n+\t\t\t\tif (bpf_ret_reg_pair(env, subprog))\n+\t\t\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\t\t/ r1-r5 are invalidated after subprog call,\n \t\t\t\t * so for global func call it shouldn't be set\n \t\t\t\t * anymore\n@@ -508,6 +518,17 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx,\n \t\t\t\treturn -ENOTSUPP;\n \t\t\t/ regular helper call sets R0 /\n \t\t\tbt_clear_reg(bt, BPF_REG_0);\n+\t\t\t/ a kfunc returning more than 8 bytes also sets R2.\n+\t\t\t * R2 is part of the args mask checked just below, so\n+\t\t\t * clear it here rather than next to R0. The prototype\n+\t\t\t * lookup is only worth doing when R2 is requested at\n+\t\t\t * all; any other call leaves R2 uninitialized, so a\n+\t\t\t * request for it is caught by the check below.\n+\t\t\t /\n+\t\t\tif (bt_is_reg_set(bt, BPF_REG_2) \u0026\u0026\n+\t\t\t insn-\u003esrc_reg == BPF_PSEUDO_KFUNC_CALL \u0026\u0026\n+\t\t\t bpf_kfunc_ret_reg_pair(env, insn))\n+\t\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\tif (bt_reg_mask(bt) \u0026 BPF_REGMASK_ARGS) {\n \t\t\t\t/ if backtracking was looking for registers R1-R5\n \t\t\t\t * they should have been found already.\n@@ -522,7 +543,41 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx,\n \t\t\t\t\treturn -EFAULT;\n \t\t\t}\n \t\t} else if (opcode == BPF_EXIT) {\n-\t\t\tbool r0_precise;\n+\t\t\tbool from_subprog_call, r0_precise, r2_precise;\n+\t\t\tstruct bpf_insn call;\n+\t\t\tint subprog;\n+\n+\t\t\t/* BPF_EXIT in subprog or callback always returns\n+\t\t\t * right after the call instruction, so by checking\n+\t\t\t * whether the instruction at subseq_idx-1 is subprog\n+\t\t\t * call or not we can distinguish actual exit from\n+\t\t\t * subprog from exit from callback. In the former\n+\t\t\t * case, we need to propagate the precision of the\n+\t\t\t * return registers, if necessary. In the latter we\n+\t\t\t * never do that.\n+\t\t\t /\n+\t\t\tfrom_subprog_call = subseq_idx - 1 \u003e= 0 \u0026\u0026\n+\t\t\t\t\t bpf_pseudo_call(\u0026env-\u003eprog-\u003einsnsi[subseq_idx - 1]);\n+\n+\t\t\t/ Sample the return registers before the callback\n+\t\t\t * handling below clears R1-R5: unlike R0, R2 is an\n+\t\t\t * argument register as well, so that clear would drop\n+\t\t\t * a pair return on the floor.\n+\t\t\t /\n+\t\t\tr0_precise = from_subprog_call \u0026\u0026 bt_is_reg_set(bt, BPF_REG_0);\n+\t\t\tr2_precise = false;\n+\t\t\tif (from_subprog_call \u0026\u0026 bt_is_reg_set(bt, BPF_REG_2)) {\n+\t\t\t\tcall = \u0026env-\u003eprog-\u003einsnsi[subseq_idx - 1];\n+\t\t\t\tsubprog = bpf_find_subprog(env, subseq_idx + call-\u003eimm);\n+\t\t\t\tif (subprog \u003c 0)\n+\t\t\t\t\treturn -EFAULT;\n+\t\t\t\t/ Only a callee that does return a pair defines\n+\t\t\t\t * R2. Leave the mask alone otherwise, so that\n+\t\t\t\t * the check below still catches an R2 that has\n+\t\t\t\t * no business being set.\n+\t\t\t\t /\n+\t\t\t\tr2_precise = bpf_ret_reg_pair(env, subprog);\n+\t\t\t}\n \n \t\t\t/ Backtracking to a nested function call, 'idx' is a part of\n \t\t\t * the inner frame 'subseq_idx' is a part of the outer frame.\n@@ -535,30 +590,27 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx,\n \t\t\tif (subseq_idx \u003e= 0 \u0026\u0026 bpf_calls_callback(env, subseq_idx))\n \t\t\t\tfor (i = BPF_REG_1; i \u003c= BPF_REG_5; i++)\n \t\t\t\t\tbt_clear_reg(bt, i);\n+\n+\t\t\t/ a callee returning more than 8 bytes sets R2 as well;\n+\t\t\t * R2 is part of the args mask checked just below, so\n+\t\t\t * clear it here rather than next to R0.\n+\t\t\t /\n+\t\t\tif (r2_precise)\n+\t\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\tif (bt_reg_mask(bt) \u0026 BPF_REGMASK_ARGS) {\n \t\t\t\tverifier_bug(env, \"backtracking exit unexpected regs %x\",\n \t\t\t\t\t bt_reg_mask(bt));\n \t\t\t\treturn -EFAULT;\n \t\t\t}\n \n-\t\t\t/ BPF_EXIT in subprog or callback always returns\n-\t\t\t * right after the call instruction, so by checking\n-\t\t\t * whether the instruction at subseq_idx-1 is subprog\n-\t\t\t * call or not we can distinguish actual exit from\n-\t\t\t * subprog from exit from callback. In the former\n-\t\t\t * case, we need to propagate r0 precision, if\n-\t\t\t * necessary. In the former we never do that.\n-\t\t\t /\n-\t\t\tr0_precise = subseq_idx - 1 \u003e= 0 \u0026\u0026\n-\t\t\t\t bpf_pseudo_call(\u0026env-\u003eprog-\u003einsnsi[subseq_idx - 1]) \u0026\u0026\n-\t\t\t\t bt_is_reg_set(bt, BPF_REG_0);\n-\n \t\t\tbt_clear_reg(bt, BPF_REG_0);\n \t\t\tif (bt_subprog_enter(bt))\n \t\t\t\treturn -EFAULT;\n \n \t\t\tif (r0_precise)\n \t\t\t\tbt_set_reg(bt, BPF_REG_0);\n+\t\t\tif (r2_precise)\n+\t\t\t\tbt_set_reg(bt, BPF_REG_2);\n \t\t\t/ r6-r9 and stack slots will stay set in caller frame\n \t\t\t * bitmasks until we return back from callee(s)\n \t\t\t /\ndiff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c\nindex 6606187ed4f43..c7718a2a56141 100644\n--- a/kernel/bpf/btf.c\n+++ b/kernel/bpf/btf.c\n@@ -7592,7 +7592,7 @@ int btf_distill_func_proto(struct bpf_verifier_log log,\n \t\treturn -EINVAL;\n \t}\n \tret = __get_type_size(btf, func-\u003etype, \u0026t);\n-\tif (ret \u003c 0 \|\| btf_type_is_struct(t)) {\n+\tif (ret \u003c 0 \|\| ret \u003e 16) {\n \t\tbpf_log(log,\n \t\t\t\"The function %s return type %s is unsupported.\\n\",\n \t\t\ttname, btf_type_str(t));\n@@ -7965,7 +7965,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env env,\n \n / Check whether the type is a valid return type. /\n static int btf_validate_return_type(struct bpf_verifier_env env, struct btf btf,\n-\t\tconst struct btf_type t, int subprog)\n+\t\tconst struct btf_type t, int subprog, bool is_global)\n {\n \tu32 tags = 0;\n \tint err;\n@@ -7988,6 +7988,19 @@ static int btf_validate_return_type(struct bpf_verifier_env env, struct btf bt\n \tif (btf_type_is_void(t) \|\| btf_type_is_int(t) \|\| btf_is_any_enum(t))\n \t\treturn 0;\n \n+\tif (btf_type_is_struct(t) \u0026\u0026 t-\u003esize \u003c= 16) {\n+\t\t/\n+\t\t * A global function's caller models the return as an opaque\n+\t\t * scalar pair, so it may only return scalars by value. A local\n+\t\t * function is verified inline, so a pointer field stays tracked\n+\t\t * and needs no such restriction.\n+\t\t /\n+\t\tbool local_func = subprog \u0026\u0026 !is_global;\n+\n+\t\tif (local_func \|\| btf_type_is_scalar_struct(env, btf, t, 0))\n+\t\t\treturn 0;\n+\t}\n+\n \treturn -EOPNOTSUPP;\n }\n \n@@ -8075,12 +8088,12 @@ int btf_prepare_func_args(struct bpf_verifier_env env, int subprog)\n \t\treturn -EINVAL;\n \t}\n \n-\terr = btf_validate_return_type(env, btf, t, subprog);\n+\terr = btf_validate_return_type(env, btf, t, subprog, is_global);\n \tif (err) {\n \t\tif (is_global) {\n \t\t\tbpf_log(log,\n-\t\t\t\t\"Global function %s() return value not void or scalar. \"\n-\t\t\t\t\"Only those are supported.\\n\",\n+\t\t\t\t\"Global function %s() has unsupported return type. \"\n+\t\t\t\t\"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\\n\",\n \t\t\t\ttname);\n \t\t}\n \t\treturn err;\ndiff --git a/kernel/bpf/core.c b/kernel/bpf/core.c\nindex 6a94370a24488..cb66a2ef52b1f 100644\n--- a/kernel/bpf/core.c\n+++ b/kernel/bpf/core.c\n@@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)\n \treturn false;\n }\n \n+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn false;\n+}\n+\n bool __weak bpf_jit_supports_stack_args(void)\n {\n \treturn false;\ndiff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c\nindex 1c997aeba6fa5..850c58aa915be 100644\n--- a/kernel/bpf/liveness.c\n+++ b/kernel/bpf/liveness.c\n@@ -2062,7 +2062,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m \u003e\u003e 16); }\n /* Compute info-\u003e{use,def} fields for the instruction /\n static void compute_insn_live_regs(struct bpf_verifier_env env,\n \t\t\t\t struct bpf_insn insn,\n-\t\t\t\t struct insn_live_regs info)\n+\t\t\t\t struct insn_live_regs info,\n+\t\t\t\t bool ret_reg_pair)\n {\n \tstruct bpf_call_summary cs;\n \tconst u8 class = BPF_CLASS(insn-\u003ecode);\n@@ -2074,6 +2075,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env,\n \tconst u32 src32 = mask_lo(src);\n \tconst u32 dst32 = mask_lo(dst);\n \tconst u32 r0 = reg64_mask(0);\n+\tconst u32 r2 = reg64_mask(BPF_REG_2);\n \tu32 def = 0;\n \tu32 use = U32_MAX;\n \n@@ -2193,7 +2195,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env,\n \t\t\tbreak;\n \t\tcase BPF_EXIT:\n \t\t\tdef = 0;\n-\t\t\tuse = r0;\n+\t\t\tuse = ret_reg_pair ? (r0 \| r2) : r0;\n \t\t\tbreak;\n \t\tcase BPF_CALL:\n \t\t\tdef = ALL_CALLER_SAVED_REGS;\n@@ -2230,8 +2232,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env env)\n \tstruct insn_live_regs state;\n \tint insn_cnt = env-\u003eprog-\u003elen;\n \tu64 pos, insn_pos;\n-\tint err = 0, i, j;\n-\tbool changed;\n+\tint err = 0, i, j, subprog, start, end;\n+\tbool changed, ret_reg_pair;\n \n \t/ Use the following algorithm:\n \t * - define the following:\n@@ -2258,8 +2260,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env env)\n \t\tgoto out;\n \t}\n \n-\tfor (i = 0; i \u003c insn_cnt; ++i)\n-\t\tcompute_insn_live_regs(env, \u0026insns[i], \u0026state[i]);\n+\tfor (subprog = 0; subprog \u003c env-\u003esubprog_cnt; subprog++) {\n+\t\tstart = env-\u003esubprog_info[subprog].start;\n+\t\tend = env-\u003esubprog_info[subprog + 1].start;\n+\t\tret_reg_pair = bpf_ret_reg_pair(env, subprog);\n+\n+\t\tfor (i = start; i \u003c end; ++i)\n+\t\t\tcompute_insn_live_regs(env, \u0026insns[i], \u0026state[i], ret_reg_pair);\n+\t}\n \n \t/ Forward pass: resolve stack access through FP-derived pointers /\n \terr = bpf_compute_subprog_arg_access(env);\ndiff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c\nindex 164d16c243ca6..98a6f702d794d 100644\n--- a/kernel/bpf/verifier.c\n+++ b/kernel/bpf/verifier.c\n@@ -382,27 +382,80 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog)\n \treturn aux \u0026\u0026 aux[subprog].linkage == BTF_FUNC_GLOBAL;\n }\n \n-static bool subprog_returns_void(struct bpf_verifier_env env, int subprog)\n+static const struct btf_type subprog_ret_type(struct bpf_verifier_env env, int subprog)\n {\n-\tconst struct btf_type type, func, func_proto;\n+\tconst struct btf_type func, func_proto;\n \tconst struct btf btf = env-\u003eprog-\u003eaux-\u003ebtf;\n \tu32 btf_id;\n \n+\tif (!btf \|\| !env-\u003eprog-\u003eaux-\u003efunc_info)\n+\t\treturn NULL;\n+\n \tbtf_id = env-\u003eprog-\u003eaux-\u003efunc_info[subprog].type_id;\n \n+\t/ Both already validated by prepare_btf_func() at prog load. /\n \tfunc = btf_type_by_id(btf, btf_id);\n-\tif (verifier_bug_if(!func, env, \"btf_id %u not found\", btf_id))\n-\t\treturn false;\n-\n \tfunc_proto = btf_type_by_id(btf, func-\u003etype);\n-\tif (!func_proto)\n-\t\treturn false;\n \n-\ttype = btf_type_skip_modifiers(btf, func_proto-\u003etype, NULL);\n-\tif (!type)\n-\t\treturn false;\n+\treturn btf_type_skip_modifiers(btf, func_proto-\u003etype, NULL);\n+}\n+\n+static bool subprog_returns_void(struct bpf_verifier_env env, int subprog)\n+{\n+\tconst struct btf_type type = subprog_ret_type(env, subprog);\n+\n+\treturn type \u0026\u0026 btf_type_is_void(type);\n+}\n+\n+static u32 ret_regs_cnt(u32 size)\n+{\n+\treturn size \u003e 8 \u0026\u0026 size \u003c= 16 ? 2 : 1;\n+}\n+\n+/ Registers holding a function return value, in order. See ret_regs_cnt(). /\n+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };\n+\n+static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env env)\n+{\n+\tconst struct btf btf = env-\u003eprog-\u003eaux-\u003ebtf;\n+\tconst struct btf_type type;\n+\tint subprog;\n+\tu32 size;\n+\n+\tfor (subprog = 0; subprog \u003c env-\u003esubprog_cnt; subprog++) {\n+\t\ttype = subprog_ret_type(env, subprog);\n+\t\tif (!type \|\| !(btf_type_is_struct(type) \|\| btf_type_is_scalar(type)))\n+\t\t\tcontinue;\n+\t\tif (IS_ERR(btf_resolve_size(btf, type, \u0026size)))\n+\t\t\tcontinue;\n+\t\tif (ret_regs_cnt(size) \u003e 1) {\n+\t\t\tsubprog_info(env, subprog)-\u003eret_reg_pair = true;\n+\t\t\t/\n+\t\t\t The R0:R2 return convention is only implemented in\n+\t\t\t * the JIT: the interpreter propagates BPF_R0 alone out\n+\t\t\t * of a subprogram, so a caller reading R2 would see a\n+\t\t\t * stale value.\n+\t\t\t /\n+\t\t\tenv-\u003eprog-\u003ejit_required = 1;\n+\t\t}\n+\t}\n+}\n \n-\treturn btf_type_is_void(type);\n+/\n+ * A \u003e8 byte BPF return changes the calling convention to R0:R2, and the\n+ * verifier derives that convention from the subprogram's BTF prototype\n+ * alone. Once that prototype is marked unreliable it is known not to\n+ * describe the compiled code, so the convention read from it cannot be\n+ * trusted either: reject the call rather than keep tracking R2 on the\n+ * strength of a signature the verifier has already discarded.\n+ /\n+static bool subprog_ret_pair_unreliable(struct bpf_verifier_env env, int subprog)\n+{\n+\tstruct bpf_prog_aux aux = env-\u003eprog-\u003eaux;\n+\n+\treturn bpf_ret_reg_pair(env, subprog) \u0026\u0026\n+\t aux-\u003efunc_info_aux \u0026\u0026\n+\t aux-\u003efunc_info_aux[subprog].unreliable;\n }\n \n static const char subprog_name(const struct bpf_verifier_env env, int subprog)\n@@ -2459,6 +2512,21 @@ find_kfunc_desc(const struct bpf_prog prog, u32 func_id, u16 offset)\n \t\t sizeof(tab-\u003edescs[0]), kfunc_desc_cmp_by_id_off);\n }\n \n+/\n+ True if the kfunc called by @insn returns its value in the R0:R2 pair.\n+ * Reads the same btf_func_model.ret_size that bpf_add_kfunc_call() validated\n+ * and that the JIT keys the second return register off, so the verifier and\n+ * the generated code cannot disagree about the convention.\n+ /\n+bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env env, struct bpf_insn insn)\n+{\n+\tconst struct bpf_kfunc_desc desc;\n+\n+\tdesc = find_kfunc_desc(env-\u003eprog, insn-\u003eimm, insn-\u003eoff);\n+\n+\treturn desc \u0026\u0026 ret_regs_cnt(desc-\u003efunc_model.ret_size) \u003e 1;\n+}\n+\n int bpf_get_kfunc_addr(const struct bpf_prog prog, u32 func_id,\n \t\t u16 btf_fd_idx, u8 func_addr)\n {\n@@ -2809,6 +2877,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env env, u32 func_id, u16 offset)\n \terr = btf_distill_func_proto(\u0026env-\u003elog, kfunc.btf, kfunc.proto, kfunc.name, \u0026func_model);\n \tif (err)\n \t\treturn err;\n+\tif (func_model.ret_size \u003e 8) {\n+\t\tif (kfunc.flags \u0026\u0026 (kfunc.flags \u0026 KF_FASTCALL)) {\n+\t\t\tverbose(env, \"kfunc %s with \u003e8-byte return is not supported with KF_FASTCALL\\n\",\n+\t\t\t\tkfunc.name);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\t\tif (!bpf_jit_supports_kfunc_ret_reg_pair()) {\n+\t\t\tverbose(env, \"kfunc %s with \u003e8-byte return is not supported by JIT\\n\",\n+\t\t\t\tkfunc.name);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\t}\n \n \tmemset(\u0026meta, 0, sizeof(meta));\n \tmeta.btf = kfunc.btf;\n@@ -9381,6 +9461,7 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn,\n \tu16 callee_incoming, stack_arg_cnt;\n \tstruct bpf_func_state caller;\n \tint err, subprog, target_insn;\n+\tu32 i, nregs;\n \n \ttarget_insn = insn_idx + insn-\u003eimm + 1;\n \tsubprog = bpf_find_subprog(env, target_insn);\n@@ -9423,9 +9504,14 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn,\n \t\tclear_caller_saved_regs(env, caller-\u003eregs);\n \t\tinvalidate_outgoing_stack_args(env, cur_func(env));\n \n-\t\t/ All non-void global functions return a 64-bit SCALAR_VALUE. /\n+\t\t/\n+\t\t * A non-void global function returns a 64-bit SCALAR_VALUE in\n+\t\t * R0, or a \u003e8 byte SCALAR_VALUE in the R0:R2 register pair.\n+\t\t /\n \t\tif (!subprog_returns_void(env, subprog)) {\n-\t\t\tmark_reg_unknown(env, caller-\u003eregs, BPF_REG_0);\n+\t\t\tnregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;\n+\t\t\tfor (i = 0; i \u003c nregs; i++)\n+\t\t\t\tmark_reg_unknown(env, caller-\u003eregs, ret_regs[i]);\n \t\t}\n \n \t\tif (env-\u003esubprog_info[subprog].might_throw) {\n@@ -9443,6 +9529,12 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn,\n \t\treturn 0;\n \t}\n \n+\tif (subprog_ret_pair_unreliable(env, subprog)) {\n+\t\tverbose(env, \"Func#%d ('%s') returns \u003e8 bytes, which requires reliable BTF\\n\",\n+\t\t\tsubprog, subprog_name(env, subprog));\n+\t\treturn -EINVAL;\n+\t}\n+\n \t/\n \t * Track caller's total stack arg count (incoming + max outgoing).\n \t * This is needed so the JIT knows how much stack arg space to allocate.\n@@ -9783,11 +9875,15 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx)\n \tstruct bpf_func_state caller, callee;\n \tstruct bpf_reg_state r0;\n \tbool in_callback_fn;\n+\tu32 i, nregs;\n \tint err;\n \n \tcallee = state-\u003eframe[state-\u003ecurframe];\n \tr0 = \u0026callee-\u003eregs[BPF_REG_0];\n-\tif (r0-\u003etype == PTR_TO_STACK) {\n+\tnregs = bpf_ret_reg_pair(env, callee-\u003esubprogno) ? 2 : 1;\n+\tfor (i = 0; i \u003c nregs; i++) {\n+\t\tif (callee-\u003eregs[ret_regs[i]].type != PTR_TO_STACK)\n+\t\t\tcontinue;\n \t\t/ technically it's ok to return caller's stack pointer\n \t\t * (or caller's caller's pointer) back to the caller,\n \t\t * since these pointers are valid. Only current stack\n@@ -9822,8 +9918,12 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx)\n \t\t\treturn -EFAULT;\n \t\t}\n \t} else {\n-\t\t/* return to the caller whatever r0 had in the callee /\n-\t\tcaller-\u003eregs[BPF_REG_0] = r0;\n+\t\t/\n+\t\t return to the caller whatever the callee had in the\n+\t\t * return register(s)\n+\t\t /\n+\t\tfor (i = 0; i \u003c nregs; i++)\n+\t\t\tcaller-\u003eregs[ret_regs[i]] = callee-\u003eregs[ret_regs[i]];\n \t}\n \n \t/ for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,\n@@ -10721,6 +10821,19 @@ static int check_helper_call(struct bpf_verifier_env env, struct bpf_insn insn\n \treturn 0;\n }\n \n+/\n+ Mark the register(s) holding a @size byte kfunc return value as unknown\n+ * scalars. Both halves of a register pair are treated the same way.\n+ /\n+static void mark_kfunc_ret_regs(struct bpf_verifier_env env,\n+\t\t\t\tstruct bpf_reg_state regs, u32 size)\n+{\n+\tu32 i, nregs = ret_regs_cnt(size);\n+\n+\tfor (i = 0; i \u003c nregs; i++)\n+\t\tmark_reg_unknown(env, regs, ret_regs[i]);\n+}\n+\n static bool is_kfunc_acquire(struct bpf_call_arg_meta meta)\n {\n \treturn meta-\u003ekfunc_flags \u0026 KF_ACQUIRE;\n@@ -10988,9 +11101,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta meta, u32 arg_\n }\n \n / Returns true if struct is composed of scalars, 4 levels of nesting allowed /\n-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env,\n-\t\t\t\t\tconst struct btf btf,\n-\t\t\t\t\tconst struct btf_type t, int rec)\n+bool btf_type_is_scalar_struct(struct bpf_verifier_env env,\n+\t\t\t const struct btf btf,\n+\t\t\t const struct btf_type t, int rec)\n {\n \tconst struct btf_type member_type;\n \tconst struct btf_member member;\n@@ -11008,7 +11121,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env,\n \t\t\t\tverbose(env, \"max struct nesting depth exceeded\\n\");\n \t\t\t\treturn false;\n \t\t\t}\n-\t\t\tif (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))\n+\t\t\tif (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1))\n \t\t\t\treturn false;\n \t\t\tcontinue;\n \t\t}\n@@ -11407,7 +11520,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta,\n \t\t (is_kfunc_arg_mem_size(meta-\u003ebtf, \u0026args[arg + 1]) \|\|\n \t\t is_kfunc_arg_const_mem_size(meta-\u003ebtf, \u0026args[arg + 1]))) {\n \t\tif (!btf_type_is_void(ref_t) \u0026\u0026 !btf_type_is_scalar(ref_t) \u0026\u0026\n-\t\t !__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t !btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\tverbose(env, \"%s pointer type %s %s must point to void, scalar, or struct with scalar\\n\",\n \t\t\t\treg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);\n \t\t\treturn -EINVAL;\n@@ -11423,7 +11536,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta,\n \t\t * scalars. The access size is derived from the pointed-to BTF type.\n \t\t /\n \t\tif (!btf_type_is_scalar(ref_t) \u0026\u0026\n-\t\t !__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t !btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\tverbose(env, \"%s pointer type %s %s must point to scalar, or struct with scalar\\n\",\n \t\t\t\treg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);\n \t\t\treturn -EINVAL;\n@@ -12394,7 +12507,7 @@ static int check_kfunc_args(struct bpf_verifier_env env, struct bpf_call_arg_me\n \t\t\t\tbreak;\n \t\t\t}\n \n-\t\t\tif (!__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t\tif (!btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\t\tenum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);\n \n \t\t\t\tverbose(env, \"%s is %s expected %s %s\",\n@@ -12858,7 +12971,7 @@ static int check_special_kfunc(struct bpf_verifier_env env, struct bpf_call_arg\n \n \t\tstruct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);\n \t\tif (is_bpf_percpu_obj_new_kfunc(meta-\u003efunc_id)) {\n-\t\t\tif (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {\n+\t\t\tif (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {\n \t\t\t\tverbose(env, \"bpf_percpu_obj_new type ID argument must be of a struct of scalars\\n\");\n \t\t\t\treturn -EINVAL;\n \t\t\t}\n@@ -13191,10 +13304,25 @@ static int check_kfunc_call(struct bpf_verifier_env env, struct bpf_insn insn,\n \t}\n \n \tif (btf_type_is_scalar(t)) {\n-\t\tmark_reg_unknown(env, regs, BPF_REG_0);\n+\t\tmark_kfunc_ret_regs(env, regs, t-\u003esize);\n \t\tif (meta.btf == btf_vmlinux \u0026\u0026 (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] \|\|\n \t\t meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))\n \t\t\t__mark_reg_const_zero(env, \u0026regs[BPF_REG_0]);\n+\t} else if (btf_type_is_struct(t)) {\n+\t\t/\n+\t\t * The returned struct comes back as raw register bits modeled\n+\t\t * as an unknown scalar, so it must contain only scalars:\n+\t\t * otherwise a pointer field would be laundered into a scalar\n+\t\t * and escape provenance and reference tracking.\n+\t\t /\n+\t\tif (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {\n+\t\t\tverbose(env,\n+\t\t\t\t\"kernel function %s returns %s %s that is not composed of scalars\\n\",\n+\t\t\t\tfunc_name, btf_type_str(t),\n+\t\t\t\tbtf_name_by_offset(desc_btf, t-\u003ename_off));\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\t\tmark_kfunc_ret_regs(env, regs, t-\u003esize);\n \t} else if (btf_type_is_ptr(t)) {\n \t\tptr_type = btf_type_skip_modifiers(desc_btf, t-\u003etype, \u0026ptr_type_id);\n \t\terr = check_special_kfunc(env, \u0026meta, regs, insn_aux, ptr_type, desc_btf);\n@@ -16641,37 +16769,54 @@ static int check_return_code(struct bpf_verifier_env env, int regno, const char\n \treturn 0;\n }\n \n-static int check_global_subprog_return_code(struct bpf_verifier_env env)\n+static int check_global_ret_scalar_reg(struct bpf_verifier_env env, u32 regno)\n {\n-\tstruct bpf_reg_state reg = reg_state(env, BPF_REG_0);\n-\tstruct bpf_func_state cur_frame = cur_func(env);\n+\tstruct bpf_reg_state reg;\n \tint err;\n \n-\tif (subprog_returns_void(env, cur_frame-\u003esubprogno))\n-\t\treturn 0;\n-\n-\terr = check_reg_arg(env, BPF_REG_0, SRC_OP);\n+\terr = check_reg_arg(env, regno, SRC_OP);\n \tif (err)\n \t\treturn err;\n \n \t/ Pointers to arena are safe to pass between subprograms. /\n-\tif (is_arena_reg(env, BPF_REG_0))\n+\tif (is_arena_reg(env, regno))\n \t\treturn 0;\n \n-\tif (is_pointer_value(env, BPF_REG_0)) {\n-\t\tverbose(env, \"R%d leaks addr as return value\\n\", BPF_REG_0);\n+\tif (is_pointer_value(env, regno)) {\n+\t\tverbose(env, \"R%d leaks addr as return value\\n\", regno);\n \t\treturn -EACCES;\n \t}\n \n+\treg = reg_state(env, regno);\n \tif (reg-\u003etype != SCALAR_VALUE) {\n-\t\tverbose(env, \"At subprogram exit the register R0 is not a scalar value (%s)\\n\",\n-\t\t\treg_type_str(env, reg-\u003etype));\n+\t\tverbose(env, \"At subprogram exit the register R%d is not a scalar value (%s)\\n\",\n+\t\t\tregno, reg_type_str(env, reg-\u003etype));\n \t\treturn -EINVAL;\n \t}\n \n \treturn 0;\n }\n \n+static int check_global_subprog_return_code(struct bpf_verifier_env env)\n+{\n+\tstruct bpf_func_state cur_frame = cur_func(env);\n+\tu32 subprog = cur_frame-\u003esubprogno;\n+\tu32 i, nregs;\n+\tint err;\n+\n+\tif (subprog_returns_void(env, subprog))\n+\t\treturn 0;\n+\n+\tnregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;\n+\tfor (i = 0; i \u003c nregs; i++) {\n+\t\terr = check_global_ret_scalar_reg(env, ret_regs[i]);\n+\t\tif (err)\n+\t\t\treturn err;\n+\t}\n+\n+\treturn 0;\n+}\n+\n / Bitmask with 1s for all caller saved registers /\n #define ALL_CALLER_SAVED_REGS ((1u \u003c\u003c CALLER_SAVED_REGS) - 1)\n \n@@ -19328,6 +19473,13 @@ int bpf_check_attach_target(struct bpf_verifier_log log,\n \t\t\treturn -EOPNOTSUPP;\n \t\t}\n \n+\t\tif (prog_extension \u0026\u0026 tgt_info-\u003efmodel.ret_size \u003e 8) {\n+\t\t\tbpf_log(log,\n+\t\t\t\t\"Cannot replace function %s with a \u003e8 byte return value\\n\",\n+\t\t\t\ttname);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\n \t\t/\n \t\t .multi programs don't need an address during program\n \t\t verification, we just take the module ref if needed.\n@@ -20317,6 +20469,9 @@ int bpf_check(struct bpf_prog *prog, union bpf_attr attr, bpfptr_t uattr,\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\n \n+\t/* must precede the first bpf_ret_reg_pair() user below /\n+\tbpf_compute_subprog_ret_regs(env);\n+\n \tret = bpf_compute_live_registers(env);\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\ndiff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c\nnew file mode 100644\nindex 0000000000000..c295adedbae85\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c\n@@ -0,0 +1,17 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003ctest_progs.h\u003e\n+#include \"aggregate_ret_int128_c.skel.h\"\n+#include \"aggregate_ret_struct_c.skel.h\"\n+#include \"aggregate_ret_kfunc_c.skel.h\"\n+#include \"aggregate_ret_func.skel.h\"\n+#include \"aggregate_ret_kfunc.skel.h\"\n+\n+void test_aggregate_ret(void)\n+{\n+\tRUN_TESTS(aggregate_ret_int128_c);\n+\tRUN_TESTS(aggregate_ret_struct_c);\n+\tRUN_TESTS(aggregate_ret_kfunc_c);\n+\tRUN_TESTS(aggregate_ret_func);\n+\tRUN_TESTS(aggregate_ret_kfunc);\n+}\ndiff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\nindex 2523c07a16c65..0b54f911015c5 100644\n--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\n+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\n@@ -441,6 +441,19 @@ static void test_func_replace_int_with_void(void)\n \t\t\t\t \" doesn't match type INT of global_func2()\");\n }\n \n+static void test_func_replace_ret_pair(void)\n+{\n+\tconst char msg = \"Cannot replace function agg_ret_target_func with a \u003e8 byte return\";\n+\n+\t/\n+\t An extension cannot replace a function whose return value comes back\n+\t * in the R0:R2 pair: the extension's own return is capped at 8 bytes,\n+\t * so it would leave R2 stale for the target's callers.\n+\t /\n+\ttest_obj_load_failure_common(\"freplace_ret_pair.bpf.o\",\n+\t\t\t\t \"./aggregate_ret_target.bpf.o\", msg);\n+}\n+\n static int find_prog_btf_id(const char name, __u32 attach_prog_fd)\n {\n \tstruct bpf_prog_info info = {};\n@@ -660,6 +673,8 @@ void serial_test_fexit_bpf2bpf(void)\n \t\ttest_func_replace_progmap();\n \tif (test__start_subtest(\"freplace_int_with_void\"))\n \t\ttest_func_replace_int_with_void();\n+\tif (test__start_subtest(\"freplace_ret_pair\"))\n+\t\ttest_func_replace_ret_pair();\n \tif (test__start_subtest(\"freplace_void\"))\n \t\ttest_func_replace_void();\n \tif (test__start_subtest(\"sleepable_fentry_to_xdp\"))\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c\nnew file mode 100644\nindex 0000000000000..cfb21bcf704be\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c\n@@ -0,0 +1,260 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+typedef unsigned __int128 u128;\n+\n+__naked u128 global_agg_good(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0x1234;\"\t/ low 64 bits /\n+\t\"r2 = 0x5678;\"\t/ high 64 bits /\n+\t\"exit;\"\n+\t);\n+}\n+\n+__naked u128 global_agg_bad(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+__naked u128 global_agg_bad_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = r10;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_global_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_bad];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_bad)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"At subprogram exit the register R2 is not a scalar value\")\n+__naked int aggregate_ret_global_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_bad_ptr];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_bad_ptr)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_bad_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = r10;\"\t/ stack pointer placed in the second return register /\n+\t\"exit;\"\n+\t);\n+}\n+\n+/\n+ * R2 is a return register once the subprogram returns a pair, so a stack\n+ * pointer left in it is rejected at the callee's exit exactly as one in R0\n+ * is: the callee frame is gone by the time the caller could use it.\n+ /\n+SEC(\"tc\")\n+__failure __msg(\"cannot return stack pointer to the caller\")\n+__naked int aggregate_ret_static_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_bad_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_bad_ptr)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_no_r2(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_static_uninit_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_no_r2];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_no_r2)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_precise(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 4;\"\t/ second half; its value is made precise below /\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 2: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 12: (95) exit\")\n+__msg(\"mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4\")\n+__naked int aggregate_ret_static_precise(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_precise];\"\n+\t\"r6 = r2;\"\t\t/ derived from the aggregate's second half /\n+\t\"r6 \u0026= 7;\"\t\t/ keep it in [0, 7] to index the stack /\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/ ptr += scalar marks r6 (hence R2) precise /\n+\t\"r0 = 0;\"\n+\t\"(u8 )(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_precise)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 2: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9\")\n+__naked int aggregate_ret_global_precise(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_good];\"\n+\t\"r6 = r2;\"\t\t/ derived from the aggregate's second half /\n+\t\"r6 \u0026= 7;\"\t\t/ keep it in [0, 7] to index the stack /\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/ ptr += scalar marks r6 (hence R2) precise /\n+\t\"r0 = 0;\"\n+\t\"(u8 )(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_good)\n+\t: __clobber_all);\n+}\n+\n+/ A by-value struct that smuggles a pointer, which must be rejected. /\n+struct with_ptr {\n+\tvoid p;\n+\t__u64 x;\n+};\n+\n+/* A by-value union that smuggles a pointer, which must be rejected too. /\n+union upair_with_ptr {\n+\tvoid p;\n+\t__u64 halves[2];\n+};\n+\n+__naked struct with_ptr global_ret_struct_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"Global function global_ret_struct_ptr() has unsupported return type\")\n+__naked int aggregate_ret_global_struct_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_ret_struct_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_ret_struct_ptr)\n+\t: __clobber_all);\n+}\n+\n+__naked union upair_with_ptr global_ret_union_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"Global function global_ret_union_ptr() has unsupported return type\")\n+__naked int aggregate_ret_global_union_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_ret_union_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_ret_union_ptr)\n+\t: __clobber_all);\n+}\n+\n+struct ptr_pair {\n+\tvoid p;\n+\t__u64 x;\n+};\n+\n+static __naked __noinline struct ptr_pair static_ret_ptr_pair(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = r1;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+__naked int aggregate_ret_static_ptr_pair(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_ret_ptr_pair];\"\n+\t\"r1 = (u32 )(r2 + 0);\"\t/ deref the returned ctx pointer /\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_ret_ptr_pair)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c\nnew file mode 100644\nindex 0000000000000..913cc374215d5\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c\n@@ -0,0 +1,49 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+#if defined(__clang_major__) \u0026\u0026 __clang_major__ \u003e= 23\n+\n+#define MIX_A\t0xdeadbeefcafef00dULL\n+#define MIX_B\t0x0123456789abcdefULL\n+\n+typedef unsigned __int128 u128;\n+\n+static __noinline u128 make_i128(__u64 a, __u64 b)\n+{\n+\treturn ((u128)(a + b) \u003c\u003c 64) \| (a - b);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_int128_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tu128 v;\n+\n+\tv = make_i128(a, b);\n+\tif ((__u64)(v \u003e\u003e 64) != a + b)\n+\t\treturn 1;\n+\tif ((__u64)v != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+SEC(\"socket\")\n+__description(\"aggregate_ret_int128_c: needs LLVM 23, dummy test\")\n+__success\n+int dummy_test(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c\nnew file mode 100644\nindex 0000000000000..c23b4beb1773c\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c\n@@ -0,0 +1,122 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+#include \"../test_kmods/bpf_testmod_kfunc.h\"\n+\n+/\n+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address\n+ * (rather than calling) avoids any dependence on the compiler lowering an\n+ * __int128 or struct return value, which the BPF backend only supports from\n+ * LLVM 23 on.\n+ /\n+void __kfunc_btf_root(void)\n+{\n+\tasm volatile (\"\"\n+\t:\n+\t: \"r\"(\u0026bpf_kfunc_call_test_i128),\n+\t \"r\"(\u0026bpf_kfunc_call_test_ret_fastcall),\n+\t \"r\"(\u0026bpf_kfunc_call_test_ret_ptr),\n+\t \"r\"(\u0026bpf_kfunc_call_test_ret_ii),\n+\t \"r\"(\u0026bpf_kfunc_call_test_ret_big));\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128\")\n+__naked int aggregate_ret_kfunc_precise(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 1;\"\n+\t\"r2 = 2;\"\n+\t\"call %[bpf_kfunc_call_test_i128];\"\n+\t\"r6 = r2;\"\t\t/ second return half /\n+\t\"r6 \u0026= 7;\"\t\t/ keep it in [0, 7] to index the stack /\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/ ptr += scalar marks r6 (hence R2) precise /\n+\t\"r0 = 0;\"\n+\t\"(u8 )(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_i128)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"kfunc bpf_kfunc_call_test_ret_fastcall with \u003e8-byte return is not supported with KF_FASTCALL\")\n+__naked int aggregate_ret_kfunc_fastcall_fail(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 1;\"\n+\t\"r2 = 2;\"\n+\t\"call %[bpf_kfunc_call_test_ret_fastcall];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_fastcall)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"is not composed of scalars\")\n+__naked int aggregate_ret_kfunc_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 0;\"\n+\t\"call %[bpf_kfunc_call_test_ret_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_ptr)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_kfunc_small_no_r2(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"call %[bpf_kfunc_call_test_ret_ii];\"\n+\t\"r0 = r2;\"\t/ R2 is not a return register for a \u003c=8 byte struct /\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_ii)\n+\t: __clobber_all);\n+}\n+\n+/\n+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by\n+ * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks,\n+ * so this behaves the same on every architecture.\n+ /\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported\")\n+__naked int aggregate_ret_kfunc_too_big_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[bpf_kfunc_call_test_ret_big];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_big)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c\nnew file mode 100644\nindex 0000000000000..2c1889fc28efd\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c\n@@ -0,0 +1,125 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"../test_kmods/bpf_testmod_kfunc.h\"\n+#include \"bpf_misc.h\"\n+\n+#if defined(__clang_major__) \u0026\u0026 __clang_major__ \u003e= 23\n+\n+#define MIX_A\t0xdeadbeefcafef00dULL\n+#define MIX_B\t0x0123456789abcdefULL\n+\n+typedef unsigned __int128 u128;\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tu128 v;\n+\n+\tv = bpf_kfunc_call_test_i128(a, b);\n+\tif ((__u64)(v \u003e\u003e 64) != a + b)\n+\t\treturn 1;\n+\tif ((__u64)v != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct prog_test_ret_pair p;\n+\n+\tp = bpf_kfunc_call_test_ret_pair(a, b);\n+\tif (p.hi != a + b)\n+\t\treturn 1;\n+\tif (p.lo != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+/ struct { u64 a; int b; }: 16 bytes, R0 = a, R2 = b. /\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_li_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\tint b = skb-\u003elen ^ MIX_B;\n+\tstruct prog_test_ret_li r;\n+\n+\tr = bpf_kfunc_call_test_ret_li(a, b);\n+\tif (r.a != a)\n+\t\treturn 1;\n+\tif (r.b != ~b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not a return reg. /\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_ii_c_test(struct __sk_buff skb)\n+{\n+\tint a = skb-\u003elen ^ MIX_A;\n+\tint b = skb-\u003elen ^ MIX_B;\n+\tstruct prog_test_ret_ii r;\n+\n+\tr = bpf_kfunc_call_test_ret_ii(a, b);\n+\tif (r.a != a)\n+\t\treturn 1;\n+\tif (r.b != b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+/* A union of 16 bytes takes the same R0:R2 path as a struct. /\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_uu_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tunion prog_test_ret_uu r;\n+\n+\tr = bpf_kfunc_call_test_ret_uu(a, b);\n+\tif (r.parts.lo != a + b)\n+\t\treturn 1;\n+\tif (r.parts.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+SEC(\"socket\")\n+__description(\"aggregate_ret_kfunc_c: needs LLVM 23, dummy test\")\n+__success\n+int dummy_test(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c\nnew file mode 100644\nindex 0000000000000..83b1a37519887\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c\n@@ -0,0 +1,114 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+#if defined(__clang_major__) \u0026\u0026 __clang_major__ \u003e= 23\n+\n+#define MIX_A\t0xdeadbeefcafef00dULL\n+#define MIX_B\t0x0123456789abcdefULL\n+\n+struct pair {\n+\t__u64 hi;\t/ R0 /\n+\t__u64 lo;\t/ R2 /\n+};\n+\n+union upair {\n+\t__u64 halves[2];\n+\tstruct {\n+\t\t__u64 lo;\t/ R0 /\n+\t\t__u64 hi;\t/ R2 /\n+\t} parts;\n+};\n+\n+static __noinline struct pair make_pair(__u64 a, __u64 b)\n+{\n+\tstruct pair p = { .hi = a + b, .lo = a - b };\n+\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_struct_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct pair p;\n+\n+\tp = make_pair(a, b);\n+\tif (p.hi != a + b)\n+\t\treturn 1;\n+\tif (p.lo != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+__noinline struct pair make_pair_global(__u64 a, __u64 b)\n+{\n+\tstruct pair p = { .hi = a + b, .lo = a - b };\n+\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_global_struct_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct pair p;\n+\n+\tp = make_pair_global(a, b);\n+\tif (p.hi != a + b)\n+\t\treturn 1;\n+\tif (p.lo != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+static __noinline union upair make_upair(__u64 a, __u64 b)\n+{\n+\tunion upair p;\n+\n+\tp.halves[0] = a + b;\n+\tp.halves[1] = a - b;\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_union_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tunion upair p;\n+\n+\tp = make_upair(a, b);\n+\tif (p.parts.lo != a + b)\n+\t\treturn 1;\n+\tif (p.parts.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+SEC(\"socket\")\n+__description(\"aggregate_ret_struct_c: needs LLVM 23, dummy test\")\n+__success\n+int dummy_test(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c\nnew file mode 100644\nindex 0000000000000..cffd8d7d3241a\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c\n@@ -0,0 +1,29 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+/ freplace target: a global subprogram returning 16 bytes in R0:R2. /\n+__naked unsigned __int128 agg_ret_target_func(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0x1234;\"\n+\t\"r2 = 0x5678;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__naked int agg_ret_target(void)\n+{\n+\tasm volatile (\n+\t\"call %[agg_ret_target_func];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(agg_ret_target_func)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c\nindex d055fc7b3b95d..0be9441ec273b 100644\n--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c\n+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c\n@@ -431,6 +431,36 @@ __naked void subprog1(void)\n \t\t::: __clobber_all);\n }\n \n+static __used __naked unsigned __int128 aux2(void)\n+{\n+\tasm volatile (\n+\t\t\"r0 = 1;\"\n+\t\t\"r2 = 2;\"\n+\t\t\"exit;\"\n+\t\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+/ A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). /\n+__load_if_JITed()\n+__log_level(2)\n+__msg(\"0: .12345.... (85) call pc+2\")\n+__msg(\"1: ..2....... (bf) r0 = r2\")\n+/ R2 is not read at the exit of this program, which returns an int, ... /\n+__msg(\"2: 0......... (95) exit\")\n+__msg(\"3: .......... (b7) r0 = 1\")\n+__msg(\"4: 0......... (b7) r2 = 2\")\n+/ ... but it is at the exit of aux2(), which returns a register pair. /\n+__msg(\"5: 0.2....... (95) exit\")\n+__naked void subprog_ret_reg_pair(void)\n+{\n+\tasm volatile (\n+\t\t\"call aux2;\"\n+\t\t\"r0 = r2;\"\n+\t\t\"exit;\"\n+\t\t::: __clobber_all);\n+}\n+\n #if defined(__TARGET_ARCH_x86) \|\| defined(__TARGET_ARCH_arm64)\n \n SEC(\"socket\")\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c\nindex ac44d60e50666..9708efb93683b 100644\n--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c\n+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c\n@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)\n \n SEC(\"?tc\")\n __exception_cb(exception_cb_bad_ret_type1)\n-__failure __msg(\"Global function exception_cb_bad_ret_type1() return value not void or scalar.\")\n+__failure __msg(\"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\")\n int reject_exception_cb_type_1(struct __sk_buff ctx)\n {\n \tbpf_throw(0);\ndiff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c\nnew file mode 100644\nindex 0000000000000..12c15d293bd79\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c\n@@ -0,0 +1,12 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+\n+SEC(\"freplace/agg_ret_target_func\")\n+__u64 new_agg_ret_target_func(void)\n+{\n+\treturn 0;\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\nindex a6133f7521f34..ad36d583d2e7f 100644\n--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\n+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\n@@ -939,6 +939,69 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)\n \treturn 0;\n }\n \n+#if defined(__x86_64__) \|\| defined(__aarch64__)\n+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)\n+{\n+\treturn (__int128)(((unsigned __int128)(a + b) \u003c\u003c 64) \| (a - b));\n+}\n+\n+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)\n+{\n+\tstruct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)\n+{\n+\tstruct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b)\n+{\n+\tstruct prog_test_ret_li r = { .a = a, .b = ~b };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b)\n+{\n+\tunion prog_test_ret_uu r;\n+\n+\tr.halves[0] = a + b;\n+\tr.halves[1] = a - b;\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)\n+{\n+\tstruct prog_test_ret_ptr r = { .p = NULL, .tag = tag };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)\n+{\n+\tstruct prog_test_ret_ii r = { .a = a, .b = b };\n+\n+\treturn r;\n+}\n+#endif / __x86_64__ \|\| __aarch64__ /\n+\n+/\n+ * Takes no argument on purpose: with no arguments there is nothing for the sret\n+ * pointer to displace, so this needs no architecture guard even though it\n+ * returns 24 bytes. See the comment on bpf_kfunc_call_test_i128() above.\n+ /\n+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)\n+{\n+\tstruct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };\n+\n+\treturn r;\n+}\n+\n __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,\n \t\t\t\t\t u64 e, u64 f, u64 g, u64 h,\n \t\t\t\t\t u64 i, u64 j)\n@@ -1472,6 +1535,16 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test3)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test4)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test5)\n+#if defined(__x86_64__) \|\| defined(__aarch64__)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)\n+#endif\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)\ndiff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\nindex c4383acb53c11..c7be973cd2860 100644\n--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\n+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\n@@ -55,6 +55,40 @@ struct prog_test_big_arg {\n \t__u64 b;\n };\n \n+struct prog_test_ret_pair {\n+\t__u64 hi;\n+\t__u64 lo;\n+};\n+\n+struct prog_test_ret_li {\t/ 16 bytes: R0:R2 /\n+\t__u64 a;\n+\tint b;\n+};\n+\n+struct prog_test_ret_ii {\t/ 8 bytes: R0 only /\n+\tint a;\n+\tint b;\n+};\n+\n+union prog_test_ret_uu {\t/ 16 bytes: R0:R2 /\n+\t__u64 halves[2];\n+\tstruct {\n+\t\t__u64 lo;\n+\t\t__u64 hi;\n+\t} parts;\n+};\n+\n+struct prog_test_ret_ptr {\t/ 16 bytes: contains a pointer /\n+\tvoid p;\n+\t__u64 tag;\n+};\n+\n+struct prog_test_ret_big {\t/* 24 bytes: too large for R0:R2 /\n+\t__u64 a;\n+\t__u64 b;\n+\t__u64 c;\n+};\n+\n struct prog_test_fail1 {\n \tvoid p;\n \tint x;\n@@ -130,6 +164,16 @@ int bpf_kfunc_call_test2(struct sock sk, __u32 a, __u32 b) __ksym;\n struct sock bpf_kfunc_call_test3(struct sock *sk) __ksym;\n long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;\n int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;\n+#ifdef __SIZEOF_INT128__\n+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;\n+#endif\n+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;\n+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;\n+struct prog_test_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym;\n+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;\n+union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym;\n+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;\n+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym;\n __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,\n \t\t\t __u64 e, __u64 f, __u64 g, __u64 h,\n \t\t\t __u64 i, __u64 j) __ksym;\n" }
2/1	2026/08/13 20:58	agent	patch-evaluator	0m Model: best-expensive Args: null Results: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT" ], "FocusSymbols": [ "bpf_kfunc_ret_reg_pair", "btf_type_is_scalar_struct", "bpf_jit_supports_kfunc_ret_reg_pair", "bpf_compute_live_registers" ], "Reasoning": "The patch introduces support for kfuncs and BPF subprograms returning larger values (up to 16 bytes) in the R0:R2 register pair. This involves changes to the BPF verifier (liveness tracking, return value checking, BTF validation) and JIT compilers (x86, arm64, riscv). These are significant functional changes to the core BPF subsystem that should be fuzzed to ensure verifier safety and JIT correctness.", "WorthFuzzing": true } Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit bafb5dc76d94a94d43e7b1575db7d4920a549687 Author: syz-cluster <triage@syzkaller.com> Date: Thu Aug 13 20:58:12 2026 +0000 syz-cluster: applied patch under review diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 10e725cbe64cd..5d840d3000338 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -575,6 +575,69 @@ is also covered by this recovery. A kfunc handed an arena pointer may therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking against the arena. Larger accesses must verify the range explicitly. +2.9 kfunc Return Values +----------------------- + +A kfunc may return a scalar, a pointer, or a small struct or union by +value. A scalar or pointer of up to 8 bytes is returned in R0, as usual. + +A struct or union returned by value must be composed only of scalars +(recursively), where a scalar is an integer or an enum; arrays of scalars are +allowed as members. Its bytes are handed back to the program as the raw +contents of R0 (and R2), so a pointer field would be laundered into a scalar +and escape the verifier's pointer provenance and reference tracking. A struct +or union with a pointer member is therefore rejected at load time, and so is +one with a floating-point member, which the ABI may not return in R0:R2 at +all. + +A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a +scalar-only struct or union, or an ``__int128``. Such a value is returned +in the register pair R0:R2, matching the convention LLVM uses for the BPF +target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or +union of 8 bytes or less is returned in R0 alone. + +:: + + struct bpf_pair { __u64 a, b; }; /* 16 bytes / + + __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void) + { + struct bpf_pair p = { .a = 1, .b = 2 }; + + return p; / p.a in R0, p.b in R2 / + } + +Returning a value in the R0:R2 pair requires the JIT to place the second +half of the return value into R2, which not every architecture supports +right now. A kfunc with a return value larger than 8 bytes is therefore +rejected at load time on a JIT that does not advertise this capability (see +``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run +by the interpreter. A return value larger than 16 bytes is not supported. + +The same R0:R2 convention applies to a BPF subprogram, global or static, +that returns an ``__int128`` or a struct or union larger than 8 bytes. Such a +program also requires the JIT, since the interpreter propagates only R0 out +of a subprogram. A global subprogram is verified in isolation, so its +by-value struct or union return is restricted to scalars just like a kfunc's; +a static subprogram is verified inline and has no such restriction. The main +program is not covered: its return value is the program's exit code, read out +of R0 alone, so a declared upper half is never looked at. + +A global subprogram must leave a scalar in every* register of the pair, so +both halves of the returned value have to be assigned. Leaving the upper half +uninitialized is not merely untidy: the compiler is then free to leave R2 +holding whatever it happened to hold, which for a subprogram taking a pointer +argument is typically that pointer. Handing the caller an unknown scalar built +from a pointer is a leak, so the verifier rejects it with:: + + At subprogram exit the register R2 is not a scalar value (...) + +Initialize the whole return value, for example ``struct pair p = {};``, to +avoid this. A static subprogram is exempt from the scalar-only rule: it is +verified inline, so an unassigned R2 is simply passed back to the caller as +uninitialized and only a caller that reads it fails. A stack pointer left in +R2 is still rejected there, just as one in R0 is. + .. _BPF_kfunc_lifecycle_expectations: 3. kfunc lifecycle expectations diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 74b4083791da3..7a4c5968976a6 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2346,6 +2346,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_stack_args(void) { return true; diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 2504df1fa1118..49220765e96d4 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -2120,6 +2120,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_ptr_xchg(void) { return true; diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index d920772af7d5f..2ec09dd02b187 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip, * arena NULL is offset 0. Return the number of emitted bytes. / static int emit_kfunc_arena_args(struct bpf_prog bpf_prog, - const struct bpf_insn insn, u8 pprog) + const struct btf_func_model fm, u8 *pprog) { - const struct btf_func_model fm; u8 prog = pprog; u8 start = prog; int i; - fm = bpf_jit_find_kfunc_model(bpf_prog, insn); - if (!fm) - return -EINVAL; - for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) { u8 flags = fm->arg_flags[i]; u32 reg = BPF_REG_1 + i; @@ -2644,6 +2639,8 @@ st: insn_off = insn->off; / call / case BPF_JMP \| BPF_CALL: { + const struct btf_func_model fm = NULL; + func = (u8 ) __bpf_call_base + imm32; if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(stack_depth); @@ -2652,7 +2649,10 @@ st: insn_off = insn->off; if (!imm32) return -EINVAL; if (src_reg == BPF_PSEUDO_KFUNC_CALL) { - err = emit_kfunc_arena_args(bpf_prog, insn, &prog); + fm = bpf_jit_find_kfunc_model(bpf_prog, insn); + if (!fm) + return -EINVAL; + err = emit_kfunc_arena_args(bpf_prog, fm, &prog); if (err < 0) return err; ip += err; @@ -2666,6 +2666,14 @@ st: insn_off = insn->off; return -EINVAL; if (priv_frame_ptr) pop_r9(&prog); + / + * A kfunc returning more than 8 bytes hands the second + * half back in RDX (the native ABI's second return reg), + * but BPF expects it in R0:R2. BPF R0 is RAX (no move + * needed), while BPF R2 is RSI, so copy RDX into RSI. + / + if (fm && fm->ret_size > 8) + emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3); break; } @@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_stack_args(void) { return true; diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 27b43fda9b178..899e5ce986432 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -814,6 +814,8 @@ struct bpf_subprog_info { bool is_async_cb: 1; bool is_exception_cb: 1; bool args_cached: 1; + / true if the return value is passed in the R0:R2 register pair / + bool ret_reg_pair: 1; / true if bpf_fastcall stack region is used by functions that can't be inlined / bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; @@ -1048,6 +1050,13 @@ static inline struct bpf_subprog_info subprog_info(struct bpf_verifier_env env return &env->subprog_info[subprog]; } +static inline bool bpf_ret_reg_pair(struct bpf_verifier_env env, int subprog) +{ + return subprog_info(env, subprog)->ret_reg_pair; +} + +bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env env, struct bpf_insn insn); + struct bpf_call_summary { u8 num_params; bool is_void; @@ -1439,6 +1448,8 @@ int bpf_jmp_offset(struct bpf_insn insn); struct bpf_iarray bpf_insn_successors(struct bpf_verifier_env env, u32 idx); void bpf_fmt_stack_mask(char buf, ssize_t buf_sz, u64 stack_mask); bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog); +bool btf_type_is_scalar_struct(struct bpf_verifier_env env, const struct btf btf, + const struct btf_type t, int rec); int bpf_find_subprog(struct bpf_verifier_env env, int off); bool bpf_is_throw_kfunc(struct bpf_insn insn); diff --git a/include/linux/filter.h b/include/linux/filter.h index 4a9bc6a848f2e..6e746b0a09306 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_percpu_insn(void); bool bpf_jit_supports_kfunc_call(void); +bool bpf_jit_supports_kfunc_ret_reg_pair(void); bool bpf_jit_supports_stack_args(void); bool bpf_jit_supports_arena_args(void); bool bpf_jit_supports_far_kfunc_call(void); diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index 40bd04421a991..3ef8a3da726de 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -425,6 +425,16 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, / verifier_bug_if(idx + 1 != subseq_idx, env, "extra insn from subprog"); + /* a global subprog returning more than 8 bytes + * sets R2 as well. R2 is part of the args mask + * checked just below, so clear it here rather + * than next to R0. Only a subprog that does + * return a pair defines R2, so leave the mask + * alone otherwise and let the check below catch + * an R2 that has no business being set. + / + if (bpf_ret_reg_pair(env, subprog)) + bt_clear_reg(bt, BPF_REG_2); / r1-r5 are invalidated after subprog call, * so for global func call it shouldn't be set * anymore @@ -508,6 +518,17 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, return -ENOTSUPP; / regular helper call sets R0 / bt_clear_reg(bt, BPF_REG_0); + / a kfunc returning more than 8 bytes also sets R2. + * R2 is part of the args mask checked just below, so + * clear it here rather than next to R0. The prototype + * lookup is only worth doing when R2 is requested at + * all; any other call leaves R2 uninitialized, so a + * request for it is caught by the check below. + / + if (bt_is_reg_set(bt, BPF_REG_2) && + insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + bpf_kfunc_ret_reg_pair(env, insn)) + bt_clear_reg(bt, BPF_REG_2); if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { / if backtracking was looking for registers R1-R5 * they should have been found already. @@ -522,7 +543,41 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, return -EFAULT; } } else if (opcode == BPF_EXIT) { - bool r0_precise; + bool from_subprog_call, r0_precise, r2_precise; + struct bpf_insn call; + int subprog; + + /* BPF_EXIT in subprog or callback always returns + * right after the call instruction, so by checking + * whether the instruction at subseq_idx-1 is subprog + * call or not we can distinguish actual exit from + * subprog from exit from callback. In the former + * case, we need to propagate the precision of the + * return registers, if necessary. In the latter we + * never do that. + / + from_subprog_call = subseq_idx - 1 >= 0 && + bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]); + + / Sample the return registers before the callback + * handling below clears R1-R5: unlike R0, R2 is an + * argument register as well, so that clear would drop + * a pair return on the floor. + / + r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0); + r2_precise = false; + if (from_subprog_call && bt_is_reg_set(bt, BPF_REG_2)) { + call = &env->prog->insnsi[subseq_idx - 1]; + subprog = bpf_find_subprog(env, subseq_idx + call->imm); + if (subprog < 0) + return -EFAULT; + / Only a callee that does return a pair defines + * R2. Leave the mask alone otherwise, so that + * the check below still catches an R2 that has + * no business being set. + / + r2_precise = bpf_ret_reg_pair(env, subprog); + } / Backtracking to a nested function call, 'idx' is a part of * the inner frame 'subseq_idx' is a part of the outer frame. @@ -535,30 +590,27 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx)) for (i = BPF_REG_1; i <= BPF_REG_5; i++) bt_clear_reg(bt, i); + + / a callee returning more than 8 bytes sets R2 as well; + * R2 is part of the args mask checked just below, so + * clear it here rather than next to R0. + / + if (r2_precise) + bt_clear_reg(bt, BPF_REG_2); if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { verifier_bug(env, "backtracking exit unexpected regs %x", bt_reg_mask(bt)); return -EFAULT; } - / BPF_EXIT in subprog or callback always returns - * right after the call instruction, so by checking - * whether the instruction at subseq_idx-1 is subprog - * call or not we can distinguish actual exit from - * subprog from exit from callback. In the former - * case, we need to propagate r0 precision, if - * necessary. In the former we never do that. - / - r0_precise = subseq_idx - 1 >= 0 && - bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) && - bt_is_reg_set(bt, BPF_REG_0); - bt_clear_reg(bt, BPF_REG_0); if (bt_subprog_enter(bt)) return -EFAULT; if (r0_precise) bt_set_reg(bt, BPF_REG_0); + if (r2_precise) + bt_set_reg(bt, BPF_REG_2); / r6-r9 and stack slots will stay set in caller frame * bitmasks until we return back from callee(s) / diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 6606187ed4f43..c7718a2a56141 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7592,7 +7592,7 @@ int btf_distill_func_proto(struct bpf_verifier_log log, return -EINVAL; } ret = __get_type_size(btf, func->type, &t); - if (ret < 0 \|\| btf_type_is_struct(t)) { + if (ret < 0 \|\| ret > 16) { bpf_log(log, "The function %s return type %s is unsupported.\n", tname, btf_type_str(t)); @@ -7965,7 +7965,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env env, / Check whether the type is a valid return type. / static int btf_validate_return_type(struct bpf_verifier_env env, struct btf btf, - const struct btf_type t, int subprog) + const struct btf_type t, int subprog, bool is_global) { u32 tags = 0; int err; @@ -7988,6 +7988,19 @@ static int btf_validate_return_type(struct bpf_verifier_env env, struct btf bt if (btf_type_is_void(t) \|\| btf_type_is_int(t) \|\| btf_is_any_enum(t)) return 0; + if (btf_type_is_struct(t) && t->size <= 16) { + / + * A global function's caller models the return as an opaque + * scalar pair, so it may only return scalars by value. A local + * function is verified inline, so a pointer field stays tracked + * and needs no such restriction. + / + bool local_func = subprog && !is_global; + + if (local_func \|\| btf_type_is_scalar_struct(env, btf, t, 0)) + return 0; + } + return -EOPNOTSUPP; } @@ -8075,12 +8088,12 @@ int btf_prepare_func_args(struct bpf_verifier_env env, int subprog) return -EINVAL; } - err = btf_validate_return_type(env, btf, t, subprog); + err = btf_validate_return_type(env, btf, t, subprog, is_global); if (err) { if (is_global) { bpf_log(log, - "Global function %s() return value not void or scalar. " - "Only those are supported.\n", + "Global function %s() has unsupported return type. " + "Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n", tname); } return err; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 6a94370a24488..cb66a2ef52b1f 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void) return false; } +bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return false; +} + bool __weak bpf_jit_supports_stack_args(void) { return false; diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index 1c997aeba6fa5..850c58aa915be 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -2062,7 +2062,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); } /* Compute info->{use,def} fields for the instruction / static void compute_insn_live_regs(struct bpf_verifier_env env, struct bpf_insn insn, - struct insn_live_regs info) + struct insn_live_regs info, + bool ret_reg_pair) { struct bpf_call_summary cs; const u8 class = BPF_CLASS(insn->code); @@ -2074,6 +2075,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env, const u32 src32 = mask_lo(src); const u32 dst32 = mask_lo(dst); const u32 r0 = reg64_mask(0); + const u32 r2 = reg64_mask(BPF_REG_2); u32 def = 0; u32 use = U32_MAX; @@ -2193,7 +2195,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env, break; case BPF_EXIT: def = 0; - use = r0; + use = ret_reg_pair ? (r0 \| r2) : r0; break; case BPF_CALL: def = ALL_CALLER_SAVED_REGS; @@ -2230,8 +2232,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env env) struct insn_live_regs state; int insn_cnt = env->prog->len; u64 pos, insn_pos; - int err = 0, i, j; - bool changed; + int err = 0, i, j, subprog, start, end; + bool changed, ret_reg_pair; / Use the following algorithm: * - define the following: @@ -2258,8 +2260,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env env) goto out; } - for (i = 0; i < insn_cnt; ++i) - compute_insn_live_regs(env, &insns[i], &state[i]); + for (subprog = 0; subprog < env->subprog_cnt; subprog++) { + start = env->subprog_info[subprog].start; + end = env->subprog_info[subprog + 1].start; + ret_reg_pair = bpf_ret_reg_pair(env, subprog); + + for (i = start; i < end; ++i) + compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair); + } / Forward pass: resolve stack access through FP-derived pointers / err = bpf_compute_subprog_arg_access(env); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 164d16c243ca6..98a6f702d794d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -382,27 +382,80 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog) return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL; } -static bool subprog_returns_void(struct bpf_verifier_env env, int subprog) +static const struct btf_type subprog_ret_type(struct bpf_verifier_env env, int subprog) { - const struct btf_type type, func, func_proto; + const struct btf_type func, func_proto; const struct btf btf = env->prog->aux->btf; u32 btf_id; + if (!btf \|\| !env->prog->aux->func_info) + return NULL; + btf_id = env->prog->aux->func_info[subprog].type_id; + / Both already validated by prepare_btf_func() at prog load. / func = btf_type_by_id(btf, btf_id); - if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id)) - return false; - func_proto = btf_type_by_id(btf, func->type); - if (!func_proto) - return false; - type = btf_type_skip_modifiers(btf, func_proto->type, NULL); - if (!type) - return false; + return btf_type_skip_modifiers(btf, func_proto->type, NULL); +} + +static bool subprog_returns_void(struct bpf_verifier_env env, int subprog) +{ + const struct btf_type type = subprog_ret_type(env, subprog); + + return type && btf_type_is_void(type); +} + +static u32 ret_regs_cnt(u32 size) +{ + return size > 8 && size <= 16 ? 2 : 1; +} + +/ Registers holding a function return value, in order. See ret_regs_cnt(). / +static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 }; + +static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env env) +{ + const struct btf btf = env->prog->aux->btf; + const struct btf_type type; + int subprog; + u32 size; + + for (subprog = 0; subprog < env->subprog_cnt; subprog++) { + type = subprog_ret_type(env, subprog); + if (!type \|\| !(btf_type_is_struct(type) \|\| btf_type_is_scalar(type))) + continue; + if (IS_ERR(btf_resolve_size(btf, type, &size))) + continue; + if (ret_regs_cnt(size) > 1) { + subprog_info(env, subprog)->ret_reg_pair = true; + /* + * The R0:R2 return convention is only implemented in + * the JIT: the interpreter propagates BPF_R0 alone out + * of a subprogram, so a caller reading R2 would see a + * stale value. + / + env->prog->jit_required = 1; + } + } +} - return btf_type_is_void(type); +/ + * A >8 byte BPF return changes the calling convention to R0:R2, and the + * verifier derives that convention from the subprogram's BTF prototype + * alone. Once that prototype is marked unreliable it is known not to + * describe the compiled code, so the convention read from it cannot be + * trusted either: reject the call rather than keep tracking R2 on the + * strength of a signature the verifier has already discarded. + / +static bool subprog_ret_pair_unreliable(struct bpf_verifier_env env, int subprog) +{ + struct bpf_prog_aux aux = env->prog->aux; + + return bpf_ret_reg_pair(env, subprog) && + aux->func_info_aux && + aux->func_info_aux[subprog].unreliable; } static const char subprog_name(const struct bpf_verifier_env env, int subprog) @@ -2459,6 +2512,21 @@ find_kfunc_desc(const struct bpf_prog prog, u32 func_id, u16 offset) sizeof(tab->descs[0]), kfunc_desc_cmp_by_id_off); } +/* + * True if the kfunc called by @insn returns its value in the R0:R2 pair. + * Reads the same btf_func_model.ret_size that bpf_add_kfunc_call() validated + * and that the JIT keys the second return register off, so the verifier and + * the generated code cannot disagree about the convention. + / +bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env env, struct bpf_insn insn) +{ + const struct bpf_kfunc_desc desc; + + desc = find_kfunc_desc(env->prog, insn->imm, insn->off); + + return desc && ret_regs_cnt(desc->func_model.ret_size) > 1; +} + int bpf_get_kfunc_addr(const struct bpf_prog prog, u32 func_id, u16 btf_fd_idx, u8 func_addr) { @@ -2809,6 +2877,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env env, u32 func_id, u16 offset) err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model); if (err) return err; + if (func_model.ret_size > 8) { + if (kfunc.flags && (kfunc.flags & KF_FASTCALL)) { + verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n", + kfunc.name); + return -EOPNOTSUPP; + } + if (!bpf_jit_supports_kfunc_ret_reg_pair()) { + verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n", + kfunc.name); + return -EOPNOTSUPP; + } + } memset(&meta, 0, sizeof(meta)); meta.btf = kfunc.btf; @@ -9381,6 +9461,7 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn, u16 callee_incoming, stack_arg_cnt; struct bpf_func_state caller; int err, subprog, target_insn; + u32 i, nregs; target_insn = insn_idx + insn->imm + 1; subprog = bpf_find_subprog(env, target_insn); @@ -9423,9 +9504,14 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn, clear_caller_saved_regs(env, caller->regs); invalidate_outgoing_stack_args(env, cur_func(env)); - / All non-void global functions return a 64-bit SCALAR_VALUE. / + / + * A non-void global function returns a 64-bit SCALAR_VALUE in + * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair. + / if (!subprog_returns_void(env, subprog)) { - mark_reg_unknown(env, caller->regs, BPF_REG_0); + nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1; + for (i = 0; i < nregs; i++) + mark_reg_unknown(env, caller->regs, ret_regs[i]); } if (env->subprog_info[subprog].might_throw) { @@ -9443,6 +9529,12 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn, return 0; } + if (subprog_ret_pair_unreliable(env, subprog)) { + verbose(env, "Func#%d ('%s') returns >8 bytes, which requires reliable BTF\n", + subprog, subprog_name(env, subprog)); + return -EINVAL; + } + / * Track caller's total stack arg count (incoming + max outgoing). * This is needed so the JIT knows how much stack arg space to allocate. @@ -9783,11 +9875,15 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx) struct bpf_func_state caller, callee; struct bpf_reg_state r0; bool in_callback_fn; + u32 i, nregs; int err; callee = state->frame[state->curframe]; r0 = &callee->regs[BPF_REG_0]; - if (r0->type == PTR_TO_STACK) { + nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1; + for (i = 0; i < nregs; i++) { + if (callee->regs[ret_regs[i]].type != PTR_TO_STACK) + continue; / technically it's ok to return caller's stack pointer * (or caller's caller's pointer) back to the caller, * since these pointers are valid. Only current stack @@ -9822,8 +9918,12 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx) return -EFAULT; } } else { - /* return to the caller whatever r0 had in the callee / - caller->regs[BPF_REG_0] = r0; + /* + * return to the caller whatever the callee had in the + * return register(s) + / + for (i = 0; i < nregs; i++) + caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]]; } / for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite, @@ -10721,6 +10821,19 @@ static int check_helper_call(struct bpf_verifier_env env, struct bpf_insn insn return 0; } +/* + * Mark the register(s) holding a @size byte kfunc return value as unknown + * scalars. Both halves of a register pair are treated the same way. + / +static void mark_kfunc_ret_regs(struct bpf_verifier_env env, + struct bpf_reg_state regs, u32 size) +{ + u32 i, nregs = ret_regs_cnt(size); + + for (i = 0; i < nregs; i++) + mark_reg_unknown(env, regs, ret_regs[i]); +} + static bool is_kfunc_acquire(struct bpf_call_arg_meta meta) { return meta->kfunc_flags & KF_ACQUIRE; @@ -10988,9 +11101,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta meta, u32 arg_ } / Returns true if struct is composed of scalars, 4 levels of nesting allowed / -static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env, - const struct btf btf, - const struct btf_type t, int rec) +bool btf_type_is_scalar_struct(struct bpf_verifier_env env, + const struct btf btf, + const struct btf_type t, int rec) { const struct btf_type member_type; const struct btf_member member; @@ -11008,7 +11121,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env, verbose(env, "max struct nesting depth exceeded\n"); return false; } - if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1)) + if (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1)) return false; continue; } @@ -11407,7 +11520,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta, (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) \|\| is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) { if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) && - !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n", reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); return -EINVAL; @@ -11423,7 +11536,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta, * scalars. The access size is derived from the pointed-to BTF type. / if (!btf_type_is_scalar(ref_t) && - !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n", reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); return -EINVAL; @@ -12394,7 +12507,7 @@ static int check_kfunc_args(struct bpf_verifier_env env, struct bpf_call_arg_me break; } - if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + if (!btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id); verbose(env, "%s is %s expected %s %s", @@ -12858,7 +12971,7 @@ static int check_special_kfunc(struct bpf_verifier_env env, struct bpf_call_arg struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id); if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) { - if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) { + if (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) { verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n"); return -EINVAL; } @@ -13191,10 +13304,25 @@ static int check_kfunc_call(struct bpf_verifier_env env, struct bpf_insn insn, } if (btf_type_is_scalar(t)) { - mark_reg_unknown(env, regs, BPF_REG_0); + mark_kfunc_ret_regs(env, regs, t->size); if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] \|\| meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave])) __mark_reg_const_zero(env, &regs[BPF_REG_0]); + } else if (btf_type_is_struct(t)) { + / + * The returned struct comes back as raw register bits modeled + * as an unknown scalar, so it must contain only scalars: + * otherwise a pointer field would be laundered into a scalar + * and escape provenance and reference tracking. + / + if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) { + verbose(env, + "kernel function %s returns %s %s that is not composed of scalars\n", + func_name, btf_type_str(t), + btf_name_by_offset(desc_btf, t->name_off)); + return -EINVAL; + } + mark_kfunc_ret_regs(env, regs, t->size); } else if (btf_type_is_ptr(t)) { ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id); err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf); @@ -16641,37 +16769,54 @@ static int check_return_code(struct bpf_verifier_env env, int regno, const char return 0; } -static int check_global_subprog_return_code(struct bpf_verifier_env env) +static int check_global_ret_scalar_reg(struct bpf_verifier_env env, u32 regno) { - struct bpf_reg_state reg = reg_state(env, BPF_REG_0); - struct bpf_func_state cur_frame = cur_func(env); + struct bpf_reg_state reg; int err; - if (subprog_returns_void(env, cur_frame->subprogno)) - return 0; - - err = check_reg_arg(env, BPF_REG_0, SRC_OP); + err = check_reg_arg(env, regno, SRC_OP); if (err) return err; / Pointers to arena are safe to pass between subprograms. / - if (is_arena_reg(env, BPF_REG_0)) + if (is_arena_reg(env, regno)) return 0; - if (is_pointer_value(env, BPF_REG_0)) { - verbose(env, "R%d leaks addr as return value\n", BPF_REG_0); + if (is_pointer_value(env, regno)) { + verbose(env, "R%d leaks addr as return value\n", regno); return -EACCES; } + reg = reg_state(env, regno); if (reg->type != SCALAR_VALUE) { - verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n", - reg_type_str(env, reg->type)); + verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n", + regno, reg_type_str(env, reg->type)); return -EINVAL; } return 0; } +static int check_global_subprog_return_code(struct bpf_verifier_env env) +{ + struct bpf_func_state cur_frame = cur_func(env); + u32 subprog = cur_frame->subprogno; + u32 i, nregs; + int err; + + if (subprog_returns_void(env, subprog)) + return 0; + + nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1; + for (i = 0; i < nregs; i++) { + err = check_global_ret_scalar_reg(env, ret_regs[i]); + if (err) + return err; + } + + return 0; +} + / Bitmask with 1s for all caller saved registers / #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1) @@ -19328,6 +19473,13 @@ int bpf_check_attach_target(struct bpf_verifier_log log, return -EOPNOTSUPP; } + if (prog_extension && tgt_info->fmodel.ret_size > 8) { + bpf_log(log, + "Cannot replace function %s with a >8 byte return value\n", + tname); + return -EOPNOTSUPP; + } + /* * .multi programs don't need an address during program verification, we just take the module ref if needed. @@ -20317,6 +20469,9 @@ int bpf_check(struct bpf_prog *prog, union bpf_attr attr, bpfptr_t uattr, if (ret < 0) goto skip_full_check; + /* must precede the first bpf_ret_reg_pair() user below / + bpf_compute_subprog_ret_regs(env); + ret = bpf_compute_live_registers(env); if (ret < 0) goto skip_full_check; diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file mode 100644 index 0000000000000..c295adedbae85 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <test_progs.h> +#include "aggregate_ret_int128_c.skel.h" +#include "aggregate_ret_struct_c.skel.h" +#include "aggregate_ret_kfunc_c.skel.h" +#include "aggregate_ret_func.skel.h" +#include "aggregate_ret_kfunc.skel.h" + +void test_aggregate_ret(void) +{ + RUN_TESTS(aggregate_ret_int128_c); + RUN_TESTS(aggregate_ret_struct_c); + RUN_TESTS(aggregate_ret_kfunc_c); + RUN_TESTS(aggregate_ret_func); + RUN_TESTS(aggregate_ret_kfunc); +} diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c index 2523c07a16c65..0b54f911015c5 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c @@ -441,6 +441,19 @@ static void test_func_replace_int_with_void(void) " doesn't match type INT of global_func2()"); } +static void test_func_replace_ret_pair(void) +{ + const char msg = "Cannot replace function agg_ret_target_func with a >8 byte return"; + + /* + * An extension cannot replace a function whose return value comes back + * in the R0:R2 pair: the extension's own return is capped at 8 bytes, + * so it would leave R2 stale for the target's callers. + / + test_obj_load_failure_common("freplace_ret_pair.bpf.o", + "./aggregate_ret_target.bpf.o", msg); +} + static int find_prog_btf_id(const char name, __u32 attach_prog_fd) { struct bpf_prog_info info = {}; @@ -660,6 +673,8 @@ void serial_test_fexit_bpf2bpf(void) test_func_replace_progmap(); if (test__start_subtest("freplace_int_with_void")) test_func_replace_int_with_void(); + if (test__start_subtest("freplace_ret_pair")) + test_func_replace_ret_pair(); if (test__start_subtest("freplace_void")) test_func_replace_void(); if (test__start_subtest("sleepable_fentry_to_xdp")) diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file mode 100644 index 0000000000000..cfb21bcf704be --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c @@ -0,0 +1,260 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +typedef unsigned __int128 u128; + +__naked u128 global_agg_good(void) +{ + asm volatile ( + "r0 = 0x1234;" / low 64 bits / + "r2 = 0x5678;" / high 64 bits / + "exit;" + ); +} + +__naked u128 global_agg_bad(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +__naked u128 global_agg_bad_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = r10;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_global_fail(void) +{ + asm volatile ( + "call %[global_agg_bad];" + "r0 = r2;" + "exit;" + : + : __imm(global_agg_bad) + : __clobber_all); +} + +SEC("tc") +__failure __msg("At subprogram exit the register R2 is not a scalar value") +__naked int aggregate_ret_global_ptr_fail(void) +{ + asm volatile ( + "call %[global_agg_bad_ptr];" + "r0 = r2;" + "exit;" + : + : __imm(global_agg_bad_ptr) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_bad_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = r10;" / stack pointer placed in the second return register / + "exit;" + ); +} + +/ + * R2 is a return register once the subprogram returns a pair, so a stack + * pointer left in it is rejected at the callee's exit exactly as one in R0 + * is: the callee frame is gone by the time the caller could use it. + / +SEC("tc") +__failure __msg("cannot return stack pointer to the caller") +__naked int aggregate_ret_static_ptr_fail(void) +{ + asm volatile ( + "call %[static_agg_bad_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(static_agg_bad_ptr) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_no_r2(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_static_uninit_fail(void) +{ + asm volatile ( + "call %[static_agg_no_r2];" + "r0 = r2;" + "exit;" + : + : __imm(static_agg_no_r2) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_precise(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 4;" / second half; its value is made precise below / + "exit;" + ); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit") +__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4") +__naked int aggregate_ret_static_precise(void) +{ + asm volatile ( + "call %[static_agg_precise];" + "r6 = r2;" / derived from the aggregate's second half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(static_agg_precise) + : __clobber_all); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9") +__naked int aggregate_ret_global_precise(void) +{ + asm volatile ( + "call %[global_agg_good];" + "r6 = r2;" / derived from the aggregate's second half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(global_agg_good) + : __clobber_all); +} + +/ A by-value struct that smuggles a pointer, which must be rejected. / +struct with_ptr { + void p; + __u64 x; +}; + +/* A by-value union that smuggles a pointer, which must be rejected too. / +union upair_with_ptr { + void p; + __u64 halves[2]; +}; + +__naked struct with_ptr global_ret_struct_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("Global function global_ret_struct_ptr() has unsupported return type") +__naked int aggregate_ret_global_struct_ptr_fail(void) +{ + asm volatile ( + "call %[global_ret_struct_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(global_ret_struct_ptr) + : __clobber_all); +} + +__naked union upair_with_ptr global_ret_union_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("Global function global_ret_union_ptr() has unsupported return type") +__naked int aggregate_ret_global_union_ptr_fail(void) +{ + asm volatile ( + "call %[global_ret_union_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(global_ret_union_ptr) + : __clobber_all); +} + +struct ptr_pair { + void p; + __u64 x; +}; + +static __naked __noinline struct ptr_pair static_ret_ptr_pair(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = r1;" + "exit;" + ); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +__naked int aggregate_ret_static_ptr_pair(void) +{ + asm volatile ( + "call %[static_ret_ptr_pair];" + "r1 = (u32 )(r2 + 0);" / deref the returned ctx pointer / + "r0 = 0;" + "exit;" + : + : __imm(static_ret_ptr_pair) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c new file mode 100644 index 0000000000000..913cc374215d5 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#if defined(__clang_major__) && __clang_major__ >= 23 + +#define MIX_A 0xdeadbeefcafef00dULL +#define MIX_B 0x0123456789abcdefULL + +typedef unsigned __int128 u128; + +static __noinline u128 make_i128(__u64 a, __u64 b) +{ + return ((u128)(a + b) << 64) \| (a - b); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_int128_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + u128 v; + + v = make_i128(a, b); + if ((__u64)(v >> 64) != a + b) + return 1; + if ((__u64)v != a - b) + return 2; + + return 0; +} + +#else + +SEC("socket") +__description("aggregate_ret_int128_c: needs LLVM 23, dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c new file mode 100644 index 0000000000000..c23b4beb1773c --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c @@ -0,0 +1,122 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" +#include "../test_kmods/bpf_testmod_kfunc.h" + +/ + * Reference kfunc addresses to force those BTF to be emitted. Taking the address + * (rather than calling) avoids any dependence on the compiler lowering an + * __int128 or struct return value, which the BPF backend only supports from + * LLVM 23 on. + / +void __kfunc_btf_root(void) +{ + asm volatile ("" + : + : "r"(&bpf_kfunc_call_test_i128), + "r"(&bpf_kfunc_call_test_ret_fastcall), + "r"(&bpf_kfunc_call_test_ret_ptr), + "r"(&bpf_kfunc_call_test_ret_ii), + "r"(&bpf_kfunc_call_test_ret_big)); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128") +__naked int aggregate_ret_kfunc_precise(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + "call %[bpf_kfunc_call_test_i128];" + "r6 = r2;" / second return half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_i128) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL") +__naked int aggregate_ret_kfunc_fastcall_fail(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + "call %[bpf_kfunc_call_test_ret_fastcall];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_fastcall) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("is not composed of scalars") +__naked int aggregate_ret_kfunc_ptr_fail(void) +{ + asm volatile ( + "r1 = 0;" + "call %[bpf_kfunc_call_test_ret_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_ptr) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_kfunc_small_no_r2(void) +{ + asm volatile ( + "r1 = 0;" + "r2 = 0;" + "call %[bpf_kfunc_call_test_ret_ii];" + "r0 = r2;" / R2 is not a return register for a <=8 byte struct / + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_ii) + : __clobber_all); +} + +/ + * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by + * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks, + * so this behaves the same on every architecture. + / +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported") +__naked int aggregate_ret_kfunc_too_big_fail(void) +{ + asm volatile ( + "call %[bpf_kfunc_call_test_ret_big];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_big) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c new file mode 100644 index 0000000000000..2c1889fc28efd --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c @@ -0,0 +1,125 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "../test_kmods/bpf_testmod_kfunc.h" +#include "bpf_misc.h" + +#if defined(__clang_major__) && __clang_major__ >= 23 + +#define MIX_A 0xdeadbeefcafef00dULL +#define MIX_B 0x0123456789abcdefULL + +typedef unsigned __int128 u128; + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_int128_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + u128 v; + + v = bpf_kfunc_call_test_i128(a, b); + if ((__u64)(v >> 64) != a + b) + return 1; + if ((__u64)v != a - b) + return 2; + + return 0; +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct prog_test_ret_pair p; + + p = bpf_kfunc_call_test_ret_pair(a, b); + if (p.hi != a + b) + return 1; + if (p.lo != a - b) + return 2; + + return 0; +} + +/ struct { u64 a; int b; }: 16 bytes, R0 = a, R2 = b. / +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_li_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + int b = skb->len ^ MIX_B; + struct prog_test_ret_li r; + + r = bpf_kfunc_call_test_ret_li(a, b); + if (r.a != a) + return 1; + if (r.b != ~b) + return 2; + + return 0; +} + +/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not a return reg. / +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_ii_c_test(struct __sk_buff skb) +{ + int a = skb->len ^ MIX_A; + int b = skb->len ^ MIX_B; + struct prog_test_ret_ii r; + + r = bpf_kfunc_call_test_ret_ii(a, b); + if (r.a != a) + return 1; + if (r.b != b) + return 2; + + return 0; +} + +/* A union of 16 bytes takes the same R0:R2 path as a struct. / +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_uu_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + union prog_test_ret_uu r; + + r = bpf_kfunc_call_test_ret_uu(a, b); + if (r.parts.lo != a + b) + return 1; + if (r.parts.hi != a - b) + return 2; + + return 0; +} + +#else + +SEC("socket") +__description("aggregate_ret_kfunc_c: needs LLVM 23, dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c new file mode 100644 index 0000000000000..83b1a37519887 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#if defined(__clang_major__) && __clang_major__ >= 23 + +#define MIX_A 0xdeadbeefcafef00dULL +#define MIX_B 0x0123456789abcdefULL + +struct pair { + __u64 hi; / R0 / + __u64 lo; / R2 / +}; + +union upair { + __u64 halves[2]; + struct { + __u64 lo; / R0 / + __u64 hi; / R2 / + } parts; +}; + +static __noinline struct pair make_pair(__u64 a, __u64 b) +{ + struct pair p = { .hi = a + b, .lo = a - b }; + + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct pair p; + + p = make_pair(a, b); + if (p.hi != a + b) + return 1; + if (p.lo != a - b) + return 2; + + return 0; +} + +__noinline struct pair make_pair_global(__u64 a, __u64 b) +{ + struct pair p = { .hi = a + b, .lo = a - b }; + + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_global_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct pair p; + + p = make_pair_global(a, b); + if (p.hi != a + b) + return 1; + if (p.lo != a - b) + return 2; + + return 0; +} + +static __noinline union upair make_upair(__u64 a, __u64 b) +{ + union upair p; + + p.halves[0] = a + b; + p.halves[1] = a - b; + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_union_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + union upair p; + + p = make_upair(a, b); + if (p.parts.lo != a + b) + return 1; + if (p.parts.hi != a - b) + return 2; + + return 0; +} + +#else + +SEC("socket") +__description("aggregate_ret_struct_c: needs LLVM 23, dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c new file mode 100644 index 0000000000000..cffd8d7d3241a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +/ freplace target: a global subprogram returning 16 bytes in R0:R2. / +__naked unsigned __int128 agg_ret_target_func(void) +{ + asm volatile ( + "r0 = 0x1234;" + "r2 = 0x5678;" + "exit;" + ); +} + +SEC("tc") +__naked int agg_ret_target(void) +{ + asm volatile ( + "call %[agg_ret_target_func];" + "r0 = 0;" + "exit;" + : + : __imm(agg_ret_target_func) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c index d055fc7b3b95d..0be9441ec273b 100644 --- a/tools/testing/selftests/bpf/progs/compute_live_registers.c +++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c @@ -431,6 +431,36 @@ __naked void subprog1(void) ::: __clobber_all); } +static __used __naked unsigned __int128 aux2(void) +{ + asm volatile ( + "r0 = 1;" + "r2 = 2;" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +/ A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). / +__load_if_JITed() +__log_level(2) +__msg("0: .12345.... (85) call pc+2") +__msg("1: ..2....... (bf) r0 = r2") +/ R2 is not read at the exit of this program, which returns an int, ... / +__msg("2: 0......... (95) exit") +__msg("3: .......... (b7) r0 = 1") +__msg("4: 0......... (b7) r2 = 2") +/ ... but it is at the exit of aux2(), which returns a register pair. / +__msg("5: 0.2....... (95) exit") +__naked void subprog_ret_reg_pair(void) +{ + asm volatile ( + "call aux2;" + "r0 = r2;" + "exit;" + ::: __clobber_all); +} + #if defined(__TARGET_ARCH_x86) \|\| defined(__TARGET_ARCH_arm64) SEC("socket") diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c index ac44d60e50666..9708efb93683b 100644 --- a/tools/testing/selftests/bpf/progs/exceptions_fail.c +++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c @@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a) SEC("?tc") __exception_cb(exception_cb_bad_ret_type1) -__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.") +__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.") int reject_exception_cb_type_1(struct __sk_buff ctx) { bpf_throw(0); diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c new file mode 100644 index 0000000000000..12c15d293bd79 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> + +SEC("freplace/agg_ret_target_func") +__u64 new_agg_ret_target_func(void) +{ + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c index a6133f7521f34..ad36d583d2e7f 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -939,6 +939,69 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c) return 0; } +#if defined(__x86_64__) \|\| defined(__aarch64__) +__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b) +{ + return (__int128)(((unsigned __int128)(a + b) << 64) \| (a - b)); +} + +__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b) +{ + struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b) +{ + struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b) +{ + struct prog_test_ret_li r = { .a = a, .b = ~b }; + + return r; +} + +__bpf_kfunc union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b) +{ + union prog_test_ret_uu r; + + r.halves[0] = a + b; + r.halves[1] = a - b; + return r; +} + +__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag) +{ + struct prog_test_ret_ptr r = { .p = NULL, .tag = tag }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) +{ + struct prog_test_ret_ii r = { .a = a, .b = b }; + + return r; +} +#endif / __x86_64__ \|\| __aarch64__ / + +/ + * Takes no argument on purpose: with no arguments there is nothing for the sret + * pointer to displace, so this needs no architecture guard even though it + * returns 24 bytes. See the comment on bpf_kfunc_call_test_i128() above. + / +__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) +{ + struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 }; + + return r; +} + __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f, u64 g, u64 h, u64 i, u64 j) @@ -1472,6 +1535,16 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2) BTF_ID_FLAGS(func, bpf_kfunc_call_test3) BTF_ID_FLAGS(func, bpf_kfunc_call_test4) BTF_ID_FLAGS(func, bpf_kfunc_call_test5) +#if defined(__x86_64__) \|\| defined(__aarch64__) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii) +#endif +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h index c4383acb53c11..c7be973cd2860 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h @@ -55,6 +55,40 @@ struct prog_test_big_arg { __u64 b; }; +struct prog_test_ret_pair { + __u64 hi; + __u64 lo; +}; + +struct prog_test_ret_li { / 16 bytes: R0:R2 / + __u64 a; + int b; +}; + +struct prog_test_ret_ii { / 8 bytes: R0 only / + int a; + int b; +}; + +union prog_test_ret_uu { / 16 bytes: R0:R2 / + __u64 halves[2]; + struct { + __u64 lo; + __u64 hi; + } parts; +}; + +struct prog_test_ret_ptr { / 16 bytes: contains a pointer / + void p; + __u64 tag; +}; + +struct prog_test_ret_big { /* 24 bytes: too large for R0:R2 / + __u64 a; + __u64 b; + __u64 c; +}; + struct prog_test_fail1 { void p; int x; @@ -130,6 +164,16 @@ int bpf_kfunc_call_test2(struct sock sk, __u32 a, __u32 b) __ksym; struct sock bpf_kfunc_call_test3(struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; +#ifdef __SIZEOF_INT128__ +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym; +#endif +struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym; +struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym; +struct prog_test_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym; +struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym; +union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym; +struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym; +struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym; __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, __u64 g, __u64 h, __u64 i, __u64 j) __ksym;
3/2	2026/08/13 20:58	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 27625 output: 200 thoughts: 970 Thoughts: Fuzzing Plan for Extended Kfunc Return Value Support in BPF Okay, so we have a patch that's expanding BPF's kfunc and subprogram return value support to handle values up to 16 bytes, returned in the `R0:R2` register pair. This is a significant change, modifying the verifier and the JITs for arm64, riscv, and x86 architectures. This absolutely demands fuzzing. My primary focus needs to be on identifying the most critical areas to target. Let's see... the patch touches a bunch of symbols, so let me break it down: First, let's identify the specific functions that are being changed: I've identified the following symbols as the key areas: * `bpf_jit_supports_kfunc_ret_reg_pair` (architecture-specific) * `emit_kfunc_arena_args` (x86 specific) * `bpf_kfunc_ret_reg_pair` (verifier) * `btf_validate_return_type` (BTF) * `compute_insn_live_regs` (liveness) * `bpf_compute_subprog_ret_regs` (verifier) * `subprog_ret_pair_unreliable` (verifier) * `mark_kfunc_ret_regs` (verifier) * `btf_type_is_scalar_struct` (verifier) * `check_global_ret_scalar_reg` (verifier) * `check_global_subprog_return_code` (verifier) Now, let me narrow the focus. These ones seem most pertinent, with regards to the function of the patch: * `bpf_kfunc_ret_reg_pair` * `bpf_compute_subprog_ret_regs` * `btf_validate_return_type` * `compute_insn_live_regs` * `check_global_subprog_return_code` * `check_global_ret_scalar_reg` * `btf_type_is_scalar_struct` * `mark_kfunc_ret_regs` * `subprog_ret_pair_unreliable` Let's check if these symbols are static or global, to understand their scope. Right, so: * `bpf_kfunc_ret_reg_pair` and `btf_type_is_scalar_struct` are global which indicates a broader impact. * The rest of those functions are static, which might imply they are contained within a particular scope. Since global symbols are more likely entry points, I should prioritize those in the initial fuzzing runs. I'll then investigate the non-static functions again. Let's see: I need to focus on these non-static functions: * `bpf_kfunc_ret_reg_pair` * `btf_type_is_scalar_struct` * `bpf_jit_supports_kfunc_ret_reg_pair` Also, let's peek at `kernel/bpf/verifier.c` and `kernel/bpf/btf.c` to see if there are other relevant functions that are not static. Okay: * `bpf_compute_live_registers` is modified. * `bpf_check` is modified. * `bpf_check_attach_target` is modified. So, I'll definitely incorporate these into my scope. Let me jot down the functions I'll definitely use: * `bpf_kfunc_ret_reg_pair` * `btf_type_is_scalar_struct` * `bpf_jit_supports_kfunc_ret_reg_pair` * `bpf_compute_live_registers` As for configurations, it's pretty clear that `BPF_SYSCALL` and `BPF_JIT` are relevant here. `CONFIG_BPF_SYSCALL` is the main one. I'll need to make sure those are enabled when I begin the fuzzing process. Now, let's fire up the `set-results` tool and get started. This should be interesting... I can't wait to see what bugs we uncover.
4/2	2026/08/13 20:58	tool	set-results	0m Args: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT" ], "FocusSymbols": [ "bpf_kfunc_ret_reg_pair", "btf_type_is_scalar_struct", "bpf_jit_supports_kfunc_ret_reg_pair", "bpf_compute_live_registers" ], "Reasoning": "The patch introduces support for kfuncs and BPF subprograms returning larger values (up to 16 bytes) in the R0:R2 register pair. This involves changes to the BPF verifier (liveness tracking, return value checking, BTF validation) and JIT compilers (x86, arm64, riscv). These are significant functional changes to the core BPF subsystem that should be fuzzed to ensure verifier safety and JIT correctness.", "WorthFuzzing": true } Results: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT" ], "FocusSymbols": [ "bpf_kfunc_ret_reg_pair", "btf_type_is_scalar_struct", "bpf_jit_supports_kfunc_ret_reg_pair", "bpf_compute_live_registers" ], "Reasoning": "The patch introduces support for kfuncs and BPF subprograms returning larger values (up to 16 bytes) in the R0:R2 register pair. This involves changes to the BPF verifier (liveness tracking, return value checking, BTF validation) and JIT compilers (x86, arm64, riscv). These are significant functional changes to the core BPF subsystem that should be fuzzed to ensure verifier safety and JIT correctness.", "WorthFuzzing": true }
5/1	2026/08/13 20:58	action	If	0m Args: { "WorthFuzzing": true } Results: null
6/2	2026/08/13 20:58	agent	kmsan-evaluator	0m Model: best-expensive Args: null Results: { "KMSANReasoning": "The patch extends the BPF verifier and JIT compilers to support returning values up to 16 bytes (e.g., `__int128` or small structs) from BPF subprograms and kfuncs using the R0 and R2 register pair. The changes primarily involve updating the verifier's liveness tracking, backtracking, and return value checks to ensure that both R0 and R2 are properly initialized before a BPF program returns. The JIT changes simply move the native ABI's second return register into BPF's R2. \n\nThere are no new kernel data structures allocated and copied to user space, nor any C-level uninitialized memory risks introduced. The verifier statically enforces that BPF programs initialize the R2 register when returning a pair, preventing uninitialized register leaks. Furthermore, KMSAN does not instrument JIT-compiled BPF code, so it would not provide additional coverage for the BPF program execution itself. Standard KASAN and verifier fuzzing are sufficient to detect any logic bugs, out-of-bounds accesses, or memory corruptions introduced by these changes.", "NeedsKMSAN": false } Instruction: You are an expert Linux kernel security engineer specializing in kernel memory error detectors (KASAN and KMSAN). Your job is to review the provided patch series and determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer) fuzzing session in addition to standard KASAN fuzzing. CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN: - Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable a comprehensive suite of debugging tools and sanitizers, including KASAN (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks. - KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap, or page allocations) and kernel-to-user memory info-leaks. Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN, LOCKDEP, OR OTHER STANDARD BUG DETECTORS. A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks that are detected ONLY by KMSAN. Look holistically at the patch series and surrounding code. Even if no direct uninitialized field accesses or new buffer allocations are added in the diff itself, a patch may alter control flow, bounds checking, or data length calculations in ways that change how the rest of the code operates on existing buffers (e.g. allowing uninitialized stack/heap memory to be read, copied to user space, or used in control flow). Do not hesitate to use your code access tools to inspect the surrounding code, called functions, and callers. Set NeedsKMSAN=true ONLY IF the patch introduces or modifies: 1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb attributes, ioctl output arguments, socket options, or BPF buffers) where fields or structure padding might not be fully initialized/zeroed. 2. Conditional logic or branching that depends on potentially uninitialized variables or struct fields. 3. Allocation or initialization of complex data structures where uninitialized fields could be read later in reachable code paths. 4. Bounds checks, lengths, or logic in a way that allows surrounding code to access uninitialized bytes of existing buffers. Set NeedsKMSAN=false IF: - The code changes primarily risk out-of-bounds access, array overflows, NULL pointer dereferences, locking deadlocks, or use-after-free bugs (these are already caught by KASAN, LOCKDEP, or standard bug detectors). - All stack/heap structures touched or introduced by the patch are fully zeroed or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied. - The patch does not introduce any risk of uninitialized memory usage or info-leaks. Use your code access tools to inspect the surrounding code if necessary, then provide detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch. Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit bafb5dc76d94a94d43e7b1575db7d4920a549687 Author: syz-cluster <triage@syzkaller.com> Date: Thu Aug 13 20:58:12 2026 +0000 syz-cluster: applied patch under review diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 10e725cbe64cd..5d840d3000338 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -575,6 +575,69 @@ is also covered by this recovery. A kfunc handed an arena pointer may therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking against the arena. Larger accesses must verify the range explicitly. +2.9 kfunc Return Values +----------------------- + +A kfunc may return a scalar, a pointer, or a small struct or union by +value. A scalar or pointer of up to 8 bytes is returned in R0, as usual. + +A struct or union returned by value must be composed only of scalars +(recursively), where a scalar is an integer or an enum; arrays of scalars are +allowed as members. Its bytes are handed back to the program as the raw +contents of R0 (and R2), so a pointer field would be laundered into a scalar +and escape the verifier's pointer provenance and reference tracking. A struct +or union with a pointer member is therefore rejected at load time, and so is +one with a floating-point member, which the ABI may not return in R0:R2 at +all. + +A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a +scalar-only struct or union, or an ``__int128``. Such a value is returned +in the register pair R0:R2, matching the convention LLVM uses for the BPF +target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or +union of 8 bytes or less is returned in R0 alone. + +:: + + struct bpf_pair { __u64 a, b; }; /* 16 bytes / + + __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void) + { + struct bpf_pair p = { .a = 1, .b = 2 }; + + return p; / p.a in R0, p.b in R2 / + } + +Returning a value in the R0:R2 pair requires the JIT to place the second +half of the return value into R2, which not every architecture supports +right now. A kfunc with a return value larger than 8 bytes is therefore +rejected at load time on a JIT that does not advertise this capability (see +``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run +by the interpreter. A return value larger than 16 bytes is not supported. + +The same R0:R2 convention applies to a BPF subprogram, global or static, +that returns an ``__int128`` or a struct or union larger than 8 bytes. Such a +program also requires the JIT, since the interpreter propagates only R0 out +of a subprogram. A global subprogram is verified in isolation, so its +by-value struct or union return is restricted to scalars just like a kfunc's; +a static subprogram is verified inline and has no such restriction. The main +program is not covered: its return value is the program's exit code, read out +of R0 alone, so a declared upper half is never looked at. + +A global subprogram must leave a scalar in every* register of the pair, so +both halves of the returned value have to be assigned. Leaving the upper half +uninitialized is not merely untidy: the compiler is then free to leave R2 +holding whatever it happened to hold, which for a subprogram taking a pointer +argument is typically that pointer. Handing the caller an unknown scalar built +from a pointer is a leak, so the verifier rejects it with:: + + At subprogram exit the register R2 is not a scalar value (...) + +Initialize the whole return value, for example ``struct pair p = {};``, to +avoid this. A static subprogram is exempt from the scalar-only rule: it is +verified inline, so an unassigned R2 is simply passed back to the caller as +uninitialized and only a caller that reads it fails. A stack pointer left in +R2 is still rejected there, just as one in R0 is. + .. _BPF_kfunc_lifecycle_expectations: 3. kfunc lifecycle expectations diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 74b4083791da3..7a4c5968976a6 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2346,6 +2346,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_stack_args(void) { return true; diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 2504df1fa1118..49220765e96d4 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -2120,6 +2120,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_ptr_xchg(void) { return true; diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index d920772af7d5f..2ec09dd02b187 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip, * arena NULL is offset 0. Return the number of emitted bytes. / static int emit_kfunc_arena_args(struct bpf_prog bpf_prog, - const struct bpf_insn insn, u8 pprog) + const struct btf_func_model fm, u8 *pprog) { - const struct btf_func_model fm; u8 prog = pprog; u8 start = prog; int i; - fm = bpf_jit_find_kfunc_model(bpf_prog, insn); - if (!fm) - return -EINVAL; - for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) { u8 flags = fm->arg_flags[i]; u32 reg = BPF_REG_1 + i; @@ -2644,6 +2639,8 @@ st: insn_off = insn->off; / call / case BPF_JMP \| BPF_CALL: { + const struct btf_func_model fm = NULL; + func = (u8 ) __bpf_call_base + imm32; if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(stack_depth); @@ -2652,7 +2649,10 @@ st: insn_off = insn->off; if (!imm32) return -EINVAL; if (src_reg == BPF_PSEUDO_KFUNC_CALL) { - err = emit_kfunc_arena_args(bpf_prog, insn, &prog); + fm = bpf_jit_find_kfunc_model(bpf_prog, insn); + if (!fm) + return -EINVAL; + err = emit_kfunc_arena_args(bpf_prog, fm, &prog); if (err < 0) return err; ip += err; @@ -2666,6 +2666,14 @@ st: insn_off = insn->off; return -EINVAL; if (priv_frame_ptr) pop_r9(&prog); + / + * A kfunc returning more than 8 bytes hands the second + * half back in RDX (the native ABI's second return reg), + * but BPF expects it in R0:R2. BPF R0 is RAX (no move + * needed), while BPF R2 is RSI, so copy RDX into RSI. + / + if (fm && fm->ret_size > 8) + emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3); break; } @@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_stack_args(void) { return true; diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 27b43fda9b178..899e5ce986432 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -814,6 +814,8 @@ struct bpf_subprog_info { bool is_async_cb: 1; bool is_exception_cb: 1; bool args_cached: 1; + / true if the return value is passed in the R0:R2 register pair / + bool ret_reg_pair: 1; / true if bpf_fastcall stack region is used by functions that can't be inlined / bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; @@ -1048,6 +1050,13 @@ static inline struct bpf_subprog_info subprog_info(struct bpf_verifier_env env return &env->subprog_info[subprog]; } +static inline bool bpf_ret_reg_pair(struct bpf_verifier_env env, int subprog) +{ + return subprog_info(env, subprog)->ret_reg_pair; +} + +bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env env, struct bpf_insn insn); + struct bpf_call_summary { u8 num_params; bool is_void; @@ -1439,6 +1448,8 @@ int bpf_jmp_offset(struct bpf_insn insn); struct bpf_iarray bpf_insn_successors(struct bpf_verifier_env env, u32 idx); void bpf_fmt_stack_mask(char buf, ssize_t buf_sz, u64 stack_mask); bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog); +bool btf_type_is_scalar_struct(struct bpf_verifier_env env, const struct btf btf, + const struct btf_type t, int rec); int bpf_find_subprog(struct bpf_verifier_env env, int off); bool bpf_is_throw_kfunc(struct bpf_insn insn); diff --git a/include/linux/filter.h b/include/linux/filter.h index 4a9bc6a848f2e..6e746b0a09306 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_percpu_insn(void); bool bpf_jit_supports_kfunc_call(void); +bool bpf_jit_supports_kfunc_ret_reg_pair(void); bool bpf_jit_supports_stack_args(void); bool bpf_jit_supports_arena_args(void); bool bpf_jit_supports_far_kfunc_call(void); diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index 40bd04421a991..3ef8a3da726de 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -425,6 +425,16 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, / verifier_bug_if(idx + 1 != subseq_idx, env, "extra insn from subprog"); + /* a global subprog returning more than 8 bytes + * sets R2 as well. R2 is part of the args mask + * checked just below, so clear it here rather + * than next to R0. Only a subprog that does + * return a pair defines R2, so leave the mask + * alone otherwise and let the check below catch + * an R2 that has no business being set. + / + if (bpf_ret_reg_pair(env, subprog)) + bt_clear_reg(bt, BPF_REG_2); / r1-r5 are invalidated after subprog call, * so for global func call it shouldn't be set * anymore @@ -508,6 +518,17 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, return -ENOTSUPP; / regular helper call sets R0 / bt_clear_reg(bt, BPF_REG_0); + / a kfunc returning more than 8 bytes also sets R2. + * R2 is part of the args mask checked just below, so + * clear it here rather than next to R0. The prototype + * lookup is only worth doing when R2 is requested at + * all; any other call leaves R2 uninitialized, so a + * request for it is caught by the check below. + / + if (bt_is_reg_set(bt, BPF_REG_2) && + insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + bpf_kfunc_ret_reg_pair(env, insn)) + bt_clear_reg(bt, BPF_REG_2); if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { / if backtracking was looking for registers R1-R5 * they should have been found already. @@ -522,7 +543,41 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, return -EFAULT; } } else if (opcode == BPF_EXIT) { - bool r0_precise; + bool from_subprog_call, r0_precise, r2_precise; + struct bpf_insn call; + int subprog; + + /* BPF_EXIT in subprog or callback always returns + * right after the call instruction, so by checking + * whether the instruction at subseq_idx-1 is subprog + * call or not we can distinguish actual exit from + * subprog from exit from callback. In the former + * case, we need to propagate the precision of the + * return registers, if necessary. In the latter we + * never do that. + / + from_subprog_call = subseq_idx - 1 >= 0 && + bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]); + + / Sample the return registers before the callback + * handling below clears R1-R5: unlike R0, R2 is an + * argument register as well, so that clear would drop + * a pair return on the floor. + / + r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0); + r2_precise = false; + if (from_subprog_call && bt_is_reg_set(bt, BPF_REG_2)) { + call = &env->prog->insnsi[subseq_idx - 1]; + subprog = bpf_find_subprog(env, subseq_idx + call->imm); + if (subprog < 0) + return -EFAULT; + / Only a callee that does return a pair defines + * R2. Leave the mask alone otherwise, so that + * the check below still catches an R2 that has + * no business being set. + / + r2_precise = bpf_ret_reg_pair(env, subprog); + } / Backtracking to a nested function call, 'idx' is a part of * the inner frame 'subseq_idx' is a part of the outer frame. @@ -535,30 +590,27 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx)) for (i = BPF_REG_1; i <= BPF_REG_5; i++) bt_clear_reg(bt, i); + + / a callee returning more than 8 bytes sets R2 as well; + * R2 is part of the args mask checked just below, so + * clear it here rather than next to R0. + / + if (r2_precise) + bt_clear_reg(bt, BPF_REG_2); if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { verifier_bug(env, "backtracking exit unexpected regs %x", bt_reg_mask(bt)); return -EFAULT; } - / BPF_EXIT in subprog or callback always returns - * right after the call instruction, so by checking - * whether the instruction at subseq_idx-1 is subprog - * call or not we can distinguish actual exit from - * subprog from exit from callback. In the former - * case, we need to propagate r0 precision, if - * necessary. In the former we never do that. - / - r0_precise = subseq_idx - 1 >= 0 && - bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) && - bt_is_reg_set(bt, BPF_REG_0); - bt_clear_reg(bt, BPF_REG_0); if (bt_subprog_enter(bt)) return -EFAULT; if (r0_precise) bt_set_reg(bt, BPF_REG_0); + if (r2_precise) + bt_set_reg(bt, BPF_REG_2); / r6-r9 and stack slots will stay set in caller frame * bitmasks until we return back from callee(s) / diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 6606187ed4f43..c7718a2a56141 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7592,7 +7592,7 @@ int btf_distill_func_proto(struct bpf_verifier_log log, return -EINVAL; } ret = __get_type_size(btf, func->type, &t); - if (ret < 0 \|\| btf_type_is_struct(t)) { + if (ret < 0 \|\| ret > 16) { bpf_log(log, "The function %s return type %s is unsupported.\n", tname, btf_type_str(t)); @@ -7965,7 +7965,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env env, / Check whether the type is a valid return type. / static int btf_validate_return_type(struct bpf_verifier_env env, struct btf btf, - const struct btf_type t, int subprog) + const struct btf_type t, int subprog, bool is_global) { u32 tags = 0; int err; @@ -7988,6 +7988,19 @@ static int btf_validate_return_type(struct bpf_verifier_env env, struct btf bt if (btf_type_is_void(t) \|\| btf_type_is_int(t) \|\| btf_is_any_enum(t)) return 0; + if (btf_type_is_struct(t) && t->size <= 16) { + / + * A global function's caller models the return as an opaque + * scalar pair, so it may only return scalars by value. A local + * function is verified inline, so a pointer field stays tracked + * and needs no such restriction. + / + bool local_func = subprog && !is_global; + + if (local_func \|\| btf_type_is_scalar_struct(env, btf, t, 0)) + return 0; + } + return -EOPNOTSUPP; } @@ -8075,12 +8088,12 @@ int btf_prepare_func_args(struct bpf_verifier_env env, int subprog) return -EINVAL; } - err = btf_validate_return_type(env, btf, t, subprog); + err = btf_validate_return_type(env, btf, t, subprog, is_global); if (err) { if (is_global) { bpf_log(log, - "Global function %s() return value not void or scalar. " - "Only those are supported.\n", + "Global function %s() has unsupported return type. " + "Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n", tname); } return err; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 6a94370a24488..cb66a2ef52b1f 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void) return false; } +bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return false; +} + bool __weak bpf_jit_supports_stack_args(void) { return false; diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index 1c997aeba6fa5..850c58aa915be 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -2062,7 +2062,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); } /* Compute info->{use,def} fields for the instruction / static void compute_insn_live_regs(struct bpf_verifier_env env, struct bpf_insn insn, - struct insn_live_regs info) + struct insn_live_regs info, + bool ret_reg_pair) { struct bpf_call_summary cs; const u8 class = BPF_CLASS(insn->code); @@ -2074,6 +2075,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env, const u32 src32 = mask_lo(src); const u32 dst32 = mask_lo(dst); const u32 r0 = reg64_mask(0); + const u32 r2 = reg64_mask(BPF_REG_2); u32 def = 0; u32 use = U32_MAX; @@ -2193,7 +2195,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env, break; case BPF_EXIT: def = 0; - use = r0; + use = ret_reg_pair ? (r0 \| r2) : r0; break; case BPF_CALL: def = ALL_CALLER_SAVED_REGS; @@ -2230,8 +2232,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env env) struct insn_live_regs state; int insn_cnt = env->prog->len; u64 pos, insn_pos; - int err = 0, i, j; - bool changed; + int err = 0, i, j, subprog, start, end; + bool changed, ret_reg_pair; / Use the following algorithm: * - define the following: @@ -2258,8 +2260,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env env) goto out; } - for (i = 0; i < insn_cnt; ++i) - compute_insn_live_regs(env, &insns[i], &state[i]); + for (subprog = 0; subprog < env->subprog_cnt; subprog++) { + start = env->subprog_info[subprog].start; + end = env->subprog_info[subprog + 1].start; + ret_reg_pair = bpf_ret_reg_pair(env, subprog); + + for (i = start; i < end; ++i) + compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair); + } / Forward pass: resolve stack access through FP-derived pointers / err = bpf_compute_subprog_arg_access(env); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 164d16c243ca6..98a6f702d794d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -382,27 +382,80 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog) return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL; } -static bool subprog_returns_void(struct bpf_verifier_env env, int subprog) +static const struct btf_type subprog_ret_type(struct bpf_verifier_env env, int subprog) { - const struct btf_type type, func, func_proto; + const struct btf_type func, func_proto; const struct btf btf = env->prog->aux->btf; u32 btf_id; + if (!btf \|\| !env->prog->aux->func_info) + return NULL; + btf_id = env->prog->aux->func_info[subprog].type_id; + / Both already validated by prepare_btf_func() at prog load. / func = btf_type_by_id(btf, btf_id); - if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id)) - return false; - func_proto = btf_type_by_id(btf, func->type); - if (!func_proto) - return false; - type = btf_type_skip_modifiers(btf, func_proto->type, NULL); - if (!type) - return false; + return btf_type_skip_modifiers(btf, func_proto->type, NULL); +} + +static bool subprog_returns_void(struct bpf_verifier_env env, int subprog) +{ + const struct btf_type type = subprog_ret_type(env, subprog); + + return type && btf_type_is_void(type); +} + +static u32 ret_regs_cnt(u32 size) +{ + return size > 8 && size <= 16 ? 2 : 1; +} + +/ Registers holding a function return value, in order. See ret_regs_cnt(). / +static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 }; + +static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env env) +{ + const struct btf btf = env->prog->aux->btf; + const struct btf_type type; + int subprog; + u32 size; + + for (subprog = 0; subprog < env->subprog_cnt; subprog++) { + type = subprog_ret_type(env, subprog); + if (!type \|\| !(btf_type_is_struct(type) \|\| btf_type_is_scalar(type))) + continue; + if (IS_ERR(btf_resolve_size(btf, type, &size))) + continue; + if (ret_regs_cnt(size) > 1) { + subprog_info(env, subprog)->ret_reg_pair = true; + /* + * The R0:R2 return convention is only implemented in + * the JIT: the interpreter propagates BPF_R0 alone out + * of a subprogram, so a caller reading R2 would see a + * stale value. + / + env->prog->jit_required = 1; + } + } +} - return btf_type_is_void(type); +/ + * A >8 byte BPF return changes the calling convention to R0:R2, and the + * verifier derives that convention from the subprogram's BTF prototype + * alone. Once that prototype is marked unreliable it is known not to + * describe the compiled code, so the convention read from it cannot be + * trusted either: reject the call rather than keep tracking R2 on the + * strength of a signature the verifier has already discarded. + / +static bool subprog_ret_pair_unreliable(struct bpf_verifier_env env, int subprog) +{ + struct bpf_prog_aux aux = env->prog->aux; + + return bpf_ret_reg_pair(env, subprog) && + aux->func_info_aux && + aux->func_info_aux[subprog].unreliable; } static const char subprog_name(const struct bpf_verifier_env env, int subprog) @@ -2459,6 +2512,21 @@ find_kfunc_desc(const struct bpf_prog prog, u32 func_id, u16 offset) sizeof(tab->descs[0]), kfunc_desc_cmp_by_id_off); } +/* + * True if the kfunc called by @insn returns its value in the R0:R2 pair. + * Reads the same btf_func_model.ret_size that bpf_add_kfunc_call() validated + * and that the JIT keys the second return register off, so the verifier and + * the generated code cannot disagree about the convention. + / +bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env env, struct bpf_insn insn) +{ + const struct bpf_kfunc_desc desc; + + desc = find_kfunc_desc(env->prog, insn->imm, insn->off); + + return desc && ret_regs_cnt(desc->func_model.ret_size) > 1; +} + int bpf_get_kfunc_addr(const struct bpf_prog prog, u32 func_id, u16 btf_fd_idx, u8 func_addr) { @@ -2809,6 +2877,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env env, u32 func_id, u16 offset) err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model); if (err) return err; + if (func_model.ret_size > 8) { + if (kfunc.flags && (kfunc.flags & KF_FASTCALL)) { + verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n", + kfunc.name); + return -EOPNOTSUPP; + } + if (!bpf_jit_supports_kfunc_ret_reg_pair()) { + verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n", + kfunc.name); + return -EOPNOTSUPP; + } + } memset(&meta, 0, sizeof(meta)); meta.btf = kfunc.btf; @@ -9381,6 +9461,7 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn, u16 callee_incoming, stack_arg_cnt; struct bpf_func_state caller; int err, subprog, target_insn; + u32 i, nregs; target_insn = insn_idx + insn->imm + 1; subprog = bpf_find_subprog(env, target_insn); @@ -9423,9 +9504,14 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn, clear_caller_saved_regs(env, caller->regs); invalidate_outgoing_stack_args(env, cur_func(env)); - / All non-void global functions return a 64-bit SCALAR_VALUE. / + / + * A non-void global function returns a 64-bit SCALAR_VALUE in + * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair. + / if (!subprog_returns_void(env, subprog)) { - mark_reg_unknown(env, caller->regs, BPF_REG_0); + nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1; + for (i = 0; i < nregs; i++) + mark_reg_unknown(env, caller->regs, ret_regs[i]); } if (env->subprog_info[subprog].might_throw) { @@ -9443,6 +9529,12 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn, return 0; } + if (subprog_ret_pair_unreliable(env, subprog)) { + verbose(env, "Func#%d ('%s') returns >8 bytes, which requires reliable BTF\n", + subprog, subprog_name(env, subprog)); + return -EINVAL; + } + / * Track caller's total stack arg count (incoming + max outgoing). * This is needed so the JIT knows how much stack arg space to allocate. @@ -9783,11 +9875,15 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx) struct bpf_func_state caller, callee; struct bpf_reg_state r0; bool in_callback_fn; + u32 i, nregs; int err; callee = state->frame[state->curframe]; r0 = &callee->regs[BPF_REG_0]; - if (r0->type == PTR_TO_STACK) { + nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1; + for (i = 0; i < nregs; i++) { + if (callee->regs[ret_regs[i]].type != PTR_TO_STACK) + continue; / technically it's ok to return caller's stack pointer * (or caller's caller's pointer) back to the caller, * since these pointers are valid. Only current stack @@ -9822,8 +9918,12 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx) return -EFAULT; } } else { - /* return to the caller whatever r0 had in the callee / - caller->regs[BPF_REG_0] = r0; + /* + * return to the caller whatever the callee had in the + * return register(s) + / + for (i = 0; i < nregs; i++) + caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]]; } / for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite, @@ -10721,6 +10821,19 @@ static int check_helper_call(struct bpf_verifier_env env, struct bpf_insn insn return 0; } +/* + * Mark the register(s) holding a @size byte kfunc return value as unknown + * scalars. Both halves of a register pair are treated the same way. + / +static void mark_kfunc_ret_regs(struct bpf_verifier_env env, + struct bpf_reg_state regs, u32 size) +{ + u32 i, nregs = ret_regs_cnt(size); + + for (i = 0; i < nregs; i++) + mark_reg_unknown(env, regs, ret_regs[i]); +} + static bool is_kfunc_acquire(struct bpf_call_arg_meta meta) { return meta->kfunc_flags & KF_ACQUIRE; @@ -10988,9 +11101,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta meta, u32 arg_ } / Returns true if struct is composed of scalars, 4 levels of nesting allowed / -static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env, - const struct btf btf, - const struct btf_type t, int rec) +bool btf_type_is_scalar_struct(struct bpf_verifier_env env, + const struct btf btf, + const struct btf_type t, int rec) { const struct btf_type member_type; const struct btf_member member; @@ -11008,7 +11121,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env, verbose(env, "max struct nesting depth exceeded\n"); return false; } - if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1)) + if (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1)) return false; continue; } @@ -11407,7 +11520,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta, (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) \|\| is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) { if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) && - !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n", reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); return -EINVAL; @@ -11423,7 +11536,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta, * scalars. The access size is derived from the pointed-to BTF type. / if (!btf_type_is_scalar(ref_t) && - !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n", reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); return -EINVAL; @@ -12394,7 +12507,7 @@ static int check_kfunc_args(struct bpf_verifier_env env, struct bpf_call_arg_me break; } - if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + if (!btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id); verbose(env, "%s is %s expected %s %s", @@ -12858,7 +12971,7 @@ static int check_special_kfunc(struct bpf_verifier_env env, struct bpf_call_arg struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id); if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) { - if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) { + if (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) { verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n"); return -EINVAL; } @@ -13191,10 +13304,25 @@ static int check_kfunc_call(struct bpf_verifier_env env, struct bpf_insn insn, } if (btf_type_is_scalar(t)) { - mark_reg_unknown(env, regs, BPF_REG_0); + mark_kfunc_ret_regs(env, regs, t->size); if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] \|\| meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave])) __mark_reg_const_zero(env, &regs[BPF_REG_0]); + } else if (btf_type_is_struct(t)) { + / + * The returned struct comes back as raw register bits modeled + * as an unknown scalar, so it must contain only scalars: + * otherwise a pointer field would be laundered into a scalar + * and escape provenance and reference tracking. + / + if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) { + verbose(env, + "kernel function %s returns %s %s that is not composed of scalars\n", + func_name, btf_type_str(t), + btf_name_by_offset(desc_btf, t->name_off)); + return -EINVAL; + } + mark_kfunc_ret_regs(env, regs, t->size); } else if (btf_type_is_ptr(t)) { ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id); err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf); @@ -16641,37 +16769,54 @@ static int check_return_code(struct bpf_verifier_env env, int regno, const char return 0; } -static int check_global_subprog_return_code(struct bpf_verifier_env env) +static int check_global_ret_scalar_reg(struct bpf_verifier_env env, u32 regno) { - struct bpf_reg_state reg = reg_state(env, BPF_REG_0); - struct bpf_func_state cur_frame = cur_func(env); + struct bpf_reg_state reg; int err; - if (subprog_returns_void(env, cur_frame->subprogno)) - return 0; - - err = check_reg_arg(env, BPF_REG_0, SRC_OP); + err = check_reg_arg(env, regno, SRC_OP); if (err) return err; / Pointers to arena are safe to pass between subprograms. / - if (is_arena_reg(env, BPF_REG_0)) + if (is_arena_reg(env, regno)) return 0; - if (is_pointer_value(env, BPF_REG_0)) { - verbose(env, "R%d leaks addr as return value\n", BPF_REG_0); + if (is_pointer_value(env, regno)) { + verbose(env, "R%d leaks addr as return value\n", regno); return -EACCES; } + reg = reg_state(env, regno); if (reg->type != SCALAR_VALUE) { - verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n", - reg_type_str(env, reg->type)); + verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n", + regno, reg_type_str(env, reg->type)); return -EINVAL; } return 0; } +static int check_global_subprog_return_code(struct bpf_verifier_env env) +{ + struct bpf_func_state cur_frame = cur_func(env); + u32 subprog = cur_frame->subprogno; + u32 i, nregs; + int err; + + if (subprog_returns_void(env, subprog)) + return 0; + + nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1; + for (i = 0; i < nregs; i++) { + err = check_global_ret_scalar_reg(env, ret_regs[i]); + if (err) + return err; + } + + return 0; +} + / Bitmask with 1s for all caller saved registers / #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1) @@ -19328,6 +19473,13 @@ int bpf_check_attach_target(struct bpf_verifier_log log, return -EOPNOTSUPP; } + if (prog_extension && tgt_info->fmodel.ret_size > 8) { + bpf_log(log, + "Cannot replace function %s with a >8 byte return value\n", + tname); + return -EOPNOTSUPP; + } + /* * .multi programs don't need an address during program verification, we just take the module ref if needed. @@ -20317,6 +20469,9 @@ int bpf_check(struct bpf_prog *prog, union bpf_attr attr, bpfptr_t uattr, if (ret < 0) goto skip_full_check; + /* must precede the first bpf_ret_reg_pair() user below / + bpf_compute_subprog_ret_regs(env); + ret = bpf_compute_live_registers(env); if (ret < 0) goto skip_full_check; diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file mode 100644 index 0000000000000..c295adedbae85 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <test_progs.h> +#include "aggregate_ret_int128_c.skel.h" +#include "aggregate_ret_struct_c.skel.h" +#include "aggregate_ret_kfunc_c.skel.h" +#include "aggregate_ret_func.skel.h" +#include "aggregate_ret_kfunc.skel.h" + +void test_aggregate_ret(void) +{ + RUN_TESTS(aggregate_ret_int128_c); + RUN_TESTS(aggregate_ret_struct_c); + RUN_TESTS(aggregate_ret_kfunc_c); + RUN_TESTS(aggregate_ret_func); + RUN_TESTS(aggregate_ret_kfunc); +} diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c index 2523c07a16c65..0b54f911015c5 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c @@ -441,6 +441,19 @@ static void test_func_replace_int_with_void(void) " doesn't match type INT of global_func2()"); } +static void test_func_replace_ret_pair(void) +{ + const char msg = "Cannot replace function agg_ret_target_func with a >8 byte return"; + + /* + * An extension cannot replace a function whose return value comes back + * in the R0:R2 pair: the extension's own return is capped at 8 bytes, + * so it would leave R2 stale for the target's callers. + / + test_obj_load_failure_common("freplace_ret_pair.bpf.o", + "./aggregate_ret_target.bpf.o", msg); +} + static int find_prog_btf_id(const char name, __u32 attach_prog_fd) { struct bpf_prog_info info = {}; @@ -660,6 +673,8 @@ void serial_test_fexit_bpf2bpf(void) test_func_replace_progmap(); if (test__start_subtest("freplace_int_with_void")) test_func_replace_int_with_void(); + if (test__start_subtest("freplace_ret_pair")) + test_func_replace_ret_pair(); if (test__start_subtest("freplace_void")) test_func_replace_void(); if (test__start_subtest("sleepable_fentry_to_xdp")) diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file mode 100644 index 0000000000000..cfb21bcf704be --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c @@ -0,0 +1,260 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +typedef unsigned __int128 u128; + +__naked u128 global_agg_good(void) +{ + asm volatile ( + "r0 = 0x1234;" / low 64 bits / + "r2 = 0x5678;" / high 64 bits / + "exit;" + ); +} + +__naked u128 global_agg_bad(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +__naked u128 global_agg_bad_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = r10;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_global_fail(void) +{ + asm volatile ( + "call %[global_agg_bad];" + "r0 = r2;" + "exit;" + : + : __imm(global_agg_bad) + : __clobber_all); +} + +SEC("tc") +__failure __msg("At subprogram exit the register R2 is not a scalar value") +__naked int aggregate_ret_global_ptr_fail(void) +{ + asm volatile ( + "call %[global_agg_bad_ptr];" + "r0 = r2;" + "exit;" + : + : __imm(global_agg_bad_ptr) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_bad_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = r10;" / stack pointer placed in the second return register / + "exit;" + ); +} + +/ + * R2 is a return register once the subprogram returns a pair, so a stack + * pointer left in it is rejected at the callee's exit exactly as one in R0 + * is: the callee frame is gone by the time the caller could use it. + / +SEC("tc") +__failure __msg("cannot return stack pointer to the caller") +__naked int aggregate_ret_static_ptr_fail(void) +{ + asm volatile ( + "call %[static_agg_bad_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(static_agg_bad_ptr) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_no_r2(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_static_uninit_fail(void) +{ + asm volatile ( + "call %[static_agg_no_r2];" + "r0 = r2;" + "exit;" + : + : __imm(static_agg_no_r2) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_precise(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 4;" / second half; its value is made precise below / + "exit;" + ); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit") +__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4") +__naked int aggregate_ret_static_precise(void) +{ + asm volatile ( + "call %[static_agg_precise];" + "r6 = r2;" / derived from the aggregate's second half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(static_agg_precise) + : __clobber_all); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9") +__naked int aggregate_ret_global_precise(void) +{ + asm volatile ( + "call %[global_agg_good];" + "r6 = r2;" / derived from the aggregate's second half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(global_agg_good) + : __clobber_all); +} + +/ A by-value struct that smuggles a pointer, which must be rejected. / +struct with_ptr { + void p; + __u64 x; +}; + +/* A by-value union that smuggles a pointer, which must be rejected too. / +union upair_with_ptr { + void p; + __u64 halves[2]; +}; + +__naked struct with_ptr global_ret_struct_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("Global function global_ret_struct_ptr() has unsupported return type") +__naked int aggregate_ret_global_struct_ptr_fail(void) +{ + asm volatile ( + "call %[global_ret_struct_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(global_ret_struct_ptr) + : __clobber_all); +} + +__naked union upair_with_ptr global_ret_union_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("Global function global_ret_union_ptr() has unsupported return type") +__naked int aggregate_ret_global_union_ptr_fail(void) +{ + asm volatile ( + "call %[global_ret_union_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(global_ret_union_ptr) + : __clobber_all); +} + +struct ptr_pair { + void p; + __u64 x; +}; + +static __naked __noinline struct ptr_pair static_ret_ptr_pair(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = r1;" + "exit;" + ); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +__naked int aggregate_ret_static_ptr_pair(void) +{ + asm volatile ( + "call %[static_ret_ptr_pair];" + "r1 = (u32 )(r2 + 0);" / deref the returned ctx pointer / + "r0 = 0;" + "exit;" + : + : __imm(static_ret_ptr_pair) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c new file mode 100644 index 0000000000000..913cc374215d5 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#if defined(__clang_major__) && __clang_major__ >= 23 + +#define MIX_A 0xdeadbeefcafef00dULL +#define MIX_B 0x0123456789abcdefULL + +typedef unsigned __int128 u128; + +static __noinline u128 make_i128(__u64 a, __u64 b) +{ + return ((u128)(a + b) << 64) \| (a - b); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_int128_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + u128 v; + + v = make_i128(a, b); + if ((__u64)(v >> 64) != a + b) + return 1; + if ((__u64)v != a - b) + return 2; + + return 0; +} + +#else + +SEC("socket") +__description("aggregate_ret_int128_c: needs LLVM 23, dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c new file mode 100644 index 0000000000000..c23b4beb1773c --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c @@ -0,0 +1,122 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" +#include "../test_kmods/bpf_testmod_kfunc.h" + +/ + * Reference kfunc addresses to force those BTF to be emitted. Taking the address + * (rather than calling) avoids any dependence on the compiler lowering an + * __int128 or struct return value, which the BPF backend only supports from + * LLVM 23 on. + / +void __kfunc_btf_root(void) +{ + asm volatile ("" + : + : "r"(&bpf_kfunc_call_test_i128), + "r"(&bpf_kfunc_call_test_ret_fastcall), + "r"(&bpf_kfunc_call_test_ret_ptr), + "r"(&bpf_kfunc_call_test_ret_ii), + "r"(&bpf_kfunc_call_test_ret_big)); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128") +__naked int aggregate_ret_kfunc_precise(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + "call %[bpf_kfunc_call_test_i128];" + "r6 = r2;" / second return half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_i128) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL") +__naked int aggregate_ret_kfunc_fastcall_fail(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + "call %[bpf_kfunc_call_test_ret_fastcall];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_fastcall) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("is not composed of scalars") +__naked int aggregate_ret_kfunc_ptr_fail(void) +{ + asm volatile ( + "r1 = 0;" + "call %[bpf_kfunc_call_test_ret_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_ptr) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_kfunc_small_no_r2(void) +{ + asm volatile ( + "r1 = 0;" + "r2 = 0;" + "call %[bpf_kfunc_call_test_ret_ii];" + "r0 = r2;" / R2 is not a return register for a <=8 byte struct / + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_ii) + : __clobber_all); +} + +/ + * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by + * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks, + * so this behaves the same on every architecture. + / +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported") +__naked int aggregate_ret_kfunc_too_big_fail(void) +{ + asm volatile ( + "call %[bpf_kfunc_call_test_ret_big];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_big) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c new file mode 100644 index 0000000000000..2c1889fc28efd --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c @@ -0,0 +1,125 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "../test_kmods/bpf_testmod_kfunc.h" +#include "bpf_misc.h" + +#if defined(__clang_major__) && __clang_major__ >= 23 + +#define MIX_A 0xdeadbeefcafef00dULL +#define MIX_B 0x0123456789abcdefULL + +typedef unsigned __int128 u128; + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_int128_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + u128 v; + + v = bpf_kfunc_call_test_i128(a, b); + if ((__u64)(v >> 64) != a + b) + return 1; + if ((__u64)v != a - b) + return 2; + + return 0; +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct prog_test_ret_pair p; + + p = bpf_kfunc_call_test_ret_pair(a, b); + if (p.hi != a + b) + return 1; + if (p.lo != a - b) + return 2; + + return 0; +} + +/ struct { u64 a; int b; }: 16 bytes, R0 = a, R2 = b. / +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_li_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + int b = skb->len ^ MIX_B; + struct prog_test_ret_li r; + + r = bpf_kfunc_call_test_ret_li(a, b); + if (r.a != a) + return 1; + if (r.b != ~b) + return 2; + + return 0; +} + +/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not a return reg. / +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_ii_c_test(struct __sk_buff skb) +{ + int a = skb->len ^ MIX_A; + int b = skb->len ^ MIX_B; + struct prog_test_ret_ii r; + + r = bpf_kfunc_call_test_ret_ii(a, b); + if (r.a != a) + return 1; + if (r.b != b) + return 2; + + return 0; +} + +/* A union of 16 bytes takes the same R0:R2 path as a struct. / +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_uu_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + union prog_test_ret_uu r; + + r = bpf_kfunc_call_test_ret_uu(a, b); + if (r.parts.lo != a + b) + return 1; + if (r.parts.hi != a - b) + return 2; + + return 0; +} + +#else + +SEC("socket") +__description("aggregate_ret_kfunc_c: needs LLVM 23, dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c new file mode 100644 index 0000000000000..83b1a37519887 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +#if defined(__clang_major__) && __clang_major__ >= 23 + +#define MIX_A 0xdeadbeefcafef00dULL +#define MIX_B 0x0123456789abcdefULL + +struct pair { + __u64 hi; / R0 / + __u64 lo; / R2 / +}; + +union upair { + __u64 halves[2]; + struct { + __u64 lo; / R0 / + __u64 hi; / R2 / + } parts; +}; + +static __noinline struct pair make_pair(__u64 a, __u64 b) +{ + struct pair p = { .hi = a + b, .lo = a - b }; + + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct pair p; + + p = make_pair(a, b); + if (p.hi != a + b) + return 1; + if (p.lo != a - b) + return 2; + + return 0; +} + +__noinline struct pair make_pair_global(__u64 a, __u64 b) +{ + struct pair p = { .hi = a + b, .lo = a - b }; + + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_global_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct pair p; + + p = make_pair_global(a, b); + if (p.hi != a + b) + return 1; + if (p.lo != a - b) + return 2; + + return 0; +} + +static __noinline union upair make_upair(__u64 a, __u64 b) +{ + union upair p; + + p.halves[0] = a + b; + p.halves[1] = a - b; + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_union_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + union upair p; + + p = make_upair(a, b); + if (p.parts.lo != a + b) + return 1; + if (p.parts.hi != a - b) + return 2; + + return 0; +} + +#else + +SEC("socket") +__description("aggregate_ret_struct_c: needs LLVM 23, dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c new file mode 100644 index 0000000000000..cffd8d7d3241a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +/ freplace target: a global subprogram returning 16 bytes in R0:R2. / +__naked unsigned __int128 agg_ret_target_func(void) +{ + asm volatile ( + "r0 = 0x1234;" + "r2 = 0x5678;" + "exit;" + ); +} + +SEC("tc") +__naked int agg_ret_target(void) +{ + asm volatile ( + "call %[agg_ret_target_func];" + "r0 = 0;" + "exit;" + : + : __imm(agg_ret_target_func) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c index d055fc7b3b95d..0be9441ec273b 100644 --- a/tools/testing/selftests/bpf/progs/compute_live_registers.c +++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c @@ -431,6 +431,36 @@ __naked void subprog1(void) ::: __clobber_all); } +static __used __naked unsigned __int128 aux2(void) +{ + asm volatile ( + "r0 = 1;" + "r2 = 2;" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +/ A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). / +__load_if_JITed() +__log_level(2) +__msg("0: .12345.... (85) call pc+2") +__msg("1: ..2....... (bf) r0 = r2") +/ R2 is not read at the exit of this program, which returns an int, ... / +__msg("2: 0......... (95) exit") +__msg("3: .......... (b7) r0 = 1") +__msg("4: 0......... (b7) r2 = 2") +/ ... but it is at the exit of aux2(), which returns a register pair. / +__msg("5: 0.2....... (95) exit") +__naked void subprog_ret_reg_pair(void) +{ + asm volatile ( + "call aux2;" + "r0 = r2;" + "exit;" + ::: __clobber_all); +} + #if defined(__TARGET_ARCH_x86) \|\| defined(__TARGET_ARCH_arm64) SEC("socket") diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c index ac44d60e50666..9708efb93683b 100644 --- a/tools/testing/selftests/bpf/progs/exceptions_fail.c +++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c @@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a) SEC("?tc") __exception_cb(exception_cb_bad_ret_type1) -__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.") +__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.") int reject_exception_cb_type_1(struct __sk_buff ctx) { bpf_throw(0); diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c new file mode 100644 index 0000000000000..12c15d293bd79 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> + +SEC("freplace/agg_ret_target_func") +__u64 new_agg_ret_target_func(void) +{ + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c index a6133f7521f34..ad36d583d2e7f 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -939,6 +939,69 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c) return 0; } +#if defined(__x86_64__) \|\| defined(__aarch64__) +__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b) +{ + return (__int128)(((unsigned __int128)(a + b) << 64) \| (a - b)); +} + +__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b) +{ + struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b) +{ + struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b) +{ + struct prog_test_ret_li r = { .a = a, .b = ~b }; + + return r; +} + +__bpf_kfunc union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b) +{ + union prog_test_ret_uu r; + + r.halves[0] = a + b; + r.halves[1] = a - b; + return r; +} + +__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag) +{ + struct prog_test_ret_ptr r = { .p = NULL, .tag = tag }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) +{ + struct prog_test_ret_ii r = { .a = a, .b = b }; + + return r; +} +#endif / __x86_64__ \|\| __aarch64__ / + +/ + * Takes no argument on purpose: with no arguments there is nothing for the sret + * pointer to displace, so this needs no architecture guard even though it + * returns 24 bytes. See the comment on bpf_kfunc_call_test_i128() above. + / +__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) +{ + struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 }; + + return r; +} + __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f, u64 g, u64 h, u64 i, u64 j) @@ -1472,6 +1535,16 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2) BTF_ID_FLAGS(func, bpf_kfunc_call_test3) BTF_ID_FLAGS(func, bpf_kfunc_call_test4) BTF_ID_FLAGS(func, bpf_kfunc_call_test5) +#if defined(__x86_64__) \|\| defined(__aarch64__) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii) +#endif +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h index c4383acb53c11..c7be973cd2860 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h @@ -55,6 +55,40 @@ struct prog_test_big_arg { __u64 b; }; +struct prog_test_ret_pair { + __u64 hi; + __u64 lo; +}; + +struct prog_test_ret_li { / 16 bytes: R0:R2 / + __u64 a; + int b; +}; + +struct prog_test_ret_ii { / 8 bytes: R0 only / + int a; + int b; +}; + +union prog_test_ret_uu { / 16 bytes: R0:R2 / + __u64 halves[2]; + struct { + __u64 lo; + __u64 hi; + } parts; +}; + +struct prog_test_ret_ptr { / 16 bytes: contains a pointer / + void p; + __u64 tag; +}; + +struct prog_test_ret_big { /* 24 bytes: too large for R0:R2 / + __u64 a; + __u64 b; + __u64 c; +}; + struct prog_test_fail1 { void p; int x; @@ -130,6 +164,16 @@ int bpf_kfunc_call_test2(struct sock sk, __u32 a, __u32 b) __ksym; struct sock bpf_kfunc_call_test3(struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; +#ifdef __SIZEOF_INT128__ +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym; +#endif +struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym; +struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym; +struct prog_test_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym; +struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym; +union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym; +struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym; +struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym; __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, __u64 g, __u64 h, __u64 i, __u64 j) __ksym;
7/3	2026/08/13 20:58	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 27622 output: 258 thoughts: 3804 Thoughts: BPF Subprogram Return Value Expansion Analysis Alright, let's break down this patch. It's all about beefing up BPF's return value capabilities, letting subprograms and kfuncs sling back values up to 16 bytes. The core is the BPF verifier, and the JIT compilers. Specifically, we're tracking a second register, R2, alongside R0 for these larger returns. The verifier's gotta be on point, ensuring R2 is properly initialized and preventing any pointer shenanigans. I see a bunch of verifier changes: `bpf_ret_reg_pair` and `bpf_kfunc_ret_reg_pair` for checking, `backtrack_insn` for R2 propagation, `btf_validate_return_type` making sure structs are scalar-only, `compute_insn_live_regs` marking R2 as used at `BPF_EXIT`, `check_global_subprog_return_code` inspecting both R0 and R2, `prepare_func_exit` for moving both, and `mark_kfunc_ret_regs` declaring them as unknown scalars. Basically, the verifier is all over it, which is good. My immediate concern is: Are we exposing uninitialized memory to user space? Not directly. The return values are in registers, within the BPF program's environment or from the kfunc to BPF. If a kfunc somehow returns an uninitialized struct, that's visible to the BPF program, but kfuncs are supposed to initialize their returns anyway. Test kfuncs are added to verify this. And if BPF itself returns uninitialized data, `check_global_subprog_return_code` checks that R0 and R2 are scalar – which means, they are initialized. But hang on… `check_global_ret_scalar_reg` checks that the register isn't `NOT_INIT`. That's how it's done: If it's uninitialized, it won't pass `check_reg_arg(env, regno, SRC_OP)`. This is a good thing. We're covered there. Okay, so no real risk of raw, uninitialized memory leaks to user space from the BPF program's point of view. The verifier is there. Kfuncs are standard C, returning structs by value and C compilers handle the initialization. Padding is the thing that concerns me. The verifier needs to make sure to enforce this. The real concern is that kfuncs are required to return structs composed of only scalar values. If there's padding in the struct, that padding is now exposed to the BPF program as part of the return value, in R0 and R2. The BPF program could read that padding, potentially leaking kernel stack memory or whatever was sitting there. Does the verifier actually check for padding? It calls `btf_type_is_scalar_struct`, but does that function look at padding specifically? Let's check… nope. It just verifies the members are scalars, but doesn't explicitly look for gaps. Consider a kfunc like this: ```c struct { u8 a; // 7 bytes padding u64 b; }; ``` That's 16 bytes. R0 gets `a` and padding; R2 gets `b`. The BPF program sees that padding in R0. If that padding is uninitialized…that's a leak! So, can KMSAN help? Normally, KMSAN would catch the use of uninitialized memory. KMSAN could trigger if a kfunc returns a struct with uninitialized padding when it is copied to the return registers, or even when the JITed BPF code attempts to access those registers. But KMSAN doesn't instrument the JIT'ed BPF code! KMSAN only instruments the kernel, so KMSAN would only catch the kfunc returning the struct. Returning it with padding on the stack would be a problem. That could go undetected. Is this patch introducing that kind of kfunc, or is it a preexisting issue? The patch itself doesn't add any real new kfuncs. It only adds some test kfuncs. The one here, `struct prog_test_ret_li`, has a 4-byte padding. When initializing it in a kfunc, the C99 standard should zero out the padding due to static storage. In reality, compiler behavior is what matters. If the initializer covers the whole struct, everything is zeroed. But, if the compiler only initializes members with an initializer, the padding might be uninitialized. The patch itself enables support for kfuncs, not
8/3	2026/08/13 20:59	tool	set-results	0m Args: { "KMSANReasoning": "The patch extends the BPF verifier and JIT compilers to support returning values up to 16 bytes (e.g., `__int128` or small structs) from BPF subprograms and kfuncs using the R0 and R2 register pair. The changes primarily involve updating the verifier's liveness tracking, backtracking, and return value checks to ensure that both R0 and R2 are properly initialized before a BPF program returns. The JIT changes simply move the native ABI's second return register into BPF's R2. \n\nThere are no new kernel data structures allocated and copied to user space, nor any C-level uninitialized memory risks introduced. The verifier statically enforces that BPF programs initialize the R2 register when returning a pair, preventing uninitialized register leaks. Furthermore, KMSAN does not instrument JIT-compiled BPF code, so it would not provide additional coverage for the BPF program execution itself. Standard KASAN and verifier fuzzing are sufficient to detect any logic bugs, out-of-bounds accesses, or memory corruptions introduced by these changes.", "NeedsKMSAN": false } Results: { "KMSANReasoning": "The patch extends the BPF verifier and JIT compilers to support returning values up to 16 bytes (e.g., `__int128` or small structs) from BPF subprograms and kfuncs using the R0 and R2 register pair. The changes primarily involve updating the verifier's liveness tracking, backtracking, and return value checks to ensure that both R0 and R2 are properly initialized before a BPF program returns. The JIT changes simply move the native ABI's second return register into BPF's R2. \n\nThere are no new kernel data structures allocated and copied to user space, nor any C-level uninitialized memory risks introduced. The verifier statically enforces that BPF programs initialize the R2 register when returning a pair, preventing uninitialized register leaks. Furthermore, KMSAN does not instrument JIT-compiled BPF code, so it would not provide additional coverage for the BPF program execution itself. Standard KASAN and verifier fuzzing are sufficient to detect any logic bugs, out-of-bounds accesses, or memory corruptions introduced by these changes.", "NeedsKMSAN": false }

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT"
  ],
  "FocusSymbols": [
    "bpf_kfunc_ret_reg_pair",
    "btf_type_is_scalar_struct",
    "bpf_jit_supports_kfunc_ret_reg_pair",
    "bpf_compute_live_registers"
  ],
  "KMSANReasoning": "The patch extends the BPF verifier and JIT compilers to support returning values up to 16 bytes (e.g., `__int128` or small structs) from BPF subprograms and kfuncs using the R0 and R2 register pair. The changes primarily involve updating the verifier's liveness tracking, backtracking, and return value checks to ensure that both R0 and R2 are properly initialized before a BPF program returns. The JIT changes simply move the native ABI's second return register into BPF's R2. \n\nThere are no new kernel data structures allocated and copied to user space, nor any C-level uninitialized memory risks introduced. The verifier statically enforces that BPF programs initialize the R2 register when returning a pair, preventing uninitialized register leaks. Furthermore, KMSAN does not instrument JIT-compiled BPF code, so it would not provide additional coverage for the BPF program execution itself. Standard KASAN and verifier fuzzing are sufficient to detect any logic bugs, out-of-bounds accesses, or memory corruptions introduced by these changes.",
  "NeedsKMSAN": false,
  "Reasoning": "The patch introduces support for kfuncs and BPF subprograms returning larger values (up to 16 bytes) in the R0:R2 register pair. This involves changes to the BPF verifier (liveness tracking, return value checking, BTF validation) and JIT compilers (x86, arm64, riscv). These are significant functional changes to the core BPF subsystem that should be fuzzed to ensure verifier safety and JIT correctness.",
  "WorthFuzzing": true
}

{
  "PatchDiff": "commit bafb5dc76d94a94d43e7b1575db7d4920a549687\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate:   Thu Aug 13 20:58:12 2026 +0000\n\n    syz-cluster: applied patch under review\n\ndiff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst\nindex 10e725cbe64cd..5d840d3000338 100644\n--- a/Documentation/bpf/kfuncs.rst\n+++ b/Documentation/bpf/kfuncs.rst\n@@ -575,6 +575,69 @@ is also covered by this recovery. A kfunc handed an arena pointer may\n therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking\n against the arena. Larger accesses must verify the range explicitly.\n \n+2.9 kfunc Return Values\n+-----------------------\n+\n+A kfunc may return a scalar, a pointer, or a small struct or union by\n+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.\n+\n+A struct or union returned by value must be composed only of scalars\n+(recursively), where a scalar is an integer or an enum; arrays of scalars are\n+allowed as members. Its bytes are handed back to the program as the raw\n+contents of R0 (and R2), so a pointer field would be laundered into a scalar\n+and escape the verifier's pointer provenance and reference tracking. A struct\n+or union with a pointer member is therefore rejected at load time, and so is\n+one with a floating-point member, which the ABI may not return in R0:R2 at\n+all.\n+\n+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a\n+scalar-only struct or union, or an ``__int128``. Such a value is returned\n+in the register pair R0:R2, matching the convention LLVM uses for the BPF\n+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or\n+union of 8 bytes or less is returned in R0 alone.\n+\n+::\n+\n+        struct bpf_pair { __u64 a, b; };   /* 16 bytes */\n+\n+        __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)\n+        {\n+                struct bpf_pair p = { .a = 1, .b = 2 };\n+\n+                return p;      /* p.a in R0, p.b in R2 */\n+        }\n+\n+Returning a value in the R0:R2 pair requires the JIT to place the second\n+half of the return value into R2, which not every architecture supports\n+right now. A kfunc with a return value larger than 8 bytes is therefore\n+rejected at load time on a JIT that does not advertise this capability (see\n+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run\n+by the interpreter. A return value larger than 16 bytes is not supported.\n+\n+The same R0:R2 convention applies to a BPF subprogram, global or static,\n+that returns an ``__int128`` or a struct or union larger than 8 bytes. Such a\n+program also requires the JIT, since the interpreter propagates only R0 out\n+of a subprogram. A global subprogram is verified in isolation, so its\n+by-value struct or union return is restricted to scalars just like a kfunc's;\n+a static subprogram is verified inline and has no such restriction. The main\n+program is not covered: its return value is the program's exit code, read out\n+of R0 alone, so a declared upper half is never looked at.\n+\n+A global subprogram must leave a scalar in *every* register of the pair, so\n+both halves of the returned value have to be assigned. Leaving the upper half\n+uninitialized is not merely untidy: the compiler is then free to leave R2\n+holding whatever it happened to hold, which for a subprogram taking a pointer\n+argument is typically that pointer. Handing the caller an unknown scalar built\n+from a pointer is a leak, so the verifier rejects it with::\n+\n+        At subprogram exit the register R2 is not a scalar value (...)\n+\n+Initialize the whole return value, for example ``struct pair p = {};``, to\n+avoid this. A static subprogram is exempt from the scalar-only rule: it is\n+verified inline, so an unassigned R2 is simply passed back to the caller as\n+uninitialized and only a caller that reads it fails. A stack pointer left in\n+R2 is still rejected there, just as one in R0 is.\n+\n .. _BPF_kfunc_lifecycle_expectations:\n \n 3. kfunc lifecycle expectations\ndiff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c\nindex 74b4083791da3..7a4c5968976a6 100644\n--- a/arch/arm64/net/bpf_jit_comp.c\n+++ b/arch/arm64/net/bpf_jit_comp.c\n@@ -2346,6 +2346,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_stack_args(void)\n {\n \treturn true;\ndiff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c\nindex 2504df1fa1118..49220765e96d4 100644\n--- a/arch/riscv/net/bpf_jit_comp64.c\n+++ b/arch/riscv/net/bpf_jit_comp64.c\n@@ -2120,6 +2120,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_ptr_xchg(void)\n {\n \treturn true;\ndiff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c\nindex d920772af7d5f..2ec09dd02b187 100644\n--- a/arch/x86/net/bpf_jit_comp.c\n+++ b/arch/x86/net/bpf_jit_comp.c\n@@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,\n  * arena NULL is offset 0. Return the number of emitted bytes.\n  */\n static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,\n-\t\t\t\t const struct bpf_insn *insn, u8 **pprog)\n+\t\t\t\t const struct btf_func_model *fm, u8 **pprog)\n {\n-\tconst struct btf_func_model *fm;\n \tu8 *prog = *pprog;\n \tu8 *start = prog;\n \tint i;\n \n-\tfm = bpf_jit_find_kfunc_model(bpf_prog, insn);\n-\tif (!fm)\n-\t\treturn -EINVAL;\n-\n \tfor (i = 0; i \u003c min_t(int, fm-\u003enr_args, MAX_BPF_FUNC_REG_ARGS); i++) {\n \t\tu8 flags = fm-\u003earg_flags[i];\n \t\tu32 reg = BPF_REG_1 + i;\n@@ -2644,6 +2639,8 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \n \t\t\t/* call */\n \t\tcase BPF_JMP | BPF_CALL: {\n+\t\t\tconst struct btf_func_model *fm = NULL;\n+\n \t\t\tfunc = (u8 *) __bpf_call_base + imm32;\n \t\t\tif (src_reg == BPF_PSEUDO_CALL \u0026\u0026 tail_call_reachable) {\n \t\t\t\tLOAD_TAIL_CALL_CNT_PTR(stack_depth);\n@@ -2652,7 +2649,10 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tif (!imm32)\n \t\t\t\treturn -EINVAL;\n \t\t\tif (src_reg == BPF_PSEUDO_KFUNC_CALL) {\n-\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, insn, \u0026prog);\n+\t\t\t\tfm = bpf_jit_find_kfunc_model(bpf_prog, insn);\n+\t\t\t\tif (!fm)\n+\t\t\t\t\treturn -EINVAL;\n+\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, fm, \u0026prog);\n \t\t\t\tif (err \u003c 0)\n \t\t\t\t\treturn err;\n \t\t\t\tip += err;\n@@ -2666,6 +2666,14 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\treturn -EINVAL;\n \t\t\tif (priv_frame_ptr)\n \t\t\t\tpop_r9(\u0026prog);\n+\t\t\t/*\n+\t\t\t * A kfunc returning more than 8 bytes hands the second\n+\t\t\t * half back in RDX (the native ABI's second return reg),\n+\t\t\t * but BPF expects it in R0:R2. BPF R0 is RAX (no move\n+\t\t\t * needed), while BPF R2 is RSI, so copy RDX into RSI.\n+\t\t\t */\n+\t\t\tif (fm \u0026\u0026 fm-\u003eret_size \u003e 8)\n+\t\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_2, BPF_REG_3);\n \t\t\tbreak;\n \t\t}\n \n@@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_stack_args(void)\n {\n \treturn true;\ndiff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h\nindex 27b43fda9b178..899e5ce986432 100644\n--- a/include/linux/bpf_verifier.h\n+++ b/include/linux/bpf_verifier.h\n@@ -814,6 +814,8 @@ struct bpf_subprog_info {\n \tbool is_async_cb: 1;\n \tbool is_exception_cb: 1;\n \tbool args_cached: 1;\n+\t/* true if the return value is passed in the R0:R2 register pair */\n+\tbool ret_reg_pair: 1;\n \t/* true if bpf_fastcall stack region is used by functions that can't be inlined */\n \tbool keep_fastcall_stack: 1;\n \tbool changes_pkt_data: 1;\n@@ -1048,6 +1050,13 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env\n \treturn \u0026env-\u003esubprog_info[subprog];\n }\n \n+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)\n+{\n+\treturn subprog_info(env, subprog)-\u003eret_reg_pair;\n+}\n+\n+bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env *env, struct bpf_insn *insn);\n+\n struct bpf_call_summary {\n \tu8 num_params;\n \tbool is_void;\n@@ -1439,6 +1448,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);\n struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);\n void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);\n bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);\n+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,\n+\t\t\t       const struct btf_type *t, int rec);\n \n int bpf_find_subprog(struct bpf_verifier_env *env, int off);\n bool bpf_is_throw_kfunc(struct bpf_insn *insn);\ndiff --git a/include/linux/filter.h b/include/linux/filter.h\nindex 4a9bc6a848f2e..6e746b0a09306 100644\n--- a/include/linux/filter.h\n+++ b/include/linux/filter.h\n@@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);\n bool bpf_jit_supports_subprog_tailcalls(void);\n bool bpf_jit_supports_percpu_insn(void);\n bool bpf_jit_supports_kfunc_call(void);\n+bool bpf_jit_supports_kfunc_ret_reg_pair(void);\n bool bpf_jit_supports_stack_args(void);\n bool bpf_jit_supports_arena_args(void);\n bool bpf_jit_supports_far_kfunc_call(void);\ndiff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c\nindex 40bd04421a991..3ef8a3da726de 100644\n--- a/kernel/bpf/backtrack.c\n+++ b/kernel/bpf/backtrack.c\n@@ -425,6 +425,16 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\t\t */\n \t\t\t\tverifier_bug_if(idx + 1 != subseq_idx, env,\n \t\t\t\t\t\t\"extra insn from subprog\");\n+\t\t\t\t/* a global subprog returning more than 8 bytes\n+\t\t\t\t * sets R2 as well. R2 is part of the args mask\n+\t\t\t\t * checked just below, so clear it here rather\n+\t\t\t\t * than next to R0. Only a subprog that does\n+\t\t\t\t * return a pair defines R2, so leave the mask\n+\t\t\t\t * alone otherwise and let the check below catch\n+\t\t\t\t * an R2 that has no business being set.\n+\t\t\t\t */\n+\t\t\t\tif (bpf_ret_reg_pair(env, subprog))\n+\t\t\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\t\t/* r1-r5 are invalidated after subprog call,\n \t\t\t\t * so for global func call it shouldn't be set\n \t\t\t\t * anymore\n@@ -508,6 +518,17 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\t\treturn -ENOTSUPP;\n \t\t\t/* regular helper call sets R0 */\n \t\t\tbt_clear_reg(bt, BPF_REG_0);\n+\t\t\t/* a kfunc returning more than 8 bytes also sets R2.\n+\t\t\t * R2 is part of the args mask checked just below, so\n+\t\t\t * clear it here rather than next to R0. The prototype\n+\t\t\t * lookup is only worth doing when R2 is requested at\n+\t\t\t * all; any other call leaves R2 uninitialized, so a\n+\t\t\t * request for it is caught by the check below.\n+\t\t\t */\n+\t\t\tif (bt_is_reg_set(bt, BPF_REG_2) \u0026\u0026\n+\t\t\t    insn-\u003esrc_reg == BPF_PSEUDO_KFUNC_CALL \u0026\u0026\n+\t\t\t    bpf_kfunc_ret_reg_pair(env, insn))\n+\t\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\tif (bt_reg_mask(bt) \u0026 BPF_REGMASK_ARGS) {\n \t\t\t\t/* if backtracking was looking for registers R1-R5\n \t\t\t\t * they should have been found already.\n@@ -522,7 +543,41 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\t\t\treturn -EFAULT;\n \t\t\t}\n \t\t} else if (opcode == BPF_EXIT) {\n-\t\t\tbool r0_precise;\n+\t\t\tbool from_subprog_call, r0_precise, r2_precise;\n+\t\t\tstruct bpf_insn *call;\n+\t\t\tint subprog;\n+\n+\t\t\t/* BPF_EXIT in subprog or callback always returns\n+\t\t\t * right after the call instruction, so by checking\n+\t\t\t * whether the instruction at subseq_idx-1 is subprog\n+\t\t\t * call or not we can distinguish actual exit from\n+\t\t\t * *subprog* from exit from *callback*. In the former\n+\t\t\t * case, we need to propagate the precision of the\n+\t\t\t * return registers, if necessary. In the latter we\n+\t\t\t * never do that.\n+\t\t\t */\n+\t\t\tfrom_subprog_call = subseq_idx - 1 \u003e= 0 \u0026\u0026\n+\t\t\t\t\t    bpf_pseudo_call(\u0026env-\u003eprog-\u003einsnsi[subseq_idx - 1]);\n+\n+\t\t\t/* Sample the return registers before the callback\n+\t\t\t * handling below clears R1-R5: unlike R0, R2 is an\n+\t\t\t * argument register as well, so that clear would drop\n+\t\t\t * a pair return on the floor.\n+\t\t\t */\n+\t\t\tr0_precise = from_subprog_call \u0026\u0026 bt_is_reg_set(bt, BPF_REG_0);\n+\t\t\tr2_precise = false;\n+\t\t\tif (from_subprog_call \u0026\u0026 bt_is_reg_set(bt, BPF_REG_2)) {\n+\t\t\t\tcall = \u0026env-\u003eprog-\u003einsnsi[subseq_idx - 1];\n+\t\t\t\tsubprog = bpf_find_subprog(env, subseq_idx + call-\u003eimm);\n+\t\t\t\tif (subprog \u003c 0)\n+\t\t\t\t\treturn -EFAULT;\n+\t\t\t\t/* Only a callee that does return a pair defines\n+\t\t\t\t * R2. Leave the mask alone otherwise, so that\n+\t\t\t\t * the check below still catches an R2 that has\n+\t\t\t\t * no business being set.\n+\t\t\t\t */\n+\t\t\t\tr2_precise = bpf_ret_reg_pair(env, subprog);\n+\t\t\t}\n \n \t\t\t/* Backtracking to a nested function call, 'idx' is a part of\n \t\t\t * the inner frame 'subseq_idx' is a part of the outer frame.\n@@ -535,30 +590,27 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\tif (subseq_idx \u003e= 0 \u0026\u0026 bpf_calls_callback(env, subseq_idx))\n \t\t\t\tfor (i = BPF_REG_1; i \u003c= BPF_REG_5; i++)\n \t\t\t\t\tbt_clear_reg(bt, i);\n+\n+\t\t\t/* a callee returning more than 8 bytes sets R2 as well;\n+\t\t\t * R2 is part of the args mask checked just below, so\n+\t\t\t * clear it here rather than next to R0.\n+\t\t\t */\n+\t\t\tif (r2_precise)\n+\t\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\tif (bt_reg_mask(bt) \u0026 BPF_REGMASK_ARGS) {\n \t\t\t\tverifier_bug(env, \"backtracking exit unexpected regs %x\",\n \t\t\t\t\t     bt_reg_mask(bt));\n \t\t\t\treturn -EFAULT;\n \t\t\t}\n \n-\t\t\t/* BPF_EXIT in subprog or callback always returns\n-\t\t\t * right after the call instruction, so by checking\n-\t\t\t * whether the instruction at subseq_idx-1 is subprog\n-\t\t\t * call or not we can distinguish actual exit from\n-\t\t\t * *subprog* from exit from *callback*. In the former\n-\t\t\t * case, we need to propagate r0 precision, if\n-\t\t\t * necessary. In the former we never do that.\n-\t\t\t */\n-\t\t\tr0_precise = subseq_idx - 1 \u003e= 0 \u0026\u0026\n-\t\t\t\t     bpf_pseudo_call(\u0026env-\u003eprog-\u003einsnsi[subseq_idx - 1]) \u0026\u0026\n-\t\t\t\t     bt_is_reg_set(bt, BPF_REG_0);\n-\n \t\t\tbt_clear_reg(bt, BPF_REG_0);\n \t\t\tif (bt_subprog_enter(bt))\n \t\t\t\treturn -EFAULT;\n \n \t\t\tif (r0_precise)\n \t\t\t\tbt_set_reg(bt, BPF_REG_0);\n+\t\t\tif (r2_precise)\n+\t\t\t\tbt_set_reg(bt, BPF_REG_2);\n \t\t\t/* r6-r9 and stack slots will stay set in caller frame\n \t\t\t * bitmasks until we return back from callee(s)\n \t\t\t */\ndiff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c\nindex 6606187ed4f43..c7718a2a56141 100644\n--- a/kernel/bpf/btf.c\n+++ b/kernel/bpf/btf.c\n@@ -7592,7 +7592,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,\n \t\treturn -EINVAL;\n \t}\n \tret = __get_type_size(btf, func-\u003etype, \u0026t);\n-\tif (ret \u003c 0 || btf_type_is_struct(t)) {\n+\tif (ret \u003c 0 || ret \u003e 16) {\n \t\tbpf_log(log,\n \t\t\t\"The function %s return type %s is unsupported.\\n\",\n \t\t\ttname, btf_type_str(t));\n@@ -7965,7 +7965,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env *env,\n \n /* Check whether the type is a valid return type. */\n static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,\n-\t\tconst struct btf_type *t, int subprog)\n+\t\tconst struct btf_type *t, int subprog, bool is_global)\n {\n \tu32 tags = 0;\n \tint err;\n@@ -7988,6 +7988,19 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt\n \tif (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))\n \t\treturn 0;\n \n+\tif (btf_type_is_struct(t) \u0026\u0026 t-\u003esize \u003c= 16) {\n+\t\t/*\n+\t\t * A global function's caller models the return as an opaque\n+\t\t * scalar pair, so it may only return scalars by value. A local\n+\t\t * function is verified inline, so a pointer field stays tracked\n+\t\t * and needs no such restriction.\n+\t\t */\n+\t\tbool local_func = subprog \u0026\u0026 !is_global;\n+\n+\t\tif (local_func || btf_type_is_scalar_struct(env, btf, t, 0))\n+\t\t\treturn 0;\n+\t}\n+\n \treturn -EOPNOTSUPP;\n }\n \n@@ -8075,12 +8088,12 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)\n \t\treturn -EINVAL;\n \t}\n \n-\terr = btf_validate_return_type(env, btf, t, subprog);\n+\terr = btf_validate_return_type(env, btf, t, subprog, is_global);\n \tif (err) {\n \t\tif (is_global) {\n \t\t\tbpf_log(log,\n-\t\t\t\t\"Global function %s() return value not void or scalar. \"\n-\t\t\t\t\"Only those are supported.\\n\",\n+\t\t\t\t\"Global function %s() has unsupported return type. \"\n+\t\t\t\t\"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\\n\",\n \t\t\t\ttname);\n \t\t}\n \t\treturn err;\ndiff --git a/kernel/bpf/core.c b/kernel/bpf/core.c\nindex 6a94370a24488..cb66a2ef52b1f 100644\n--- a/kernel/bpf/core.c\n+++ b/kernel/bpf/core.c\n@@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)\n \treturn false;\n }\n \n+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn false;\n+}\n+\n bool __weak bpf_jit_supports_stack_args(void)\n {\n \treturn false;\ndiff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c\nindex 1c997aeba6fa5..850c58aa915be 100644\n--- a/kernel/bpf/liveness.c\n+++ b/kernel/bpf/liveness.c\n@@ -2062,7 +2062,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m \u003e\u003e 16); }\n /* Compute info-\u003e{use,def} fields for the instruction */\n static void compute_insn_live_regs(struct bpf_verifier_env *env,\n \t\t\t\t   struct bpf_insn *insn,\n-\t\t\t\t   struct insn_live_regs *info)\n+\t\t\t\t   struct insn_live_regs *info,\n+\t\t\t\t   bool ret_reg_pair)\n {\n \tstruct bpf_call_summary cs;\n \tconst u8 class = BPF_CLASS(insn-\u003ecode);\n@@ -2074,6 +2075,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,\n \tconst u32 src32 = mask_lo(src);\n \tconst u32 dst32 = mask_lo(dst);\n \tconst u32 r0  = reg64_mask(0);\n+\tconst u32 r2  = reg64_mask(BPF_REG_2);\n \tu32 def = 0;\n \tu32 use = U32_MAX;\n \n@@ -2193,7 +2195,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,\n \t\t\tbreak;\n \t\tcase BPF_EXIT:\n \t\t\tdef = 0;\n-\t\t\tuse = r0;\n+\t\t\tuse = ret_reg_pair ? (r0 | r2) : r0;\n \t\t\tbreak;\n \t\tcase BPF_CALL:\n \t\t\tdef = ALL_CALLER_SAVED_REGS;\n@@ -2230,8 +2232,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)\n \tstruct insn_live_regs *state;\n \tint insn_cnt = env-\u003eprog-\u003elen;\n \tu64 pos, insn_pos;\n-\tint err = 0, i, j;\n-\tbool changed;\n+\tint err = 0, i, j, subprog, start, end;\n+\tbool changed, ret_reg_pair;\n \n \t/* Use the following algorithm:\n \t * - define the following:\n@@ -2258,8 +2260,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)\n \t\tgoto out;\n \t}\n \n-\tfor (i = 0; i \u003c insn_cnt; ++i)\n-\t\tcompute_insn_live_regs(env, \u0026insns[i], \u0026state[i]);\n+\tfor (subprog = 0; subprog \u003c env-\u003esubprog_cnt; subprog++) {\n+\t\tstart = env-\u003esubprog_info[subprog].start;\n+\t\tend = env-\u003esubprog_info[subprog + 1].start;\n+\t\tret_reg_pair = bpf_ret_reg_pair(env, subprog);\n+\n+\t\tfor (i = start; i \u003c end; ++i)\n+\t\t\tcompute_insn_live_regs(env, \u0026insns[i], \u0026state[i], ret_reg_pair);\n+\t}\n \n \t/* Forward pass: resolve stack access through FP-derived pointers */\n \terr = bpf_compute_subprog_arg_access(env);\ndiff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c\nindex 164d16c243ca6..98a6f702d794d 100644\n--- a/kernel/bpf/verifier.c\n+++ b/kernel/bpf/verifier.c\n@@ -382,27 +382,80 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)\n \treturn aux \u0026\u0026 aux[subprog].linkage == BTF_FUNC_GLOBAL;\n }\n \n-static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)\n+static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)\n {\n-\tconst struct btf_type *type, *func, *func_proto;\n+\tconst struct btf_type *func, *func_proto;\n \tconst struct btf *btf = env-\u003eprog-\u003eaux-\u003ebtf;\n \tu32 btf_id;\n \n+\tif (!btf || !env-\u003eprog-\u003eaux-\u003efunc_info)\n+\t\treturn NULL;\n+\n \tbtf_id = env-\u003eprog-\u003eaux-\u003efunc_info[subprog].type_id;\n \n+\t/* Both already validated by prepare_btf_func() at prog load. */\n \tfunc = btf_type_by_id(btf, btf_id);\n-\tif (verifier_bug_if(!func, env, \"btf_id %u not found\", btf_id))\n-\t\treturn false;\n-\n \tfunc_proto = btf_type_by_id(btf, func-\u003etype);\n-\tif (!func_proto)\n-\t\treturn false;\n \n-\ttype = btf_type_skip_modifiers(btf, func_proto-\u003etype, NULL);\n-\tif (!type)\n-\t\treturn false;\n+\treturn btf_type_skip_modifiers(btf, func_proto-\u003etype, NULL);\n+}\n+\n+static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)\n+{\n+\tconst struct btf_type *type = subprog_ret_type(env, subprog);\n+\n+\treturn type \u0026\u0026 btf_type_is_void(type);\n+}\n+\n+static u32 ret_regs_cnt(u32 size)\n+{\n+\treturn size \u003e 8 \u0026\u0026 size \u003c= 16 ? 2 : 1;\n+}\n+\n+/* Registers holding a function return value, in order. See ret_regs_cnt(). */\n+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };\n+\n+static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)\n+{\n+\tconst struct btf *btf = env-\u003eprog-\u003eaux-\u003ebtf;\n+\tconst struct btf_type *type;\n+\tint subprog;\n+\tu32 size;\n+\n+\tfor (subprog = 0; subprog \u003c env-\u003esubprog_cnt; subprog++) {\n+\t\ttype = subprog_ret_type(env, subprog);\n+\t\tif (!type || !(btf_type_is_struct(type) || btf_type_is_scalar(type)))\n+\t\t\tcontinue;\n+\t\tif (IS_ERR(btf_resolve_size(btf, type, \u0026size)))\n+\t\t\tcontinue;\n+\t\tif (ret_regs_cnt(size) \u003e 1) {\n+\t\t\tsubprog_info(env, subprog)-\u003eret_reg_pair = true;\n+\t\t\t/*\n+\t\t\t * The R0:R2 return convention is only implemented in\n+\t\t\t * the JIT: the interpreter propagates BPF_R0 alone out\n+\t\t\t * of a subprogram, so a caller reading R2 would see a\n+\t\t\t * stale value.\n+\t\t\t */\n+\t\t\tenv-\u003eprog-\u003ejit_required = 1;\n+\t\t}\n+\t}\n+}\n \n-\treturn btf_type_is_void(type);\n+/*\n+ * A \u003e8 byte BPF return changes the calling convention to R0:R2, and the\n+ * verifier derives that convention from the subprogram's BTF prototype\n+ * alone. Once that prototype is marked unreliable it is known not to\n+ * describe the compiled code, so the convention read from it cannot be\n+ * trusted either: reject the call rather than keep tracking R2 on the\n+ * strength of a signature the verifier has already discarded.\n+ */\n+static bool subprog_ret_pair_unreliable(struct bpf_verifier_env *env, int subprog)\n+{\n+\tstruct bpf_prog_aux *aux = env-\u003eprog-\u003eaux;\n+\n+\treturn bpf_ret_reg_pair(env, subprog) \u0026\u0026\n+\t       aux-\u003efunc_info_aux \u0026\u0026\n+\t       aux-\u003efunc_info_aux[subprog].unreliable;\n }\n \n static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)\n@@ -2459,6 +2512,21 @@ find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset)\n \t\t       sizeof(tab-\u003edescs[0]), kfunc_desc_cmp_by_id_off);\n }\n \n+/*\n+ * True if the kfunc called by @insn returns its value in the R0:R2 pair.\n+ * Reads the same btf_func_model.ret_size that bpf_add_kfunc_call() validated\n+ * and that the JIT keys the second return register off, so the verifier and\n+ * the generated code cannot disagree about the convention.\n+ */\n+bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env *env, struct bpf_insn *insn)\n+{\n+\tconst struct bpf_kfunc_desc *desc;\n+\n+\tdesc = find_kfunc_desc(env-\u003eprog, insn-\u003eimm, insn-\u003eoff);\n+\n+\treturn desc \u0026\u0026 ret_regs_cnt(desc-\u003efunc_model.ret_size) \u003e 1;\n+}\n+\n int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,\n \t\t       u16 btf_fd_idx, u8 **func_addr)\n {\n@@ -2809,6 +2877,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)\n \terr = btf_distill_func_proto(\u0026env-\u003elog, kfunc.btf, kfunc.proto, kfunc.name, \u0026func_model);\n \tif (err)\n \t\treturn err;\n+\tif (func_model.ret_size \u003e 8) {\n+\t\tif (kfunc.flags \u0026\u0026 (*kfunc.flags \u0026 KF_FASTCALL)) {\n+\t\t\tverbose(env, \"kfunc %s with \u003e8-byte return is not supported with KF_FASTCALL\\n\",\n+\t\t\t\tkfunc.name);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\t\tif (!bpf_jit_supports_kfunc_ret_reg_pair()) {\n+\t\t\tverbose(env, \"kfunc %s with \u003e8-byte return is not supported by JIT\\n\",\n+\t\t\t\tkfunc.name);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\t}\n \n \tmemset(\u0026meta, 0, sizeof(meta));\n \tmeta.btf = kfunc.btf;\n@@ -9381,6 +9461,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n \tu16 callee_incoming, stack_arg_cnt;\n \tstruct bpf_func_state *caller;\n \tint err, subprog, target_insn;\n+\tu32 i, nregs;\n \n \ttarget_insn = *insn_idx + insn-\u003eimm + 1;\n \tsubprog = bpf_find_subprog(env, target_insn);\n@@ -9423,9 +9504,14 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n \t\tclear_caller_saved_regs(env, caller-\u003eregs);\n \t\tinvalidate_outgoing_stack_args(env, cur_func(env));\n \n-\t\t/* All non-void global functions return a 64-bit SCALAR_VALUE. */\n+\t\t/*\n+\t\t * A non-void global function returns a 64-bit SCALAR_VALUE in\n+\t\t * R0, or a \u003e8 byte SCALAR_VALUE in the R0:R2 register pair.\n+\t\t */\n \t\tif (!subprog_returns_void(env, subprog)) {\n-\t\t\tmark_reg_unknown(env, caller-\u003eregs, BPF_REG_0);\n+\t\t\tnregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;\n+\t\t\tfor (i = 0; i \u003c nregs; i++)\n+\t\t\t\tmark_reg_unknown(env, caller-\u003eregs, ret_regs[i]);\n \t\t}\n \n \t\tif (env-\u003esubprog_info[subprog].might_throw) {\n@@ -9443,6 +9529,12 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n \t\treturn 0;\n \t}\n \n+\tif (subprog_ret_pair_unreliable(env, subprog)) {\n+\t\tverbose(env, \"Func#%d ('%s') returns \u003e8 bytes, which requires reliable BTF\\n\",\n+\t\t\tsubprog, subprog_name(env, subprog));\n+\t\treturn -EINVAL;\n+\t}\n+\n \t/*\n \t * Track caller's total stack arg count (incoming + max outgoing).\n \t * This is needed so the JIT knows how much stack arg space to allocate.\n@@ -9783,11 +9875,15 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)\n \tstruct bpf_func_state *caller, *callee;\n \tstruct bpf_reg_state *r0;\n \tbool in_callback_fn;\n+\tu32 i, nregs;\n \tint err;\n \n \tcallee = state-\u003eframe[state-\u003ecurframe];\n \tr0 = \u0026callee-\u003eregs[BPF_REG_0];\n-\tif (r0-\u003etype == PTR_TO_STACK) {\n+\tnregs = bpf_ret_reg_pair(env, callee-\u003esubprogno) ? 2 : 1;\n+\tfor (i = 0; i \u003c nregs; i++) {\n+\t\tif (callee-\u003eregs[ret_regs[i]].type != PTR_TO_STACK)\n+\t\t\tcontinue;\n \t\t/* technically it's ok to return caller's stack pointer\n \t\t * (or caller's caller's pointer) back to the caller,\n \t\t * since these pointers are valid. Only current stack\n@@ -9822,8 +9918,12 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)\n \t\t\treturn -EFAULT;\n \t\t}\n \t} else {\n-\t\t/* return to the caller whatever r0 had in the callee */\n-\t\tcaller-\u003eregs[BPF_REG_0] = *r0;\n+\t\t/*\n+\t\t * return to the caller whatever the callee had in the\n+\t\t * return register(s)\n+\t\t */\n+\t\tfor (i = 0; i \u003c nregs; i++)\n+\t\t\tcaller-\u003eregs[ret_regs[i]] = callee-\u003eregs[ret_regs[i]];\n \t}\n \n \t/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,\n@@ -10721,6 +10821,19 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn\n \treturn 0;\n }\n \n+/*\n+ * Mark the register(s) holding a @size byte kfunc return value as unknown\n+ * scalars. Both halves of a register pair are treated the same way.\n+ */\n+static void mark_kfunc_ret_regs(struct bpf_verifier_env *env,\n+\t\t\t\tstruct bpf_reg_state *regs, u32 size)\n+{\n+\tu32 i, nregs = ret_regs_cnt(size);\n+\n+\tfor (i = 0; i \u003c nregs; i++)\n+\t\tmark_reg_unknown(env, regs, ret_regs[i]);\n+}\n+\n static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta)\n {\n \treturn meta-\u003ekfunc_flags \u0026 KF_ACQUIRE;\n@@ -10988,9 +11101,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_\n }\n \n /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */\n-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,\n-\t\t\t\t\tconst struct btf *btf,\n-\t\t\t\t\tconst struct btf_type *t, int rec)\n+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env,\n+\t\t\t       const struct btf *btf,\n+\t\t\t       const struct btf_type *t, int rec)\n {\n \tconst struct btf_type *member_type;\n \tconst struct btf_member *member;\n@@ -11008,7 +11121,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,\n \t\t\t\tverbose(env, \"max struct nesting depth exceeded\\n\");\n \t\t\t\treturn false;\n \t\t\t}\n-\t\t\tif (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))\n+\t\t\tif (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1))\n \t\t\t\treturn false;\n \t\t\tcontinue;\n \t\t}\n@@ -11407,7 +11520,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,\n \t\t (is_kfunc_arg_mem_size(meta-\u003ebtf, \u0026args[arg + 1]) ||\n \t\t  is_kfunc_arg_const_mem_size(meta-\u003ebtf, \u0026args[arg + 1]))) {\n \t\tif (!btf_type_is_void(ref_t) \u0026\u0026 !btf_type_is_scalar(ref_t) \u0026\u0026\n-\t\t    !__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t    !btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\tverbose(env, \"%s pointer type %s %s must point to void, scalar, or struct with scalar\\n\",\n \t\t\t\treg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);\n \t\t\treturn -EINVAL;\n@@ -11423,7 +11536,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,\n \t\t * scalars. The access size is derived from the pointed-to BTF type.\n \t\t */\n \t\tif (!btf_type_is_scalar(ref_t) \u0026\u0026\n-\t\t    !__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t    !btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\tverbose(env, \"%s pointer type %s %s must point to scalar, or struct with scalar\\n\",\n \t\t\t\treg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);\n \t\t\treturn -EINVAL;\n@@ -12394,7 +12507,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me\n \t\t\t\tbreak;\n \t\t\t}\n \n-\t\t\tif (!__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t\tif (!btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\t\tenum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);\n \n \t\t\t\tverbose(env, \"%s is %s expected %s %s\",\n@@ -12858,7 +12971,7 @@ static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg\n \n \t\tstruct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);\n \t\tif (is_bpf_percpu_obj_new_kfunc(meta-\u003efunc_id)) {\n-\t\t\tif (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {\n+\t\t\tif (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {\n \t\t\t\tverbose(env, \"bpf_percpu_obj_new type ID argument must be of a struct of scalars\\n\");\n \t\t\t\treturn -EINVAL;\n \t\t\t}\n@@ -13191,10 +13304,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n \t}\n \n \tif (btf_type_is_scalar(t)) {\n-\t\tmark_reg_unknown(env, regs, BPF_REG_0);\n+\t\tmark_kfunc_ret_regs(env, regs, t-\u003esize);\n \t\tif (meta.btf == btf_vmlinux \u0026\u0026 (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||\n \t\t    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))\n \t\t\t__mark_reg_const_zero(env, \u0026regs[BPF_REG_0]);\n+\t} else if (btf_type_is_struct(t)) {\n+\t\t/*\n+\t\t * The returned struct comes back as raw register bits modeled\n+\t\t * as an unknown scalar, so it must contain only scalars:\n+\t\t * otherwise a pointer field would be laundered into a scalar\n+\t\t * and escape provenance and reference tracking.\n+\t\t */\n+\t\tif (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {\n+\t\t\tverbose(env,\n+\t\t\t\t\"kernel function %s returns %s %s that is not composed of scalars\\n\",\n+\t\t\t\tfunc_name, btf_type_str(t),\n+\t\t\t\tbtf_name_by_offset(desc_btf, t-\u003ename_off));\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\t\tmark_kfunc_ret_regs(env, regs, t-\u003esize);\n \t} else if (btf_type_is_ptr(t)) {\n \t\tptr_type = btf_type_skip_modifiers(desc_btf, t-\u003etype, \u0026ptr_type_id);\n \t\terr = check_special_kfunc(env, \u0026meta, regs, insn_aux, ptr_type, desc_btf);\n@@ -16641,37 +16769,54 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char\n \treturn 0;\n }\n \n-static int check_global_subprog_return_code(struct bpf_verifier_env *env)\n+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)\n {\n-\tstruct bpf_reg_state *reg = reg_state(env, BPF_REG_0);\n-\tstruct bpf_func_state *cur_frame = cur_func(env);\n+\tstruct bpf_reg_state *reg;\n \tint err;\n \n-\tif (subprog_returns_void(env, cur_frame-\u003esubprogno))\n-\t\treturn 0;\n-\n-\terr = check_reg_arg(env, BPF_REG_0, SRC_OP);\n+\terr = check_reg_arg(env, regno, SRC_OP);\n \tif (err)\n \t\treturn err;\n \n \t/* Pointers to arena are safe to pass between subprograms. */\n-\tif (is_arena_reg(env, BPF_REG_0))\n+\tif (is_arena_reg(env, regno))\n \t\treturn 0;\n \n-\tif (is_pointer_value(env, BPF_REG_0)) {\n-\t\tverbose(env, \"R%d leaks addr as return value\\n\", BPF_REG_0);\n+\tif (is_pointer_value(env, regno)) {\n+\t\tverbose(env, \"R%d leaks addr as return value\\n\", regno);\n \t\treturn -EACCES;\n \t}\n \n+\treg = reg_state(env, regno);\n \tif (reg-\u003etype != SCALAR_VALUE) {\n-\t\tverbose(env, \"At subprogram exit the register R0 is not a scalar value (%s)\\n\",\n-\t\t\treg_type_str(env, reg-\u003etype));\n+\t\tverbose(env, \"At subprogram exit the register R%d is not a scalar value (%s)\\n\",\n+\t\t\tregno, reg_type_str(env, reg-\u003etype));\n \t\treturn -EINVAL;\n \t}\n \n \treturn 0;\n }\n \n+static int check_global_subprog_return_code(struct bpf_verifier_env *env)\n+{\n+\tstruct bpf_func_state *cur_frame = cur_func(env);\n+\tu32 subprog = cur_frame-\u003esubprogno;\n+\tu32 i, nregs;\n+\tint err;\n+\n+\tif (subprog_returns_void(env, subprog))\n+\t\treturn 0;\n+\n+\tnregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;\n+\tfor (i = 0; i \u003c nregs; i++) {\n+\t\terr = check_global_ret_scalar_reg(env, ret_regs[i]);\n+\t\tif (err)\n+\t\t\treturn err;\n+\t}\n+\n+\treturn 0;\n+}\n+\n /* Bitmask with 1s for all caller saved registers */\n #define ALL_CALLER_SAVED_REGS ((1u \u003c\u003c CALLER_SAVED_REGS) - 1)\n \n@@ -19328,6 +19473,13 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,\n \t\t\treturn -EOPNOTSUPP;\n \t\t}\n \n+\t\tif (prog_extension \u0026\u0026 tgt_info-\u003efmodel.ret_size \u003e 8) {\n+\t\t\tbpf_log(log,\n+\t\t\t\t\"Cannot replace function %s with a \u003e8 byte return value\\n\",\n+\t\t\t\ttname);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\n \t\t/*\n \t\t * *.multi programs don't need an address during program\n \t\t * verification, we just take the module ref if needed.\n@@ -20317,6 +20469,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\n \n+\t/* must precede the first bpf_ret_reg_pair() user below */\n+\tbpf_compute_subprog_ret_regs(env);\n+\n \tret = bpf_compute_live_registers(env);\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\ndiff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c\nnew file mode 100644\nindex 0000000000000..c295adedbae85\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c\n@@ -0,0 +1,17 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003ctest_progs.h\u003e\n+#include \"aggregate_ret_int128_c.skel.h\"\n+#include \"aggregate_ret_struct_c.skel.h\"\n+#include \"aggregate_ret_kfunc_c.skel.h\"\n+#include \"aggregate_ret_func.skel.h\"\n+#include \"aggregate_ret_kfunc.skel.h\"\n+\n+void test_aggregate_ret(void)\n+{\n+\tRUN_TESTS(aggregate_ret_int128_c);\n+\tRUN_TESTS(aggregate_ret_struct_c);\n+\tRUN_TESTS(aggregate_ret_kfunc_c);\n+\tRUN_TESTS(aggregate_ret_func);\n+\tRUN_TESTS(aggregate_ret_kfunc);\n+}\ndiff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\nindex 2523c07a16c65..0b54f911015c5 100644\n--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\n+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\n@@ -441,6 +441,19 @@ static void test_func_replace_int_with_void(void)\n \t\t\t\t     \" doesn't match type INT of global_func2()\");\n }\n \n+static void test_func_replace_ret_pair(void)\n+{\n+\tconst char *msg = \"Cannot replace function agg_ret_target_func with a \u003e8 byte return\";\n+\n+\t/*\n+\t * An extension cannot replace a function whose return value comes back\n+\t * in the R0:R2 pair: the extension's own return is capped at 8 bytes,\n+\t * so it would leave R2 stale for the target's callers.\n+\t */\n+\ttest_obj_load_failure_common(\"freplace_ret_pair.bpf.o\",\n+\t\t\t\t     \"./aggregate_ret_target.bpf.o\", msg);\n+}\n+\n static int find_prog_btf_id(const char *name, __u32 attach_prog_fd)\n {\n \tstruct bpf_prog_info info = {};\n@@ -660,6 +673,8 @@ void serial_test_fexit_bpf2bpf(void)\n \t\ttest_func_replace_progmap();\n \tif (test__start_subtest(\"freplace_int_with_void\"))\n \t\ttest_func_replace_int_with_void();\n+\tif (test__start_subtest(\"freplace_ret_pair\"))\n+\t\ttest_func_replace_ret_pair();\n \tif (test__start_subtest(\"freplace_void\"))\n \t\ttest_func_replace_void();\n \tif (test__start_subtest(\"sleepable_fentry_to_xdp\"))\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c\nnew file mode 100644\nindex 0000000000000..cfb21bcf704be\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c\n@@ -0,0 +1,260 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+typedef unsigned __int128 u128;\n+\n+__naked u128 global_agg_good(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0x1234;\"\t/* low 64 bits */\n+\t\"r2 = 0x5678;\"\t/* high 64 bits */\n+\t\"exit;\"\n+\t);\n+}\n+\n+__naked u128 global_agg_bad(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+__naked u128 global_agg_bad_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = r10;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_global_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_bad];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_bad)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"At subprogram exit the register R2 is not a scalar value\")\n+__naked int aggregate_ret_global_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_bad_ptr];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_bad_ptr)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_bad_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = r10;\"\t/* stack pointer placed in the second return register */\n+\t\"exit;\"\n+\t);\n+}\n+\n+/*\n+ * R2 is a return register once the subprogram returns a pair, so a stack\n+ * pointer left in it is rejected at the callee's exit exactly as one in R0\n+ * is: the callee frame is gone by the time the caller could use it.\n+ */\n+SEC(\"tc\")\n+__failure __msg(\"cannot return stack pointer to the caller\")\n+__naked int aggregate_ret_static_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_bad_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_bad_ptr)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_no_r2(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_static_uninit_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_no_r2];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_no_r2)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_precise(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 4;\"\t/* second half; its value is made precise below */\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 2: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 12: (95) exit\")\n+__msg(\"mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4\")\n+__naked int aggregate_ret_static_precise(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_precise];\"\n+\t\"r6 = r2;\"\t\t/* derived from the aggregate's second half */\n+\t\"r6 \u0026= 7;\"\t\t/* keep it in [0, 7] to index the stack */\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/* ptr += scalar marks r6 (hence R2) precise */\n+\t\"r0 = 0;\"\n+\t\"*(u8 *)(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_precise)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 2: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9\")\n+__naked int aggregate_ret_global_precise(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_good];\"\n+\t\"r6 = r2;\"\t\t/* derived from the aggregate's second half */\n+\t\"r6 \u0026= 7;\"\t\t/* keep it in [0, 7] to index the stack */\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/* ptr += scalar marks r6 (hence R2) precise */\n+\t\"r0 = 0;\"\n+\t\"*(u8 *)(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_good)\n+\t: __clobber_all);\n+}\n+\n+/* A by-value struct that smuggles a pointer, which must be rejected. */\n+struct with_ptr {\n+\tvoid *p;\n+\t__u64 x;\n+};\n+\n+/* A by-value union that smuggles a pointer, which must be rejected too. */\n+union upair_with_ptr {\n+\tvoid *p;\n+\t__u64 halves[2];\n+};\n+\n+__naked struct with_ptr global_ret_struct_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"Global function global_ret_struct_ptr() has unsupported return type\")\n+__naked int aggregate_ret_global_struct_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_ret_struct_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_ret_struct_ptr)\n+\t: __clobber_all);\n+}\n+\n+__naked union upair_with_ptr global_ret_union_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"Global function global_ret_union_ptr() has unsupported return type\")\n+__naked int aggregate_ret_global_union_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_ret_union_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_ret_union_ptr)\n+\t: __clobber_all);\n+}\n+\n+struct ptr_pair {\n+\tvoid *p;\n+\t__u64 x;\n+};\n+\n+static __naked __noinline struct ptr_pair static_ret_ptr_pair(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = r1;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+__naked int aggregate_ret_static_ptr_pair(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_ret_ptr_pair];\"\n+\t\"r1 = *(u32 *)(r2 + 0);\"\t/* deref the returned ctx pointer */\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_ret_ptr_pair)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c\nnew file mode 100644\nindex 0000000000000..913cc374215d5\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c\n@@ -0,0 +1,49 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+#if defined(__clang_major__) \u0026\u0026 __clang_major__ \u003e= 23\n+\n+#define MIX_A\t0xdeadbeefcafef00dULL\n+#define MIX_B\t0x0123456789abcdefULL\n+\n+typedef unsigned __int128 u128;\n+\n+static __noinline u128 make_i128(__u64 a, __u64 b)\n+{\n+\treturn ((u128)(a + b) \u003c\u003c 64) | (a - b);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_int128_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tu128 v;\n+\n+\tv = make_i128(a, b);\n+\tif ((__u64)(v \u003e\u003e 64) != a + b)\n+\t\treturn 1;\n+\tif ((__u64)v != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+SEC(\"socket\")\n+__description(\"aggregate_ret_int128_c: needs LLVM 23, dummy test\")\n+__success\n+int dummy_test(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c\nnew file mode 100644\nindex 0000000000000..c23b4beb1773c\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c\n@@ -0,0 +1,122 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+#include \"../test_kmods/bpf_testmod_kfunc.h\"\n+\n+/*\n+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address\n+ * (rather than calling) avoids any dependence on the compiler lowering an\n+ * __int128 or struct return value, which the BPF backend only supports from\n+ * LLVM 23 on.\n+ */\n+void __kfunc_btf_root(void)\n+{\n+\tasm volatile (\"\"\n+\t:\n+\t: \"r\"(\u0026bpf_kfunc_call_test_i128),\n+\t  \"r\"(\u0026bpf_kfunc_call_test_ret_fastcall),\n+\t  \"r\"(\u0026bpf_kfunc_call_test_ret_ptr),\n+\t  \"r\"(\u0026bpf_kfunc_call_test_ret_ii),\n+\t  \"r\"(\u0026bpf_kfunc_call_test_ret_big));\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128\")\n+__naked int aggregate_ret_kfunc_precise(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 1;\"\n+\t\"r2 = 2;\"\n+\t\"call %[bpf_kfunc_call_test_i128];\"\n+\t\"r6 = r2;\"\t\t/* second return half */\n+\t\"r6 \u0026= 7;\"\t\t/* keep it in [0, 7] to index the stack */\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/* ptr += scalar marks r6 (hence R2) precise */\n+\t\"r0 = 0;\"\n+\t\"*(u8 *)(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_i128)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"kfunc bpf_kfunc_call_test_ret_fastcall with \u003e8-byte return is not supported with KF_FASTCALL\")\n+__naked int aggregate_ret_kfunc_fastcall_fail(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 1;\"\n+\t\"r2 = 2;\"\n+\t\"call %[bpf_kfunc_call_test_ret_fastcall];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_fastcall)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"is not composed of scalars\")\n+__naked int aggregate_ret_kfunc_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 0;\"\n+\t\"call %[bpf_kfunc_call_test_ret_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_ptr)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_kfunc_small_no_r2(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"call %[bpf_kfunc_call_test_ret_ii];\"\n+\t\"r0 = r2;\"\t/* R2 is not a return register for a \u003c=8 byte struct */\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_ii)\n+\t: __clobber_all);\n+}\n+\n+/*\n+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by\n+ * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks,\n+ * so this behaves the same on every architecture.\n+ */\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported\")\n+__naked int aggregate_ret_kfunc_too_big_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[bpf_kfunc_call_test_ret_big];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_big)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c\nnew file mode 100644\nindex 0000000000000..2c1889fc28efd\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c\n@@ -0,0 +1,125 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"../test_kmods/bpf_testmod_kfunc.h\"\n+#include \"bpf_misc.h\"\n+\n+#if defined(__clang_major__) \u0026\u0026 __clang_major__ \u003e= 23\n+\n+#define MIX_A\t0xdeadbeefcafef00dULL\n+#define MIX_B\t0x0123456789abcdefULL\n+\n+typedef unsigned __int128 u128;\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tu128 v;\n+\n+\tv = bpf_kfunc_call_test_i128(a, b);\n+\tif ((__u64)(v \u003e\u003e 64) != a + b)\n+\t\treturn 1;\n+\tif ((__u64)v != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct prog_test_ret_pair p;\n+\n+\tp = bpf_kfunc_call_test_ret_pair(a, b);\n+\tif (p.hi != a + b)\n+\t\treturn 1;\n+\tif (p.lo != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+/* struct { u64 a; int b; }: 16 bytes, R0 = a, R2 = b. */\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_li_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\tint b = skb-\u003elen ^ MIX_B;\n+\tstruct prog_test_ret_li r;\n+\n+\tr = bpf_kfunc_call_test_ret_li(a, b);\n+\tif (r.a != a)\n+\t\treturn 1;\n+\tif (r.b != ~b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not a return reg. */\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_ii_c_test(struct __sk_buff *skb)\n+{\n+\tint a = skb-\u003elen ^ MIX_A;\n+\tint b = skb-\u003elen ^ MIX_B;\n+\tstruct prog_test_ret_ii r;\n+\n+\tr = bpf_kfunc_call_test_ret_ii(a, b);\n+\tif (r.a != a)\n+\t\treturn 1;\n+\tif (r.b != b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+/* A union of 16 bytes takes the same R0:R2 path as a struct. */\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_uu_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tunion prog_test_ret_uu r;\n+\n+\tr = bpf_kfunc_call_test_ret_uu(a, b);\n+\tif (r.parts.lo != a + b)\n+\t\treturn 1;\n+\tif (r.parts.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+SEC(\"socket\")\n+__description(\"aggregate_ret_kfunc_c: needs LLVM 23, dummy test\")\n+__success\n+int dummy_test(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c\nnew file mode 100644\nindex 0000000000000..83b1a37519887\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c\n@@ -0,0 +1,114 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+#if defined(__clang_major__) \u0026\u0026 __clang_major__ \u003e= 23\n+\n+#define MIX_A\t0xdeadbeefcafef00dULL\n+#define MIX_B\t0x0123456789abcdefULL\n+\n+struct pair {\n+\t__u64 hi;\t/* R0 */\n+\t__u64 lo;\t/* R2 */\n+};\n+\n+union upair {\n+\t__u64 halves[2];\n+\tstruct {\n+\t\t__u64 lo;\t/* R0 */\n+\t\t__u64 hi;\t/* R2 */\n+\t} parts;\n+};\n+\n+static __noinline struct pair make_pair(__u64 a, __u64 b)\n+{\n+\tstruct pair p = { .hi = a + b, .lo = a - b };\n+\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_struct_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct pair p;\n+\n+\tp = make_pair(a, b);\n+\tif (p.hi != a + b)\n+\t\treturn 1;\n+\tif (p.lo != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+__noinline struct pair make_pair_global(__u64 a, __u64 b)\n+{\n+\tstruct pair p = { .hi = a + b, .lo = a - b };\n+\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct pair p;\n+\n+\tp = make_pair_global(a, b);\n+\tif (p.hi != a + b)\n+\t\treturn 1;\n+\tif (p.lo != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+static __noinline union upair make_upair(__u64 a, __u64 b)\n+{\n+\tunion upair p;\n+\n+\tp.halves[0] = a + b;\n+\tp.halves[1] = a - b;\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_union_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tunion upair p;\n+\n+\tp = make_upair(a, b);\n+\tif (p.parts.lo != a + b)\n+\t\treturn 1;\n+\tif (p.parts.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+SEC(\"socket\")\n+__description(\"aggregate_ret_struct_c: needs LLVM 23, dummy test\")\n+__success\n+int dummy_test(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c\nnew file mode 100644\nindex 0000000000000..cffd8d7d3241a\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c\n@@ -0,0 +1,29 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+/* freplace target: a global subprogram returning 16 bytes in R0:R2. */\n+__naked unsigned __int128 agg_ret_target_func(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0x1234;\"\n+\t\"r2 = 0x5678;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__naked int agg_ret_target(void)\n+{\n+\tasm volatile (\n+\t\"call %[agg_ret_target_func];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(agg_ret_target_func)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c\nindex d055fc7b3b95d..0be9441ec273b 100644\n--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c\n+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c\n@@ -431,6 +431,36 @@ __naked void subprog1(void)\n \t\t::: __clobber_all);\n }\n \n+static __used __naked unsigned __int128 aux2(void)\n+{\n+\tasm volatile (\n+\t\t\"r0 = 1;\"\n+\t\t\"r2 = 2;\"\n+\t\t\"exit;\"\n+\t\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */\n+__load_if_JITed()\n+__log_level(2)\n+__msg(\"0: .12345.... (85) call pc+2\")\n+__msg(\"1: ..2....... (bf) r0 = r2\")\n+/* R2 is not read at the exit of this program, which returns an int, ... */\n+__msg(\"2: 0......... (95) exit\")\n+__msg(\"3: .......... (b7) r0 = 1\")\n+__msg(\"4: 0......... (b7) r2 = 2\")\n+/* ... but it is at the exit of aux2(), which returns a register pair. */\n+__msg(\"5: 0.2....... (95) exit\")\n+__naked void subprog_ret_reg_pair(void)\n+{\n+\tasm volatile (\n+\t\t\"call aux2;\"\n+\t\t\"r0 = r2;\"\n+\t\t\"exit;\"\n+\t\t::: __clobber_all);\n+}\n+\n #if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)\n \n SEC(\"socket\")\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c\nindex ac44d60e50666..9708efb93683b 100644\n--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c\n+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c\n@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)\n \n SEC(\"?tc\")\n __exception_cb(exception_cb_bad_ret_type1)\n-__failure __msg(\"Global function exception_cb_bad_ret_type1() return value not void or scalar.\")\n+__failure __msg(\"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\")\n int reject_exception_cb_type_1(struct __sk_buff *ctx)\n {\n \tbpf_throw(0);\ndiff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c\nnew file mode 100644\nindex 0000000000000..12c15d293bd79\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c\n@@ -0,0 +1,12 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+\n+SEC(\"freplace/agg_ret_target_func\")\n+__u64 new_agg_ret_target_func(void)\n+{\n+\treturn 0;\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\nindex a6133f7521f34..ad36d583d2e7f 100644\n--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\n+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\n@@ -939,6 +939,69 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)\n \treturn 0;\n }\n \n+#if defined(__x86_64__) || defined(__aarch64__)\n+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)\n+{\n+\treturn (__int128)(((unsigned __int128)(a + b) \u003c\u003c 64) | (a - b));\n+}\n+\n+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)\n+{\n+\tstruct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)\n+{\n+\tstruct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b)\n+{\n+\tstruct prog_test_ret_li r = { .a = a, .b = ~b };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b)\n+{\n+\tunion prog_test_ret_uu r;\n+\n+\tr.halves[0] = a + b;\n+\tr.halves[1] = a - b;\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)\n+{\n+\tstruct prog_test_ret_ptr r = { .p = NULL, .tag = tag };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)\n+{\n+\tstruct prog_test_ret_ii r = { .a = a, .b = b };\n+\n+\treturn r;\n+}\n+#endif /* __x86_64__ || __aarch64__ */\n+\n+/*\n+ * Takes no argument on purpose: with no arguments there is nothing for the sret\n+ * pointer to displace, so this needs no architecture guard even though it\n+ * returns 24 bytes. See the comment on bpf_kfunc_call_test_i128() above.\n+ */\n+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)\n+{\n+\tstruct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };\n+\n+\treturn r;\n+}\n+\n __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,\n \t\t\t\t\t u64 e, u64 f, u64 g, u64 h,\n \t\t\t\t\t u64 i, u64 j)\n@@ -1472,6 +1535,16 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test3)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test4)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test5)\n+#if defined(__x86_64__) || defined(__aarch64__)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)\n+#endif\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)\ndiff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\nindex c4383acb53c11..c7be973cd2860 100644\n--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\n+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\n@@ -55,6 +55,40 @@ struct prog_test_big_arg {\n \t__u64 b;\n };\n \n+struct prog_test_ret_pair {\n+\t__u64 hi;\n+\t__u64 lo;\n+};\n+\n+struct prog_test_ret_li {\t/* 16 bytes: R0:R2 */\n+\t__u64 a;\n+\tint b;\n+};\n+\n+struct prog_test_ret_ii {\t/* 8 bytes: R0 only */\n+\tint a;\n+\tint b;\n+};\n+\n+union prog_test_ret_uu {\t/* 16 bytes: R0:R2 */\n+\t__u64 halves[2];\n+\tstruct {\n+\t\t__u64 lo;\n+\t\t__u64 hi;\n+\t} parts;\n+};\n+\n+struct prog_test_ret_ptr {\t/* 16 bytes: contains a pointer */\n+\tvoid *p;\n+\t__u64 tag;\n+};\n+\n+struct prog_test_ret_big {\t/* 24 bytes: too large for R0:R2 */\n+\t__u64 a;\n+\t__u64 b;\n+\t__u64 c;\n+};\n+\n struct prog_test_fail1 {\n \tvoid *p;\n \tint x;\n@@ -130,6 +164,16 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;\n struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;\n long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;\n int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;\n+#ifdef __SIZEOF_INT128__\n+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;\n+#endif\n+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;\n+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;\n+struct prog_test_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym;\n+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;\n+union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym;\n+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;\n+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym;\n __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,\n \t\t\t       __u64 e, __u64 f, __u64 g, __u64 h,\n \t\t\t       __u64 i, __u64 j) __ksym;\n"
}

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT"
  ],
  "FocusSymbols": [
    "bpf_kfunc_ret_reg_pair",
    "btf_type_is_scalar_struct",
    "bpf_jit_supports_kfunc_ret_reg_pair",
    "bpf_compute_live_registers"
  ],
  "Reasoning": "The patch introduces support for kfuncs and BPF subprograms returning larger values (up to 16 bytes) in the R0:R2 register pair. This involves changes to the BPF verifier (liveness tracking, return value checking, BTF validation) and JIT compilers (x86, arm64, riscv). These are significant functional changes to the core BPF subsystem that should be fuzzed to ensure verifier safety and JIT correctness.",
  "WorthFuzzing": true
}

You are an expert Linux kernel maintainer.
Your job is to review a provided patch series and determine
if it makes functional changes to the kernel that should be fuzzed.

IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively
use your code access tools to examine the actual source code and confirm any assumptions.

Return WorthFuzzing=false if the patch only contains:
- Modifications to Documentation/, Kconfig files, or code comments.
- Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints.
- Changes to numeric constants or macros that do not functionally alter execution flow.
- Code paths that are impossible to reach in virtualized environments like GCE or QEMU,
  even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim).
- Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed,
  ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU.
- Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove)
  that are executed only during PCI hot-unplug or sysfs driver unbind operations.

If it modifies reachable core kernel logic, drivers, or architectures, use your code search
tools to verify the code can be executed, then return WorthFuzzing=true.

When returning WorthFuzzing=true, you MUST ALSO:
1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols.
   Avoid listing generic hot-path functions to prevent skewed test distributions.
   Prefer non-static, non-inlined API entrypoint functions over internal static helper functions
   (which are inlined by the compiler and do not have distinct symbol addresses).
2. Identify any specific CONFIG_ options required to properly test this new/modified feature.
   Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions.
   Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs
   needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs
   output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit bafb5dc76d94a94d43e7b1575db7d4920a549687
Author: syz-cluster <triage@syzkaller.com>
Date:   Thu Aug 13 20:58:12 2026 +0000

    syz-cluster: applied patch under review

diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index 10e725cbe64cd..5d840d3000338 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -575,6 +575,69 @@ is also covered by this recovery. A kfunc handed an arena pointer may
 therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
 against the arena. Larger accesses must verify the range explicitly.
 
+2.9 kfunc Return Values
+-----------------------
+
+A kfunc may return a scalar, a pointer, or a small struct or union by
+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
+
+A struct or union returned by value must be composed only of scalars
+(recursively), where a scalar is an integer or an enum; arrays of scalars are
+allowed as members. Its bytes are handed back to the program as the raw
+contents of R0 (and R2), so a pointer field would be laundered into a scalar
+and escape the verifier's pointer provenance and reference tracking. A struct
+or union with a pointer member is therefore rejected at load time, and so is
+one with a floating-point member, which the ABI may not return in R0:R2 at
+all.
+
+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
+scalar-only struct or union, or an ``__int128``. Such a value is returned
+in the register pair R0:R2, matching the convention LLVM uses for the BPF
+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or
+union of 8 bytes or less is returned in R0 alone.
+
+::
+
+        struct bpf_pair { __u64 a, b; };   /* 16 bytes */
+
+        __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
+        {
+                struct bpf_pair p = { .a = 1, .b = 2 };
+
+                return p;      /* p.a in R0, p.b in R2 */
+        }
+
+Returning a value in the R0:R2 pair requires the JIT to place the second
+half of the return value into R2, which not every architecture supports
+right now. A kfunc with a return value larger than 8 bytes is therefore
+rejected at load time on a JIT that does not advertise this capability (see
+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
+by the interpreter. A return value larger than 16 bytes is not supported.
+
+The same R0:R2 convention applies to a BPF subprogram, global or static,
+that returns an ``__int128`` or a struct or union larger than 8 bytes. Such a
+program also requires the JIT, since the interpreter propagates only R0 out
+of a subprogram. A global subprogram is verified in isolation, so its
+by-value struct or union return is restricted to scalars just like a kfunc's;
+a static subprogram is verified inline and has no such restriction. The main
+program is not covered: its return value is the program's exit code, read out
+of R0 alone, so a declared upper half is never looked at.
+
+A global subprogram must leave a scalar in *every* register of the pair, so
+both halves of the returned value have to be assigned. Leaving the upper half
+uninitialized is not merely untidy: the compiler is then free to leave R2
+holding whatever it happened to hold, which for a subprogram taking a pointer
+argument is typically that pointer. Handing the caller an unknown scalar built
+from a pointer is a leak, so the verifier rejects it with::
+
+        At subprogram exit the register R2 is not a scalar value (...)
+
+Initialize the whole return value, for example ``struct pair p = {};``, to
+avoid this. A static subprogram is exempt from the scalar-only rule: it is
+verified inline, so an unassigned R2 is simply passed back to the caller as
+uninitialized and only a caller that reads it fails. A stack pointer left in
+R2 is still rejected there, just as one in R0 is.
+
 .. _BPF_kfunc_lifecycle_expectations:
 
 3. kfunc lifecycle expectations
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 74b4083791da3..7a4c5968976a6 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2346,6 +2346,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index 2504df1fa1118..49220765e96d4 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -2120,6 +2120,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_ptr_xchg(void)
 {
 	return true;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d920772af7d5f..2ec09dd02b187 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
  * arena NULL is offset 0. Return the number of emitted bytes.
  */
 static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
-				 const struct bpf_insn *insn, u8 **pprog)
+				 const struct btf_func_model *fm, u8 **pprog)
 {
-	const struct btf_func_model *fm;
 	u8 *prog = *pprog;
 	u8 *start = prog;
 	int i;
 
-	fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
-	if (!fm)
-		return -EINVAL;
-
 	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
 		u8 flags = fm->arg_flags[i];
 		u32 reg = BPF_REG_1 + i;
@@ -2644,6 +2639,8 @@ st:			insn_off = insn->off;
 
 			/* call */
 		case BPF_JMP | BPF_CALL: {
+			const struct btf_func_model *fm = NULL;
+
 			func = (u8 *) __bpf_call_base + imm32;
 			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
 				LOAD_TAIL_CALL_CNT_PTR(stack_depth);
@@ -2652,7 +2649,10 @@ st:			insn_off = insn->off;
 			if (!imm32)
 				return -EINVAL;
 			if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
-				err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
+				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
+				if (!fm)
+					return -EINVAL;
+				err = emit_kfunc_arena_args(bpf_prog, fm, &prog);
 				if (err < 0)
 					return err;
 				ip += err;
@@ -2666,6 +2666,14 @@ st:			insn_off = insn->off;
 				return -EINVAL;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
+			/*
+			 * A kfunc returning more than 8 bytes hands the second
+			 * half back in RDX (the native ABI's second return reg),
+			 * but BPF expects it in R0:R2. BPF R0 is RAX (no move
+			 * needed), while BPF R2 is RSI, so copy RDX into RSI.
+			 */
+			if (fm && fm->ret_size > 8)
+				emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
 			break;
 		}
 
@@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 27b43fda9b178..899e5ce986432 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -814,6 +814,8 @@ struct bpf_subprog_info {
 	bool is_async_cb: 1;
 	bool is_exception_cb: 1;
 	bool args_cached: 1;
+	/* true if the return value is passed in the R0:R2 register pair */
+	bool ret_reg_pair: 1;
 	/* true if bpf_fastcall stack region is used by functions that can't be inlined */
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
@@ -1048,6 +1050,13 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
 	return &env->subprog_info[subprog];
 }
 
+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	return subprog_info(env, subprog)->ret_reg_pair;
+}
+
+bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env *env, struct bpf_insn *insn);
+
 struct bpf_call_summary {
 	u8 num_params;
 	bool is_void;
@@ -1439,6 +1448,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);
 struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+			       const struct btf_type *t, int rec);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 4a9bc6a848f2e..6e746b0a09306 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_kfunc_ret_reg_pair(void);
 bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 40bd04421a991..3ef8a3da726de 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -425,6 +425,16 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				 */
 				verifier_bug_if(idx + 1 != subseq_idx, env,
 						"extra insn from subprog");
+				/* a global subprog returning more than 8 bytes
+				 * sets R2 as well. R2 is part of the args mask
+				 * checked just below, so clear it here rather
+				 * than next to R0. Only a subprog that does
+				 * return a pair defines R2, so leave the mask
+				 * alone otherwise and let the check below catch
+				 * an R2 that has no business being set.
+				 */
+				if (bpf_ret_reg_pair(env, subprog))
+					bt_clear_reg(bt, BPF_REG_2);
 				/* r1-r5 are invalidated after subprog call,
 				 * so for global func call it shouldn't be set
 				 * anymore
@@ -508,6 +518,17 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				return -ENOTSUPP;
 			/* regular helper call sets R0 */
 			bt_clear_reg(bt, BPF_REG_0);
+			/* a kfunc returning more than 8 bytes also sets R2.
+			 * R2 is part of the args mask checked just below, so
+			 * clear it here rather than next to R0. The prototype
+			 * lookup is only worth doing when R2 is requested at
+			 * all; any other call leaves R2 uninitialized, so a
+			 * request for it is caught by the check below.
+			 */
+			if (bt_is_reg_set(bt, BPF_REG_2) &&
+			    insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+			    bpf_kfunc_ret_reg_pair(env, insn))
+				bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				/* if backtracking was looking for registers R1-R5
 				 * they should have been found already.
@@ -522,7 +543,41 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 					return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			bool r0_precise;
+			bool from_subprog_call, r0_precise, r2_precise;
+			struct bpf_insn *call;
+			int subprog;
+
+			/* BPF_EXIT in subprog or callback always returns
+			 * right after the call instruction, so by checking
+			 * whether the instruction at subseq_idx-1 is subprog
+			 * call or not we can distinguish actual exit from
+			 * *subprog* from exit from *callback*. In the former
+			 * case, we need to propagate the precision of the
+			 * return registers, if necessary. In the latter we
+			 * never do that.
+			 */
+			from_subprog_call = subseq_idx - 1 >= 0 &&
+					    bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
+
+			/* Sample the return registers before the callback
+			 * handling below clears R1-R5: unlike R0, R2 is an
+			 * argument register as well, so that clear would drop
+			 * a pair return on the floor.
+			 */
+			r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
+			r2_precise = false;
+			if (from_subprog_call && bt_is_reg_set(bt, BPF_REG_2)) {
+				call = &env->prog->insnsi[subseq_idx - 1];
+				subprog = bpf_find_subprog(env, subseq_idx + call->imm);
+				if (subprog < 0)
+					return -EFAULT;
+				/* Only a callee that does return a pair defines
+				 * R2. Leave the mask alone otherwise, so that
+				 * the check below still catches an R2 that has
+				 * no business being set.
+				 */
+				r2_precise = bpf_ret_reg_pair(env, subprog);
+			}
 
 			/* Backtracking to a nested function call, 'idx' is a part of
 			 * the inner frame 'subseq_idx' is a part of the outer frame.
@@ -535,30 +590,27 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
 				for (i = BPF_REG_1; i <= BPF_REG_5; i++)
 					bt_clear_reg(bt, i);
+
+			/* a callee returning more than 8 bytes sets R2 as well;
+			 * R2 is part of the args mask checked just below, so
+			 * clear it here rather than next to R0.
+			 */
+			if (r2_precise)
+				bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				verifier_bug(env, "backtracking exit unexpected regs %x",
 					     bt_reg_mask(bt));
 				return -EFAULT;
 			}
 
-			/* BPF_EXIT in subprog or callback always returns
-			 * right after the call instruction, so by checking
-			 * whether the instruction at subseq_idx-1 is subprog
-			 * call or not we can distinguish actual exit from
-			 * *subprog* from exit from *callback*. In the former
-			 * case, we need to propagate r0 precision, if
-			 * necessary. In the former we never do that.
-			 */
-			r0_precise = subseq_idx - 1 >= 0 &&
-				     bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
-				     bt_is_reg_set(bt, BPF_REG_0);
-
 			bt_clear_reg(bt, BPF_REG_0);
 			if (bt_subprog_enter(bt))
 				return -EFAULT;
 
 			if (r0_precise)
 				bt_set_reg(bt, BPF_REG_0);
+			if (r2_precise)
+				bt_set_reg(bt, BPF_REG_2);
 			/* r6-r9 and stack slots will stay set in caller frame
 			 * bitmasks until we return back from callee(s)
 			 */
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 6606187ed4f43..c7718a2a56141 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7592,7 +7592,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
 		return -EINVAL;
 	}
 	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	if (ret < 0 || ret > 16) {
 		bpf_log(log,
 			"The function %s return type %s is unsupported.\n",
 			tname, btf_type_str(t));
@@ -7965,7 +7965,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env *env,
 
 /* Check whether the type is a valid return type. */
 static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
-		const struct btf_type *t, int subprog)
+		const struct btf_type *t, int subprog, bool is_global)
 {
 	u32 tags = 0;
 	int err;
@@ -7988,6 +7988,19 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
 	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
 		return 0;
 
+	if (btf_type_is_struct(t) && t->size <= 16) {
+		/*
+		 * A global function's caller models the return as an opaque
+		 * scalar pair, so it may only return scalars by value. A local
+		 * function is verified inline, so a pointer field stays tracked
+		 * and needs no such restriction.
+		 */
+		bool local_func = subprog && !is_global;
+
+		if (local_func || btf_type_is_scalar_struct(env, btf, t, 0))
+			return 0;
+	}
+
 	return -EOPNOTSUPP;
 }
 
@@ -8075,12 +8088,12 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 		return -EINVAL;
 	}
 
-	err = btf_validate_return_type(env, btf, t, subprog);
+	err = btf_validate_return_type(env, btf, t, subprog, is_global);
 	if (err) {
 		if (is_global) {
 			bpf_log(log,
-				"Global function %s() return value not void or scalar. "
-				"Only those are supported.\n",
+				"Global function %s() has unsupported return type. "
+				"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n",
 				tname);
 		}
 		return err;
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 6a94370a24488..cb66a2ef52b1f 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_stack_args(void)
 {
 	return false;
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 1c997aeba6fa5..850c58aa915be 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2062,7 +2062,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); }
 /* Compute info->{use,def} fields for the instruction */
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
-				   struct insn_live_regs *info)
+				   struct insn_live_regs *info,
+				   bool ret_reg_pair)
 {
 	struct bpf_call_summary cs;
 	const u8 class = BPF_CLASS(insn->code);
@@ -2074,6 +2075,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 	const u32 src32 = mask_lo(src);
 	const u32 dst32 = mask_lo(dst);
 	const u32 r0  = reg64_mask(0);
+	const u32 r2  = reg64_mask(BPF_REG_2);
 	u32 def = 0;
 	u32 use = U32_MAX;
 
@@ -2193,7 +2195,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			break;
 		case BPF_EXIT:
 			def = 0;
-			use = r0;
+			use = ret_reg_pair ? (r0 | r2) : r0;
 			break;
 		case BPF_CALL:
 			def = ALL_CALLER_SAVED_REGS;
@@ -2230,8 +2232,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 	struct insn_live_regs *state;
 	int insn_cnt = env->prog->len;
 	u64 pos, insn_pos;
-	int err = 0, i, j;
-	bool changed;
+	int err = 0, i, j, subprog, start, end;
+	bool changed, ret_reg_pair;
 
 	/* Use the following algorithm:
 	 * - define the following:
@@ -2258,8 +2260,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		goto out;
 	}
 
-	for (i = 0; i < insn_cnt; ++i)
-		compute_insn_live_regs(env, &insns[i], &state[i]);
+	for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+		start = env->subprog_info[subprog].start;
+		end = env->subprog_info[subprog + 1].start;
+		ret_reg_pair = bpf_ret_reg_pair(env, subprog);
+
+		for (i = start; i < end; ++i)
+			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
+	}
 
 	/* Forward pass: resolve stack access through FP-derived pointers */
 	err = bpf_compute_subprog_arg_access(env);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 164d16c243ca6..98a6f702d794d 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -382,27 +382,80 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
 	return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
 }
 
-static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
 {
-	const struct btf_type *type, *func, *func_proto;
+	const struct btf_type *func, *func_proto;
 	const struct btf *btf = env->prog->aux->btf;
 	u32 btf_id;
 
+	if (!btf || !env->prog->aux->func_info)
+		return NULL;
+
 	btf_id = env->prog->aux->func_info[subprog].type_id;
 
+	/* Both already validated by prepare_btf_func() at prog load. */
 	func = btf_type_by_id(btf, btf_id);
-	if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
-		return false;
-
 	func_proto = btf_type_by_id(btf, func->type);
-	if (!func_proto)
-		return false;
 
-	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
-	if (!type)
-		return false;
+	return btf_type_skip_modifiers(btf, func_proto->type, NULL);
+}
+
+static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type = subprog_ret_type(env, subprog);
+
+	return type && btf_type_is_void(type);
+}
+
+static u32 ret_regs_cnt(u32 size)
+{
+	return size > 8 && size <= 16 ? 2 : 1;
+}
+
+/* Registers holding a function return value, in order. See ret_regs_cnt(). */
+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };
+
+static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
+{
+	const struct btf *btf = env->prog->aux->btf;
+	const struct btf_type *type;
+	int subprog;
+	u32 size;
+
+	for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+		type = subprog_ret_type(env, subprog);
+		if (!type || !(btf_type_is_struct(type) || btf_type_is_scalar(type)))
+			continue;
+		if (IS_ERR(btf_resolve_size(btf, type, &size)))
+			continue;
+		if (ret_regs_cnt(size) > 1) {
+			subprog_info(env, subprog)->ret_reg_pair = true;
+			/*
+			 * The R0:R2 return convention is only implemented in
+			 * the JIT: the interpreter propagates BPF_R0 alone out
+			 * of a subprogram, so a caller reading R2 would see a
+			 * stale value.
+			 */
+			env->prog->jit_required = 1;
+		}
+	}
+}
 
-	return btf_type_is_void(type);
+/*
+ * A >8 byte BPF return changes the calling convention to R0:R2, and the
+ * verifier derives that convention from the subprogram's BTF prototype
+ * alone. Once that prototype is marked unreliable it is known not to
+ * describe the compiled code, so the convention read from it cannot be
+ * trusted either: reject the call rather than keep tracking R2 on the
+ * strength of a signature the verifier has already discarded.
+ */
+static bool subprog_ret_pair_unreliable(struct bpf_verifier_env *env, int subprog)
+{
+	struct bpf_prog_aux *aux = env->prog->aux;
+
+	return bpf_ret_reg_pair(env, subprog) &&
+	       aux->func_info_aux &&
+	       aux->func_info_aux[subprog].unreliable;
 }
 
 static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
@@ -2459,6 +2512,21 @@ find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset)
 		       sizeof(tab->descs[0]), kfunc_desc_cmp_by_id_off);
 }
 
+/*
+ * True if the kfunc called by @insn returns its value in the R0:R2 pair.
+ * Reads the same btf_func_model.ret_size that bpf_add_kfunc_call() validated
+ * and that the JIT keys the second return register off, so the verifier and
+ * the generated code cannot disagree about the convention.
+ */
+bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+	const struct bpf_kfunc_desc *desc;
+
+	desc = find_kfunc_desc(env->prog, insn->imm, insn->off);
+
+	return desc && ret_regs_cnt(desc->func_model.ret_size) > 1;
+}
+
 int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
 		       u16 btf_fd_idx, u8 **func_addr)
 {
@@ -2809,6 +2877,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
 	if (err)
 		return err;
+	if (func_model.ret_size > 8) {
+		if (kfunc.flags && (*kfunc.flags & KF_FASTCALL)) {
+			verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n",
+				kfunc.name);
+			return -EOPNOTSUPP;
+		}
+		if (!bpf_jit_supports_kfunc_ret_reg_pair()) {
+			verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
+				kfunc.name);
+			return -EOPNOTSUPP;
+		}
+	}
 
 	memset(&meta, 0, sizeof(meta));
 	meta.btf = kfunc.btf;
@@ -9381,6 +9461,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	u16 callee_incoming, stack_arg_cnt;
 	struct bpf_func_state *caller;
 	int err, subprog, target_insn;
+	u32 i, nregs;
 
 	target_insn = *insn_idx + insn->imm + 1;
 	subprog = bpf_find_subprog(env, target_insn);
@@ -9423,9 +9504,14 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		clear_caller_saved_regs(env, caller->regs);
 		invalidate_outgoing_stack_args(env, cur_func(env));
 
-		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
+		/*
+		 * A non-void global function returns a 64-bit SCALAR_VALUE in
+		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+		 */
 		if (!subprog_returns_void(env, subprog)) {
-			mark_reg_unknown(env, caller->regs, BPF_REG_0);
+			nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+			for (i = 0; i < nregs; i++)
+				mark_reg_unknown(env, caller->regs, ret_regs[i]);
 		}
 
 		if (env->subprog_info[subprog].might_throw) {
@@ -9443,6 +9529,12 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		return 0;
 	}
 
+	if (subprog_ret_pair_unreliable(env, subprog)) {
+		verbose(env, "Func#%d ('%s') returns >8 bytes, which requires reliable BTF\n",
+			subprog, subprog_name(env, subprog));
+		return -EINVAL;
+	}
+
 	/*
 	 * Track caller's total stack arg count (incoming + max outgoing).
 	 * This is needed so the JIT knows how much stack arg space to allocate.
@@ -9783,11 +9875,15 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	struct bpf_func_state *caller, *callee;
 	struct bpf_reg_state *r0;
 	bool in_callback_fn;
+	u32 i, nregs;
 	int err;
 
 	callee = state->frame[state->curframe];
 	r0 = &callee->regs[BPF_REG_0];
-	if (r0->type == PTR_TO_STACK) {
+	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
+			continue;
 		/* technically it's ok to return caller's stack pointer
 		 * (or caller's caller's pointer) back to the caller,
 		 * since these pointers are valid. Only current stack
@@ -9822,8 +9918,12 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 			return -EFAULT;
 		}
 	} else {
-		/* return to the caller whatever r0 had in the callee */
-		caller->regs[BPF_REG_0] = *r0;
+		/*
+		 * return to the caller whatever the callee had in the
+		 * return register(s)
+		 */
+		for (i = 0; i < nregs; i++)
+			caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]];
 	}
 
 	/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
@@ -10721,6 +10821,19 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 	return 0;
 }
 
+/*
+ * Mark the register(s) holding a @size byte kfunc return value as unknown
+ * scalars. Both halves of a register pair are treated the same way.
+ */
+static void mark_kfunc_ret_regs(struct bpf_verifier_env *env,
+				struct bpf_reg_state *regs, u32 size)
+{
+	u32 i, nregs = ret_regs_cnt(size);
+
+	for (i = 0; i < nregs; i++)
+		mark_reg_unknown(env, regs, ret_regs[i]);
+}
+
 static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ACQUIRE;
@@ -10988,9 +11101,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_
 }
 
 /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
-					const struct btf *btf,
-					const struct btf_type *t, int rec)
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env,
+			       const struct btf *btf,
+			       const struct btf_type *t, int rec)
 {
 	const struct btf_type *member_type;
 	const struct btf_member *member;
@@ -11008,7 +11121,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
 				verbose(env, "max struct nesting depth exceeded\n");
 				return false;
 			}
-			if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
+			if (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
 				return false;
 			continue;
 		}
@@ -11407,7 +11520,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		 (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) ||
 		  is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
 		if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) &&
-		    !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+		    !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 			verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n",
 				reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
 			return -EINVAL;
@@ -11423,7 +11536,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		 * scalars. The access size is derived from the pointed-to BTF type.
 		 */
 		if (!btf_type_is_scalar(ref_t) &&
-		    !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+		    !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 			verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n",
 				reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
 			return -EINVAL;
@@ -12394,7 +12507,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 				break;
 			}
 
-			if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+			if (!btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 				enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);
 
 				verbose(env, "%s is %s expected %s %s",
@@ -12858,7 +12971,7 @@ static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg
 
 		struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);
 		if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) {
-			if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
+			if (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
 				verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n");
 				return -EINVAL;
 			}
@@ -13191,10 +13304,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	}
 
 	if (btf_type_is_scalar(t)) {
-		mark_reg_unknown(env, regs, BPF_REG_0);
+		mark_kfunc_ret_regs(env, regs, t->size);
 		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
 		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
 			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
+	} else if (btf_type_is_struct(t)) {
+		/*
+		 * The returned struct comes back as raw register bits modeled
+		 * as an unknown scalar, so it must contain only scalars:
+		 * otherwise a pointer field would be laundered into a scalar
+		 * and escape provenance and reference tracking.
+		 */
+		if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+			verbose(env,
+				"kernel function %s returns %s %s that is not composed of scalars\n",
+				func_name, btf_type_str(t),
+				btf_name_by_offset(desc_btf, t->name_off));
+			return -EINVAL;
+		}
+		mark_kfunc_ret_regs(env, regs, t->size);
 	} else if (btf_type_is_ptr(t)) {
 		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
 		err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
@@ -16641,37 +16769,54 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	return 0;
 }
 
-static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
 {
-	struct bpf_reg_state *reg = reg_state(env, BPF_REG_0);
-	struct bpf_func_state *cur_frame = cur_func(env);
+	struct bpf_reg_state *reg;
 	int err;
 
-	if (subprog_returns_void(env, cur_frame->subprogno))
-		return 0;
-
-	err = check_reg_arg(env, BPF_REG_0, SRC_OP);
+	err = check_reg_arg(env, regno, SRC_OP);
 	if (err)
 		return err;
 
 	/* Pointers to arena are safe to pass between subprograms. */
-	if (is_arena_reg(env, BPF_REG_0))
+	if (is_arena_reg(env, regno))
 		return 0;
 
-	if (is_pointer_value(env, BPF_REG_0)) {
-		verbose(env, "R%d leaks addr as return value\n", BPF_REG_0);
+	if (is_pointer_value(env, regno)) {
+		verbose(env, "R%d leaks addr as return value\n", regno);
 		return -EACCES;
 	}
 
+	reg = reg_state(env, regno);
 	if (reg->type != SCALAR_VALUE) {
-		verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n",
-			reg_type_str(env, reg->type));
+		verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
+			regno, reg_type_str(env, reg->type));
 		return -EINVAL;
 	}
 
 	return 0;
 }
 
+static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+{
+	struct bpf_func_state *cur_frame = cur_func(env);
+	u32 subprog = cur_frame->subprogno;
+	u32 i, nregs;
+	int err;
+
+	if (subprog_returns_void(env, subprog))
+		return 0;
+
+	nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		err = check_global_ret_scalar_reg(env, ret_regs[i]);
+		if (err)
+			return err;
+	}
+
+	return 0;
+}
+
 /* Bitmask with 1s for all caller saved registers */
 #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
 
@@ -19328,6 +19473,13 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 			return -EOPNOTSUPP;
 		}
 
+		if (prog_extension && tgt_info->fmodel.ret_size > 8) {
+			bpf_log(log,
+				"Cannot replace function %s with a >8 byte return value\n",
+				tname);
+			return -EOPNOTSUPP;
+		}
+
 		/*
 		 * *.multi programs don't need an address during program
 		 * verification, we just take the module ref if needed.
@@ -20317,6 +20469,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* must precede the first bpf_ret_reg_pair() user below */
+	bpf_compute_subprog_ret_regs(env);
+
 	ret = bpf_compute_live_registers(env);
 	if (ret < 0)
 		goto skip_full_check;
diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
new file mode 100644
index 0000000000000..c295adedbae85
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "aggregate_ret_int128_c.skel.h"
+#include "aggregate_ret_struct_c.skel.h"
+#include "aggregate_ret_kfunc_c.skel.h"
+#include "aggregate_ret_func.skel.h"
+#include "aggregate_ret_kfunc.skel.h"
+
+void test_aggregate_ret(void)
+{
+	RUN_TESTS(aggregate_ret_int128_c);
+	RUN_TESTS(aggregate_ret_struct_c);
+	RUN_TESTS(aggregate_ret_kfunc_c);
+	RUN_TESTS(aggregate_ret_func);
+	RUN_TESTS(aggregate_ret_kfunc);
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
index 2523c07a16c65..0b54f911015c5 100644
--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
@@ -441,6 +441,19 @@ static void test_func_replace_int_with_void(void)
 				     " doesn't match type INT of global_func2()");
 }
 
+static void test_func_replace_ret_pair(void)
+{
+	const char *msg = "Cannot replace function agg_ret_target_func with a >8 byte return";
+
+	/*
+	 * An extension cannot replace a function whose return value comes back
+	 * in the R0:R2 pair: the extension's own return is capped at 8 bytes,
+	 * so it would leave R2 stale for the target's callers.
+	 */
+	test_obj_load_failure_common("freplace_ret_pair.bpf.o",
+				     "./aggregate_ret_target.bpf.o", msg);
+}
+
 static int find_prog_btf_id(const char *name, __u32 attach_prog_fd)
 {
 	struct bpf_prog_info info = {};
@@ -660,6 +673,8 @@ void serial_test_fexit_bpf2bpf(void)
 		test_func_replace_progmap();
 	if (test__start_subtest("freplace_int_with_void"))
 		test_func_replace_int_with_void();
+	if (test__start_subtest("freplace_ret_pair"))
+		test_func_replace_ret_pair();
 	if (test__start_subtest("freplace_void"))
 		test_func_replace_void();
 	if (test__start_subtest("sleepable_fentry_to_xdp"))
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
new file mode 100644
index 0000000000000..cfb21bcf704be
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -0,0 +1,260 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+typedef unsigned __int128 u128;
+
+__naked u128 global_agg_good(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"	/* low 64 bits */
+	"r2 = 0x5678;"	/* high 64 bits */
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_global_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad)
+	: __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("At subprogram exit the register R2 is not a scalar value")
+__naked int aggregate_ret_global_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad_ptr];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"	/* stack pointer placed in the second return register */
+	"exit;"
+	);
+}
+
+/*
+ * R2 is a return register once the subprogram returns a pair, so a stack
+ * pointer left in it is rejected at the callee's exit exactly as one in R0
+ * is: the callee frame is gone by the time the caller could use it.
+ */
+SEC("tc")
+__failure __msg("cannot return stack pointer to the caller")
+__naked int aggregate_ret_static_ptr_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_bad_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_no_r2(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_static_uninit_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_no_r2];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(static_agg_no_r2)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_precise(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 4;"	/* second half; its value is made precise below */
+	"exit;"
+	);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
+__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
+__naked int aggregate_ret_static_precise(void)
+{
+	asm volatile (
+	"call %[static_agg_precise];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_precise)
+	: __clobber_all);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
+__naked int aggregate_ret_global_precise(void)
+{
+	asm volatile (
+	"call %[global_agg_good];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_agg_good)
+	: __clobber_all);
+}
+
+/* A by-value struct that smuggles a pointer, which must be rejected. */
+struct with_ptr {
+	void *p;
+	__u64 x;
+};
+
+/* A by-value union that smuggles a pointer, which must be rejected too. */
+union upair_with_ptr {
+	void *p;
+	__u64 halves[2];
+};
+
+__naked struct with_ptr global_ret_struct_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_struct_ptr() has unsupported return type")
+__naked int aggregate_ret_global_struct_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_struct_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_struct_ptr)
+	: __clobber_all);
+}
+
+__naked union upair_with_ptr global_ret_union_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_union_ptr() has unsupported return type")
+__naked int aggregate_ret_global_union_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_union_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_union_ptr)
+	: __clobber_all);
+}
+
+struct ptr_pair {
+	void *p;
+	__u64 x;
+};
+
+static __naked __noinline struct ptr_pair static_ret_ptr_pair(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r1;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__naked int aggregate_ret_static_ptr_pair(void)
+{
+	asm volatile (
+	"call %[static_ret_ptr_pair];"
+	"r1 = *(u32 *)(r2 + 0);"	/* deref the returned ctx pointer */
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_ret_ptr_pair)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
new file mode 100644
index 0000000000000..913cc374215d5
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
@@ -0,0 +1,49 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+static __noinline u128 make_i128(__u64 a, __u64 b)
+{
+	return ((u128)(a + b) << 64) | (a - b);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v;
+
+	v = make_i128(a, b);
+	if ((__u64)(v >> 64) != a + b)
+		return 1;
+	if ((__u64)v != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+SEC("socket")
+__description("aggregate_ret_int128_c: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
new file mode 100644
index 0000000000000..c23b4beb1773c
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -0,0 +1,122 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+/*
+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address
+ * (rather than calling) avoids any dependence on the compiler lowering an
+ * __int128 or struct return value, which the BPF backend only supports from
+ * LLVM 23 on.
+ */
+void __kfunc_btf_root(void)
+{
+	asm volatile (""
+	:
+	: "r"(&bpf_kfunc_call_test_i128),
+	  "r"(&bpf_kfunc_call_test_ret_fastcall),
+	  "r"(&bpf_kfunc_call_test_ret_ptr),
+	  "r"(&bpf_kfunc_call_test_ret_ii),
+	  "r"(&bpf_kfunc_call_test_ret_big));
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
+__naked int aggregate_ret_kfunc_precise(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_i128];"
+	"r6 = r2;"		/* second return half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_i128)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL")
+__naked int aggregate_ret_kfunc_fastcall_fail(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_ret_fastcall];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_fastcall)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("is not composed of scalars")
+__naked int aggregate_ret_kfunc_ptr_fail(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ptr)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_kfunc_small_no_r2(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"r2 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ii];"
+	"r0 = r2;"	/* R2 is not a return register for a <=8 byte struct */
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ii)
+	: __clobber_all);
+}
+
+/*
+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
+ * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks,
+ * so this behaves the same on every architecture.
+ */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported")
+__naked int aggregate_ret_kfunc_too_big_fail(void)
+{
+	asm volatile (
+	"call %[bpf_kfunc_call_test_ret_big];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_big)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
new file mode 100644
index 0000000000000..2c1889fc28efd
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
@@ -0,0 +1,125 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v;
+
+	v = bpf_kfunc_call_test_i128(a, b);
+	if ((__u64)(v >> 64) != a + b)
+		return 1;
+	if ((__u64)v != a - b)
+		return 2;
+
+	return 0;
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct prog_test_ret_pair p;
+
+	p = bpf_kfunc_call_test_ret_pair(a, b);
+	if (p.hi != a + b)
+		return 1;
+	if (p.lo != a - b)
+		return 2;
+
+	return 0;
+}
+
+/* struct { u64 a; int b; }: 16 bytes, R0 = a, R2 = b. */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_li_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	int b = skb->len ^ MIX_B;
+	struct prog_test_ret_li r;
+
+	r = bpf_kfunc_call_test_ret_li(a, b);
+	if (r.a != a)
+		return 1;
+	if (r.b != ~b)
+		return 2;
+
+	return 0;
+}
+
+/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not a return reg. */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_ii_c_test(struct __sk_buff *skb)
+{
+	int a = skb->len ^ MIX_A;
+	int b = skb->len ^ MIX_B;
+	struct prog_test_ret_ii r;
+
+	r = bpf_kfunc_call_test_ret_ii(a, b);
+	if (r.a != a)
+		return 1;
+	if (r.b != b)
+		return 2;
+
+	return 0;
+}
+
+/* A union of 16 bytes takes the same R0:R2 path as a struct. */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_uu_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	union prog_test_ret_uu r;
+
+	r = bpf_kfunc_call_test_ret_uu(a, b);
+	if (r.parts.lo != a + b)
+		return 1;
+	if (r.parts.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+SEC("socket")
+__description("aggregate_ret_kfunc_c: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
new file mode 100644
index 0000000000000..83b1a37519887
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
@@ -0,0 +1,114 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+struct pair {
+	__u64 hi;	/* R0 */
+	__u64 lo;	/* R2 */
+};
+
+union upair {
+	__u64 halves[2];
+	struct {
+		__u64 lo;	/* R0 */
+		__u64 hi;	/* R2 */
+	} parts;
+};
+
+static __noinline struct pair make_pair(__u64 a, __u64 b)
+{
+	struct pair p = { .hi = a + b, .lo = a - b };
+
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p;
+
+	p = make_pair(a, b);
+	if (p.hi != a + b)
+		return 1;
+	if (p.lo != a - b)
+		return 2;
+
+	return 0;
+}
+
+__noinline struct pair make_pair_global(__u64 a, __u64 b)
+{
+	struct pair p = { .hi = a + b, .lo = a - b };
+
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p;
+
+	p = make_pair_global(a, b);
+	if (p.hi != a + b)
+		return 1;
+	if (p.lo != a - b)
+		return 2;
+
+	return 0;
+}
+
+static __noinline union upair make_upair(__u64 a, __u64 b)
+{
+	union upair p;
+
+	p.halves[0] = a + b;
+	p.halves[1] = a - b;
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_union_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	union upair p;
+
+	p = make_upair(a, b);
+	if (p.parts.lo != a + b)
+		return 1;
+	if (p.parts.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+SEC("socket")
+__description("aggregate_ret_struct_c: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
new file mode 100644
index 0000000000000..cffd8d7d3241a
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* freplace target: a global subprogram returning 16 bytes in R0:R2. */
+__naked unsigned __int128 agg_ret_target_func(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"
+	"r2 = 0x5678;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__naked int agg_ret_target(void)
+{
+	asm volatile (
+	"call %[agg_ret_target_func];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(agg_ret_target_func)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index d055fc7b3b95d..0be9441ec273b 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -431,6 +431,36 @@ __naked void subprog1(void)
 		::: __clobber_all);
 }
 
+static __used __naked unsigned __int128 aux2(void)
+{
+	asm volatile (
+		"r0 = 1;"
+		"r2 = 2;"
+		"exit;"
+		::: __clobber_all);
+}
+
+SEC("socket")
+/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
+__load_if_JITed()
+__log_level(2)
+__msg("0: .12345.... (85) call pc+2")
+__msg("1: ..2....... (bf) r0 = r2")
+/* R2 is not read at the exit of this program, which returns an int, ... */
+__msg("2: 0......... (95) exit")
+__msg("3: .......... (b7) r0 = 1")
+__msg("4: 0......... (b7) r2 = 2")
+/* ... but it is at the exit of aux2(), which returns a register pair. */
+__msg("5: 0.2....... (95) exit")
+__naked void subprog_ret_reg_pair(void)
+{
+	asm volatile (
+		"call aux2;"
+		"r0 = r2;"
+		"exit;"
+		::: __clobber_all);
+}
+
 #if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
 
 SEC("socket")
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index ac44d60e50666..9708efb93683b 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)
 
 SEC("?tc")
 __exception_cb(exception_cb_bad_ret_type1)
-__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.")
+__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.")
 int reject_exception_cb_type_1(struct __sk_buff *ctx)
 {
 	bpf_throw(0);
diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
new file mode 100644
index 0000000000000..12c15d293bd79
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+SEC("freplace/agg_ret_target_func")
+__u64 new_agg_ret_target_func(void)
+{
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index a6133f7521f34..ad36d583d2e7f 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -939,6 +939,69 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
 	return 0;
 }
 
+#if defined(__x86_64__) || defined(__aarch64__)
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+	return (__int128)(((unsigned __int128)(a + b) << 64) | (a - b));
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)
+{
+	struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)
+{
+	struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b)
+{
+	struct prog_test_ret_li r = { .a = a, .b = ~b };
+
+	return r;
+}
+
+__bpf_kfunc union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b)
+{
+	union prog_test_ret_uu r;
+
+	r.halves[0] = a + b;
+	r.halves[1] = a - b;
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
+{
+	struct prog_test_ret_ptr r = { .p = NULL, .tag = tag };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
+{
+	struct prog_test_ret_ii r = { .a = a, .b = b };
+
+	return r;
+}
+#endif /* __x86_64__ || __aarch64__ */
+
+/*
+ * Takes no argument on purpose: with no arguments there is nothing for the sret
+ * pointer to displace, so this needs no architecture guard even though it
+ * returns 24 bytes. See the comment on bpf_kfunc_call_test_i128() above.
+ */
+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)
+{
+	struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };
+
+	return r;
+}
+
 __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
 					 u64 e, u64 f, u64 g, u64 h,
 					 u64 i, u64 j)
@@ -1472,6 +1535,16 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+#if defined(__x86_64__) || defined(__aarch64__)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
+#endif
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index c4383acb53c11..c7be973cd2860 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -55,6 +55,40 @@ struct prog_test_big_arg {
 	__u64 b;
 };
 
+struct prog_test_ret_pair {
+	__u64 hi;
+	__u64 lo;
+};
+
+struct prog_test_ret_li {	/* 16 bytes: R0:R2 */
+	__u64 a;
+	int b;
+};
+
+struct prog_test_ret_ii {	/* 8 bytes: R0 only */
+	int a;
+	int b;
+};
+
+union prog_test_ret_uu {	/* 16 bytes: R0:R2 */
+	__u64 halves[2];
+	struct {
+		__u64 lo;
+		__u64 hi;
+	} parts;
+};
+
+struct prog_test_ret_ptr {	/* 16 bytes: contains a pointer */
+	void *p;
+	__u64 tag;
+};
+
+struct prog_test_ret_big {	/* 24 bytes: too large for R0:R2 */
+	__u64 a;
+	__u64 b;
+	__u64 c;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -130,6 +164,16 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+#ifdef __SIZEOF_INT128__
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
+#endif
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym;
+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
+union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;
+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym;
 __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
 			       __u64 e, __u64 f, __u64 g, __u64 h,
 			       __u64 i, __u64 j) __ksym;

gemini-3.1-pro-preview

							input: 27625
							output: 200
							thoughts: 970

**Fuzzing Plan for Extended Kfunc Return Value Support in BPF**

Okay, so we have a patch that's expanding BPF's kfunc and subprogram return value support to handle values up to 16 bytes, returned in the `R0:R2` register pair. This is a significant change, modifying the verifier and the JITs for arm64, riscv, and x86 architectures. This *absolutely* demands fuzzing.

My primary focus needs to be on identifying the most critical areas to target. Let's see... the patch touches a bunch of symbols, so let me break it down:

First, let's identify the specific functions that are being changed: I've identified the following symbols as the key areas:

*   `bpf_jit_supports_kfunc_ret_reg_pair` (architecture-specific)
*   `emit_kfunc_arena_args` (x86 specific)
*   `bpf_kfunc_ret_reg_pair` (verifier)
*   `btf_validate_return_type` (BTF)
*   `compute_insn_live_regs` (liveness)
*   `bpf_compute_subprog_ret_regs` (verifier)
*   `subprog_ret_pair_unreliable` (verifier)
*   `mark_kfunc_ret_regs` (verifier)
*   `btf_type_is_scalar_struct` (verifier)
*   `check_global_ret_scalar_reg` (verifier)
*   `check_global_subprog_return_code` (verifier)

Now, let me narrow the focus. These ones seem most pertinent, with regards to the function of the patch:

*   `bpf_kfunc_ret_reg_pair`
*   `bpf_compute_subprog_ret_regs`
*   `btf_validate_return_type`
*   `compute_insn_live_regs`
*   `check_global_subprog_return_code`
*   `check_global_ret_scalar_reg`
*   `btf_type_is_scalar_struct`
*   `mark_kfunc_ret_regs`
*   `subprog_ret_pair_unreliable`

Let's check if these symbols are static or global, to understand their scope. Right, so:

*   `bpf_kfunc_ret_reg_pair` and `btf_type_is_scalar_struct` are global which indicates a broader impact.
*   The rest of those functions are static, which might imply they are contained within a particular scope.

Since global symbols are more likely entry points, I should prioritize those in the initial fuzzing runs.  I'll then investigate the non-static functions again.  Let's see:

I need to focus on these non-static functions:

*   `bpf_kfunc_ret_reg_pair`
*   `btf_type_is_scalar_struct`
*   `bpf_jit_supports_kfunc_ret_reg_pair`

Also, let's peek at `kernel/bpf/verifier.c` and `kernel/bpf/btf.c` to see if there are other relevant functions that are not static. Okay:

*   `bpf_compute_live_registers` is modified.
*   `bpf_check` is modified.
*   `bpf_check_attach_target` is modified.

So, I'll definitely incorporate these into my scope. Let me jot down the functions I'll definitely use:

*   `bpf_kfunc_ret_reg_pair`
*   `btf_type_is_scalar_struct`
*   `bpf_jit_supports_kfunc_ret_reg_pair`
*   `bpf_compute_live_registers`

As for configurations, it's pretty clear that `BPF_SYSCALL` and `BPF_JIT` are relevant here. `CONFIG_BPF_SYSCALL` is the main one. I'll need to make sure those are enabled when I begin the fuzzing process.

Now, let's fire up the `set-results` tool and get started. This should be interesting... I can't wait to see what bugs we uncover.

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT"
  ],
  "FocusSymbols": [
    "bpf_kfunc_ret_reg_pair",
    "btf_type_is_scalar_struct",
    "bpf_jit_supports_kfunc_ret_reg_pair",
    "bpf_compute_live_registers"
  ],
  "Reasoning": "The patch introduces support for kfuncs and BPF subprograms returning larger values (up to 16 bytes) in the R0:R2 register pair. This involves changes to the BPF verifier (liveness tracking, return value checking, BTF validation) and JIT compilers (x86, arm64, riscv). These are significant functional changes to the core BPF subsystem that should be fuzzed to ensure verifier safety and JIT correctness.",
  "WorthFuzzing": true
}

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT"
  ],
  "FocusSymbols": [
    "bpf_kfunc_ret_reg_pair",
    "btf_type_is_scalar_struct",
    "bpf_jit_supports_kfunc_ret_reg_pair",
    "bpf_compute_live_registers"
  ],
  "Reasoning": "The patch introduces support for kfuncs and BPF subprograms returning larger values (up to 16 bytes) in the R0:R2 register pair. This involves changes to the BPF verifier (liveness tracking, return value checking, BTF validation) and JIT compilers (x86, arm64, riscv). These are significant functional changes to the core BPF subsystem that should be fuzzed to ensure verifier safety and JIT correctness.",
  "WorthFuzzing": true
}

{
  "WorthFuzzing": true
}

{
  "KMSANReasoning": "The patch extends the BPF verifier and JIT compilers to support returning values up to 16 bytes (e.g., `__int128` or small structs) from BPF subprograms and kfuncs using the R0 and R2 register pair. The changes primarily involve updating the verifier's liveness tracking, backtracking, and return value checks to ensure that both R0 and R2 are properly initialized before a BPF program returns. The JIT changes simply move the native ABI's second return register into BPF's R2. \n\nThere are no new kernel data structures allocated and copied to user space, nor any C-level uninitialized memory risks introduced. The verifier statically enforces that BPF programs initialize the R2 register when returning a pair, preventing uninitialized register leaks. Furthermore, KMSAN does not instrument JIT-compiled BPF code, so it would not provide additional coverage for the BPF program execution itself. Standard KASAN and verifier fuzzing are sufficient to detect any logic bugs, out-of-bounds accesses, or memory corruptions introduced by these changes.",
  "NeedsKMSAN": false
}

You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.

CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
  a comprehensive suite of debugging tools and sanitizers, including KASAN
  (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
  (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
  or page allocations) and kernel-to-user memory info-leaks.

Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.

Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.

Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
   attributes, ioctl output arguments, socket options, or BPF buffers) where fields
   or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
   or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
   could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
   uninitialized bytes of existing buffers.

Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
  dereferences, locking deadlocks, or use-after-free bugs (these are already caught
  by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
  or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.

Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit bafb5dc76d94a94d43e7b1575db7d4920a549687
Author: syz-cluster <triage@syzkaller.com>
Date:   Thu Aug 13 20:58:12 2026 +0000

    syz-cluster: applied patch under review

diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index 10e725cbe64cd..5d840d3000338 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -575,6 +575,69 @@ is also covered by this recovery. A kfunc handed an arena pointer may
 therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
 against the arena. Larger accesses must verify the range explicitly.
 
+2.9 kfunc Return Values
+-----------------------
+
+A kfunc may return a scalar, a pointer, or a small struct or union by
+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
+
+A struct or union returned by value must be composed only of scalars
+(recursively), where a scalar is an integer or an enum; arrays of scalars are
+allowed as members. Its bytes are handed back to the program as the raw
+contents of R0 (and R2), so a pointer field would be laundered into a scalar
+and escape the verifier's pointer provenance and reference tracking. A struct
+or union with a pointer member is therefore rejected at load time, and so is
+one with a floating-point member, which the ABI may not return in R0:R2 at
+all.
+
+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
+scalar-only struct or union, or an ``__int128``. Such a value is returned
+in the register pair R0:R2, matching the convention LLVM uses for the BPF
+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or
+union of 8 bytes or less is returned in R0 alone.
+
+::
+
+        struct bpf_pair { __u64 a, b; };   /* 16 bytes */
+
+        __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
+        {
+                struct bpf_pair p = { .a = 1, .b = 2 };
+
+                return p;      /* p.a in R0, p.b in R2 */
+        }
+
+Returning a value in the R0:R2 pair requires the JIT to place the second
+half of the return value into R2, which not every architecture supports
+right now. A kfunc with a return value larger than 8 bytes is therefore
+rejected at load time on a JIT that does not advertise this capability (see
+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
+by the interpreter. A return value larger than 16 bytes is not supported.
+
+The same R0:R2 convention applies to a BPF subprogram, global or static,
+that returns an ``__int128`` or a struct or union larger than 8 bytes. Such a
+program also requires the JIT, since the interpreter propagates only R0 out
+of a subprogram. A global subprogram is verified in isolation, so its
+by-value struct or union return is restricted to scalars just like a kfunc's;
+a static subprogram is verified inline and has no such restriction. The main
+program is not covered: its return value is the program's exit code, read out
+of R0 alone, so a declared upper half is never looked at.
+
+A global subprogram must leave a scalar in *every* register of the pair, so
+both halves of the returned value have to be assigned. Leaving the upper half
+uninitialized is not merely untidy: the compiler is then free to leave R2
+holding whatever it happened to hold, which for a subprogram taking a pointer
+argument is typically that pointer. Handing the caller an unknown scalar built
+from a pointer is a leak, so the verifier rejects it with::
+
+        At subprogram exit the register R2 is not a scalar value (...)
+
+Initialize the whole return value, for example ``struct pair p = {};``, to
+avoid this. A static subprogram is exempt from the scalar-only rule: it is
+verified inline, so an unassigned R2 is simply passed back to the caller as
+uninitialized and only a caller that reads it fails. A stack pointer left in
+R2 is still rejected there, just as one in R0 is.
+
 .. _BPF_kfunc_lifecycle_expectations:
 
 3. kfunc lifecycle expectations
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 74b4083791da3..7a4c5968976a6 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2346,6 +2346,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index 2504df1fa1118..49220765e96d4 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -2120,6 +2120,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_ptr_xchg(void)
 {
 	return true;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d920772af7d5f..2ec09dd02b187 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
  * arena NULL is offset 0. Return the number of emitted bytes.
  */
 static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
-				 const struct bpf_insn *insn, u8 **pprog)
+				 const struct btf_func_model *fm, u8 **pprog)
 {
-	const struct btf_func_model *fm;
 	u8 *prog = *pprog;
 	u8 *start = prog;
 	int i;
 
-	fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
-	if (!fm)
-		return -EINVAL;
-
 	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
 		u8 flags = fm->arg_flags[i];
 		u32 reg = BPF_REG_1 + i;
@@ -2644,6 +2639,8 @@ st:			insn_off = insn->off;
 
 			/* call */
 		case BPF_JMP | BPF_CALL: {
+			const struct btf_func_model *fm = NULL;
+
 			func = (u8 *) __bpf_call_base + imm32;
 			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
 				LOAD_TAIL_CALL_CNT_PTR(stack_depth);
@@ -2652,7 +2649,10 @@ st:			insn_off = insn->off;
 			if (!imm32)
 				return -EINVAL;
 			if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
-				err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
+				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
+				if (!fm)
+					return -EINVAL;
+				err = emit_kfunc_arena_args(bpf_prog, fm, &prog);
 				if (err < 0)
 					return err;
 				ip += err;
@@ -2666,6 +2666,14 @@ st:			insn_off = insn->off;
 				return -EINVAL;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
+			/*
+			 * A kfunc returning more than 8 bytes hands the second
+			 * half back in RDX (the native ABI's second return reg),
+			 * but BPF expects it in R0:R2. BPF R0 is RAX (no move
+			 * needed), while BPF R2 is RSI, so copy RDX into RSI.
+			 */
+			if (fm && fm->ret_size > 8)
+				emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
 			break;
 		}
 
@@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 27b43fda9b178..899e5ce986432 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -814,6 +814,8 @@ struct bpf_subprog_info {
 	bool is_async_cb: 1;
 	bool is_exception_cb: 1;
 	bool args_cached: 1;
+	/* true if the return value is passed in the R0:R2 register pair */
+	bool ret_reg_pair: 1;
 	/* true if bpf_fastcall stack region is used by functions that can't be inlined */
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
@@ -1048,6 +1050,13 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
 	return &env->subprog_info[subprog];
 }
 
+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	return subprog_info(env, subprog)->ret_reg_pair;
+}
+
+bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env *env, struct bpf_insn *insn);
+
 struct bpf_call_summary {
 	u8 num_params;
 	bool is_void;
@@ -1439,6 +1448,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);
 struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+			       const struct btf_type *t, int rec);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 4a9bc6a848f2e..6e746b0a09306 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_kfunc_ret_reg_pair(void);
 bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 40bd04421a991..3ef8a3da726de 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -425,6 +425,16 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				 */
 				verifier_bug_if(idx + 1 != subseq_idx, env,
 						"extra insn from subprog");
+				/* a global subprog returning more than 8 bytes
+				 * sets R2 as well. R2 is part of the args mask
+				 * checked just below, so clear it here rather
+				 * than next to R0. Only a subprog that does
+				 * return a pair defines R2, so leave the mask
+				 * alone otherwise and let the check below catch
+				 * an R2 that has no business being set.
+				 */
+				if (bpf_ret_reg_pair(env, subprog))
+					bt_clear_reg(bt, BPF_REG_2);
 				/* r1-r5 are invalidated after subprog call,
 				 * so for global func call it shouldn't be set
 				 * anymore
@@ -508,6 +518,17 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				return -ENOTSUPP;
 			/* regular helper call sets R0 */
 			bt_clear_reg(bt, BPF_REG_0);
+			/* a kfunc returning more than 8 bytes also sets R2.
+			 * R2 is part of the args mask checked just below, so
+			 * clear it here rather than next to R0. The prototype
+			 * lookup is only worth doing when R2 is requested at
+			 * all; any other call leaves R2 uninitialized, so a
+			 * request for it is caught by the check below.
+			 */
+			if (bt_is_reg_set(bt, BPF_REG_2) &&
+			    insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+			    bpf_kfunc_ret_reg_pair(env, insn))
+				bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				/* if backtracking was looking for registers R1-R5
 				 * they should have been found already.
@@ -522,7 +543,41 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 					return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			bool r0_precise;
+			bool from_subprog_call, r0_precise, r2_precise;
+			struct bpf_insn *call;
+			int subprog;
+
+			/* BPF_EXIT in subprog or callback always returns
+			 * right after the call instruction, so by checking
+			 * whether the instruction at subseq_idx-1 is subprog
+			 * call or not we can distinguish actual exit from
+			 * *subprog* from exit from *callback*. In the former
+			 * case, we need to propagate the precision of the
+			 * return registers, if necessary. In the latter we
+			 * never do that.
+			 */
+			from_subprog_call = subseq_idx - 1 >= 0 &&
+					    bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
+
+			/* Sample the return registers before the callback
+			 * handling below clears R1-R5: unlike R0, R2 is an
+			 * argument register as well, so that clear would drop
+			 * a pair return on the floor.
+			 */
+			r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
+			r2_precise = false;
+			if (from_subprog_call && bt_is_reg_set(bt, BPF_REG_2)) {
+				call = &env->prog->insnsi[subseq_idx - 1];
+				subprog = bpf_find_subprog(env, subseq_idx + call->imm);
+				if (subprog < 0)
+					return -EFAULT;
+				/* Only a callee that does return a pair defines
+				 * R2. Leave the mask alone otherwise, so that
+				 * the check below still catches an R2 that has
+				 * no business being set.
+				 */
+				r2_precise = bpf_ret_reg_pair(env, subprog);
+			}
 
 			/* Backtracking to a nested function call, 'idx' is a part of
 			 * the inner frame 'subseq_idx' is a part of the outer frame.
@@ -535,30 +590,27 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
 				for (i = BPF_REG_1; i <= BPF_REG_5; i++)
 					bt_clear_reg(bt, i);
+
+			/* a callee returning more than 8 bytes sets R2 as well;
+			 * R2 is part of the args mask checked just below, so
+			 * clear it here rather than next to R0.
+			 */
+			if (r2_precise)
+				bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				verifier_bug(env, "backtracking exit unexpected regs %x",
 					     bt_reg_mask(bt));
 				return -EFAULT;
 			}
 
-			/* BPF_EXIT in subprog or callback always returns
-			 * right after the call instruction, so by checking
-			 * whether the instruction at subseq_idx-1 is subprog
-			 * call or not we can distinguish actual exit from
-			 * *subprog* from exit from *callback*. In the former
-			 * case, we need to propagate r0 precision, if
-			 * necessary. In the former we never do that.
-			 */
-			r0_precise = subseq_idx - 1 >= 0 &&
-				     bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
-				     bt_is_reg_set(bt, BPF_REG_0);
-
 			bt_clear_reg(bt, BPF_REG_0);
 			if (bt_subprog_enter(bt))
 				return -EFAULT;
 
 			if (r0_precise)
 				bt_set_reg(bt, BPF_REG_0);
+			if (r2_precise)
+				bt_set_reg(bt, BPF_REG_2);
 			/* r6-r9 and stack slots will stay set in caller frame
 			 * bitmasks until we return back from callee(s)
 			 */
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 6606187ed4f43..c7718a2a56141 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7592,7 +7592,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
 		return -EINVAL;
 	}
 	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	if (ret < 0 || ret > 16) {
 		bpf_log(log,
 			"The function %s return type %s is unsupported.\n",
 			tname, btf_type_str(t));
@@ -7965,7 +7965,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env *env,
 
 /* Check whether the type is a valid return type. */
 static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
-		const struct btf_type *t, int subprog)
+		const struct btf_type *t, int subprog, bool is_global)
 {
 	u32 tags = 0;
 	int err;
@@ -7988,6 +7988,19 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
 	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
 		return 0;
 
+	if (btf_type_is_struct(t) && t->size <= 16) {
+		/*
+		 * A global function's caller models the return as an opaque
+		 * scalar pair, so it may only return scalars by value. A local
+		 * function is verified inline, so a pointer field stays tracked
+		 * and needs no such restriction.
+		 */
+		bool local_func = subprog && !is_global;
+
+		if (local_func || btf_type_is_scalar_struct(env, btf, t, 0))
+			return 0;
+	}
+
 	return -EOPNOTSUPP;
 }
 
@@ -8075,12 +8088,12 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 		return -EINVAL;
 	}
 
-	err = btf_validate_return_type(env, btf, t, subprog);
+	err = btf_validate_return_type(env, btf, t, subprog, is_global);
 	if (err) {
 		if (is_global) {
 			bpf_log(log,
-				"Global function %s() return value not void or scalar. "
-				"Only those are supported.\n",
+				"Global function %s() has unsupported return type. "
+				"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n",
 				tname);
 		}
 		return err;
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 6a94370a24488..cb66a2ef52b1f 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_stack_args(void)
 {
 	return false;
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 1c997aeba6fa5..850c58aa915be 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2062,7 +2062,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); }
 /* Compute info->{use,def} fields for the instruction */
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
-				   struct insn_live_regs *info)
+				   struct insn_live_regs *info,
+				   bool ret_reg_pair)
 {
 	struct bpf_call_summary cs;
 	const u8 class = BPF_CLASS(insn->code);
@@ -2074,6 +2075,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 	const u32 src32 = mask_lo(src);
 	const u32 dst32 = mask_lo(dst);
 	const u32 r0  = reg64_mask(0);
+	const u32 r2  = reg64_mask(BPF_REG_2);
 	u32 def = 0;
 	u32 use = U32_MAX;
 
@@ -2193,7 +2195,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			break;
 		case BPF_EXIT:
 			def = 0;
-			use = r0;
+			use = ret_reg_pair ? (r0 | r2) : r0;
 			break;
 		case BPF_CALL:
 			def = ALL_CALLER_SAVED_REGS;
@@ -2230,8 +2232,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 	struct insn_live_regs *state;
 	int insn_cnt = env->prog->len;
 	u64 pos, insn_pos;
-	int err = 0, i, j;
-	bool changed;
+	int err = 0, i, j, subprog, start, end;
+	bool changed, ret_reg_pair;
 
 	/* Use the following algorithm:
 	 * - define the following:
@@ -2258,8 +2260,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		goto out;
 	}
 
-	for (i = 0; i < insn_cnt; ++i)
-		compute_insn_live_regs(env, &insns[i], &state[i]);
+	for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+		start = env->subprog_info[subprog].start;
+		end = env->subprog_info[subprog + 1].start;
+		ret_reg_pair = bpf_ret_reg_pair(env, subprog);
+
+		for (i = start; i < end; ++i)
+			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
+	}
 
 	/* Forward pass: resolve stack access through FP-derived pointers */
 	err = bpf_compute_subprog_arg_access(env);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 164d16c243ca6..98a6f702d794d 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -382,27 +382,80 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
 	return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
 }
 
-static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
 {
-	const struct btf_type *type, *func, *func_proto;
+	const struct btf_type *func, *func_proto;
 	const struct btf *btf = env->prog->aux->btf;
 	u32 btf_id;
 
+	if (!btf || !env->prog->aux->func_info)
+		return NULL;
+
 	btf_id = env->prog->aux->func_info[subprog].type_id;
 
+	/* Both already validated by prepare_btf_func() at prog load. */
 	func = btf_type_by_id(btf, btf_id);
-	if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
-		return false;
-
 	func_proto = btf_type_by_id(btf, func->type);
-	if (!func_proto)
-		return false;
 
-	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
-	if (!type)
-		return false;
+	return btf_type_skip_modifiers(btf, func_proto->type, NULL);
+}
+
+static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type = subprog_ret_type(env, subprog);
+
+	return type && btf_type_is_void(type);
+}
+
+static u32 ret_regs_cnt(u32 size)
+{
+	return size > 8 && size <= 16 ? 2 : 1;
+}
+
+/* Registers holding a function return value, in order. See ret_regs_cnt(). */
+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };
+
+static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
+{
+	const struct btf *btf = env->prog->aux->btf;
+	const struct btf_type *type;
+	int subprog;
+	u32 size;
+
+	for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+		type = subprog_ret_type(env, subprog);
+		if (!type || !(btf_type_is_struct(type) || btf_type_is_scalar(type)))
+			continue;
+		if (IS_ERR(btf_resolve_size(btf, type, &size)))
+			continue;
+		if (ret_regs_cnt(size) > 1) {
+			subprog_info(env, subprog)->ret_reg_pair = true;
+			/*
+			 * The R0:R2 return convention is only implemented in
+			 * the JIT: the interpreter propagates BPF_R0 alone out
+			 * of a subprogram, so a caller reading R2 would see a
+			 * stale value.
+			 */
+			env->prog->jit_required = 1;
+		}
+	}
+}
 
-	return btf_type_is_void(type);
+/*
+ * A >8 byte BPF return changes the calling convention to R0:R2, and the
+ * verifier derives that convention from the subprogram's BTF prototype
+ * alone. Once that prototype is marked unreliable it is known not to
+ * describe the compiled code, so the convention read from it cannot be
+ * trusted either: reject the call rather than keep tracking R2 on the
+ * strength of a signature the verifier has already discarded.
+ */
+static bool subprog_ret_pair_unreliable(struct bpf_verifier_env *env, int subprog)
+{
+	struct bpf_prog_aux *aux = env->prog->aux;
+
+	return bpf_ret_reg_pair(env, subprog) &&
+	       aux->func_info_aux &&
+	       aux->func_info_aux[subprog].unreliable;
 }
 
 static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
@@ -2459,6 +2512,21 @@ find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset)
 		       sizeof(tab->descs[0]), kfunc_desc_cmp_by_id_off);
 }
 
+/*
+ * True if the kfunc called by @insn returns its value in the R0:R2 pair.
+ * Reads the same btf_func_model.ret_size that bpf_add_kfunc_call() validated
+ * and that the JIT keys the second return register off, so the verifier and
+ * the generated code cannot disagree about the convention.
+ */
+bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+	const struct bpf_kfunc_desc *desc;
+
+	desc = find_kfunc_desc(env->prog, insn->imm, insn->off);
+
+	return desc && ret_regs_cnt(desc->func_model.ret_size) > 1;
+}
+
 int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
 		       u16 btf_fd_idx, u8 **func_addr)
 {
@@ -2809,6 +2877,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
 	if (err)
 		return err;
+	if (func_model.ret_size > 8) {
+		if (kfunc.flags && (*kfunc.flags & KF_FASTCALL)) {
+			verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n",
+				kfunc.name);
+			return -EOPNOTSUPP;
+		}
+		if (!bpf_jit_supports_kfunc_ret_reg_pair()) {
+			verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
+				kfunc.name);
+			return -EOPNOTSUPP;
+		}
+	}
 
 	memset(&meta, 0, sizeof(meta));
 	meta.btf = kfunc.btf;
@@ -9381,6 +9461,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	u16 callee_incoming, stack_arg_cnt;
 	struct bpf_func_state *caller;
 	int err, subprog, target_insn;
+	u32 i, nregs;
 
 	target_insn = *insn_idx + insn->imm + 1;
 	subprog = bpf_find_subprog(env, target_insn);
@@ -9423,9 +9504,14 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		clear_caller_saved_regs(env, caller->regs);
 		invalidate_outgoing_stack_args(env, cur_func(env));
 
-		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
+		/*
+		 * A non-void global function returns a 64-bit SCALAR_VALUE in
+		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+		 */
 		if (!subprog_returns_void(env, subprog)) {
-			mark_reg_unknown(env, caller->regs, BPF_REG_0);
+			nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+			for (i = 0; i < nregs; i++)
+				mark_reg_unknown(env, caller->regs, ret_regs[i]);
 		}
 
 		if (env->subprog_info[subprog].might_throw) {
@@ -9443,6 +9529,12 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		return 0;
 	}
 
+	if (subprog_ret_pair_unreliable(env, subprog)) {
+		verbose(env, "Func#%d ('%s') returns >8 bytes, which requires reliable BTF\n",
+			subprog, subprog_name(env, subprog));
+		return -EINVAL;
+	}
+
 	/*
 	 * Track caller's total stack arg count (incoming + max outgoing).
 	 * This is needed so the JIT knows how much stack arg space to allocate.
@@ -9783,11 +9875,15 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	struct bpf_func_state *caller, *callee;
 	struct bpf_reg_state *r0;
 	bool in_callback_fn;
+	u32 i, nregs;
 	int err;
 
 	callee = state->frame[state->curframe];
 	r0 = &callee->regs[BPF_REG_0];
-	if (r0->type == PTR_TO_STACK) {
+	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
+			continue;
 		/* technically it's ok to return caller's stack pointer
 		 * (or caller's caller's pointer) back to the caller,
 		 * since these pointers are valid. Only current stack
@@ -9822,8 +9918,12 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 			return -EFAULT;
 		}
 	} else {
-		/* return to the caller whatever r0 had in the callee */
-		caller->regs[BPF_REG_0] = *r0;
+		/*
+		 * return to the caller whatever the callee had in the
+		 * return register(s)
+		 */
+		for (i = 0; i < nregs; i++)
+			caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]];
 	}
 
 	/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
@@ -10721,6 +10821,19 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 	return 0;
 }
 
+/*
+ * Mark the register(s) holding a @size byte kfunc return value as unknown
+ * scalars. Both halves of a register pair are treated the same way.
+ */
+static void mark_kfunc_ret_regs(struct bpf_verifier_env *env,
+				struct bpf_reg_state *regs, u32 size)
+{
+	u32 i, nregs = ret_regs_cnt(size);
+
+	for (i = 0; i < nregs; i++)
+		mark_reg_unknown(env, regs, ret_regs[i]);
+}
+
 static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ACQUIRE;
@@ -10988,9 +11101,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_
 }
 
 /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
-					const struct btf *btf,
-					const struct btf_type *t, int rec)
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env,
+			       const struct btf *btf,
+			       const struct btf_type *t, int rec)
 {
 	const struct btf_type *member_type;
 	const struct btf_member *member;
@@ -11008,7 +11121,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
 				verbose(env, "max struct nesting depth exceeded\n");
 				return false;
 			}
-			if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
+			if (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
 				return false;
 			continue;
 		}
@@ -11407,7 +11520,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		 (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) ||
 		  is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
 		if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) &&
-		    !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+		    !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 			verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n",
 				reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
 			return -EINVAL;
@@ -11423,7 +11536,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		 * scalars. The access size is derived from the pointed-to BTF type.
 		 */
 		if (!btf_type_is_scalar(ref_t) &&
-		    !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+		    !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 			verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n",
 				reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
 			return -EINVAL;
@@ -12394,7 +12507,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 				break;
 			}
 
-			if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+			if (!btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 				enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);
 
 				verbose(env, "%s is %s expected %s %s",
@@ -12858,7 +12971,7 @@ static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg
 
 		struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);
 		if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) {
-			if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
+			if (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
 				verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n");
 				return -EINVAL;
 			}
@@ -13191,10 +13304,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	}
 
 	if (btf_type_is_scalar(t)) {
-		mark_reg_unknown(env, regs, BPF_REG_0);
+		mark_kfunc_ret_regs(env, regs, t->size);
 		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
 		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
 			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
+	} else if (btf_type_is_struct(t)) {
+		/*
+		 * The returned struct comes back as raw register bits modeled
+		 * as an unknown scalar, so it must contain only scalars:
+		 * otherwise a pointer field would be laundered into a scalar
+		 * and escape provenance and reference tracking.
+		 */
+		if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+			verbose(env,
+				"kernel function %s returns %s %s that is not composed of scalars\n",
+				func_name, btf_type_str(t),
+				btf_name_by_offset(desc_btf, t->name_off));
+			return -EINVAL;
+		}
+		mark_kfunc_ret_regs(env, regs, t->size);
 	} else if (btf_type_is_ptr(t)) {
 		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
 		err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
@@ -16641,37 +16769,54 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	return 0;
 }
 
-static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
 {
-	struct bpf_reg_state *reg = reg_state(env, BPF_REG_0);
-	struct bpf_func_state *cur_frame = cur_func(env);
+	struct bpf_reg_state *reg;
 	int err;
 
-	if (subprog_returns_void(env, cur_frame->subprogno))
-		return 0;
-
-	err = check_reg_arg(env, BPF_REG_0, SRC_OP);
+	err = check_reg_arg(env, regno, SRC_OP);
 	if (err)
 		return err;
 
 	/* Pointers to arena are safe to pass between subprograms. */
-	if (is_arena_reg(env, BPF_REG_0))
+	if (is_arena_reg(env, regno))
 		return 0;
 
-	if (is_pointer_value(env, BPF_REG_0)) {
-		verbose(env, "R%d leaks addr as return value\n", BPF_REG_0);
+	if (is_pointer_value(env, regno)) {
+		verbose(env, "R%d leaks addr as return value\n", regno);
 		return -EACCES;
 	}
 
+	reg = reg_state(env, regno);
 	if (reg->type != SCALAR_VALUE) {
-		verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n",
-			reg_type_str(env, reg->type));
+		verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
+			regno, reg_type_str(env, reg->type));
 		return -EINVAL;
 	}
 
 	return 0;
 }
 
+static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+{
+	struct bpf_func_state *cur_frame = cur_func(env);
+	u32 subprog = cur_frame->subprogno;
+	u32 i, nregs;
+	int err;
+
+	if (subprog_returns_void(env, subprog))
+		return 0;
+
+	nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		err = check_global_ret_scalar_reg(env, ret_regs[i]);
+		if (err)
+			return err;
+	}
+
+	return 0;
+}
+
 /* Bitmask with 1s for all caller saved registers */
 #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
 
@@ -19328,6 +19473,13 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 			return -EOPNOTSUPP;
 		}
 
+		if (prog_extension && tgt_info->fmodel.ret_size > 8) {
+			bpf_log(log,
+				"Cannot replace function %s with a >8 byte return value\n",
+				tname);
+			return -EOPNOTSUPP;
+		}
+
 		/*
 		 * *.multi programs don't need an address during program
 		 * verification, we just take the module ref if needed.
@@ -20317,6 +20469,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* must precede the first bpf_ret_reg_pair() user below */
+	bpf_compute_subprog_ret_regs(env);
+
 	ret = bpf_compute_live_registers(env);
 	if (ret < 0)
 		goto skip_full_check;
diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
new file mode 100644
index 0000000000000..c295adedbae85
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "aggregate_ret_int128_c.skel.h"
+#include "aggregate_ret_struct_c.skel.h"
+#include "aggregate_ret_kfunc_c.skel.h"
+#include "aggregate_ret_func.skel.h"
+#include "aggregate_ret_kfunc.skel.h"
+
+void test_aggregate_ret(void)
+{
+	RUN_TESTS(aggregate_ret_int128_c);
+	RUN_TESTS(aggregate_ret_struct_c);
+	RUN_TESTS(aggregate_ret_kfunc_c);
+	RUN_TESTS(aggregate_ret_func);
+	RUN_TESTS(aggregate_ret_kfunc);
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
index 2523c07a16c65..0b54f911015c5 100644
--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
@@ -441,6 +441,19 @@ static void test_func_replace_int_with_void(void)
 				     " doesn't match type INT of global_func2()");
 }
 
+static void test_func_replace_ret_pair(void)
+{
+	const char *msg = "Cannot replace function agg_ret_target_func with a >8 byte return";
+
+	/*
+	 * An extension cannot replace a function whose return value comes back
+	 * in the R0:R2 pair: the extension's own return is capped at 8 bytes,
+	 * so it would leave R2 stale for the target's callers.
+	 */
+	test_obj_load_failure_common("freplace_ret_pair.bpf.o",
+				     "./aggregate_ret_target.bpf.o", msg);
+}
+
 static int find_prog_btf_id(const char *name, __u32 attach_prog_fd)
 {
 	struct bpf_prog_info info = {};
@@ -660,6 +673,8 @@ void serial_test_fexit_bpf2bpf(void)
 		test_func_replace_progmap();
 	if (test__start_subtest("freplace_int_with_void"))
 		test_func_replace_int_with_void();
+	if (test__start_subtest("freplace_ret_pair"))
+		test_func_replace_ret_pair();
 	if (test__start_subtest("freplace_void"))
 		test_func_replace_void();
 	if (test__start_subtest("sleepable_fentry_to_xdp"))
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
new file mode 100644
index 0000000000000..cfb21bcf704be
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -0,0 +1,260 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+typedef unsigned __int128 u128;
+
+__naked u128 global_agg_good(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"	/* low 64 bits */
+	"r2 = 0x5678;"	/* high 64 bits */
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_global_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad)
+	: __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("At subprogram exit the register R2 is not a scalar value")
+__naked int aggregate_ret_global_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad_ptr];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"	/* stack pointer placed in the second return register */
+	"exit;"
+	);
+}
+
+/*
+ * R2 is a return register once the subprogram returns a pair, so a stack
+ * pointer left in it is rejected at the callee's exit exactly as one in R0
+ * is: the callee frame is gone by the time the caller could use it.
+ */
+SEC("tc")
+__failure __msg("cannot return stack pointer to the caller")
+__naked int aggregate_ret_static_ptr_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_bad_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_no_r2(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_static_uninit_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_no_r2];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(static_agg_no_r2)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_precise(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 4;"	/* second half; its value is made precise below */
+	"exit;"
+	);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
+__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
+__naked int aggregate_ret_static_precise(void)
+{
+	asm volatile (
+	"call %[static_agg_precise];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_precise)
+	: __clobber_all);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
+__naked int aggregate_ret_global_precise(void)
+{
+	asm volatile (
+	"call %[global_agg_good];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_agg_good)
+	: __clobber_all);
+}
+
+/* A by-value struct that smuggles a pointer, which must be rejected. */
+struct with_ptr {
+	void *p;
+	__u64 x;
+};
+
+/* A by-value union that smuggles a pointer, which must be rejected too. */
+union upair_with_ptr {
+	void *p;
+	__u64 halves[2];
+};
+
+__naked struct with_ptr global_ret_struct_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_struct_ptr() has unsupported return type")
+__naked int aggregate_ret_global_struct_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_struct_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_struct_ptr)
+	: __clobber_all);
+}
+
+__naked union upair_with_ptr global_ret_union_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_union_ptr() has unsupported return type")
+__naked int aggregate_ret_global_union_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_union_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_union_ptr)
+	: __clobber_all);
+}
+
+struct ptr_pair {
+	void *p;
+	__u64 x;
+};
+
+static __naked __noinline struct ptr_pair static_ret_ptr_pair(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r1;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__naked int aggregate_ret_static_ptr_pair(void)
+{
+	asm volatile (
+	"call %[static_ret_ptr_pair];"
+	"r1 = *(u32 *)(r2 + 0);"	/* deref the returned ctx pointer */
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_ret_ptr_pair)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
new file mode 100644
index 0000000000000..913cc374215d5
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
@@ -0,0 +1,49 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+static __noinline u128 make_i128(__u64 a, __u64 b)
+{
+	return ((u128)(a + b) << 64) | (a - b);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v;
+
+	v = make_i128(a, b);
+	if ((__u64)(v >> 64) != a + b)
+		return 1;
+	if ((__u64)v != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+SEC("socket")
+__description("aggregate_ret_int128_c: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
new file mode 100644
index 0000000000000..c23b4beb1773c
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -0,0 +1,122 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+/*
+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address
+ * (rather than calling) avoids any dependence on the compiler lowering an
+ * __int128 or struct return value, which the BPF backend only supports from
+ * LLVM 23 on.
+ */
+void __kfunc_btf_root(void)
+{
+	asm volatile (""
+	:
+	: "r"(&bpf_kfunc_call_test_i128),
+	  "r"(&bpf_kfunc_call_test_ret_fastcall),
+	  "r"(&bpf_kfunc_call_test_ret_ptr),
+	  "r"(&bpf_kfunc_call_test_ret_ii),
+	  "r"(&bpf_kfunc_call_test_ret_big));
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
+__naked int aggregate_ret_kfunc_precise(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_i128];"
+	"r6 = r2;"		/* second return half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_i128)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL")
+__naked int aggregate_ret_kfunc_fastcall_fail(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_ret_fastcall];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_fastcall)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("is not composed of scalars")
+__naked int aggregate_ret_kfunc_ptr_fail(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ptr)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_kfunc_small_no_r2(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"r2 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ii];"
+	"r0 = r2;"	/* R2 is not a return register for a <=8 byte struct */
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ii)
+	: __clobber_all);
+}
+
+/*
+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
+ * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks,
+ * so this behaves the same on every architecture.
+ */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported")
+__naked int aggregate_ret_kfunc_too_big_fail(void)
+{
+	asm volatile (
+	"call %[bpf_kfunc_call_test_ret_big];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_big)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
new file mode 100644
index 0000000000000..2c1889fc28efd
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
@@ -0,0 +1,125 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v;
+
+	v = bpf_kfunc_call_test_i128(a, b);
+	if ((__u64)(v >> 64) != a + b)
+		return 1;
+	if ((__u64)v != a - b)
+		return 2;
+
+	return 0;
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct prog_test_ret_pair p;
+
+	p = bpf_kfunc_call_test_ret_pair(a, b);
+	if (p.hi != a + b)
+		return 1;
+	if (p.lo != a - b)
+		return 2;
+
+	return 0;
+}
+
+/* struct { u64 a; int b; }: 16 bytes, R0 = a, R2 = b. */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_li_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	int b = skb->len ^ MIX_B;
+	struct prog_test_ret_li r;
+
+	r = bpf_kfunc_call_test_ret_li(a, b);
+	if (r.a != a)
+		return 1;
+	if (r.b != ~b)
+		return 2;
+
+	return 0;
+}
+
+/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not a return reg. */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_ii_c_test(struct __sk_buff *skb)
+{
+	int a = skb->len ^ MIX_A;
+	int b = skb->len ^ MIX_B;
+	struct prog_test_ret_ii r;
+
+	r = bpf_kfunc_call_test_ret_ii(a, b);
+	if (r.a != a)
+		return 1;
+	if (r.b != b)
+		return 2;
+
+	return 0;
+}
+
+/* A union of 16 bytes takes the same R0:R2 path as a struct. */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_uu_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	union prog_test_ret_uu r;
+
+	r = bpf_kfunc_call_test_ret_uu(a, b);
+	if (r.parts.lo != a + b)
+		return 1;
+	if (r.parts.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+SEC("socket")
+__description("aggregate_ret_kfunc_c: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
new file mode 100644
index 0000000000000..83b1a37519887
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
@@ -0,0 +1,114 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+struct pair {
+	__u64 hi;	/* R0 */
+	__u64 lo;	/* R2 */
+};
+
+union upair {
+	__u64 halves[2];
+	struct {
+		__u64 lo;	/* R0 */
+		__u64 hi;	/* R2 */
+	} parts;
+};
+
+static __noinline struct pair make_pair(__u64 a, __u64 b)
+{
+	struct pair p = { .hi = a + b, .lo = a - b };
+
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p;
+
+	p = make_pair(a, b);
+	if (p.hi != a + b)
+		return 1;
+	if (p.lo != a - b)
+		return 2;
+
+	return 0;
+}
+
+__noinline struct pair make_pair_global(__u64 a, __u64 b)
+{
+	struct pair p = { .hi = a + b, .lo = a - b };
+
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p;
+
+	p = make_pair_global(a, b);
+	if (p.hi != a + b)
+		return 1;
+	if (p.lo != a - b)
+		return 2;
+
+	return 0;
+}
+
+static __noinline union upair make_upair(__u64 a, __u64 b)
+{
+	union upair p;
+
+	p.halves[0] = a + b;
+	p.halves[1] = a - b;
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_union_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	union upair p;
+
+	p = make_upair(a, b);
+	if (p.parts.lo != a + b)
+		return 1;
+	if (p.parts.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+SEC("socket")
+__description("aggregate_ret_struct_c: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
new file mode 100644
index 0000000000000..cffd8d7d3241a
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* freplace target: a global subprogram returning 16 bytes in R0:R2. */
+__naked unsigned __int128 agg_ret_target_func(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"
+	"r2 = 0x5678;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__naked int agg_ret_target(void)
+{
+	asm volatile (
+	"call %[agg_ret_target_func];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(agg_ret_target_func)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index d055fc7b3b95d..0be9441ec273b 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -431,6 +431,36 @@ __naked void subprog1(void)
 		::: __clobber_all);
 }
 
+static __used __naked unsigned __int128 aux2(void)
+{
+	asm volatile (
+		"r0 = 1;"
+		"r2 = 2;"
+		"exit;"
+		::: __clobber_all);
+}
+
+SEC("socket")
+/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
+__load_if_JITed()
+__log_level(2)
+__msg("0: .12345.... (85) call pc+2")
+__msg("1: ..2....... (bf) r0 = r2")
+/* R2 is not read at the exit of this program, which returns an int, ... */
+__msg("2: 0......... (95) exit")
+__msg("3: .......... (b7) r0 = 1")
+__msg("4: 0......... (b7) r2 = 2")
+/* ... but it is at the exit of aux2(), which returns a register pair. */
+__msg("5: 0.2....... (95) exit")
+__naked void subprog_ret_reg_pair(void)
+{
+	asm volatile (
+		"call aux2;"
+		"r0 = r2;"
+		"exit;"
+		::: __clobber_all);
+}
+
 #if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
 
 SEC("socket")
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index ac44d60e50666..9708efb93683b 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)
 
 SEC("?tc")
 __exception_cb(exception_cb_bad_ret_type1)
-__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.")
+__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.")
 int reject_exception_cb_type_1(struct __sk_buff *ctx)
 {
 	bpf_throw(0);
diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
new file mode 100644
index 0000000000000..12c15d293bd79
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+SEC("freplace/agg_ret_target_func")
+__u64 new_agg_ret_target_func(void)
+{
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index a6133f7521f34..ad36d583d2e7f 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -939,6 +939,69 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
 	return 0;
 }
 
+#if defined(__x86_64__) || defined(__aarch64__)
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+	return (__int128)(((unsigned __int128)(a + b) << 64) | (a - b));
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)
+{
+	struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)
+{
+	struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b)
+{
+	struct prog_test_ret_li r = { .a = a, .b = ~b };
+
+	return r;
+}
+
+__bpf_kfunc union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b)
+{
+	union prog_test_ret_uu r;
+
+	r.halves[0] = a + b;
+	r.halves[1] = a - b;
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
+{
+	struct prog_test_ret_ptr r = { .p = NULL, .tag = tag };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
+{
+	struct prog_test_ret_ii r = { .a = a, .b = b };
+
+	return r;
+}
+#endif /* __x86_64__ || __aarch64__ */
+
+/*
+ * Takes no argument on purpose: with no arguments there is nothing for the sret
+ * pointer to displace, so this needs no architecture guard even though it
+ * returns 24 bytes. See the comment on bpf_kfunc_call_test_i128() above.
+ */
+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)
+{
+	struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };
+
+	return r;
+}
+
 __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
 					 u64 e, u64 f, u64 g, u64 h,
 					 u64 i, u64 j)
@@ -1472,6 +1535,16 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+#if defined(__x86_64__) || defined(__aarch64__)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
+#endif
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index c4383acb53c11..c7be973cd2860 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -55,6 +55,40 @@ struct prog_test_big_arg {
 	__u64 b;
 };
 
+struct prog_test_ret_pair {
+	__u64 hi;
+	__u64 lo;
+};
+
+struct prog_test_ret_li {	/* 16 bytes: R0:R2 */
+	__u64 a;
+	int b;
+};
+
+struct prog_test_ret_ii {	/* 8 bytes: R0 only */
+	int a;
+	int b;
+};
+
+union prog_test_ret_uu {	/* 16 bytes: R0:R2 */
+	__u64 halves[2];
+	struct {
+		__u64 lo;
+		__u64 hi;
+	} parts;
+};
+
+struct prog_test_ret_ptr {	/* 16 bytes: contains a pointer */
+	void *p;
+	__u64 tag;
+};
+
+struct prog_test_ret_big {	/* 24 bytes: too large for R0:R2 */
+	__u64 a;
+	__u64 b;
+	__u64 c;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -130,6 +164,16 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+#ifdef __SIZEOF_INT128__
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
+#endif
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym;
+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
+union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;
+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym;
 __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
 			       __u64 e, __u64 f, __u64 g, __u64 h,
 			       __u64 i, __u64 j) __ksym;

gemini-3.1-pro-preview

							input: 27622
							output: 258
							thoughts: 3804

**BPF Subprogram Return Value Expansion Analysis**

Alright, let's break down this patch. It's all about beefing up BPF's return value capabilities, letting subprograms and kfuncs sling back values up to 16 bytes. The core is the BPF verifier, and the JIT compilers. Specifically, we're tracking a second register, R2, alongside R0 for these larger returns. The verifier's gotta be on point, ensuring R2 is properly initialized and preventing any pointer shenanigans.

I see a bunch of verifier changes: `bpf_ret_reg_pair` and `bpf_kfunc_ret_reg_pair` for checking, `backtrack_insn` for R2 propagation, `btf_validate_return_type` making sure structs are scalar-only, `compute_insn_live_regs` marking R2 as used at `BPF_EXIT`, `check_global_subprog_return_code` inspecting both R0 and R2, `prepare_func_exit` for moving both, and `mark_kfunc_ret_regs` declaring them as unknown scalars. Basically, the verifier is all over it, which is good.

My immediate concern is: Are we exposing uninitialized memory to user space? Not directly. The return values are *in registers*, within the BPF program's environment or from the kfunc to BPF. If a kfunc *somehow* returns an uninitialized struct, that's visible to the BPF program, but kfuncs are supposed to initialize their returns anyway. Test kfuncs are added to verify this. And if *BPF* itself returns uninitialized data, `check_global_subprog_return_code` checks that R0 and R2 are scalar – which means, they are initialized.

But hang on… `check_global_ret_scalar_reg` checks that the register isn't `NOT_INIT`. That's how it's done: If it's uninitialized, it won't pass `check_reg_arg(env, regno, SRC_OP)`. This is a *good* thing. We're covered there.

Okay, so no real risk of raw, uninitialized memory leaks to user space *from the BPF program's point of view*. The verifier is there. Kfuncs are standard C, returning structs by value and C compilers handle the initialization. Padding is the thing that concerns me. The verifier needs to make sure to enforce this.

The real concern is that kfuncs are required to return structs composed of *only* scalar values. If there's padding in the struct, that padding is now exposed to the BPF program as part of the return value, in R0 and R2. The BPF program *could* read that padding, potentially leaking kernel stack memory or whatever was sitting there.

Does the verifier actually check for padding? It calls `btf_type_is_scalar_struct`, but does that function look at padding specifically? Let's check… nope. It just verifies the *members* are scalars, but doesn't explicitly look for gaps.

Consider a kfunc like this:
```c
struct {
    u8 a;
    // 7 bytes padding
    u64 b;
};
```
That's 16 bytes. R0 gets `a` and padding; R2 gets `b`. The BPF program *sees* that padding in R0. If that padding is *uninitialized*…that's a leak!

So, can KMSAN help? Normally, KMSAN would catch the use of uninitialized memory. KMSAN could trigger if a kfunc returns a struct with uninitialized padding when it is copied to the return registers, or even when the JITed BPF code attempts to access those registers. But KMSAN *doesn't* instrument the JIT'ed BPF code! KMSAN only instruments the kernel, so KMSAN would only catch the kfunc returning the struct. Returning it with padding on the stack would be a problem. That could go undetected.

Is this patch *introducing* that kind of kfunc, or is it a preexisting issue? The patch itself doesn't add any real new kfuncs. It only adds some test kfuncs. The one here, `struct prog_test_ret_li`, has a 4-byte padding. When initializing it in a kfunc, the C99 standard *should* zero out the padding due to static storage. In reality, compiler behavior is what matters. If the initializer covers the whole struct, everything is zeroed. But, if the compiler only initializes members with an initializer, the padding *might* be uninitialized.

The patch itself enables support for kfuncs, not

{
  "KMSANReasoning": "The patch extends the BPF verifier and JIT compilers to support returning values up to 16 bytes (e.g., `__int128` or small structs) from BPF subprograms and kfuncs using the R0 and R2 register pair. The changes primarily involve updating the verifier's liveness tracking, backtracking, and return value checks to ensure that both R0 and R2 are properly initialized before a BPF program returns. The JIT changes simply move the native ABI's second return register into BPF's R2. \n\nThere are no new kernel data structures allocated and copied to user space, nor any C-level uninitialized memory risks introduced. The verifier statically enforces that BPF programs initialize the R2 register when returning a pair, preventing uninitialized register leaks. Furthermore, KMSAN does not instrument JIT-compiled BPF code, so it would not provide additional coverage for the BPF program execution itself. Standard KASAN and verifier fuzzing are sufficient to detect any logic bugs, out-of-bounds accesses, or memory corruptions introduced by these changes.",
  "NeedsKMSAN": false
}

{
  "KMSANReasoning": "The patch extends the BPF verifier and JIT compilers to support returning values up to 16 bytes (e.g., `__int128` or small structs) from BPF subprograms and kfuncs using the R0 and R2 register pair. The changes primarily involve updating the verifier's liveness tracking, backtracking, and return value checks to ensure that both R0 and R2 are properly initialized before a BPF program returns. The JIT changes simply move the native ABI's second return register into BPF's R2. \n\nThere are no new kernel data structures allocated and copied to user space, nor any C-level uninitialized memory risks introduced. The verifier statically enforces that BPF programs initialize the R2 register when returning a pair, preventing uninitialized register leaks. Furthermore, KMSAN does not instrument JIT-compiled BPF code, so it would not provide additional coverage for the BPF program execution itself. Standard KASAN and verifier fuzzing are sufficient to detect any logic bugs, out-of-bounds accesses, or memory corruptions introduced by these changes.",
  "NeedsKMSAN": false
}

LLM Calls Summary:
Total Calls	Total Tokens	Avg Tokens	Total Duration (Seconds)	Avg Duration (Seconds)

Tool Calls Summary:
Total Calls	Total Duration (Seconds)	Avg Duration (Seconds)