AI Trajectory

Trajectory: (Expand/Collapse All)
Seq	Timestamp	Type	Name	Duration
0/0	2026/08/19 06:42	flow	patch-triage	1m Args: null Results: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT" ], "FocusSymbols": [ "btf_type_is_scalar_struct", "btf_distill_func_proto", "btf_prepare_func_args", "bpf_compute_live_registers", "bpf_jit_supports_kfunc_ret_reg_pair" ], "KMSANReasoning": "The patch series introduces support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or small structs) in the register pair R0:R2. The changes are confined to the BPF verifier's static analysis (tracking liveness and initialization state of R2, type checking BTF return types) and architecture-specific JIT compilers (moving the native second return register into BPF R2). \n\nThere are no allocations of complex data structures, no new kernel-to-user memory copies, and no modifications to bounds checking of memory buffers. The verifier statically enforces that BPF programs cannot read uninitialized BPF registers, and any logical bugs in this tracking would lead to verifier bypasses rather than C-level uninitialized memory uses. Furthermore, KMSAN instruments kernel C code and tracks memory, but it does not instrument JIT-ed BPF code or track BPF register states directly. Therefore, any potential issues introduced by this patch (such as verifier flaws allowing stale register reads) would not be detected by KMSAN. Standard KASAN, verifier tests, and syzkaller's BPF fuzzing are the appropriate tools for validating these changes.", "NeedsKMSAN": false, "Reasoning": "The patch series adds support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or scalar-only structs/unions) in the register pair R0:R2. This involves modifications to the BPF verifier to track the return values in both registers, changes to the liveness and backtracking logic, and updates to the JIT compilers (x86, arm64, riscv) to support this new calling convention. These are functional changes to reachable core kernel logic (BPF subsystem) that should be fuzzed to ensure the verifier correctly tracks the register states and prevents any leaks or invalid memory accesses.", "WorthFuzzing": true }
1/1	2026/08/19 06:42	action	read-patch-diff	0m Args: null Results: { "PatchDiff": "commit cc236f68937f8479bce51601bd02ac7647affe1d\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Wed Aug 19 06:41:58 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst\nindex 85f73e0bbd0ff..89dea6b0b0241 100644\n--- a/Documentation/bpf/kfuncs.rst\n+++ b/Documentation/bpf/kfuncs.rst\n@@ -575,6 +575,71 @@ is also covered by this recovery. A kfunc handed an arena pointer may\n therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking\n against the arena. Larger accesses must verify the range explicitly.\n \n+2.9 kfunc Return Values\n+-----------------------\n+\n+A kfunc may return a scalar, a pointer, or a small struct or union by\n+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.\n+\n+A struct or union returned by value must be composed only of scalars\n+(recursively), where a scalar is an integer or an enum; arrays of scalars are\n+allowed as members. Its bytes are handed back to the program as the raw\n+contents of R0 (and R2), so a pointer field would be laundered into a scalar\n+and escape the verifier's pointer provenance and reference tracking. A struct\n+or union with a pointer member is therefore rejected at load time, and so is\n+one with a floating-point member, which the ABI may not return in R0:R2 at\n+all.\n+\n+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a\n+scalar-only struct or union, or an ``__int128``. Such a value is returned\n+in the register pair R0:R2, matching the convention LLVM uses for the BPF\n+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or\n+union of 8 bytes or less is returned in R0 alone.\n+\n+::\n+\n+ struct bpf_pair { __u64 a, b; }; /* 16 bytes /\n+\n+ __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)\n+ {\n+ struct bpf_pair p = { .a = 1, .b = 2 };\n+\n+ return p; / p.a in R0, p.b in R2 /\n+ }\n+\n+Returning a value in the R0:R2 pair requires the JIT to place the second\n+half of the return value into R2, which not every architecture supports\n+right now. A kfunc with a return value larger than 8 bytes is therefore\n+rejected at load time on a JIT that does not advertise this capability (see\n+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run\n+by the interpreter. A return value larger than 16 bytes is not supported.\n+\n+The same R0:R2 convention applies to a BPF subprogram, global or static, that\n+returns an ``__int128`` or a struct or union larger than 8 bytes. It is only\n+used when the program is JITed, since the interpreter propagates only R0 out of\n+a subprogram: without a JIT the return value stays in R0 alone, and a caller\n+reading R2 is rejected for reading an uninitialized register. A global\n+subprogram is verified in isolation, so its by-value struct or union return is\n+restricted to scalars just like a kfunc's; a static subprogram is verified\n+inline and has no such restriction. The main program is not covered: its return\n+value is the program's exit code, read out of R0 alone, so a declared upper\n+half is never looked at.\n+\n+A global subprogram must leave a scalar in every* register of the pair, so\n+both halves of the returned value have to be assigned. Leaving the upper half\n+uninitialized is not merely untidy: the compiler is then free to leave R2\n+holding whatever it happened to hold, which for a subprogram taking a pointer\n+argument is typically that pointer. Handing the caller an unknown scalar built\n+from a pointer is a leak, so the verifier rejects it with::\n+\n+ At subprogram exit the register R2 is not a scalar value (...)\n+\n+Initialize the whole return value, for example ``struct pair p = {};``, to\n+avoid this. A static subprogram is exempt from the scalar-only rule: it is\n+verified inline, so an unassigned R2 is simply passed back to the caller as\n+uninitialized and only a caller that reads it fails. A stack pointer left in\n+R2 is still rejected there, just as one in R0 is.\n+\n .. _BPF_kfunc_lifecycle_expectations:\n \n 3. kfunc lifecycle expectations\ndiff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c\nindex c18e005a41dbe..3aa3ea0bc30bb 100644\n--- a/arch/arm64/net/bpf_jit_comp.c\n+++ b/arch/arm64/net/bpf_jit_comp.c\n@@ -2388,6 +2388,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_stack_args(void)\n {\n \treturn true;\ndiff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c\nindex 74efe4b138d2a..47c7bf431ba85 100644\n--- a/arch/riscv/net/bpf_jit_comp64.c\n+++ b/arch/riscv/net/bpf_jit_comp64.c\n@@ -2121,6 +2121,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_ptr_xchg(void)\n {\n \treturn true;\ndiff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c\nindex 1a9fb530adc3c..48429fae06410 100644\n--- a/arch/x86/net/bpf_jit_comp.c\n+++ b/arch/x86/net/bpf_jit_comp.c\n@@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip,\n * arena NULL is offset 0. Return the number of emitted bytes.\n /\n static int emit_kfunc_arena_args(struct bpf_prog bpf_prog,\n-\t\t\t\t const struct bpf_insn insn, u8 pprog)\n+\t\t\t\t const struct btf_func_model fm, u8 *pprog)\n {\n-\tconst struct btf_func_model fm;\n \tu8 prog = pprog;\n \tu8 start = prog;\n \tint i;\n \n-\tfm = bpf_jit_find_kfunc_model(bpf_prog, insn);\n-\tif (!fm)\n-\t\treturn -EINVAL;\n-\n \tfor (i = 0; i \u003c min_t(int, fm-\u003enr_args, MAX_BPF_FUNC_REG_ARGS); i++) {\n \t\tu8 flags = fm-\u003earg_flags[i];\n \t\tu32 reg = BPF_REG_1 + i;\n@@ -2644,6 +2639,8 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \n \t\t\t/ call /\n \t\tcase BPF_JMP \| BPF_CALL: {\n+\t\t\tconst struct btf_func_model fm = NULL;\n+\n \t\t\tfunc = (u8 ) __bpf_call_base + imm32;\n \t\t\tif (src_reg == BPF_PSEUDO_CALL \u0026\u0026 tail_call_reachable) {\n \t\t\t\tLOAD_TAIL_CALL_CNT_PTR(stack_depth);\n@@ -2652,7 +2649,10 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tif (!imm32)\n \t\t\t\treturn -EINVAL;\n \t\t\tif (src_reg == BPF_PSEUDO_KFUNC_CALL) {\n-\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, insn, \u0026prog);\n+\t\t\t\tfm = bpf_jit_find_kfunc_model(bpf_prog, insn);\n+\t\t\t\tif (!fm)\n+\t\t\t\t\treturn -EINVAL;\n+\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, fm, \u0026prog);\n \t\t\t\tif (err \u003c 0)\n \t\t\t\t\treturn err;\n \t\t\t\tip += err;\n@@ -2666,6 +2666,14 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\treturn -EINVAL;\n \t\t\tif (priv_frame_ptr)\n \t\t\t\tpop_r9(\u0026prog);\n+\t\t\t/\n+\t\t\t * A kfunc returning more than 8 bytes hands the second\n+\t\t\t * half back in RDX (the native ABI's second return reg),\n+\t\t\t * but BPF expects it in R0:R2. BPF R0 is RAX (no move\n+\t\t\t * needed), while BPF R2 is RSI, so copy RDX into RSI.\n+\t\t\t /\n+\t\t\tif (fm \u0026\u0026 fm-\u003eret_size \u003e 8)\n+\t\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_2, BPF_REG_3);\n \t\t\tbreak;\n \t\t}\n \n@@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_stack_args(void)\n {\n \treturn true;\ndiff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h\nindex 5fad59fdab0d0..004b06785521c 100644\n--- a/include/linux/bpf_verifier.h\n+++ b/include/linux/bpf_verifier.h\n@@ -819,6 +819,8 @@ struct bpf_subprog_info {\n \tbool is_async_cb: 1;\n \tbool is_exception_cb: 1;\n \tbool args_cached: 1;\n+\t/ true if the return value is passed in the R0:R2 register pair /\n+\tbool ret_reg_pair: 1;\n \t/ true if bpf_fastcall stack region is used by functions that can't be inlined /\n \tbool keep_fastcall_stack: 1;\n \tbool changes_pkt_data: 1;\n@@ -1055,6 +1057,11 @@ static inline struct bpf_subprog_info subprog_info(struct bpf_verifier_env env\n \treturn \u0026env-\u003esubprog_info[subprog];\n }\n \n+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env env, int subprog)\n+{\n+\treturn subprog_info(env, subprog)-\u003eret_reg_pair;\n+}\n+\n struct bpf_call_summary {\n \tu8 num_params;\n \tbool is_void;\n@@ -1481,6 +1488,8 @@ int bpf_jmp_offset(struct bpf_insn insn);\n struct bpf_iarray bpf_insn_successors(struct bpf_verifier_env env, u32 idx);\n void bpf_fmt_stack_mask(char buf, ssize_t buf_sz, u64 stack_mask);\n bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog);\n+bool btf_type_is_scalar_struct(struct bpf_verifier_env env, const struct btf btf,\n+\t\t\t const struct btf_type t, int rec);\n \n int bpf_find_subprog(struct bpf_verifier_env env, int off);\n bool bpf_is_throw_kfunc(struct bpf_insn insn);\ndiff --git a/include/linux/filter.h b/include/linux/filter.h\nindex 4a9bc6a848f2e..6e746b0a09306 100644\n--- a/include/linux/filter.h\n+++ b/include/linux/filter.h\n@@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);\n bool bpf_jit_supports_subprog_tailcalls(void);\n bool bpf_jit_supports_percpu_insn(void);\n bool bpf_jit_supports_kfunc_call(void);\n+bool bpf_jit_supports_kfunc_ret_reg_pair(void);\n bool bpf_jit_supports_stack_args(void);\n bool bpf_jit_supports_arena_args(void);\n bool bpf_jit_supports_far_kfunc_call(void);\ndiff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c\nindex a2b18a9f1694c..653db80bcc47b 100644\n--- a/kernel/bpf/backtrack.c\n+++ b/kernel/bpf/backtrack.c\n@@ -423,6 +423,10 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx,\n \t\t\t\t /\n \t\t\t\tverifier_bug_if(idx + 1 != subseq_idx, env,\n \t\t\t\t\t\t\"extra insn from subprog\");\n+\t\t\t\t/* global subprog always sets R0 /\n+\t\t\t\tbt_clear_reg(bt, BPF_REG_0);\n+\t\t\t\t/ and if it does not set R2, main pass would catch it /\n+\t\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\t\t/ r1-r5 are invalidated after subprog call,\n \t\t\t\t * so for global func call it shouldn't be set\n \t\t\t\t * anymore\n@@ -432,8 +436,6 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx,\n \t\t\t\t\t\t bt_reg_mask(bt));\n \t\t\t\t\treturn -EFAULT;\n \t\t\t\t}\n-\t\t\t\t/ global subprog always sets R0 /\n-\t\t\t\tbt_clear_reg(bt, BPF_REG_0);\n \t\t\t\treturn 0;\n \t\t\t} else {\n \t\t\t\t/ static subprog call instruction, which\n@@ -506,6 +508,8 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx,\n \t\t\t\treturn -ENOTSUPP;\n \t\t\t/ regular helper call sets R0 /\n \t\t\tbt_clear_reg(bt, BPF_REG_0);\n+\t\t\t/ kfunc might also set R2 /\n+\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\tif (bt_reg_mask(bt) \u0026 BPF_REGMASK_ARGS) {\n \t\t\t\t/ if backtracking was looking for registers R1-R5\n \t\t\t\t * they should have been found already.\n@@ -520,7 +524,25 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx,\n \t\t\t\t\treturn -EFAULT;\n \t\t\t}\n \t\t} else if (opcode == BPF_EXIT) {\n-\t\t\tbool r0_precise;\n+\t\t\tbool from_subprog_call, r0_precise, r2_precise;\n+\n+\t\t\t/ BPF_EXIT in subprog or callback always returns\n+\t\t\t * right after the call instruction, so by checking\n+\t\t\t * whether the instruction at subseq_idx-1 is subprog\n+\t\t\t * call or not we can distinguish actual exit from\n+\t\t\t * subprog from exit from callback. In the former\n+\t\t\t * case, we need to propagate the precision of the\n+\t\t\t * return registers, if necessary. In the latter we\n+\t\t\t * never do that.\n+\t\t\t /\n+\t\t\tfrom_subprog_call = subseq_idx - 1 \u003e= 0 \u0026\u0026\n+\t\t\t\t\t bpf_pseudo_call(\u0026env-\u003eprog-\u003einsnsi[subseq_idx - 1]);\n+\n+\t\t\t/ Sample the return registers before the callback\n+\t\t\t * handling below clears R1-R5.\n+\t\t\t /\n+\t\t\tr0_precise = from_subprog_call \u0026\u0026 bt_is_reg_set(bt, BPF_REG_0);\n+\t\t\tr2_precise = from_subprog_call \u0026\u0026 bt_is_reg_set(bt, BPF_REG_2);\n \n \t\t\t/ Backtracking to a nested function call, 'idx' is a part of\n \t\t\t * the inner frame 'subseq_idx' is a part of the outer frame.\n@@ -533,30 +555,22 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx,\n \t\t\tif (subseq_idx \u003e= 0 \u0026\u0026 bpf_calls_callback(env, subseq_idx))\n \t\t\t\tfor (i = BPF_REG_1; i \u003c= BPF_REG_5; i++)\n \t\t\t\t\tbt_clear_reg(bt, i);\n+\n+\t\t\tbt_clear_reg(bt, BPF_REG_0);\n+\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\tif (bt_reg_mask(bt) \u0026 BPF_REGMASK_ARGS) {\n \t\t\t\tverifier_bug(env, \"backtracking exit unexpected regs %x\",\n \t\t\t\t\t bt_reg_mask(bt));\n \t\t\t\treturn -EFAULT;\n \t\t\t}\n \n-\t\t\t/ BPF_EXIT in subprog or callback always returns\n-\t\t\t * right after the call instruction, so by checking\n-\t\t\t * whether the instruction at subseq_idx-1 is subprog\n-\t\t\t * call or not we can distinguish actual exit from\n-\t\t\t * subprog from exit from callback. In the former\n-\t\t\t * case, we need to propagate r0 precision, if\n-\t\t\t * necessary. In the former we never do that.\n-\t\t\t /\n-\t\t\tr0_precise = subseq_idx - 1 \u003e= 0 \u0026\u0026\n-\t\t\t\t bpf_pseudo_call(\u0026env-\u003eprog-\u003einsnsi[subseq_idx - 1]) \u0026\u0026\n-\t\t\t\t bt_is_reg_set(bt, BPF_REG_0);\n-\n-\t\t\tbt_clear_reg(bt, BPF_REG_0);\n \t\t\tif (bt_subprog_enter(bt))\n \t\t\t\treturn -EFAULT;\n \n \t\t\tif (r0_precise)\n \t\t\t\tbt_set_reg(bt, BPF_REG_0);\n+\t\t\tif (r2_precise)\n+\t\t\t\tbt_set_reg(bt, BPF_REG_2);\n \t\t\t/ r6-r9 and stack slots will stay set in caller frame\n \t\t\t * bitmasks until we return back from callee(s)\n \t\t\t /\ndiff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c\nindex 6967d48bba49d..58b81c6f5288c 100644\n--- a/kernel/bpf/btf.c\n+++ b/kernel/bpf/btf.c\n@@ -7591,7 +7591,7 @@ int btf_distill_func_proto(struct bpf_verifier_log log,\n \t\treturn -EINVAL;\n \t}\n \tret = __get_type_size(btf, func-\u003etype, \u0026t);\n-\tif (ret \u003c 0 \|\| btf_type_is_struct(t)) {\n+\tif (ret \u003c 0 \|\| ret \u003e 16) {\n \t\tbpf_log(log,\n \t\t\t\"The function %s return type %s is unsupported.\\n\",\n \t\t\ttname, btf_type_str(t));\n@@ -7684,6 +7684,12 @@ static int btf_check_func_type_match(struct bpf_verifier_log log,\n \t\t\tbtf_type_str(t2), fn2);\n \t\treturn -EINVAL;\n \t}\n+\tif (btf_type_has_size(t1) \u0026\u0026 (t1-\u003esize \u003e 8 \|\| t2-\u003esize \u003e 8)) {\n+\t\tbpf_log(log,\n+\t\t\t\"Return type of %s() has size %u and of %s() size %u, and a size above 8 bytes cannot be replaced\\n\",\n+\t\t\tfn1, t1-\u003esize, fn2, t2-\u003esize);\n+\t\treturn -EINVAL;\n+\t}\n \n \tfor (i = 0; i \u003c nargs1; i++) {\n \t\tt1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL);\n@@ -7964,7 +7970,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env env,\n \n /* Check whether the type is a valid return type. /\n static int btf_validate_return_type(struct bpf_verifier_env env, struct btf btf,\n-\t\tconst struct btf_type t, int subprog)\n+\t\tconst struct btf_type t, int subprog, bool is_global)\n {\n \tu32 tags = 0;\n \tint err;\n@@ -7987,6 +7993,19 @@ static int btf_validate_return_type(struct bpf_verifier_env env, struct btf bt\n \tif (btf_type_is_void(t) \|\| btf_type_is_int(t) \|\| btf_is_any_enum(t))\n \t\treturn 0;\n \n+\tif (btf_type_is_struct(t) \u0026\u0026 t-\u003esize \u003c= 16) {\n+\t\t/\n+\t\t * A global function's caller models the return as an opaque\n+\t\t * scalar pair, so it may only return scalars by value. A local\n+\t\t * function is verified inline, so a pointer field stays tracked\n+\t\t * and needs no such restriction.\n+\t\t /\n+\t\tbool local_func = subprog \u0026\u0026 !is_global;\n+\n+\t\tif (local_func \|\| btf_type_is_scalar_struct(env, btf, t, 0))\n+\t\t\treturn 0;\n+\t}\n+\n \treturn -EOPNOTSUPP;\n }\n \n@@ -8074,12 +8093,12 @@ int btf_prepare_func_args(struct bpf_verifier_env env, int subprog)\n \t\treturn -EINVAL;\n \t}\n \n-\terr = btf_validate_return_type(env, btf, t, subprog);\n+\terr = btf_validate_return_type(env, btf, t, subprog, is_global);\n \tif (err) {\n \t\tif (is_global) {\n \t\t\tbpf_log(log,\n-\t\t\t\t\"Global function %s() return value not void or scalar. \"\n-\t\t\t\t\"Only those are supported.\\n\",\n+\t\t\t\t\"Global function %s() has unsupported return type. \"\n+\t\t\t\t\"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\\n\",\n \t\t\t\ttname);\n \t\t}\n \t\treturn err;\ndiff --git a/kernel/bpf/core.c b/kernel/bpf/core.c\nindex d55e737ed75ad..5db77d7915dfc 100644\n--- a/kernel/bpf/core.c\n+++ b/kernel/bpf/core.c\n@@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)\n \treturn false;\n }\n \n+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn false;\n+}\n+\n bool __weak bpf_jit_supports_stack_args(void)\n {\n \treturn false;\ndiff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c\nindex 74fc4b3f80d6e..71f998c6eb888 100644\n--- a/kernel/bpf/liveness.c\n+++ b/kernel/bpf/liveness.c\n@@ -2060,7 +2060,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m \u003e\u003e 16); }\n /* Compute info-\u003e{use,def} fields for the instruction /\n static void compute_insn_live_regs(struct bpf_verifier_env env,\n \t\t\t\t struct bpf_insn insn,\n-\t\t\t\t struct insn_live_regs info)\n+\t\t\t\t struct insn_live_regs info,\n+\t\t\t\t bool ret_reg_pair)\n {\n \tstruct bpf_call_summary cs;\n \tconst u8 class = BPF_CLASS(insn-\u003ecode);\n@@ -2072,6 +2073,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env,\n \tconst u32 src32 = mask_lo(src);\n \tconst u32 dst32 = mask_lo(dst);\n \tconst u32 r0 = reg64_mask(0);\n+\tconst u32 r2 = reg64_mask(BPF_REG_2);\n \tu32 def = 0;\n \tu32 use = U32_MAX;\n \n@@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env,\n \t\t\tbreak;\n \t\tcase BPF_EXIT:\n \t\t\tdef = 0;\n-\t\t\tuse = r0;\n+\t\t\tuse = ret_reg_pair ? (r0 \| r2) : r0;\n \t\t\tbreak;\n \t\tcase BPF_CALL:\n \t\t\tdef = ALL_CALLER_SAVED_REGS;\n@@ -2228,8 +2230,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env env)\n \tstruct insn_live_regs state;\n \tint insn_cnt = env-\u003eprog-\u003elen;\n \tu64 pos, insn_pos;\n-\tint err = 0, i, j;\n-\tbool changed;\n+\tint err = 0, i, j, subprog, start, end;\n+\tbool changed, ret_reg_pair;\n \n \t/ Use the following algorithm:\n \t * - define the following:\n@@ -2256,8 +2258,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env env)\n \t\tgoto out;\n \t}\n \n-\tfor (i = 0; i \u003c insn_cnt; ++i)\n-\t\tcompute_insn_live_regs(env, \u0026insns[i], \u0026state[i]);\n+\tfor (subprog = 0; subprog \u003c env-\u003esubprog_cnt; subprog++) {\n+\t\tstart = env-\u003esubprog_info[subprog].start;\n+\t\tend = env-\u003esubprog_info[subprog + 1].start;\n+\t\tret_reg_pair = bpf_ret_reg_pair(env, subprog);\n+\n+\t\tfor (i = start; i \u003c end; ++i)\n+\t\t\tcompute_insn_live_regs(env, \u0026insns[i], \u0026state[i], ret_reg_pair);\n+\t}\n \n \t/ Forward pass: resolve stack access through FP-derived pointers /\n \terr = bpf_compute_subprog_arg_access(env);\ndiff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c\nindex e421ea2b80c3a..7a939dfa3797b 100644\n--- a/kernel/bpf/verifier.c\n+++ b/kernel/bpf/verifier.c\n@@ -385,27 +385,69 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog)\n \treturn aux \u0026\u0026 aux[subprog].linkage == BTF_FUNC_GLOBAL;\n }\n \n-static bool subprog_returns_void(struct bpf_verifier_env env, int subprog)\n+static const struct btf_type subprog_ret_type(struct bpf_verifier_env env, int subprog)\n {\n-\tconst struct btf_type type, func, func_proto;\n+\tconst struct btf_type func, func_proto;\n \tconst struct btf btf = env-\u003eprog-\u003eaux-\u003ebtf;\n \tu32 btf_id;\n \n+\tif (!btf \|\| !env-\u003eprog-\u003eaux-\u003efunc_info)\n+\t\treturn NULL;\n+\n \tbtf_id = env-\u003eprog-\u003eaux-\u003efunc_info[subprog].type_id;\n \n+\t/ Both already validated by prepare_btf_func() at prog load. /\n \tfunc = btf_type_by_id(btf, btf_id);\n-\tif (verifier_bug_if(!func, env, \"btf_id %u not found\", btf_id))\n-\t\treturn false;\n-\n \tfunc_proto = btf_type_by_id(btf, func-\u003etype);\n-\tif (!func_proto)\n-\t\treturn false;\n \n-\ttype = btf_type_skip_modifiers(btf, func_proto-\u003etype, NULL);\n-\tif (!type)\n-\t\treturn false;\n+\treturn btf_type_skip_modifiers(btf, func_proto-\u003etype, NULL);\n+}\n \n-\treturn btf_type_is_void(type);\n+static bool subprog_returns_void(struct bpf_verifier_env env, int subprog)\n+{\n+\tconst struct btf_type type = subprog_ret_type(env, subprog);\n+\n+\treturn type \u0026\u0026 btf_type_is_void(type);\n+}\n+\n+static u32 ret_regs_cnt(u32 size)\n+{\n+\treturn size \u003e 8 \u0026\u0026 size \u003c= 16 ? 2 : 1;\n+}\n+\n+/ Registers holding a function return value, in order. See ret_regs_cnt(). /\n+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };\n+\n+static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env env)\n+{\n+\tconst struct btf btf = env-\u003eprog-\u003eaux-\u003ebtf;\n+\tconst struct btf_type type;\n+\tint subprog;\n+\tu32 size;\n+\n+\tif (!env-\u003eprog-\u003ejit_requested \|\| bpf_prog_is_offloaded(env-\u003eprog-\u003eaux))\n+\t\treturn 0;\n+\n+\t/\n+\t Skip the main program: its return value is the program's exit code,\n+\t * read out of R0, so it never uses the register pair. An extension does\n+\t * have a real prototype for subprog 0, but bpf_check_attach_target()\n+\t * refuses to replace a function returning more than 8 bytes.\n+\t /\n+\tfor (subprog = 1; subprog \u003c env-\u003esubprog_cnt; subprog++) {\n+\t\ttype = subprog_ret_type(env, subprog);\n+\t\tif (!type \|\| btf_type_is_void(type))\n+\t\t\tcontinue;\n+\t\tif (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, \u0026size)), env,\n+\t\t\t\t \"cannot size return type of subprog %d\", subprog))\n+\t\t\treturn -EFAULT;\n+\t\tif (ret_regs_cnt(size) \u003e 1) {\n+\t\t\tsubprog_info(env, subprog)-\u003eret_reg_pair = true;\n+\t\t\tenv-\u003eprog-\u003ejit_required = 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n }\n \n const char bpf_subprog_name(const struct bpf_verifier_env env, int subprog)\n@@ -2877,6 +2919,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env env, u32 func_id, u16 offset)\n \terr = btf_distill_func_proto(\u0026env-\u003elog, kfunc.btf, kfunc.proto, kfunc.name, \u0026func_model);\n \tif (err)\n \t\treturn err;\n+\tif (func_model.ret_size \u003e 8) {\n+\t\tif (kfunc.flags \u0026\u0026 (kfunc.flags \u0026 KF_FASTCALL)) {\n+\t\t\tverbose(env, \"kfunc %s with \u003e8-byte return is not supported with KF_FASTCALL\\n\",\n+\t\t\t\tkfunc.name);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\t\tif (!bpf_jit_supports_kfunc_ret_reg_pair()) {\n+\t\t\tverbose(env, \"kfunc %s with \u003e8-byte return is not supported by JIT\\n\",\n+\t\t\t\tkfunc.name);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\t}\n \n \tmemset(\u0026meta, 0, sizeof(meta));\n \tmeta.btf = kfunc.btf;\n@@ -9853,6 +9907,7 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn,\n \tu16 callee_incoming, stack_arg_cnt;\n \tstruct bpf_func_state caller;\n \tint err, subprog, target_insn;\n+\tu32 i, nregs;\n \n \ttarget_insn = insn_idx + insn-\u003eimm + 1;\n \tsubprog = bpf_find_subprog(env, target_insn);\n@@ -9908,9 +9963,14 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn,\n \t\tclear_caller_saved_regs(env, caller-\u003eregs);\n \t\tinvalidate_outgoing_stack_args(env, cur_func(env));\n \n-\t\t/ All non-void global functions return a 64-bit SCALAR_VALUE. /\n+\t\t/\n+\t\t * A non-void global function returns a 64-bit SCALAR_VALUE in\n+\t\t * R0, or a \u003e8 byte SCALAR_VALUE in the R0:R2 register pair.\n+\t\t /\n \t\tif (!returns_void) {\n-\t\t\tmark_reg_unknown(env, caller-\u003eregs, BPF_REG_0);\n+\t\t\tnregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;\n+\t\t\tfor (i = 0; i \u003c nregs; i++)\n+\t\t\t\tmark_reg_unknown(env, caller-\u003eregs, ret_regs[i]);\n \t\t\tbpf_diag_mod_end(env);\n \t\t}\n \n@@ -10270,11 +10330,15 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx)\n \tstruct bpf_func_state caller, callee;\n \tstruct bpf_reg_state r0;\n \tbool in_callback_fn;\n+\tu32 i, nregs;\n \tint err;\n \n \tcallee = state-\u003eframe[state-\u003ecurframe];\n \tr0 = \u0026callee-\u003eregs[BPF_REG_0];\n-\tif (r0-\u003etype == PTR_TO_STACK) {\n+\tnregs = bpf_ret_reg_pair(env, callee-\u003esubprogno) ? 2 : 1;\n+\tfor (i = 0; i \u003c nregs; i++) {\n+\t\tif (callee-\u003eregs[ret_regs[i]].type != PTR_TO_STACK)\n+\t\t\tcontinue;\n \t\t/* technically it's ok to return caller's stack pointer\n \t\t * (or caller's caller's pointer) back to the caller,\n \t\t * since these pointers are valid. Only current stack\n@@ -10309,9 +10373,13 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx)\n \t\t\treturn -EFAULT;\n \t\t}\n \t} else {\n-\t\t/* return to the caller whatever r0 had in the callee /\n+\t\t/\n+\t\t * return to the caller whatever the callee had in the\n+\t\t * return register(s)\n+\t\t /\n \t\tbpf_diag_mod_begin(env, \u0026caller-\u003eregs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE);\n-\t\tcaller-\u003eregs[BPF_REG_0] = r0;\n+\t\tfor (i = 0; i \u003c nregs; i++)\n+\t\t\tcaller-\u003eregs[ret_regs[i]] = callee-\u003eregs[ret_regs[i]];\n \t\tbpf_diag_mod_end(env);\n \t}\n \n@@ -11246,6 +11314,19 @@ static int check_helper_call(struct bpf_verifier_env env, struct bpf_insn insn\n \treturn 0;\n }\n \n+/\n+ Mark the register(s) holding a @size byte kfunc return value as unknown\n+ * scalars. Both halves of a register pair are treated the same way.\n+ /\n+static void mark_kfunc_ret_regs(struct bpf_verifier_env env,\n+\t\t\t\tstruct bpf_reg_state regs, u32 size)\n+{\n+\tu32 i, nregs = ret_regs_cnt(size);\n+\n+\tfor (i = 0; i \u003c nregs; i++)\n+\t\tmark_reg_unknown(env, regs, ret_regs[i]);\n+}\n+\n static bool is_kfunc_acquire(struct bpf_call_arg_meta meta)\n {\n \treturn meta-\u003ekfunc_flags \u0026 KF_ACQUIRE;\n@@ -11513,9 +11594,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta meta, u32 arg_\n }\n \n / Returns true if struct is composed of scalars, 4 levels of nesting allowed /\n-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env,\n-\t\t\t\t\tconst struct btf btf,\n-\t\t\t\t\tconst struct btf_type t, int rec)\n+bool btf_type_is_scalar_struct(struct bpf_verifier_env env,\n+\t\t\t const struct btf btf,\n+\t\t\t const struct btf_type t, int rec)\n {\n \tconst struct btf_type member_type;\n \tconst struct btf_member member;\n@@ -11533,7 +11614,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env,\n \t\t\t\tverbose(env, \"max struct nesting depth exceeded\\n\");\n \t\t\t\treturn false;\n \t\t\t}\n-\t\t\tif (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))\n+\t\t\tif (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1))\n \t\t\t\treturn false;\n \t\t\tcontinue;\n \t\t}\n@@ -11932,7 +12013,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta,\n \t\t (is_kfunc_arg_mem_size(meta-\u003ebtf, \u0026args[arg + 1]) \|\|\n \t\t is_kfunc_arg_const_mem_size(meta-\u003ebtf, \u0026args[arg + 1]))) {\n \t\tif (!btf_type_is_void(ref_t) \u0026\u0026 !btf_type_is_scalar(ref_t) \u0026\u0026\n-\t\t !__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t !btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\tverbose(env, \"%s pointer type %s %s must point to void, scalar, or struct with scalar\\n\",\n \t\t\t\treg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);\n \t\t\treturn -EINVAL;\n@@ -11948,7 +12029,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta,\n \t\t * scalars. The access size is derived from the pointed-to BTF type.\n \t\t /\n \t\tif (!btf_type_is_scalar(ref_t) \u0026\u0026\n-\t\t !__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t !btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\tverbose(env, \"%s pointer type %s %s must point to scalar, or struct with scalar\\n\",\n \t\t\t\treg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);\n \t\t\treturn -EINVAL;\n@@ -13004,7 +13085,7 @@ static int check_kfunc_args(struct bpf_verifier_env env, struct bpf_call_arg_me\n \t\t\t\tbreak;\n \t\t\t}\n \n-\t\t\tif (!__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t\tif (!btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\t\tenum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);\n \t\t\t\tconst char expected_type;\n \n@@ -13546,7 +13627,7 @@ static int check_special_kfunc(struct bpf_verifier_env env, struct bpf_call_arg\n \n \t\tstruct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);\n \t\tif (is_bpf_percpu_obj_new_kfunc(meta-\u003efunc_id)) {\n-\t\t\tif (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {\n+\t\t\tif (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {\n \t\t\t\tverbose(env, \"bpf_percpu_obj_new type ID argument must be of a struct of scalars\\n\");\n \t\t\t\treturn -EINVAL;\n \t\t\t}\n@@ -13914,10 +13995,25 @@ static int check_kfunc_call(struct bpf_verifier_env env, struct bpf_insn insn,\n \t}\n \n \tif (btf_type_is_scalar(t)) {\n-\t\tmark_reg_unknown(env, regs, BPF_REG_0);\n+\t\tmark_kfunc_ret_regs(env, regs, t-\u003esize);\n \t\tif (meta.btf == btf_vmlinux \u0026\u0026 (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] \|\|\n \t\t meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))\n \t\t\t__mark_reg_const_zero(env, \u0026regs[BPF_REG_0]);\n+\t} else if (btf_type_is_struct(t)) {\n+\t\t/\n+\t\t The returned struct comes back as raw register bits modeled\n+\t\t * as an unknown scalar, so it must contain only scalars:\n+\t\t * otherwise a pointer field would be laundered into a scalar\n+\t\t * and escape provenance and reference tracking.\n+\t\t /\n+\t\tif (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {\n+\t\t\tverbose(env,\n+\t\t\t\t\"kernel function %s returns %s %s that is not composed of scalars\\n\",\n+\t\t\t\tfunc_name, btf_type_str(t),\n+\t\t\t\tbtf_name_by_offset(desc_btf, t-\u003ename_off));\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\t\tmark_kfunc_ret_regs(env, regs, t-\u003esize);\n \t} else if (btf_type_is_ptr(t)) {\n \t\tptr_type = btf_type_skip_modifiers(desc_btf, t-\u003etype, \u0026ptr_type_id);\n \t\terr = check_special_kfunc(env, \u0026meta, regs, insn_aux, ptr_type, desc_btf);\n@@ -17453,37 +17549,54 @@ static int check_return_code(struct bpf_verifier_env env, int regno, const char\n \treturn 0;\n }\n \n-static int check_global_subprog_return_code(struct bpf_verifier_env env)\n+static int check_global_ret_scalar_reg(struct bpf_verifier_env env, u32 regno)\n {\n-\tstruct bpf_reg_state reg = reg_state(env, BPF_REG_0);\n-\tstruct bpf_func_state cur_frame = cur_func(env);\n+\tstruct bpf_reg_state reg;\n \tint err;\n \n-\tif (subprog_returns_void(env, cur_frame-\u003esubprogno))\n-\t\treturn 0;\n-\n-\terr = check_reg_arg(env, BPF_REG_0, SRC_OP);\n+\terr = check_reg_arg(env, regno, SRC_OP);\n \tif (err)\n \t\treturn err;\n \n \t/ Pointers to arena are safe to pass between subprograms. /\n-\tif (is_arena_reg(env, BPF_REG_0))\n+\tif (is_arena_reg(env, regno))\n \t\treturn 0;\n \n-\tif (is_pointer_value(env, BPF_REG_0)) {\n-\t\tverbose(env, \"R%d leaks addr as return value\\n\", BPF_REG_0);\n+\tif (is_pointer_value(env, regno)) {\n+\t\tverbose(env, \"R%d leaks addr as return value\\n\", regno);\n \t\treturn -EACCES;\n \t}\n \n+\treg = reg_state(env, regno);\n \tif (reg-\u003etype != SCALAR_VALUE) {\n-\t\tverbose(env, \"At subprogram exit the register R0 is not a scalar value (%s)\\n\",\n-\t\t\treg_type_str(env, reg-\u003etype));\n+\t\tverbose(env, \"At subprogram exit the register R%d is not a scalar value (%s)\\n\",\n+\t\t\tregno, reg_type_str(env, reg-\u003etype));\n \t\treturn -EINVAL;\n \t}\n \n \treturn 0;\n }\n \n+static int check_global_subprog_return_code(struct bpf_verifier_env env)\n+{\n+\tstruct bpf_func_state cur_frame = cur_func(env);\n+\tu32 subprog = cur_frame-\u003esubprogno;\n+\tu32 i, nregs;\n+\tint err;\n+\n+\tif (subprog_returns_void(env, subprog))\n+\t\treturn 0;\n+\n+\tnregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;\n+\tfor (i = 0; i \u003c nregs; i++) {\n+\t\terr = check_global_ret_scalar_reg(env, ret_regs[i]);\n+\t\tif (err)\n+\t\t\treturn err;\n+\t}\n+\n+\treturn 0;\n+}\n+\n / Bitmask with 1s for all caller saved registers /\n #define ALL_CALLER_SAVED_REGS ((1u \u003c\u003c CALLER_SAVED_REGS) - 1)\n \n@@ -21151,6 +21264,11 @@ int bpf_check(struct bpf_prog prog, union bpf_attr attr, bpfptr_t uattr,\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\n \n+\t/* must precede the first bpf_ret_reg_pair() user below /\n+\tret = bpf_compute_subprog_ret_regs(env);\n+\tif (ret \u003c 0)\n+\t\tgoto skip_full_check;\n+\n \tret = bpf_compute_live_registers(env);\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\ndiff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c\nnew file mode 100644\nindex 0000000000000..e0b94ed10f941\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c\n@@ -0,0 +1,11 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003ctest_progs.h\u003e\n+#include \"aggregate_ret_func.skel.h\"\n+#include \"aggregate_ret_kfunc.skel.h\"\n+\n+void test_aggregate_ret(void)\n+{\n+\tRUN_TESTS(aggregate_ret_func);\n+\tRUN_TESTS(aggregate_ret_kfunc);\n+}\ndiff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\nindex 2523c07a16c65..6262c81cdaa9f 100644\n--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\n+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\n@@ -441,6 +441,20 @@ static void test_func_replace_int_with_void(void)\n \t\t\t\t \" doesn't match type INT of global_func2()\");\n }\n \n+static void test_func_replace_ret_pair(void)\n+{\n+\tconst char msg = \"Return type of new_agg_ret_target_func() has size 8 \"\n+\t\t\t \"and of agg_ret_target_func() size 16\";\n+\n+\t/\n+\t An extension cannot replace a function whose return value comes back\n+\t * in the R0:R2 pair: only R0 is checked at the extension's exit, so it\n+\t * would leave R2 stale for the target's callers.\n+\t /\n+\ttest_obj_load_failure_common(\"freplace_ret_pair.bpf.o\",\n+\t\t\t\t \"./aggregate_ret_target.bpf.o\", msg);\n+}\n+\n static int find_prog_btf_id(const char name, __u32 attach_prog_fd)\n {\n \tstruct bpf_prog_info info = {};\n@@ -660,6 +674,8 @@ void serial_test_fexit_bpf2bpf(void)\n \t\ttest_func_replace_progmap();\n \tif (test__start_subtest(\"freplace_int_with_void\"))\n \t\ttest_func_replace_int_with_void();\n+\tif (test__start_subtest(\"freplace_ret_pair\"))\n+\t\ttest_func_replace_ret_pair();\n \tif (test__start_subtest(\"freplace_void\"))\n \t\ttest_func_replace_void();\n \tif (test__start_subtest(\"sleepable_fentry_to_xdp\"))\ndiff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c\nindex 64ac49ad67e63..f7f94ccebce27 100644\n--- a/tools/testing/selftests/bpf/prog_tests/verifier.c\n+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c\n@@ -5,6 +5,7 @@\n #include \"arena_kfunc.skel.h\"\n #include \"arena_kfunc_jit.skel.h\"\n #include \"cap_helpers.h\"\n+#include \"verifier_aggregate_ret.skel.h\"\n #include \"verifier_align.skel.h\"\n #include \"verifier_and.skel.h\"\n #include \"verifier_arena.skel.h\"\n@@ -170,6 +171,7 @@ void test_arena_kfunc(void) { RUN_TESTS(arena_kfunc); }\n \n void test_arena_kfunc_jit(void) { RUN_TESTS(arena_kfunc_jit); }\n \n+void test_verifier_aggregate_ret(void) { RUN_TESTS(verifier_aggregate_ret); }\n void test_verifier_align(void) { RUN(verifier_align); }\n void test_verifier_and(void) { RUN(verifier_and); }\n void test_verifier_arena(void) { RUN(verifier_arena); }\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c\nnew file mode 100644\nindex 0000000000000..6f66fc822cede\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c\n@@ -0,0 +1,237 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+typedef unsigned __int128 u128;\n+\n+__naked u128 global_agg_good(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0x1234;\"\t/ low 64 bits /\n+\t\"r2 = 0x5678;\"\t/ high 64 bits /\n+\t\"exit;\"\n+\t);\n+}\n+\n+__naked u128 global_agg_bad(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+__naked u128 global_agg_bad_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = r10;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_global_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_bad];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_bad)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__failure __msg(\"At subprogram exit the register R2 is not a scalar value\")\n+__naked int aggregate_ret_global_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_bad_ptr];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_bad_ptr)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_bad_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = r10;\"\t/ stack pointer placed in the second return register /\n+\t\"exit;\"\n+\t);\n+}\n+\n+/\n+ * R2 is a return register once the subprogram returns a pair, so a stack\n+ * pointer left in it is rejected at the callee's exit exactly as one in R0\n+ * is: the callee frame is gone by the time the caller could use it.\n+ /\n+SEC(\"tc\")\n+__load_if_JITed()\n+__failure __msg(\"cannot return stack pointer to the caller\")\n+__naked int aggregate_ret_static_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_bad_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_bad_ptr)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_no_r2(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_static_uninit_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_no_r2];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_no_r2)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_precise(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 4;\"\t/ second half; its value is made precise below /\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 2: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 12: (95) exit\")\n+__msg(\"mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4\")\n+__naked int aggregate_ret_static_precise(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_precise];\"\n+\t\"r6 = r2;\"\t\t/ derived from the aggregate's second half /\n+\t\"r6 \u0026= 7;\"\t\t/ keep it in [0, 7] to index the stack /\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/ ptr += scalar marks r6 (hence R2) precise /\n+\t\"r0 = 0;\"\n+\t\"(u8 )(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_precise)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 2: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9\")\n+__naked int aggregate_ret_global_precise(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_good];\"\n+\t\"r6 = r2;\"\t\t/ derived from the aggregate's second half /\n+\t\"r6 \u0026= 7;\"\t\t/ keep it in [0, 7] to index the stack /\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/ ptr += scalar marks r6 (hence R2) precise /\n+\t\"r0 = 0;\"\n+\t\"(u8 )(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_good)\n+\t: __clobber_all);\n+}\n+\n+#if defined(__clang_major__) \u0026\u0026 __clang_major__ \u003e= 23\n+\n+/ A by-value struct that smuggles a pointer, which must be rejected. /\n+struct with_ptr {\n+\tvoid p;\n+\t__u64 x;\n+};\n+\n+/* A by-value union that smuggles a pointer, which must be rejected too. /\n+union upair_with_ptr {\n+\tvoid p;\n+\t__u64 halves[2];\n+};\n+\n+__naked struct with_ptr global_ret_struct_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"Global function global_ret_struct_ptr() has unsupported return type\")\n+__naked int aggregate_ret_global_struct_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_ret_struct_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_ret_struct_ptr)\n+\t: __clobber_all);\n+}\n+\n+__naked union upair_with_ptr global_ret_union_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"Global function global_ret_union_ptr() has unsupported return type\")\n+__naked int aggregate_ret_global_union_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_ret_union_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_ret_union_ptr)\n+\t: __clobber_all);\n+}\n+\n+#endif\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c\nnew file mode 100644\nindex 0000000000000..d6b422ae9784b\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c\n@@ -0,0 +1,121 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+#include \"../test_kmods/bpf_testmod_kfunc.h\"\n+\n+/\n+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address\n+ * (rather than calling) avoids any dependence on the compiler lowering an\n+ * __int128 or struct return value, which the BPF backend only supports from\n+ * LLVM 23 on.\n+ /\n+void __kfunc_btf_root(void)\n+{\n+\tasm volatile (\"\"\n+\t:\n+\t: \"r\"(\u0026bpf_kfunc_call_test_i128),\n+\t \"r\"(\u0026bpf_kfunc_call_test_ret_fastcall),\n+\t \"r\"(\u0026bpf_kfunc_call_test_ret_ptr),\n+\t \"r\"(\u0026bpf_kfunc_call_test_ret_ii),\n+\t \"r\"(\u0026bpf_kfunc_call_test_ret_big));\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128\")\n+__naked int aggregate_ret_kfunc_precise(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 1;\"\n+\t\"r2 = 2;\"\n+\t\"call %[bpf_kfunc_call_test_i128];\"\n+\t\"r6 = r2;\"\t\t/ second return half /\n+\t\"r6 \u0026= 7;\"\t\t/ keep it in [0, 7] to index the stack /\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/ ptr += scalar marks r6 (hence R2) precise /\n+\t\"r0 = 0;\"\n+\t\"(u8 )(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_i128)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"kfunc bpf_kfunc_call_test_ret_fastcall with \u003e8-byte return is not supported with KF_FASTCALL\")\n+__naked int aggregate_ret_kfunc_fastcall_fail(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 1;\"\n+\t\"r2 = 2;\"\n+\t\"call %[bpf_kfunc_call_test_ret_fastcall];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_fastcall)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"is not composed of scalars\")\n+__naked int aggregate_ret_kfunc_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 0;\"\n+\t\"call %[bpf_kfunc_call_test_ret_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_ptr)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_kfunc_small_no_r2(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"call %[bpf_kfunc_call_test_ret_ii];\"\n+\t\"r0 = r2;\"\t/ R2 is not a return register for a \u003c=8 byte struct /\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_ii)\n+\t: __clobber_all);\n+}\n+\n+/\n+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by\n+ * btf_distill_func_proto() before the KF_FASTCALL and JIT-capability checks.\n+ /\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported\")\n+__naked int aggregate_ret_kfunc_too_big_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[bpf_kfunc_call_test_ret_big];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_big)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c\nnew file mode 100644\nindex 0000000000000..cffd8d7d3241a\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c\n@@ -0,0 +1,29 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+/ freplace target: a global subprogram returning 16 bytes in R0:R2. /\n+__naked unsigned __int128 agg_ret_target_func(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0x1234;\"\n+\t\"r2 = 0x5678;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__naked int agg_ret_target(void)\n+{\n+\tasm volatile (\n+\t\"call %[agg_ret_target_func];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(agg_ret_target_func)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c\nindex d055fc7b3b95d..338e67cd8856a 100644\n--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c\n+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c\n@@ -431,6 +431,34 @@ __naked void subprog1(void)\n \t\t::: __clobber_all);\n }\n \n+static __used __naked unsigned __int128 aux2(void)\n+{\n+\tasm volatile (\n+\t\t\"r0 = 1;\"\n+\t\t\"r2 = 2;\"\n+\t\t\"exit;\"\n+\t\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+/ A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). /\n+__load_if_JITed()\n+__log_level(2)\n+__msg(\"0: .12345.... (85) call pc+2\")\n+__msg(\"1: ..2....... (bf) r0 = r2\")\n+__msg(\"2: 0......... (95) exit\")\n+__msg(\"3: .......... (b7) r0 = 1\")\n+__msg(\"4: 0......... (b7) r2 = 2\")\n+__msg(\"5: 0.2....... (95) exit\")\n+__naked void subprog_ret_reg_pair(void)\n+{\n+\tasm volatile (\n+\t\t\"call aux2;\"\n+\t\t\"r0 = r2;\"\n+\t\t\"exit;\"\n+\t\t::: __clobber_all);\n+}\n+\n #if defined(__TARGET_ARCH_x86) \|\| defined(__TARGET_ARCH_arm64)\n \n SEC(\"socket\")\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c\nindex ac44d60e50666..9708efb93683b 100644\n--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c\n+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c\n@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)\n \n SEC(\"?tc\")\n __exception_cb(exception_cb_bad_ret_type1)\n-__failure __msg(\"Global function exception_cb_bad_ret_type1() return value not void or scalar.\")\n+__failure __msg(\"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\")\n int reject_exception_cb_type_1(struct __sk_buff ctx)\n {\n \tbpf_throw(0);\ndiff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c\nnew file mode 100644\nindex 0000000000000..12c15d293bd79\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c\n@@ -0,0 +1,12 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+\n+SEC(\"freplace/agg_ret_target_func\")\n+__u64 new_agg_ret_target_func(void)\n+{\n+\treturn 0;\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c\nnew file mode 100644\nindex 0000000000000..7851bade2b406\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c\n@@ -0,0 +1,178 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. /\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"../test_kmods/bpf_testmod_kfunc.h\"\n+#include \"bpf_misc.h\"\n+\n+#if defined(__clang_major__) \u0026\u0026 __clang_major__ \u003e= 23\n+\n+#define MIX_A\t0xdeadbeefcafef00dULL\n+#define MIX_B\t0x0123456789abcdefULL\n+\n+typedef unsigned __int128 u128;\n+\n+struct pair {\n+\t__u64 lo;\t/ R0 /\n+\t__u64 hi;\t/ R2 /\n+};\n+\n+union upair {\n+\t__u64 halves[2];\n+\tstruct {\n+\t\t__u64 lo;\t/ R0 /\n+\t\t__u64 hi;\t/ R2 /\n+\t} parts;\n+};\n+\n+static __noinline u128 make_i128(__u64 a, __u64 b)\n+{\n+\treturn ((u128)(a + b) \u003c\u003c 64) \| (a - b);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_int128_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tu128 v;\n+\n+\tv = make_i128(a, b);\n+\tif ((__u64)(v \u003e\u003e 64) != a + b)\n+\t\treturn 1;\n+\tif ((__u64)v != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+static __noinline struct pair make_pair(__u64 a, __u64 b)\n+{\n+\tstruct pair p = { .lo = a + b, .hi = a - b };\n+\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_struct_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct pair p;\n+\n+\tp = make_pair(a, b);\n+\tif (p.lo != a + b)\n+\t\treturn 1;\n+\tif (p.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+__noinline struct pair make_pair_global(__u64 a, __u64 b)\n+{\n+\tstruct pair p = { .lo = a + b, .hi = a - b };\n+\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_global_struct_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct pair p;\n+\n+\tp = make_pair_global(a, b);\n+\tif (p.lo != a + b)\n+\t\treturn 1;\n+\tif (p.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+static __noinline union upair make_upair(__u64 a, __u64 b)\n+{\n+\tunion upair p;\n+\n+\tp.halves[0] = a + b;\n+\tp.halves[1] = a - b;\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_union_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tunion upair p;\n+\n+\tp = make_upair(a, b);\n+\tif (p.parts.lo != a + b)\n+\t\treturn 1;\n+\tif (p.parts.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tu128 v;\n+\n+\tv = bpf_kfunc_call_test_i128(a, b);\n+\tif ((__u64)(v \u003e\u003e 64) != a + b)\n+\t\treturn 1;\n+\tif ((__u64)v != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct prog_test_ret_pair p;\n+\n+\tp = bpf_kfunc_call_test_ret_pair(a, b);\n+\tif (p.lo != a + b)\n+\t\treturn 1;\n+\tif (p.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+SEC(\"socket\")\n+__description(\"verifier_aggregate_ret: needs LLVM 23, dummy test\")\n+__success\n+int dummy_test(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\nindex 9366a3c578f13..850cf4f830c48 100644\n--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\n+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\n@@ -954,6 +954,53 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)\n \treturn 0;\n }\n \n+#if defined(__x86_64__) \|\| defined(__aarch64__)\n+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)\n+{\n+\treturn (__int128)(((unsigned __int128)(a + b) \u003c\u003c 64) \| (a - b));\n+}\n+\n+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)\n+{\n+\tstruct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)\n+{\n+\tstruct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)\n+{\n+\tstruct prog_test_ret_ptr r = { .p = NULL, .tag = tag };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)\n+{\n+\tstruct prog_test_ret_ii r = { .a = a, .b = b };\n+\n+\treturn r;\n+}\n+#endif / __x86_64__ \|\| __aarch64__ /\n+\n+/\n+ * Takes no argument on purpose: with no arguments there is nothing for the sret\n+ * pointer to displace, so this needs no architecture guard even though it\n+ * returns 24 bytes.\n+ /\n+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)\n+{\n+\tstruct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };\n+\n+\treturn r;\n+}\n+\n __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,\n \t\t\t\t\t u64 e, u64 f, u64 g, u64 h,\n \t\t\t\t\t u64 i, u64 j)\n@@ -1487,6 +1534,14 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test3)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test4)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test5)\n+#if defined(__x86_64__) \|\| defined(__aarch64__)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)\n+#endif\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)\ndiff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\nindex 7d81070eefe7f..65e693ada7364 100644\n--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\n+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\n@@ -55,6 +55,27 @@ struct prog_test_big_arg {\n \t__u64 b;\n };\n \n+struct prog_test_ret_pair {\t/ 16 bytes: R0:R2 /\n+\t__u64 lo;\n+\t__u64 hi;\n+};\n+\n+struct prog_test_ret_ii {\t/ 8 bytes: R0 only /\n+\tint a;\n+\tint b;\n+};\n+\n+struct prog_test_ret_ptr {\t/ 16 bytes: contains a pointer /\n+\tvoid p;\n+\t__u64 tag;\n+};\n+\n+struct prog_test_ret_big {\t/* 24 bytes: too large for R0:R2 /\n+\t__u64 a;\n+\t__u64 b;\n+\t__u64 c;\n+};\n+\n struct prog_test_fail1 {\n \tvoid p;\n \tint x;\n@@ -131,6 +152,14 @@ int bpf_kfunc_call_test2(struct sock sk, __u32 a, __u32 b) __ksym;\n struct sock bpf_kfunc_call_test3(struct sock *sk) __ksym;\n long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;\n int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;\n+#ifdef __SIZEOF_INT128__\n+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;\n+#endif\n+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;\n+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;\n+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;\n+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;\n+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym;\n __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,\n \t\t\t __u64 e, __u64 f, __u64 g, __u64 h,\n \t\t\t __u64 i, __u64 j) __ksym;\n" }
2/1	2026/08/19 06:42	agent	patch-evaluator	0m Model: best-expensive Args: null Results: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT" ], "FocusSymbols": [ "btf_type_is_scalar_struct", "btf_distill_func_proto", "btf_prepare_func_args", "bpf_compute_live_registers", "bpf_jit_supports_kfunc_ret_reg_pair" ], "Reasoning": "The patch series adds support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or scalar-only structs/unions) in the register pair R0:R2. This involves modifications to the BPF verifier to track the return values in both registers, changes to the liveness and backtracking logic, and updates to the JIT compilers (x86, arm64, riscv) to support this new calling convention. These are functional changes to reachable core kernel logic (BPF subsystem) that should be fuzzed to ensure the verifier correctly tracks the register states and prevents any leaks or invalid memory accesses.", "WorthFuzzing": true } Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit cc236f68937f8479bce51601bd02ac7647affe1d Author: syz-cluster <triage@syzkaller.com> Date: Wed Aug 19 06:41:58 2026 +0000 syz-cluster: applied patch under review diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 85f73e0bbd0ff..89dea6b0b0241 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -575,6 +575,71 @@ is also covered by this recovery. A kfunc handed an arena pointer may therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking against the arena. Larger accesses must verify the range explicitly. +2.9 kfunc Return Values +----------------------- + +A kfunc may return a scalar, a pointer, or a small struct or union by +value. A scalar or pointer of up to 8 bytes is returned in R0, as usual. + +A struct or union returned by value must be composed only of scalars +(recursively), where a scalar is an integer or an enum; arrays of scalars are +allowed as members. Its bytes are handed back to the program as the raw +contents of R0 (and R2), so a pointer field would be laundered into a scalar +and escape the verifier's pointer provenance and reference tracking. A struct +or union with a pointer member is therefore rejected at load time, and so is +one with a floating-point member, which the ABI may not return in R0:R2 at +all. + +A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a +scalar-only struct or union, or an ``__int128``. Such a value is returned +in the register pair R0:R2, matching the convention LLVM uses for the BPF +target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or +union of 8 bytes or less is returned in R0 alone. + +:: + + struct bpf_pair { __u64 a, b; }; /* 16 bytes / + + __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void) + { + struct bpf_pair p = { .a = 1, .b = 2 }; + + return p; / p.a in R0, p.b in R2 / + } + +Returning a value in the R0:R2 pair requires the JIT to place the second +half of the return value into R2, which not every architecture supports +right now. A kfunc with a return value larger than 8 bytes is therefore +rejected at load time on a JIT that does not advertise this capability (see +``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run +by the interpreter. A return value larger than 16 bytes is not supported. + +The same R0:R2 convention applies to a BPF subprogram, global or static, that +returns an ``__int128`` or a struct or union larger than 8 bytes. It is only +used when the program is JITed, since the interpreter propagates only R0 out of +a subprogram: without a JIT the return value stays in R0 alone, and a caller +reading R2 is rejected for reading an uninitialized register. A global +subprogram is verified in isolation, so its by-value struct or union return is +restricted to scalars just like a kfunc's; a static subprogram is verified +inline and has no such restriction. The main program is not covered: its return +value is the program's exit code, read out of R0 alone, so a declared upper +half is never looked at. + +A global subprogram must leave a scalar in every* register of the pair, so +both halves of the returned value have to be assigned. Leaving the upper half +uninitialized is not merely untidy: the compiler is then free to leave R2 +holding whatever it happened to hold, which for a subprogram taking a pointer +argument is typically that pointer. Handing the caller an unknown scalar built +from a pointer is a leak, so the verifier rejects it with:: + + At subprogram exit the register R2 is not a scalar value (...) + +Initialize the whole return value, for example ``struct pair p = {};``, to +avoid this. A static subprogram is exempt from the scalar-only rule: it is +verified inline, so an unassigned R2 is simply passed back to the caller as +uninitialized and only a caller that reads it fails. A stack pointer left in +R2 is still rejected there, just as one in R0 is. + .. _BPF_kfunc_lifecycle_expectations: 3. kfunc lifecycle expectations diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index c18e005a41dbe..3aa3ea0bc30bb 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2388,6 +2388,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_stack_args(void) { return true; diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 74efe4b138d2a..47c7bf431ba85 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -2121,6 +2121,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_ptr_xchg(void) { return true; diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 1a9fb530adc3c..48429fae06410 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip, * arena NULL is offset 0. Return the number of emitted bytes. / static int emit_kfunc_arena_args(struct bpf_prog bpf_prog, - const struct bpf_insn insn, u8 pprog) + const struct btf_func_model fm, u8 *pprog) { - const struct btf_func_model fm; u8 prog = pprog; u8 start = prog; int i; - fm = bpf_jit_find_kfunc_model(bpf_prog, insn); - if (!fm) - return -EINVAL; - for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) { u8 flags = fm->arg_flags[i]; u32 reg = BPF_REG_1 + i; @@ -2644,6 +2639,8 @@ st: insn_off = insn->off; / call / case BPF_JMP \| BPF_CALL: { + const struct btf_func_model fm = NULL; + func = (u8 ) __bpf_call_base + imm32; if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(stack_depth); @@ -2652,7 +2649,10 @@ st: insn_off = insn->off; if (!imm32) return -EINVAL; if (src_reg == BPF_PSEUDO_KFUNC_CALL) { - err = emit_kfunc_arena_args(bpf_prog, insn, &prog); + fm = bpf_jit_find_kfunc_model(bpf_prog, insn); + if (!fm) + return -EINVAL; + err = emit_kfunc_arena_args(bpf_prog, fm, &prog); if (err < 0) return err; ip += err; @@ -2666,6 +2666,14 @@ st: insn_off = insn->off; return -EINVAL; if (priv_frame_ptr) pop_r9(&prog); + / + * A kfunc returning more than 8 bytes hands the second + * half back in RDX (the native ABI's second return reg), + * but BPF expects it in R0:R2. BPF R0 is RAX (no move + * needed), while BPF R2 is RSI, so copy RDX into RSI. + / + if (fm && fm->ret_size > 8) + emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3); break; } @@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_stack_args(void) { return true; diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 5fad59fdab0d0..004b06785521c 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -819,6 +819,8 @@ struct bpf_subprog_info { bool is_async_cb: 1; bool is_exception_cb: 1; bool args_cached: 1; + / true if the return value is passed in the R0:R2 register pair / + bool ret_reg_pair: 1; / true if bpf_fastcall stack region is used by functions that can't be inlined / bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; @@ -1055,6 +1057,11 @@ static inline struct bpf_subprog_info subprog_info(struct bpf_verifier_env env return &env->subprog_info[subprog]; } +static inline bool bpf_ret_reg_pair(struct bpf_verifier_env env, int subprog) +{ + return subprog_info(env, subprog)->ret_reg_pair; +} + struct bpf_call_summary { u8 num_params; bool is_void; @@ -1481,6 +1488,8 @@ int bpf_jmp_offset(struct bpf_insn insn); struct bpf_iarray bpf_insn_successors(struct bpf_verifier_env env, u32 idx); void bpf_fmt_stack_mask(char buf, ssize_t buf_sz, u64 stack_mask); bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog); +bool btf_type_is_scalar_struct(struct bpf_verifier_env env, const struct btf btf, + const struct btf_type t, int rec); int bpf_find_subprog(struct bpf_verifier_env env, int off); bool bpf_is_throw_kfunc(struct bpf_insn insn); diff --git a/include/linux/filter.h b/include/linux/filter.h index 4a9bc6a848f2e..6e746b0a09306 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_percpu_insn(void); bool bpf_jit_supports_kfunc_call(void); +bool bpf_jit_supports_kfunc_ret_reg_pair(void); bool bpf_jit_supports_stack_args(void); bool bpf_jit_supports_arena_args(void); bool bpf_jit_supports_far_kfunc_call(void); diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index a2b18a9f1694c..653db80bcc47b 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -423,6 +423,10 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, / verifier_bug_if(idx + 1 != subseq_idx, env, "extra insn from subprog"); + /* global subprog always sets R0 / + bt_clear_reg(bt, BPF_REG_0); + / and if it does not set R2, main pass would catch it / + bt_clear_reg(bt, BPF_REG_2); / r1-r5 are invalidated after subprog call, * so for global func call it shouldn't be set * anymore @@ -432,8 +436,6 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, bt_reg_mask(bt)); return -EFAULT; } - / global subprog always sets R0 / - bt_clear_reg(bt, BPF_REG_0); return 0; } else { / static subprog call instruction, which @@ -506,6 +508,8 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, return -ENOTSUPP; / regular helper call sets R0 / bt_clear_reg(bt, BPF_REG_0); + / kfunc might also set R2 / + bt_clear_reg(bt, BPF_REG_2); if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { / if backtracking was looking for registers R1-R5 * they should have been found already. @@ -520,7 +524,25 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, return -EFAULT; } } else if (opcode == BPF_EXIT) { - bool r0_precise; + bool from_subprog_call, r0_precise, r2_precise; + + / BPF_EXIT in subprog or callback always returns + * right after the call instruction, so by checking + * whether the instruction at subseq_idx-1 is subprog + * call or not we can distinguish actual exit from + * subprog from exit from callback. In the former + * case, we need to propagate the precision of the + * return registers, if necessary. In the latter we + * never do that. + / + from_subprog_call = subseq_idx - 1 >= 0 && + bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]); + + / Sample the return registers before the callback + * handling below clears R1-R5. + / + r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0); + r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2); / Backtracking to a nested function call, 'idx' is a part of * the inner frame 'subseq_idx' is a part of the outer frame. @@ -533,30 +555,22 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx)) for (i = BPF_REG_1; i <= BPF_REG_5; i++) bt_clear_reg(bt, i); + + bt_clear_reg(bt, BPF_REG_0); + bt_clear_reg(bt, BPF_REG_2); if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { verifier_bug(env, "backtracking exit unexpected regs %x", bt_reg_mask(bt)); return -EFAULT; } - / BPF_EXIT in subprog or callback always returns - * right after the call instruction, so by checking - * whether the instruction at subseq_idx-1 is subprog - * call or not we can distinguish actual exit from - * subprog from exit from callback. In the former - * case, we need to propagate r0 precision, if - * necessary. In the former we never do that. - / - r0_precise = subseq_idx - 1 >= 0 && - bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) && - bt_is_reg_set(bt, BPF_REG_0); - - bt_clear_reg(bt, BPF_REG_0); if (bt_subprog_enter(bt)) return -EFAULT; if (r0_precise) bt_set_reg(bt, BPF_REG_0); + if (r2_precise) + bt_set_reg(bt, BPF_REG_2); / r6-r9 and stack slots will stay set in caller frame * bitmasks until we return back from callee(s) / diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 6967d48bba49d..58b81c6f5288c 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7591,7 +7591,7 @@ int btf_distill_func_proto(struct bpf_verifier_log log, return -EINVAL; } ret = __get_type_size(btf, func->type, &t); - if (ret < 0 \|\| btf_type_is_struct(t)) { + if (ret < 0 \|\| ret > 16) { bpf_log(log, "The function %s return type %s is unsupported.\n", tname, btf_type_str(t)); @@ -7684,6 +7684,12 @@ static int btf_check_func_type_match(struct bpf_verifier_log log, btf_type_str(t2), fn2); return -EINVAL; } + if (btf_type_has_size(t1) && (t1->size > 8 \|\| t2->size > 8)) { + bpf_log(log, + "Return type of %s() has size %u and of %s() size %u, and a size above 8 bytes cannot be replaced\n", + fn1, t1->size, fn2, t2->size); + return -EINVAL; + } for (i = 0; i < nargs1; i++) { t1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL); @@ -7964,7 +7970,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env env, /* Check whether the type is a valid return type. / static int btf_validate_return_type(struct bpf_verifier_env env, struct btf btf, - const struct btf_type t, int subprog) + const struct btf_type t, int subprog, bool is_global) { u32 tags = 0; int err; @@ -7987,6 +7993,19 @@ static int btf_validate_return_type(struct bpf_verifier_env env, struct btf bt if (btf_type_is_void(t) \|\| btf_type_is_int(t) \|\| btf_is_any_enum(t)) return 0; + if (btf_type_is_struct(t) && t->size <= 16) { + / + * A global function's caller models the return as an opaque + * scalar pair, so it may only return scalars by value. A local + * function is verified inline, so a pointer field stays tracked + * and needs no such restriction. + / + bool local_func = subprog && !is_global; + + if (local_func \|\| btf_type_is_scalar_struct(env, btf, t, 0)) + return 0; + } + return -EOPNOTSUPP; } @@ -8074,12 +8093,12 @@ int btf_prepare_func_args(struct bpf_verifier_env env, int subprog) return -EINVAL; } - err = btf_validate_return_type(env, btf, t, subprog); + err = btf_validate_return_type(env, btf, t, subprog, is_global); if (err) { if (is_global) { bpf_log(log, - "Global function %s() return value not void or scalar. " - "Only those are supported.\n", + "Global function %s() has unsupported return type. " + "Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n", tname); } return err; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index d55e737ed75ad..5db77d7915dfc 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void) return false; } +bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return false; +} + bool __weak bpf_jit_supports_stack_args(void) { return false; diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index 74fc4b3f80d6e..71f998c6eb888 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -2060,7 +2060,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); } /* Compute info->{use,def} fields for the instruction / static void compute_insn_live_regs(struct bpf_verifier_env env, struct bpf_insn insn, - struct insn_live_regs info) + struct insn_live_regs info, + bool ret_reg_pair) { struct bpf_call_summary cs; const u8 class = BPF_CLASS(insn->code); @@ -2072,6 +2073,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env, const u32 src32 = mask_lo(src); const u32 dst32 = mask_lo(dst); const u32 r0 = reg64_mask(0); + const u32 r2 = reg64_mask(BPF_REG_2); u32 def = 0; u32 use = U32_MAX; @@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env, break; case BPF_EXIT: def = 0; - use = r0; + use = ret_reg_pair ? (r0 \| r2) : r0; break; case BPF_CALL: def = ALL_CALLER_SAVED_REGS; @@ -2228,8 +2230,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env env) struct insn_live_regs state; int insn_cnt = env->prog->len; u64 pos, insn_pos; - int err = 0, i, j; - bool changed; + int err = 0, i, j, subprog, start, end; + bool changed, ret_reg_pair; / Use the following algorithm: * - define the following: @@ -2256,8 +2258,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env env) goto out; } - for (i = 0; i < insn_cnt; ++i) - compute_insn_live_regs(env, &insns[i], &state[i]); + for (subprog = 0; subprog < env->subprog_cnt; subprog++) { + start = env->subprog_info[subprog].start; + end = env->subprog_info[subprog + 1].start; + ret_reg_pair = bpf_ret_reg_pair(env, subprog); + + for (i = start; i < end; ++i) + compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair); + } / Forward pass: resolve stack access through FP-derived pointers / err = bpf_compute_subprog_arg_access(env); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e421ea2b80c3a..7a939dfa3797b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -385,27 +385,69 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog) return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL; } -static bool subprog_returns_void(struct bpf_verifier_env env, int subprog) +static const struct btf_type subprog_ret_type(struct bpf_verifier_env env, int subprog) { - const struct btf_type type, func, func_proto; + const struct btf_type func, func_proto; const struct btf btf = env->prog->aux->btf; u32 btf_id; + if (!btf \|\| !env->prog->aux->func_info) + return NULL; + btf_id = env->prog->aux->func_info[subprog].type_id; + / Both already validated by prepare_btf_func() at prog load. / func = btf_type_by_id(btf, btf_id); - if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id)) - return false; - func_proto = btf_type_by_id(btf, func->type); - if (!func_proto) - return false; - type = btf_type_skip_modifiers(btf, func_proto->type, NULL); - if (!type) - return false; + return btf_type_skip_modifiers(btf, func_proto->type, NULL); +} - return btf_type_is_void(type); +static bool subprog_returns_void(struct bpf_verifier_env env, int subprog) +{ + const struct btf_type type = subprog_ret_type(env, subprog); + + return type && btf_type_is_void(type); +} + +static u32 ret_regs_cnt(u32 size) +{ + return size > 8 && size <= 16 ? 2 : 1; +} + +/ Registers holding a function return value, in order. See ret_regs_cnt(). / +static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 }; + +static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env env) +{ + const struct btf btf = env->prog->aux->btf; + const struct btf_type type; + int subprog; + u32 size; + + if (!env->prog->jit_requested \|\| bpf_prog_is_offloaded(env->prog->aux)) + return 0; + + /* + * Skip the main program: its return value is the program's exit code, + * read out of R0, so it never uses the register pair. An extension does + * have a real prototype for subprog 0, but bpf_check_attach_target() + * refuses to replace a function returning more than 8 bytes. + / + for (subprog = 1; subprog < env->subprog_cnt; subprog++) { + type = subprog_ret_type(env, subprog); + if (!type \|\| btf_type_is_void(type)) + continue; + if (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, &size)), env, + "cannot size return type of subprog %d", subprog)) + return -EFAULT; + if (ret_regs_cnt(size) > 1) { + subprog_info(env, subprog)->ret_reg_pair = true; + env->prog->jit_required = 1; + } + } + + return 0; } const char bpf_subprog_name(const struct bpf_verifier_env env, int subprog) @@ -2877,6 +2919,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env env, u32 func_id, u16 offset) err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model); if (err) return err; + if (func_model.ret_size > 8) { + if (kfunc.flags && (kfunc.flags & KF_FASTCALL)) { + verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n", + kfunc.name); + return -EOPNOTSUPP; + } + if (!bpf_jit_supports_kfunc_ret_reg_pair()) { + verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n", + kfunc.name); + return -EOPNOTSUPP; + } + } memset(&meta, 0, sizeof(meta)); meta.btf = kfunc.btf; @@ -9853,6 +9907,7 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn, u16 callee_incoming, stack_arg_cnt; struct bpf_func_state caller; int err, subprog, target_insn; + u32 i, nregs; target_insn = insn_idx + insn->imm + 1; subprog = bpf_find_subprog(env, target_insn); @@ -9908,9 +9963,14 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn, clear_caller_saved_regs(env, caller->regs); invalidate_outgoing_stack_args(env, cur_func(env)); - / All non-void global functions return a 64-bit SCALAR_VALUE. / + / + * A non-void global function returns a 64-bit SCALAR_VALUE in + * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair. + / if (!returns_void) { - mark_reg_unknown(env, caller->regs, BPF_REG_0); + nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1; + for (i = 0; i < nregs; i++) + mark_reg_unknown(env, caller->regs, ret_regs[i]); bpf_diag_mod_end(env); } @@ -10270,11 +10330,15 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx) struct bpf_func_state caller, callee; struct bpf_reg_state r0; bool in_callback_fn; + u32 i, nregs; int err; callee = state->frame[state->curframe]; r0 = &callee->regs[BPF_REG_0]; - if (r0->type == PTR_TO_STACK) { + nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1; + for (i = 0; i < nregs; i++) { + if (callee->regs[ret_regs[i]].type != PTR_TO_STACK) + continue; /* technically it's ok to return caller's stack pointer * (or caller's caller's pointer) back to the caller, * since these pointers are valid. Only current stack @@ -10309,9 +10373,13 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx) return -EFAULT; } } else { - /* return to the caller whatever r0 had in the callee / + / + * return to the caller whatever the callee had in the + * return register(s) + / bpf_diag_mod_begin(env, &caller->regs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE); - caller->regs[BPF_REG_0] = r0; + for (i = 0; i < nregs; i++) + caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]]; bpf_diag_mod_end(env); } @@ -11246,6 +11314,19 @@ static int check_helper_call(struct bpf_verifier_env env, struct bpf_insn insn return 0; } +/* + * Mark the register(s) holding a @size byte kfunc return value as unknown + * scalars. Both halves of a register pair are treated the same way. + / +static void mark_kfunc_ret_regs(struct bpf_verifier_env env, + struct bpf_reg_state regs, u32 size) +{ + u32 i, nregs = ret_regs_cnt(size); + + for (i = 0; i < nregs; i++) + mark_reg_unknown(env, regs, ret_regs[i]); +} + static bool is_kfunc_acquire(struct bpf_call_arg_meta meta) { return meta->kfunc_flags & KF_ACQUIRE; @@ -11513,9 +11594,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta meta, u32 arg_ } / Returns true if struct is composed of scalars, 4 levels of nesting allowed / -static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env, - const struct btf btf, - const struct btf_type t, int rec) +bool btf_type_is_scalar_struct(struct bpf_verifier_env env, + const struct btf btf, + const struct btf_type t, int rec) { const struct btf_type member_type; const struct btf_member member; @@ -11533,7 +11614,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env, verbose(env, "max struct nesting depth exceeded\n"); return false; } - if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1)) + if (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1)) return false; continue; } @@ -11932,7 +12013,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta, (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) \|\| is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) { if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) && - !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n", reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); return -EINVAL; @@ -11948,7 +12029,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta, * scalars. The access size is derived from the pointed-to BTF type. / if (!btf_type_is_scalar(ref_t) && - !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n", reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); return -EINVAL; @@ -13004,7 +13085,7 @@ static int check_kfunc_args(struct bpf_verifier_env env, struct bpf_call_arg_me break; } - if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + if (!btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id); const char expected_type; @@ -13546,7 +13627,7 @@ static int check_special_kfunc(struct bpf_verifier_env env, struct bpf_call_arg struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id); if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) { - if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) { + if (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) { verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n"); return -EINVAL; } @@ -13914,10 +13995,25 @@ static int check_kfunc_call(struct bpf_verifier_env env, struct bpf_insn insn, } if (btf_type_is_scalar(t)) { - mark_reg_unknown(env, regs, BPF_REG_0); + mark_kfunc_ret_regs(env, regs, t->size); if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] \|\| meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave])) __mark_reg_const_zero(env, &regs[BPF_REG_0]); + } else if (btf_type_is_struct(t)) { + /* + * The returned struct comes back as raw register bits modeled + * as an unknown scalar, so it must contain only scalars: + * otherwise a pointer field would be laundered into a scalar + * and escape provenance and reference tracking. + / + if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) { + verbose(env, + "kernel function %s returns %s %s that is not composed of scalars\n", + func_name, btf_type_str(t), + btf_name_by_offset(desc_btf, t->name_off)); + return -EINVAL; + } + mark_kfunc_ret_regs(env, regs, t->size); } else if (btf_type_is_ptr(t)) { ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id); err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf); @@ -17453,37 +17549,54 @@ static int check_return_code(struct bpf_verifier_env env, int regno, const char return 0; } -static int check_global_subprog_return_code(struct bpf_verifier_env env) +static int check_global_ret_scalar_reg(struct bpf_verifier_env env, u32 regno) { - struct bpf_reg_state reg = reg_state(env, BPF_REG_0); - struct bpf_func_state cur_frame = cur_func(env); + struct bpf_reg_state reg; int err; - if (subprog_returns_void(env, cur_frame->subprogno)) - return 0; - - err = check_reg_arg(env, BPF_REG_0, SRC_OP); + err = check_reg_arg(env, regno, SRC_OP); if (err) return err; / Pointers to arena are safe to pass between subprograms. / - if (is_arena_reg(env, BPF_REG_0)) + if (is_arena_reg(env, regno)) return 0; - if (is_pointer_value(env, BPF_REG_0)) { - verbose(env, "R%d leaks addr as return value\n", BPF_REG_0); + if (is_pointer_value(env, regno)) { + verbose(env, "R%d leaks addr as return value\n", regno); return -EACCES; } + reg = reg_state(env, regno); if (reg->type != SCALAR_VALUE) { - verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n", - reg_type_str(env, reg->type)); + verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n", + regno, reg_type_str(env, reg->type)); return -EINVAL; } return 0; } +static int check_global_subprog_return_code(struct bpf_verifier_env env) +{ + struct bpf_func_state cur_frame = cur_func(env); + u32 subprog = cur_frame->subprogno; + u32 i, nregs; + int err; + + if (subprog_returns_void(env, subprog)) + return 0; + + nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1; + for (i = 0; i < nregs; i++) { + err = check_global_ret_scalar_reg(env, ret_regs[i]); + if (err) + return err; + } + + return 0; +} + / Bitmask with 1s for all caller saved registers / #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1) @@ -21151,6 +21264,11 @@ int bpf_check(struct bpf_prog prog, union bpf_attr attr, bpfptr_t uattr, if (ret < 0) goto skip_full_check; + /* must precede the first bpf_ret_reg_pair() user below / + ret = bpf_compute_subprog_ret_regs(env); + if (ret < 0) + goto skip_full_check; + ret = bpf_compute_live_registers(env); if (ret < 0) goto skip_full_check; diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file mode 100644 index 0000000000000..e0b94ed10f941 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c @@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <test_progs.h> +#include "aggregate_ret_func.skel.h" +#include "aggregate_ret_kfunc.skel.h" + +void test_aggregate_ret(void) +{ + RUN_TESTS(aggregate_ret_func); + RUN_TESTS(aggregate_ret_kfunc); +} diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c index 2523c07a16c65..6262c81cdaa9f 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c @@ -441,6 +441,20 @@ static void test_func_replace_int_with_void(void) " doesn't match type INT of global_func2()"); } +static void test_func_replace_ret_pair(void) +{ + const char msg = "Return type of new_agg_ret_target_func() has size 8 " + "and of agg_ret_target_func() size 16"; + + /* + * An extension cannot replace a function whose return value comes back + * in the R0:R2 pair: only R0 is checked at the extension's exit, so it + * would leave R2 stale for the target's callers. + / + test_obj_load_failure_common("freplace_ret_pair.bpf.o", + "./aggregate_ret_target.bpf.o", msg); +} + static int find_prog_btf_id(const char name, __u32 attach_prog_fd) { struct bpf_prog_info info = {}; @@ -660,6 +674,8 @@ void serial_test_fexit_bpf2bpf(void) test_func_replace_progmap(); if (test__start_subtest("freplace_int_with_void")) test_func_replace_int_with_void(); + if (test__start_subtest("freplace_ret_pair")) + test_func_replace_ret_pair(); if (test__start_subtest("freplace_void")) test_func_replace_void(); if (test__start_subtest("sleepable_fentry_to_xdp")) diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 64ac49ad67e63..f7f94ccebce27 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -5,6 +5,7 @@ #include "arena_kfunc.skel.h" #include "arena_kfunc_jit.skel.h" #include "cap_helpers.h" +#include "verifier_aggregate_ret.skel.h" #include "verifier_align.skel.h" #include "verifier_and.skel.h" #include "verifier_arena.skel.h" @@ -170,6 +171,7 @@ void test_arena_kfunc(void) { RUN_TESTS(arena_kfunc); } void test_arena_kfunc_jit(void) { RUN_TESTS(arena_kfunc_jit); } +void test_verifier_aggregate_ret(void) { RUN_TESTS(verifier_aggregate_ret); } void test_verifier_align(void) { RUN(verifier_align); } void test_verifier_and(void) { RUN(verifier_and); } void test_verifier_arena(void) { RUN(verifier_arena); } diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file mode 100644 index 0000000000000..6f66fc822cede --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c @@ -0,0 +1,237 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +typedef unsigned __int128 u128; + +__naked u128 global_agg_good(void) +{ + asm volatile ( + "r0 = 0x1234;" / low 64 bits / + "r2 = 0x5678;" / high 64 bits / + "exit;" + ); +} + +__naked u128 global_agg_bad(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +__naked u128 global_agg_bad_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = r10;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_global_fail(void) +{ + asm volatile ( + "call %[global_agg_bad];" + "r0 = r2;" + "exit;" + : + : __imm(global_agg_bad) + : __clobber_all); +} + +SEC("tc") +__load_if_JITed() +__failure __msg("At subprogram exit the register R2 is not a scalar value") +__naked int aggregate_ret_global_ptr_fail(void) +{ + asm volatile ( + "call %[global_agg_bad_ptr];" + "r0 = r2;" + "exit;" + : + : __imm(global_agg_bad_ptr) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_bad_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = r10;" / stack pointer placed in the second return register / + "exit;" + ); +} + +/ + * R2 is a return register once the subprogram returns a pair, so a stack + * pointer left in it is rejected at the callee's exit exactly as one in R0 + * is: the callee frame is gone by the time the caller could use it. + / +SEC("tc") +__load_if_JITed() +__failure __msg("cannot return stack pointer to the caller") +__naked int aggregate_ret_static_ptr_fail(void) +{ + asm volatile ( + "call %[static_agg_bad_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(static_agg_bad_ptr) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_no_r2(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_static_uninit_fail(void) +{ + asm volatile ( + "call %[static_agg_no_r2];" + "r0 = r2;" + "exit;" + : + : __imm(static_agg_no_r2) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_precise(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 4;" / second half; its value is made precise below / + "exit;" + ); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit") +__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4") +__naked int aggregate_ret_static_precise(void) +{ + asm volatile ( + "call %[static_agg_precise];" + "r6 = r2;" / derived from the aggregate's second half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(static_agg_precise) + : __clobber_all); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9") +__naked int aggregate_ret_global_precise(void) +{ + asm volatile ( + "call %[global_agg_good];" + "r6 = r2;" / derived from the aggregate's second half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(global_agg_good) + : __clobber_all); +} + +#if defined(__clang_major__) && __clang_major__ >= 23 + +/ A by-value struct that smuggles a pointer, which must be rejected. / +struct with_ptr { + void p; + __u64 x; +}; + +/* A by-value union that smuggles a pointer, which must be rejected too. / +union upair_with_ptr { + void p; + __u64 halves[2]; +}; + +__naked struct with_ptr global_ret_struct_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("Global function global_ret_struct_ptr() has unsupported return type") +__naked int aggregate_ret_global_struct_ptr_fail(void) +{ + asm volatile ( + "call %[global_ret_struct_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(global_ret_struct_ptr) + : __clobber_all); +} + +__naked union upair_with_ptr global_ret_union_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("Global function global_ret_union_ptr() has unsupported return type") +__naked int aggregate_ret_global_union_ptr_fail(void) +{ + asm volatile ( + "call %[global_ret_union_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(global_ret_union_ptr) + : __clobber_all); +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c new file mode 100644 index 0000000000000..d6b422ae9784b --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c @@ -0,0 +1,121 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" +#include "../test_kmods/bpf_testmod_kfunc.h" + +/ + * Reference kfunc addresses to force those BTF to be emitted. Taking the address + * (rather than calling) avoids any dependence on the compiler lowering an + * __int128 or struct return value, which the BPF backend only supports from + * LLVM 23 on. + / +void __kfunc_btf_root(void) +{ + asm volatile ("" + : + : "r"(&bpf_kfunc_call_test_i128), + "r"(&bpf_kfunc_call_test_ret_fastcall), + "r"(&bpf_kfunc_call_test_ret_ptr), + "r"(&bpf_kfunc_call_test_ret_ii), + "r"(&bpf_kfunc_call_test_ret_big)); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128") +__naked int aggregate_ret_kfunc_precise(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + "call %[bpf_kfunc_call_test_i128];" + "r6 = r2;" / second return half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_i128) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL") +__naked int aggregate_ret_kfunc_fastcall_fail(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + "call %[bpf_kfunc_call_test_ret_fastcall];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_fastcall) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("is not composed of scalars") +__naked int aggregate_ret_kfunc_ptr_fail(void) +{ + asm volatile ( + "r1 = 0;" + "call %[bpf_kfunc_call_test_ret_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_ptr) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_kfunc_small_no_r2(void) +{ + asm volatile ( + "r1 = 0;" + "r2 = 0;" + "call %[bpf_kfunc_call_test_ret_ii];" + "r0 = r2;" / R2 is not a return register for a <=8 byte struct / + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_ii) + : __clobber_all); +} + +/ + * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by + * btf_distill_func_proto() before the KF_FASTCALL and JIT-capability checks. + / +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported") +__naked int aggregate_ret_kfunc_too_big_fail(void) +{ + asm volatile ( + "call %[bpf_kfunc_call_test_ret_big];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_big) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c new file mode 100644 index 0000000000000..cffd8d7d3241a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +/ freplace target: a global subprogram returning 16 bytes in R0:R2. / +__naked unsigned __int128 agg_ret_target_func(void) +{ + asm volatile ( + "r0 = 0x1234;" + "r2 = 0x5678;" + "exit;" + ); +} + +SEC("tc") +__naked int agg_ret_target(void) +{ + asm volatile ( + "call %[agg_ret_target_func];" + "r0 = 0;" + "exit;" + : + : __imm(agg_ret_target_func) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c index d055fc7b3b95d..338e67cd8856a 100644 --- a/tools/testing/selftests/bpf/progs/compute_live_registers.c +++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c @@ -431,6 +431,34 @@ __naked void subprog1(void) ::: __clobber_all); } +static __used __naked unsigned __int128 aux2(void) +{ + asm volatile ( + "r0 = 1;" + "r2 = 2;" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +/ A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). / +__load_if_JITed() +__log_level(2) +__msg("0: .12345.... (85) call pc+2") +__msg("1: ..2....... (bf) r0 = r2") +__msg("2: 0......... (95) exit") +__msg("3: .......... (b7) r0 = 1") +__msg("4: 0......... (b7) r2 = 2") +__msg("5: 0.2....... (95) exit") +__naked void subprog_ret_reg_pair(void) +{ + asm volatile ( + "call aux2;" + "r0 = r2;" + "exit;" + ::: __clobber_all); +} + #if defined(__TARGET_ARCH_x86) \|\| defined(__TARGET_ARCH_arm64) SEC("socket") diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c index ac44d60e50666..9708efb93683b 100644 --- a/tools/testing/selftests/bpf/progs/exceptions_fail.c +++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c @@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a) SEC("?tc") __exception_cb(exception_cb_bad_ret_type1) -__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.") +__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.") int reject_exception_cb_type_1(struct __sk_buff ctx) { bpf_throw(0); diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c new file mode 100644 index 0000000000000..12c15d293bd79 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> + +SEC("freplace/agg_ret_target_func") +__u64 new_agg_ret_target_func(void) +{ + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c new file mode 100644 index 0000000000000..7851bade2b406 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c @@ -0,0 +1,178 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "../test_kmods/bpf_testmod_kfunc.h" +#include "bpf_misc.h" + +#if defined(__clang_major__) && __clang_major__ >= 23 + +#define MIX_A 0xdeadbeefcafef00dULL +#define MIX_B 0x0123456789abcdefULL + +typedef unsigned __int128 u128; + +struct pair { + __u64 lo; / R0 / + __u64 hi; / R2 / +}; + +union upair { + __u64 halves[2]; + struct { + __u64 lo; / R0 / + __u64 hi; / R2 / + } parts; +}; + +static __noinline u128 make_i128(__u64 a, __u64 b) +{ + return ((u128)(a + b) << 64) \| (a - b); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_int128_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + u128 v; + + v = make_i128(a, b); + if ((__u64)(v >> 64) != a + b) + return 1; + if ((__u64)v != a - b) + return 2; + + return 0; +} + +static __noinline struct pair make_pair(__u64 a, __u64 b) +{ + struct pair p = { .lo = a + b, .hi = a - b }; + + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct pair p; + + p = make_pair(a, b); + if (p.lo != a + b) + return 1; + if (p.hi != a - b) + return 2; + + return 0; +} + +__noinline struct pair make_pair_global(__u64 a, __u64 b) +{ + struct pair p = { .lo = a + b, .hi = a - b }; + + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_global_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct pair p; + + p = make_pair_global(a, b); + if (p.lo != a + b) + return 1; + if (p.hi != a - b) + return 2; + + return 0; +} + +static __noinline union upair make_upair(__u64 a, __u64 b) +{ + union upair p; + + p.halves[0] = a + b; + p.halves[1] = a - b; + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_union_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + union upair p; + + p = make_upair(a, b); + if (p.parts.lo != a + b) + return 1; + if (p.parts.hi != a - b) + return 2; + + return 0; +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_int128_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + u128 v; + + v = bpf_kfunc_call_test_i128(a, b); + if ((__u64)(v >> 64) != a + b) + return 1; + if ((__u64)v != a - b) + return 2; + + return 0; +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct prog_test_ret_pair p; + + p = bpf_kfunc_call_test_ret_pair(a, b); + if (p.lo != a + b) + return 1; + if (p.hi != a - b) + return 2; + + return 0; +} + +#else + +SEC("socket") +__description("verifier_aggregate_ret: needs LLVM 23, dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c index 9366a3c578f13..850cf4f830c48 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -954,6 +954,53 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c) return 0; } +#if defined(__x86_64__) \|\| defined(__aarch64__) +__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b) +{ + return (__int128)(((unsigned __int128)(a + b) << 64) \| (a - b)); +} + +__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b) +{ + struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b) +{ + struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag) +{ + struct prog_test_ret_ptr r = { .p = NULL, .tag = tag }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) +{ + struct prog_test_ret_ii r = { .a = a, .b = b }; + + return r; +} +#endif / __x86_64__ \|\| __aarch64__ / + +/ + * Takes no argument on purpose: with no arguments there is nothing for the sret + * pointer to displace, so this needs no architecture guard even though it + * returns 24 bytes. + / +__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) +{ + struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 }; + + return r; +} + __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f, u64 g, u64 h, u64 i, u64 j) @@ -1487,6 +1534,14 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2) BTF_ID_FLAGS(func, bpf_kfunc_call_test3) BTF_ID_FLAGS(func, bpf_kfunc_call_test4) BTF_ID_FLAGS(func, bpf_kfunc_call_test5) +#if defined(__x86_64__) \|\| defined(__aarch64__) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii) +#endif +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h index 7d81070eefe7f..65e693ada7364 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h @@ -55,6 +55,27 @@ struct prog_test_big_arg { __u64 b; }; +struct prog_test_ret_pair { / 16 bytes: R0:R2 / + __u64 lo; + __u64 hi; +}; + +struct prog_test_ret_ii { / 8 bytes: R0 only / + int a; + int b; +}; + +struct prog_test_ret_ptr { / 16 bytes: contains a pointer / + void p; + __u64 tag; +}; + +struct prog_test_ret_big { /* 24 bytes: too large for R0:R2 / + __u64 a; + __u64 b; + __u64 c; +}; + struct prog_test_fail1 { void p; int x; @@ -131,6 +152,14 @@ int bpf_kfunc_call_test2(struct sock sk, __u32 a, __u32 b) __ksym; struct sock bpf_kfunc_call_test3(struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; +#ifdef __SIZEOF_INT128__ +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym; +#endif +struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym; +struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym; +struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym; +struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym; +struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym; __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, __u64 g, __u64 h, __u64 i, __u64 j) __ksym;
3/2	2026/08/19 06:42	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 24923 output: 61 thoughts: 974 Thoughts: Fuzzing Strategy for BPF Kfunc Return Value Expansion Okay, so we have this patch series introducing support for BPF kfuncs to return up to 16 bytes, using the R0:R2 register pair. This includes support for `__int128` and scalar-only structs/unions. This has definitely made significant changes, and my initial thought is that this needs to be fuzzed heavily. I've got a good handle on where the changes are: the BPF verifier, JIT compilers (x86, arm64, riscv), and some backtracking logic. Specifically, I need to focus my fuzzing efforts on the kernel functions that have been modified. I've compiled a list: In `kernel/bpf/verifier.c` I'll need to look at: - `bpf_compute_subprog_ret_regs` - `check_global_subprog_return_code` - `check_global_ret_scalar_reg` - `mark_kfunc_ret_regs` - `check_kfunc_call` - `check_func_call` - `prepare_func_exit` In `kernel/bpf/btf.c`: - `btf_validate_return_type` - `btf_check_func_type_match` In `kernel/bpf/backtrack.c`: - `backtrack_insn` In `kernel/bpf/liveness.c`: - `compute_insn_live_regs` - `bpf_compute_live_registers` And in `arch/x86/net/bpf_jit_comp.c`: - `emit_kfunc_arena_args` - `do_jit` - specifically the `BPF_JMP \| BPF_CALL` case. I need to quickly check the visibility of some of these, as I'm sure some are declared static or local. `btf_type_is_scalar_struct` is global. `bpf_compute_subprog_ret_regs` is static. `bpf_compute_live_registers` is global. `btf_validate_return_type` is static. `backtrack_insn` is static. Based on this, my first target list would be: - `bpf_check` - the main entry point - `bpf_compute_live_registers` - `btf_type_is_scalar_struct` - `btf_check_func_type_match` - `bpf_jit_supports_kfunc_ret_reg_pair` (although this is a simple boolean check, so might not need extensive fuzzing) Now, I'm reconsidering. Wait, those last three are static! So, I need to add those to my focus: - `check_kfunc_call` - `check_func_call` - `prepare_func_exit` I'll use `grepper` to find the non-static functions again to make sure I've got the most relevant ones to target.
4/2	2026/08/19 06:42	tool	grepper	0m Args: { "Expression": "^(int\|void\|bool\|struct [a-zA-Z0-9_]+ \\) [a-zA-Z0-9_]+\\(", "PathPrefix": "kernel/bpf/verifier.c" } Results:* { "Output": "kernel/bpf/verifier.c=366=static struct btf_record reg_btf_record(const struct bpf_reg_state reg)\n--\nkernel/bpf/verifier.c-380-\nkernel/bpf/verifier.c:381:bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog)\nkernel/bpf/verifier.c-382-{\n--\nkernel/bpf/verifier.c=453=const char bpf_subprog_name(const struct bpf_verifier_env env, int subprog)\n--\nkernel/bpf/verifier.c-463-\nkernel/bpf/verifier.c:464:void bpf_mark_subprog_exc_cb(struct bpf_verifier_env env, int subprog)\nkernel/bpf/verifier.c-465-{\n--\nkernel/bpf/verifier.c=540=static bool is_callback_calling_function(enum bpf_func_id func_id)\n--\nkernel/bpf/verifier.c-545-\nkernel/bpf/verifier.c:546:bool bpf_is_sync_callback_calling_insn(struct bpf_insn insn)\nkernel/bpf/verifier.c-547-{\n--\nkernel/bpf/verifier.c-551-\nkernel/bpf/verifier.c:552:bool bpf_is_async_callback_calling_insn(struct bpf_insn insn)\nkernel/bpf/verifier.c-553-{\n--\nkernel/bpf/verifier.c=558=static bool is_async_cb_sleepable(struct bpf_verifier_env env, struct bpf_insn insn)\n--\nkernel/bpf/verifier.c-572-\nkernel/bpf/verifier.c:573:bool bpf_is_may_goto_insn(struct bpf_insn insn)\nkernel/bpf/verifier.c-574-{\n--\nkernel/bpf/verifier.c=1635=static void free_func_state(struct bpf_func_state state)\n--\nkernel/bpf/verifier.c-1643-\nkernel/bpf/verifier.c:1644:void bpf_clear_jmp_history(struct bpf_verifier_state state)\nkernel/bpf/verifier.c-1645-{\n--\nkernel/bpf/verifier.c-1650-\nkernel/bpf/verifier.c:1651:void bpf_free_verifier_state(struct bpf_verifier_state state,\nkernel/bpf/verifier.c-1652-\t\t\t bool free_self)\n--\nkernel/bpf/verifier.c=1669=static int copy_func_state(struct bpf_func_state dst,\n--\nkernel/bpf/verifier.c-1677-\nkernel/bpf/verifier.c:1678:int bpf_copy_verifier_state(struct bpf_verifier_state dst_state,\nkernel/bpf/verifier.c-1679-\t\t\t const struct bpf_verifier_state src)\n--\nkernel/bpf/verifier.c=1740=static bool same_callsites(struct bpf_verifier_state a, struct bpf_verifier_state b)\n--\nkernel/bpf/verifier.c-1753-\nkernel/bpf/verifier.c:1754:void bpf_free_backedges(struct bpf_scc_visit visit)\nkernel/bpf/verifier.c-1755-{\n--\nkernel/bpf/verifier.c=2189=static int reg_bounds_sanity_check(struct bpf_verifier_env env,\n--\nkernel/bpf/verifier.c-2223-/ Mark a register as having a completely unknown (scalar) value. /\nkernel/bpf/verifier.c:2224:void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state reg)\nkernel/bpf/verifier.c-2225-{\n--\nkernel/bpf/verifier.c=2248=static int __mark_reg_s32_range(struct bpf_verifier_env env,\n--\nkernel/bpf/verifier.c-2267-\nkernel/bpf/verifier.c:2268:void bpf_mark_reg_not_init(const struct bpf_verifier_env env,\nkernel/bpf/verifier.c-2269-\t\t\t struct bpf_reg_state reg)\n--\nkernel/bpf/verifier.c=2392=struct bpf_subprog_info bpf_find_containing_subprog(struct bpf_verifier_env env, int off)\n--\nkernel/bpf/verifier.c-2412-/ Find subprogram that starts exactly at 'off' /\nkernel/bpf/verifier.c:2413:int bpf_find_subprog(struct bpf_verifier_env env, int off)\nkernel/bpf/verifier.c-2414-{\n--\nkernel/bpf/verifier.c=2539=find_kfunc_desc(const struct bpf_prog prog, u32 func_id, u16 offset)\n--\nkernel/bpf/verifier.c-2551-\nkernel/bpf/verifier.c:2552:int bpf_get_kfunc_addr(const struct bpf_prog prog, u32 func_id,\nkernel/bpf/verifier.c-2553-\t\t u16 btf_fd_idx, u8 *func_addr)\n--\nkernel/bpf/verifier.c=2642=static struct btf __find_kfunc_desc_btf(struct bpf_verifier_env env,\n--\nkernel/bpf/verifier.c-2691-\nkernel/bpf/verifier.c:2692:void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab tab)\nkernel/bpf/verifier.c-2693-{\n--\nkernel/bpf/verifier.c=2832=static int gen_kfunc_arg_proto(struct bpf_verifier_env env, struct bpf_call_arg_meta meta,\n--\nkernel/bpf/verifier.c-2834-\nkernel/bpf/verifier.c:2835:int bpf_add_kfunc_call(struct bpf_verifier_env env, u32 func_id, u16 offset)\nkernel/bpf/verifier.c-2836-{\n--\nkernel/bpf/verifier.c=3358=const char bpf_disasm_kfunc_name(void data, const struct bpf_insn insn)\n--\nkernel/bpf/verifier.c-3375-\nkernel/bpf/verifier.c:3376:void bpf_verbose_insn(struct bpf_verifier_env env, struct bpf_insn insn)\nkernel/bpf/verifier.c-3377-{\n--\nkernel/bpf/verifier.c-3389- /\nkernel/bpf/verifier.c:3390:void bpf_bt_sync_linked_regs(struct backtrack_state bt, struct bpf_jmp_history_entry hist)\nkernel/bpf/verifier.c-3391-{\n--\nkernel/bpf/verifier.c-3422-\nkernel/bpf/verifier.c:3423:int mark_chain_precision(struct bpf_verifier_env env, int regno)\nkernel/bpf/verifier.c-3424-{\n--\nkernel/bpf/verifier.c=5766=static void coerce_subreg_to_size_sx(struct bpf_reg_state reg, int size)\n--\nkernel/bpf/verifier.c-5811-\nkernel/bpf/verifier.c:5812:bool bpf_map_is_rdonly(const struct bpf_map map)\nkernel/bpf/verifier.c-5813-{\n--\nkernel/bpf/verifier.c-5831-\nkernel/bpf/verifier.c:5832:int bpf_map_direct_read(struct bpf_map map, int off, int size, u64 val,\nkernel/bpf/verifier.c-5833-\t\t\tbool is_ldsx)\n--\nkernel/bpf/verifier.c=8958=static bool may_update_sockmap(struct bpf_verifier_env env, int func_id)\n--\nkernel/bpf/verifier.c-8990-\nkernel/bpf/verifier.c:8991:bool bpf_allow_tail_call_in_subprogs(struct bpf_verifier_env env)\nkernel/bpf/verifier.c-8992-{\n--\nkernel/bpf/verifier.c=9902=static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn,\n--\nkernel/bpf/verifier.c-10024-\nkernel/bpf/verifier.c:10025:int map_set_for_each_callback_args(struct bpf_verifier_env env,\nkernel/bpf/verifier.c-10026-\t\t\t\t struct bpf_func_state caller,\n--\nkernel/bpf/verifier.c=10734=static bool can_elide_value_nullness(const struct bpf_map map)\n--\nkernel/bpf/verifier.c-10747-\nkernel/bpf/verifier.c:10748:int bpf_get_helper_proto(struct bpf_verifier_env env, int func_id,\nkernel/bpf/verifier.c-10749-\t\t\t const struct bpf_func_proto *ptr)\n--\nkernel/bpf/verifier.c=11581=static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta meta, u32 arg_idx)\n--\nkernel/bpf/verifier.c-11596-/* Returns true if struct is composed of scalars, 4 levels of nesting allowed /\nkernel/bpf/verifier.c:11597:bool btf_type_is_scalar_struct(struct bpf_verifier_env env,\nkernel/bpf/verifier.c-11598-\t\t\t const struct btf btf,\n--\nkernel/bpf/verifier.c=11901=static bool is_kfunc_bpf_preempt_enable(struct bpf_call_arg_meta meta)\n--\nkernel/bpf/verifier.c-11905-\nkernel/bpf/verifier.c:11906:bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta meta)\nkernel/bpf/verifier.c-11907-{\n--\nkernel/bpf/verifier.c=12388=static bool is_async_callback_calling_kfunc(u32 btf_id)\n--\nkernel/bpf/verifier.c-12393-\nkernel/bpf/verifier.c:12394:bool bpf_is_throw_kfunc(struct bpf_insn insn)\nkernel/bpf/verifier.c-12395-{\n--\nkernel/bpf/verifier.c=12645=static int check_kfunc_args(struct bpf_verifier_env env, struct bpf_call_arg_meta meta,\n--\nkernel/bpf/verifier.c-13336-\nkernel/bpf/verifier.c:13337:int bpf_fetch_kfunc_arg_meta(struct bpf_verifier_env env,\nkernel/bpf/verifier.c-13338-\t\t\t s32 func_id,\n--\nkernel/bpf/verifier.c=17580=static int check_global_subprog_return_code(struct bpf_verifier_env env)\n--\nkernel/bpf/verifier.c-17606- /\nkernel/bpf/verifier.c:17607:bool bpf_verifier_inlines_helper_call(struct bpf_verifier_env env, s32 imm)\nkernel/bpf/verifier.c-17608-{\n--\nkernel/bpf/verifier.c-17625- /\nkernel/bpf/verifier.c:17626:bool bpf_get_call_summary(struct bpf_verifier_env env, struct bpf_insn call,\nkernel/bpf/verifier.c-17627-\t\t\t struct bpf_call_summary cs)\n--\nkernel/bpf/verifier.c=17888=static u32 idset_cnt_get(struct bpf_idset idset, u32 id)\n--\nkernel/bpf/verifier.c-17903- /\nkernel/bpf/verifier.c:17904:void bpf_clear_singular_ids(struct bpf_verifier_env env,\nkernel/bpf/verifier.c-17905-\t\t\t struct bpf_verifier_state st)\n--\nkernel/bpf/verifier.c=19626=static void print_verification_stats(struct bpf_verifier_env env)\n--\nkernel/bpf/verifier.c-19655-\nkernel/bpf/verifier.c:19656:int bpf_prog_ctx_arg_info_init(struct bpf_prog prog,\nkernel/bpf/verifier.c-19657-\t\t\t const struct bpf_ctx_arg_aux info, u32 cnt)\n--\nkernel/bpf/verifier.c=19979=static bool attach_uses_trampoline_retval(enum bpf_attach_type type)\n--\nkernel/bpf/verifier.c-19992-\nkernel/bpf/verifier.c:19993:int bpf_check_attach_target(struct bpf_verifier_log log,\nkernel/bpf/verifier.c-19994-\t\t\t const struct bpf_prog prog,\n--\nkernel/bpf/verifier.c=20416=static int check_attach_btf_id(struct bpf_verifier_env env)\n--\nkernel/bpf/verifier.c-20515-\nkernel/bpf/verifier.c:20516:int bpf_check_attach_btf_id_multi(struct btf btf, struct bpf_prog prog, u32 btf_id,\nkernel/bpf/verifier.c-20517-\t\t\t\t struct bpf_attach_target_info tgt_info)\n--\nkernel/bpf/verifier.c=20743=static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data insn_aux,\n--\nkernel/bpf/verifier.c-20759-\nkernel/bpf/verifier.c:20760:int bpf_fixup_kfunc_call(struct bpf_verifier_env env, struct bpf_insn insn,\nkernel/bpf/verifier.c-20761-\t\t struct bpf_insn insn_buf, int insn_idx, int cnt)\n--\nkernel/bpf/verifier.c=20972=static int bpf_prog_verify_signature(struct bpf_verifier_env env,\n--\nkernel/bpf/verifier.c-21088-\nkernel/bpf/verifier.c:21089:int bpf_check(struct bpf_prog prog, union bpf_attr attr, bpfptr_t uattr,\nkernel/bpf/verifier.c-21090-\t struct bpf_log_attr *attr_log)\n" }
5/2	2026/08/19 06:42	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 30702 output: 62 thoughts: 466 Thoughts: My Analysis of the Code Changes Okay, let's break down these function changes. I need to get a handle on what's being touched here. First, I'm focusing on the non-static, global functions, and what is being modified. I see `btf_type_is_scalar_struct` and `bpf_jit_supports_kfunc_ret_reg_pair` popping up. `bpf_jit_supports_kfunc_ret_reg_pair` is architecture-specific, and shows up in the JIT compiler, which makes sense. I have to look into this more closely to see how it's being used. Then there's `bpf_check` - a hot-path entry point. That's a good place to start, even though it's generic. I'll need to see how the interactions are happening there. I see `bpf_compute_live_registers` is being flagged, even though it's part of the verifier, I can't overlook it. I see the function `bpf_add_kfunc_call` is modified, which warrants another look. `bpf_check_attach_target` is related and I see `btf_check_func_type_match` is a call from there, so I'll need to keep this in mind as well. Now, let's take another pass at the patch. I'm noting the static helper functions: `bpf_compute_subprog_ret_regs`, `check_global_subprog_return_code`, `check_kfunc_call`, `check_func_call`, and `prepare_func_exit`. I'll want to understand their roles within the broader context. So, let's create a list to track: I should definitely include `btf_type_is_scalar_struct` and `bpf_jit_supports_kfunc_ret_reg_pair` in `FocusSymbols`. Then, from the modified non-static functions, I see the ones I need to examine: `bpf_compute_live_registers`, `btf_distill_func_proto`, and `btf_prepare_func_args`. I have to dig into those now, particularly `btf_distill_func_proto` and `btf_prepare_func_args`.
6/2	2026/08/19 06:42	tool	grepper	0m Args: { "Expression": "^(int\|void\|bool\|struct [a-zA-Z0-9_]+ \\) [a-zA-Z0-9_]+\\(", "PathPrefix": "kernel/bpf/btf.c" } Results:* { "Output": "kernel/bpf/btf.c=500=static int btf_start_id(const struct btf btf)\n--\nkernel/bpf/btf.c-504-\nkernel/bpf/btf.c:505:bool btf_type_is_void(const struct btf_type t)\nkernel/bpf/btf.c-506-{\n--\nkernel/bpf/btf.c=532=static bool btf_type_is_decl_tag_target(const struct btf_type t)\n--\nkernel/bpf/btf.c-537-\nkernel/bpf/btf.c:538:bool btf_is_vmlinux(const struct btf btf)\nkernel/bpf/btf.c-539-{\n--\nkernel/bpf/btf.c=991=static bool btf_type_int_is_regular(const struct btf_type t)\n--\nkernel/bpf/btf.c-995-\nkernel/bpf/btf.c:996:bool btf_type_is_i32(const struct btf_type t)\nkernel/bpf/btf.c-997-{\n--\nkernel/bpf/btf.c-1000-\nkernel/bpf/btf.c:1001:bool btf_type_is_i64(const struct btf_type t)\nkernel/bpf/btf.c-1002-{\n--\nkernel/bpf/btf.c-1005-\nkernel/bpf/btf.c:1006:bool btf_type_is_primitive(const struct btf_type t)\nkernel/bpf/btf.c-1007-{\n--\nkernel/bpf/btf.c-1015- /\nkernel/bpf/btf.c:1016:bool btf_member_is_reg_int(const struct btf btf, const struct btf_type s,\nkernel/bpf/btf.c-1017-\t\t\t const struct btf_member m,\n--\nkernel/bpf/btf.c=1891=const char btf_get_name(const struct btf btf)\n--\nkernel/bpf/btf.c-1895-\nkernel/bpf/btf.c:1896:void btf_get(struct btf btf)\nkernel/bpf/btf.c-1897-{\n--\nkernel/bpf/btf.c-1900-\nkernel/bpf/btf.c:1901:void btf_put(struct btf btf)\nkernel/bpf/btf.c-1902-{\n--\nkernel/bpf/btf.c=1914=const struct btf_header btf_header(const struct btf btf)\n--\nkernel/bpf/btf.c-1918-\nkernel/bpf/btf.c:1919:void btf_set_base_btf(struct btf btf, const struct btf base_btf)\nkernel/bpf/btf.c-1920-{\n--\nkernel/bpf/btf.c=3526=static int btf_find_kptr(const struct btf btf, const struct btf_type t,\n--\nkernel/bpf/btf.c-3569-\nkernel/bpf/btf.c:3570:int btf_find_next_decl_tag(const struct btf btf, const struct btf_type pt,\nkernel/bpf/btf.c-3571-\t\t\t int comp_idx, const char tag_key, int last_id)\n--\nkernel/bpf/btf.c=4119=struct btf_record btf_parse_fields(const struct btf btf, const struct btf_type t,\n--\nkernel/bpf/btf.c-4251-\nkernel/bpf/btf.c:4252:int btf_check_and_fixup_fields(const struct btf btf, struct btf_record rec)\nkernel/bpf/btf.c-4253-{\n--\nkernel/bpf/btf.c=6106=static int find_kern_ctx_type_id(enum bpf_prog_type prog_type)\n--\nkernel/bpf/btf.c-6121-\nkernel/bpf/btf.c:6122:bool btf_is_projection_of(const char pname, const char tname)\nkernel/bpf/btf.c-6123-{\n--\nkernel/bpf/btf.c-6130-\nkernel/bpf/btf.c:6131:bool btf_is_prog_ctx_type(struct bpf_verifier_log log, const struct btf btf,\nkernel/bpf/btf.c-6132-\t\t\t const struct btf_type t, enum bpf_prog_type prog_type,\n--\nkernel/bpf/btf.c=6352=static int btf_translate_to_vmlinux(struct bpf_verifier_log log,\n--\nkernel/bpf/btf.c-6362-\nkernel/bpf/btf.c:6363:int get_kern_ctx_btf_id(struct bpf_verifier_log log, enum bpf_prog_type prog_type)\nkernel/bpf/btf.c-6364-{\n--\nkernel/bpf/btf.c=6616=static bool prog_args_trusted(const struct bpf_prog prog)\n--\nkernel/bpf/btf.c-6631-\nkernel/bpf/btf.c:6632:int btf_ctx_arg_offset(const struct btf btf, const struct btf_type func_proto,\nkernel/bpf/btf.c-6633-\t\t u32 arg_no)\n--\nkernel/bpf/btf.c=6657=static const struct bpf_raw_tp_null_args raw_tp_null_args[] = {\n--\nkernel/bpf/btf.c-6848-\nkernel/bpf/btf.c:6849:bool btf_ctx_access(int off, int size, enum bpf_access_type type,\nkernel/bpf/btf.c-6850-\t\t const struct bpf_prog prog,\n--\nkernel/bpf/btf.c=7113=static int btf_struct_walk(struct bpf_verifier_log log, const struct btf btf,\n--\nkernel/bpf/btf.c-7378-\nkernel/bpf/btf.c:7379:int btf_struct_access(struct bpf_verifier_log log,\nkernel/bpf/btf.c-7380-\t\t const struct bpf_reg_state reg,\n--\nkernel/bpf/btf.c-7461- /\nkernel/bpf/btf.c:7462:bool btf_types_are_same(const struct btf btf1, u32 id1,\nkernel/bpf/btf.c-7463-\t\t\tconst struct btf btf2, u32 id2)\n--\nkernel/bpf/btf.c-7471-\nkernel/bpf/btf.c:7472:bool btf_struct_ids_match(struct bpf_verifier_log log,\nkernel/bpf/btf.c-7473-\t\t\t const struct btf btf, u32 id, int off,\n--\nkernel/bpf/btf.c=7545=static u8 __get_arg_fmodel_flags(const struct btf btf,\n--\nkernel/bpf/btf.c-7560-\nkernel/bpf/btf.c:7561:int btf_distill_func_proto(struct bpf_verifier_log log,\nkernel/bpf/btf.c-7562-\t\t\t struct btf btf,\n--\nkernel/bpf/btf.c=7640=static int btf_check_func_type_match(struct bpf_verifier_log log,\n--\nkernel/bpf/btf.c-7756-/* Compare BTFs of given program with BTF of target program /\nkernel/bpf/btf.c:7757:int btf_check_type_match(struct bpf_verifier_log log, const struct bpf_prog prog,\nkernel/bpf/btf.c-7758-\t\t\t struct btf btf2, const struct btf_type t2)\n--\nkernel/bpf/btf.c=7972=static int btf_validate_return_type(struct bpf_verifier_env env, struct btf btf,\n--\nkernel/bpf/btf.c-8019- /\nkernel/bpf/btf.c:8020:int btf_prepare_func_args(struct bpf_verifier_env env, int subprog)\nkernel/bpf/btf.c-8021-{\n--\nkernel/bpf/btf.c=8261=__printf(2, 0) static void btf_seq_show(struct btf_show show, const char fmt,\n--\nkernel/bpf/btf.c-8266-\nkernel/bpf/btf.c:8267:int btf_type_seq_show_flags(const struct btf btf, u32 type_id,\nkernel/bpf/btf.c-8268-\t\t\t void obj, struct seq_file m, u64 flags)\n--\nkernel/bpf/btf.c-8280-\nkernel/bpf/btf.c:8281:void btf_type_seq_show(const struct btf btf, u32 type_id, void obj,\nkernel/bpf/btf.c-8282-\t\t struct seq_file m)\n--\nkernel/bpf/btf.c=8295=__printf(2, 0) static void btf_snprintf_show(struct btf_show show, const char fmt,\n--\nkernel/bpf/btf.c-8316-\nkernel/bpf/btf.c:8317:int btf_type_snprintf_show(const struct btf btf, u32 type_id, void obj,\nkernel/bpf/btf.c-8318-\t\t\t char buf, int len, u64 flags)\n--\nkernel/bpf/btf.c-8337-\nkernel/bpf/btf.c:8338:int btf_type_name_to_buf(const struct btf btf, u32 type_id, char buf, int len)\nkernel/bpf/btf.c-8339-{\n--\nkernel/bpf/btf.c=8370=static int __btf_new_fd(struct btf btf)\n--\nkernel/bpf/btf.c-8374-\nkernel/bpf/btf.c:8375:int btf_new_fd(const union bpf_attr attr, bpfptr_t uattr, struct bpf_log_attr attr_log)\nkernel/bpf/btf.c-8376-{\n--\nkernel/bpf/btf.c=8403=struct btf btf_get_by_fd(int fd)\n--\nkernel/bpf/btf.c-8414-\nkernel/bpf/btf.c:8415:int btf_get_info_by_fd(const struct btf btf,\nkernel/bpf/btf.c-8416-\t\t const union bpf_attr attr,\n--\nkernel/bpf/btf.c-8474-\nkernel/bpf/btf.c:8475:int btf_get_fd_by_id(u32 id)\nkernel/bpf/btf.c-8476-{\n--\nkernel/bpf/btf.c=8496=u32 btf_obj_id(const struct btf btf)\n--\nkernel/bpf/btf.c-8500-\nkernel/bpf/btf.c:8501:bool btf_is_kernel(const struct btf btf)\nkernel/bpf/btf.c-8502-{\n--\nkernel/bpf/btf.c-8505-\nkernel/bpf/btf.c:8506:bool btf_is_module(const struct btf btf)\nkernel/bpf/btf.c-8507-{\n--\nkernel/bpf/btf.c=8765=BTF_TRACING_TYPE_xxx\n--\nkernel/bpf/btf.c-8771- /\nkernel/bpf/btf.c:8772:int btf_check_iter_arg(struct btf btf, const struct btf_type func, int arg_idx)\nkernel/bpf/btf.c-8773-{\n--\nkernel/bpf/btf.c=9082=static int bpf_prog_type_to_kfunc_hook(enum bpf_prog_type prog_type)\n--\nkernel/bpf/btf.c-9128-\nkernel/bpf/btf.c:9129:bool btf_kfunc_is_allowed(const struct btf btf,\nkernel/bpf/btf.c-9130-\t\t\t u32 kfunc_btf_id,\n--\nkernel/bpf/btf.c=9156=u32 btf_kfunc_flags(const struct btf btf, u32 kfunc_btf_id, const struct bpf_prog prog)\n--\nkernel/bpf/btf.c-9177- /\nkernel/bpf/btf.c:9178:int btf_kfunc_check_flag(const struct btf btf, u32 kfunc_btf_id, u32 flag)\nkernel/bpf/btf.c-9179-{\n--\nkernel/bpf/btf.c=9208=static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook,\n--\nkernel/bpf/btf.c-9234-/* This function must be invoked only from initcalls/module init functions /\nkernel/bpf/btf.c:9235:int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,\nkernel/bpf/btf.c-9236-\t\t\t const struct btf_kfunc_id_set kset)\n--\nkernel/bpf/btf.c=9251=EXPORT_SYMBOL_GPL(register_btf_kfunc_id_set);\n--\nkernel/bpf/btf.c-9253-/* This function must be invoked only from initcalls/module init functions /\nkernel/bpf/btf.c:9254:int register_btf_fmodret_id_set(const struct btf_kfunc_id_set kset)\nkernel/bpf/btf.c-9255-{\n--\nkernel/bpf/btf.c=9277=static int btf_check_dtor_kfuncs(struct btf btf, const struct btf_id_dtor_kfunc dtors, u32 cnt)\n--\nkernel/bpf/btf.c-9321-/* This function must be invoked only from initcalls/module init functions /\nkernel/bpf/btf.c:9322:int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc dtors, u32 add_cnt,\nkernel/bpf/btf.c-9323-\t\t\t\tstruct module owner)\n--\nkernel/bpf/btf.c=9395=EXPORT_SYMBOL_GPL(register_btf_id_dtor_kfuncs);\n--\nkernel/bpf/btf.c-9417- /\nkernel/bpf/btf.c:9418:int bpf_core_types_are_compat(const struct btf local_btf, __u32 local_id,\nkernel/bpf/btf.c-9419-\t\t\t const struct btf targ_btf, __u32 targ_id)\n--\nkernel/bpf/btf.c-9426-\nkernel/bpf/btf.c:9427:int bpf_core_types_match(const struct btf local_btf, u32 local_id,\nkernel/bpf/btf.c-9428-\t\t\t const struct btf targ_btf, u32 targ_id)\n--\nkernel/bpf/btf.c=9647=bpf_core_find_cands(struct bpf_core_ctx ctx, u32 local_type_id)\n--\nkernel/bpf/btf.c-9733-\nkernel/bpf/btf.c:9734:int bpf_core_apply(struct bpf_core_ctx ctx, const struct bpf_core_relo relo,\nkernel/bpf/btf.c-9735-\t\t int relo_idx, void insn)\n--\nkernel/bpf/btf.c-9812-\nkernel/bpf/btf.c:9813:bool btf_nested_type_is_trusted(struct bpf_verifier_log log,\nkernel/bpf/btf.c-9814-\t\t\t\tconst struct bpf_reg_state reg,\n--\nkernel/bpf/btf.c-9859-\nkernel/bpf/btf.c:9860:bool btf_type_ids_nocast_alias(struct bpf_verifier_log log,\nkernel/bpf/btf.c-9861-\t\t\t const struct btf reg_btf, u32 reg_id,\n--\nkernel/bpf/btf.c=9981=bpf_struct_ops_find(struct btf btf, u32 type_id)\n--\nkernel/bpf/btf.c-10001-\nkernel/bpf/btf.c:10002:int __register_bpf_struct_ops(struct bpf_struct_ops st_ops)\nkernel/bpf/btf.c-10003-{\n--\nkernel/bpf/btf.c=10030=EXPORT_SYMBOL_GPL(__register_bpf_struct_ops);\n--\nkernel/bpf/btf.c-10032-\nkernel/bpf/btf.c:10033:bool btf_param_match_suffix(const struct btf btf,\nkernel/bpf/btf.c-10034-\t\t\t const struct btf_param arg,\n" }
7/2	2026/08/19 06:42	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 36671 output: 262 thoughts: 1424 Thoughts: Analysis of BPF Kfunc Return Value Extension Patch Okay, let's break this down. It looks like we're enabling BPF kfuncs and subprograms to return 16-byte values using a register pair (R0:R2). This touches a bunch of key areas. First, `btf_distill_func_proto` and `btf_prepare_func_args` have been modified, and are now non-static. `btf_distill_func_proto` is crucial for extracting the function prototype for kfuncs, and `btf_prepare_func_args` deals with preparing the arguments for subprograms. I see that `bpf_jit_supports_kfunc_ret_reg_pair` is also modified. It's added in `kernel/bpf/core.c` and overridden in the architecture-specific JIT compilers (`x86`, `arm64`, `riscv`). Good. Seems it is enabled by default for these architectures. No new config options are added, which is nice. We just need the standard BPF configs – `BPF_SYSCALL` and `BPF_JIT`. Now, let's look at the verifier changes in `kernel/bpf/verifier.c`. We have new functions like `bpf_compute_subprog_ret_regs`, `check_global_ret_scalar_reg`, and `mark_kfunc_ret_regs`. Several existing functions are modified, including `check_kfunc_call`, `check_func_call`, and `prepare_func_exit`. The addition of `btf_type_is_scalar_struct` is interesting, and it's a global function. I need to focus on a few key symbols here. I'll put `btf_type_is_scalar_struct`, `btf_distill_func_proto`, and `btf_prepare_func_args` in `FocusSymbols`. Considering performance, `bpf_compute_live_registers` is a very hot path function called for every BPF program load, so that's a crucial one to monitor and is added to `FocusSymbols`. `bpf_jit_supports_kfunc_ret_reg_pair` is also important since it's called when adding a kfunc call and can impact every architecture. Let's double-check that `btf_type_is_scalar_struct` and `bpf_compute_live_registers` are properly visible. `btf_type_is_scalar_struct` is declared in a header, defined in the kernel and is not module exported, and `bpf_compute_live_registers` is also global. Alright, the core of this patch series is adding support for kfuncs to return 16-byte values. This involves substantial changes to the BPF verifier for tracking return values, modifications to the JIT compilers (x86, arm64, riscv), and adjustments to the liveness and backtracking logic. This functionality touches critical, reachable code within the BPF subsystem and demands fuzzing to ensure the verifier correctly handles register states and prevents potential leaks or memory access errors. So, here's my reasoning and the resulting data for the fuzzing: FocusSymbols: * `btf_type_is_scalar_struct` * `btf_distill_func_proto` * `btf_prepare_func_args` * `bpf_compute_live_registers` * `bpf_jit_supports_kfunc_ret_reg_pair` EnableConfigs: * `BPF_SYSCALL` * `BPF_JIT` Now, let's execute `set-results`.
8/2	2026/08/19 06:42	tool	set-results	0m Args: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT" ], "FocusSymbols": [ "btf_type_is_scalar_struct", "btf_distill_func_proto", "btf_prepare_func_args", "bpf_compute_live_registers", "bpf_jit_supports_kfunc_ret_reg_pair" ], "Reasoning": "The patch series adds support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or scalar-only structs/unions) in the register pair R0:R2. This involves modifications to the BPF verifier to track the return values in both registers, changes to the liveness and backtracking logic, and updates to the JIT compilers (x86, arm64, riscv) to support this new calling convention. These are functional changes to reachable core kernel logic (BPF subsystem) that should be fuzzed to ensure the verifier correctly tracks the register states and prevents any leaks or invalid memory accesses.", "WorthFuzzing": true } Results: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT" ], "FocusSymbols": [ "btf_type_is_scalar_struct", "btf_distill_func_proto", "btf_prepare_func_args", "bpf_compute_live_registers", "bpf_jit_supports_kfunc_ret_reg_pair" ], "Reasoning": "The patch series adds support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or scalar-only structs/unions) in the register pair R0:R2. This involves modifications to the BPF verifier to track the return values in both registers, changes to the liveness and backtracking logic, and updates to the JIT compilers (x86, arm64, riscv) to support this new calling convention. These are functional changes to reachable core kernel logic (BPF subsystem) that should be fuzzed to ensure the verifier correctly tracks the register states and prevents any leaks or invalid memory accesses.", "WorthFuzzing": true }
9/1	2026/08/19 06:42	action	If	0m Args: { "WorthFuzzing": true } Results: null
10/2	2026/08/19 06:42	agent	kmsan-evaluator	0m Model: best-expensive Args: null Results: { "KMSANReasoning": "The patch series introduces support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or small structs) in the register pair R0:R2. The changes are confined to the BPF verifier's static analysis (tracking liveness and initialization state of R2, type checking BTF return types) and architecture-specific JIT compilers (moving the native second return register into BPF R2). \n\nThere are no allocations of complex data structures, no new kernel-to-user memory copies, and no modifications to bounds checking of memory buffers. The verifier statically enforces that BPF programs cannot read uninitialized BPF registers, and any logical bugs in this tracking would lead to verifier bypasses rather than C-level uninitialized memory uses. Furthermore, KMSAN instruments kernel C code and tracks memory, but it does not instrument JIT-ed BPF code or track BPF register states directly. Therefore, any potential issues introduced by this patch (such as verifier flaws allowing stale register reads) would not be detected by KMSAN. Standard KASAN, verifier tests, and syzkaller's BPF fuzzing are the appropriate tools for validating these changes.", "NeedsKMSAN": false } Instruction: You are an expert Linux kernel security engineer specializing in kernel memory error detectors (KASAN and KMSAN). Your job is to review the provided patch series and determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer) fuzzing session in addition to standard KASAN fuzzing. CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN: - Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable a comprehensive suite of debugging tools and sanitizers, including KASAN (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks. - KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap, or page allocations) and kernel-to-user memory info-leaks. Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN, LOCKDEP, OR OTHER STANDARD BUG DETECTORS. A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks that are detected ONLY by KMSAN. Look holistically at the patch series and surrounding code. Even if no direct uninitialized field accesses or new buffer allocations are added in the diff itself, a patch may alter control flow, bounds checking, or data length calculations in ways that change how the rest of the code operates on existing buffers (e.g. allowing uninitialized stack/heap memory to be read, copied to user space, or used in control flow). Do not hesitate to use your code access tools to inspect the surrounding code, called functions, and callers. Set NeedsKMSAN=true ONLY IF the patch introduces or modifies: 1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb attributes, ioctl output arguments, socket options, or BPF buffers) where fields or structure padding might not be fully initialized/zeroed. 2. Conditional logic or branching that depends on potentially uninitialized variables or struct fields. 3. Allocation or initialization of complex data structures where uninitialized fields could be read later in reachable code paths. 4. Bounds checks, lengths, or logic in a way that allows surrounding code to access uninitialized bytes of existing buffers. Set NeedsKMSAN=false IF: - The code changes primarily risk out-of-bounds access, array overflows, NULL pointer dereferences, locking deadlocks, or use-after-free bugs (these are already caught by KASAN, LOCKDEP, or standard bug detectors). - All stack/heap structures touched or introduced by the patch are fully zeroed or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied. - The patch does not introduce any risk of uninitialized memory usage or info-leaks. Use your code access tools to inspect the surrounding code if necessary, then provide detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch. Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit cc236f68937f8479bce51601bd02ac7647affe1d Author: syz-cluster <triage@syzkaller.com> Date: Wed Aug 19 06:41:58 2026 +0000 syz-cluster: applied patch under review diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 85f73e0bbd0ff..89dea6b0b0241 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -575,6 +575,71 @@ is also covered by this recovery. A kfunc handed an arena pointer may therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking against the arena. Larger accesses must verify the range explicitly. +2.9 kfunc Return Values +----------------------- + +A kfunc may return a scalar, a pointer, or a small struct or union by +value. A scalar or pointer of up to 8 bytes is returned in R0, as usual. + +A struct or union returned by value must be composed only of scalars +(recursively), where a scalar is an integer or an enum; arrays of scalars are +allowed as members. Its bytes are handed back to the program as the raw +contents of R0 (and R2), so a pointer field would be laundered into a scalar +and escape the verifier's pointer provenance and reference tracking. A struct +or union with a pointer member is therefore rejected at load time, and so is +one with a floating-point member, which the ABI may not return in R0:R2 at +all. + +A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a +scalar-only struct or union, or an ``__int128``. Such a value is returned +in the register pair R0:R2, matching the convention LLVM uses for the BPF +target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or +union of 8 bytes or less is returned in R0 alone. + +:: + + struct bpf_pair { __u64 a, b; }; /* 16 bytes / + + __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void) + { + struct bpf_pair p = { .a = 1, .b = 2 }; + + return p; / p.a in R0, p.b in R2 / + } + +Returning a value in the R0:R2 pair requires the JIT to place the second +half of the return value into R2, which not every architecture supports +right now. A kfunc with a return value larger than 8 bytes is therefore +rejected at load time on a JIT that does not advertise this capability (see +``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run +by the interpreter. A return value larger than 16 bytes is not supported. + +The same R0:R2 convention applies to a BPF subprogram, global or static, that +returns an ``__int128`` or a struct or union larger than 8 bytes. It is only +used when the program is JITed, since the interpreter propagates only R0 out of +a subprogram: without a JIT the return value stays in R0 alone, and a caller +reading R2 is rejected for reading an uninitialized register. A global +subprogram is verified in isolation, so its by-value struct or union return is +restricted to scalars just like a kfunc's; a static subprogram is verified +inline and has no such restriction. The main program is not covered: its return +value is the program's exit code, read out of R0 alone, so a declared upper +half is never looked at. + +A global subprogram must leave a scalar in every* register of the pair, so +both halves of the returned value have to be assigned. Leaving the upper half +uninitialized is not merely untidy: the compiler is then free to leave R2 +holding whatever it happened to hold, which for a subprogram taking a pointer +argument is typically that pointer. Handing the caller an unknown scalar built +from a pointer is a leak, so the verifier rejects it with:: + + At subprogram exit the register R2 is not a scalar value (...) + +Initialize the whole return value, for example ``struct pair p = {};``, to +avoid this. A static subprogram is exempt from the scalar-only rule: it is +verified inline, so an unassigned R2 is simply passed back to the caller as +uninitialized and only a caller that reads it fails. A stack pointer left in +R2 is still rejected there, just as one in R0 is. + .. _BPF_kfunc_lifecycle_expectations: 3. kfunc lifecycle expectations diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index c18e005a41dbe..3aa3ea0bc30bb 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2388,6 +2388,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_stack_args(void) { return true; diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 74efe4b138d2a..47c7bf431ba85 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -2121,6 +2121,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_ptr_xchg(void) { return true; diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 1a9fb530adc3c..48429fae06410 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip, * arena NULL is offset 0. Return the number of emitted bytes. / static int emit_kfunc_arena_args(struct bpf_prog bpf_prog, - const struct bpf_insn insn, u8 pprog) + const struct btf_func_model fm, u8 *pprog) { - const struct btf_func_model fm; u8 prog = pprog; u8 start = prog; int i; - fm = bpf_jit_find_kfunc_model(bpf_prog, insn); - if (!fm) - return -EINVAL; - for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) { u8 flags = fm->arg_flags[i]; u32 reg = BPF_REG_1 + i; @@ -2644,6 +2639,8 @@ st: insn_off = insn->off; / call / case BPF_JMP \| BPF_CALL: { + const struct btf_func_model fm = NULL; + func = (u8 ) __bpf_call_base + imm32; if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(stack_depth); @@ -2652,7 +2649,10 @@ st: insn_off = insn->off; if (!imm32) return -EINVAL; if (src_reg == BPF_PSEUDO_KFUNC_CALL) { - err = emit_kfunc_arena_args(bpf_prog, insn, &prog); + fm = bpf_jit_find_kfunc_model(bpf_prog, insn); + if (!fm) + return -EINVAL; + err = emit_kfunc_arena_args(bpf_prog, fm, &prog); if (err < 0) return err; ip += err; @@ -2666,6 +2666,14 @@ st: insn_off = insn->off; return -EINVAL; if (priv_frame_ptr) pop_r9(&prog); + / + * A kfunc returning more than 8 bytes hands the second + * half back in RDX (the native ABI's second return reg), + * but BPF expects it in R0:R2. BPF R0 is RAX (no move + * needed), while BPF R2 is RSI, so copy RDX into RSI. + / + if (fm && fm->ret_size > 8) + emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3); break; } @@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void) return true; } +bool bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return true; +} + bool bpf_jit_supports_stack_args(void) { return true; diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 5fad59fdab0d0..004b06785521c 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -819,6 +819,8 @@ struct bpf_subprog_info { bool is_async_cb: 1; bool is_exception_cb: 1; bool args_cached: 1; + / true if the return value is passed in the R0:R2 register pair / + bool ret_reg_pair: 1; / true if bpf_fastcall stack region is used by functions that can't be inlined / bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; @@ -1055,6 +1057,11 @@ static inline struct bpf_subprog_info subprog_info(struct bpf_verifier_env env return &env->subprog_info[subprog]; } +static inline bool bpf_ret_reg_pair(struct bpf_verifier_env env, int subprog) +{ + return subprog_info(env, subprog)->ret_reg_pair; +} + struct bpf_call_summary { u8 num_params; bool is_void; @@ -1481,6 +1488,8 @@ int bpf_jmp_offset(struct bpf_insn insn); struct bpf_iarray bpf_insn_successors(struct bpf_verifier_env env, u32 idx); void bpf_fmt_stack_mask(char buf, ssize_t buf_sz, u64 stack_mask); bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog); +bool btf_type_is_scalar_struct(struct bpf_verifier_env env, const struct btf btf, + const struct btf_type t, int rec); int bpf_find_subprog(struct bpf_verifier_env env, int off); bool bpf_is_throw_kfunc(struct bpf_insn insn); diff --git a/include/linux/filter.h b/include/linux/filter.h index 4a9bc6a848f2e..6e746b0a09306 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_percpu_insn(void); bool bpf_jit_supports_kfunc_call(void); +bool bpf_jit_supports_kfunc_ret_reg_pair(void); bool bpf_jit_supports_stack_args(void); bool bpf_jit_supports_arena_args(void); bool bpf_jit_supports_far_kfunc_call(void); diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index a2b18a9f1694c..653db80bcc47b 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -423,6 +423,10 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, / verifier_bug_if(idx + 1 != subseq_idx, env, "extra insn from subprog"); + /* global subprog always sets R0 / + bt_clear_reg(bt, BPF_REG_0); + / and if it does not set R2, main pass would catch it / + bt_clear_reg(bt, BPF_REG_2); / r1-r5 are invalidated after subprog call, * so for global func call it shouldn't be set * anymore @@ -432,8 +436,6 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, bt_reg_mask(bt)); return -EFAULT; } - / global subprog always sets R0 / - bt_clear_reg(bt, BPF_REG_0); return 0; } else { / static subprog call instruction, which @@ -506,6 +508,8 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, return -ENOTSUPP; / regular helper call sets R0 / bt_clear_reg(bt, BPF_REG_0); + / kfunc might also set R2 / + bt_clear_reg(bt, BPF_REG_2); if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { / if backtracking was looking for registers R1-R5 * they should have been found already. @@ -520,7 +524,25 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, return -EFAULT; } } else if (opcode == BPF_EXIT) { - bool r0_precise; + bool from_subprog_call, r0_precise, r2_precise; + + / BPF_EXIT in subprog or callback always returns + * right after the call instruction, so by checking + * whether the instruction at subseq_idx-1 is subprog + * call or not we can distinguish actual exit from + * subprog from exit from callback. In the former + * case, we need to propagate the precision of the + * return registers, if necessary. In the latter we + * never do that. + / + from_subprog_call = subseq_idx - 1 >= 0 && + bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]); + + / Sample the return registers before the callback + * handling below clears R1-R5. + / + r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0); + r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2); / Backtracking to a nested function call, 'idx' is a part of * the inner frame 'subseq_idx' is a part of the outer frame. @@ -533,30 +555,22 @@ static int backtrack_insn(struct bpf_verifier_env env, int idx, int subseq_idx, if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx)) for (i = BPF_REG_1; i <= BPF_REG_5; i++) bt_clear_reg(bt, i); + + bt_clear_reg(bt, BPF_REG_0); + bt_clear_reg(bt, BPF_REG_2); if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { verifier_bug(env, "backtracking exit unexpected regs %x", bt_reg_mask(bt)); return -EFAULT; } - / BPF_EXIT in subprog or callback always returns - * right after the call instruction, so by checking - * whether the instruction at subseq_idx-1 is subprog - * call or not we can distinguish actual exit from - * subprog from exit from callback. In the former - * case, we need to propagate r0 precision, if - * necessary. In the former we never do that. - / - r0_precise = subseq_idx - 1 >= 0 && - bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) && - bt_is_reg_set(bt, BPF_REG_0); - - bt_clear_reg(bt, BPF_REG_0); if (bt_subprog_enter(bt)) return -EFAULT; if (r0_precise) bt_set_reg(bt, BPF_REG_0); + if (r2_precise) + bt_set_reg(bt, BPF_REG_2); / r6-r9 and stack slots will stay set in caller frame * bitmasks until we return back from callee(s) / diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 6967d48bba49d..58b81c6f5288c 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7591,7 +7591,7 @@ int btf_distill_func_proto(struct bpf_verifier_log log, return -EINVAL; } ret = __get_type_size(btf, func->type, &t); - if (ret < 0 \|\| btf_type_is_struct(t)) { + if (ret < 0 \|\| ret > 16) { bpf_log(log, "The function %s return type %s is unsupported.\n", tname, btf_type_str(t)); @@ -7684,6 +7684,12 @@ static int btf_check_func_type_match(struct bpf_verifier_log log, btf_type_str(t2), fn2); return -EINVAL; } + if (btf_type_has_size(t1) && (t1->size > 8 \|\| t2->size > 8)) { + bpf_log(log, + "Return type of %s() has size %u and of %s() size %u, and a size above 8 bytes cannot be replaced\n", + fn1, t1->size, fn2, t2->size); + return -EINVAL; + } for (i = 0; i < nargs1; i++) { t1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL); @@ -7964,7 +7970,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env env, /* Check whether the type is a valid return type. / static int btf_validate_return_type(struct bpf_verifier_env env, struct btf btf, - const struct btf_type t, int subprog) + const struct btf_type t, int subprog, bool is_global) { u32 tags = 0; int err; @@ -7987,6 +7993,19 @@ static int btf_validate_return_type(struct bpf_verifier_env env, struct btf bt if (btf_type_is_void(t) \|\| btf_type_is_int(t) \|\| btf_is_any_enum(t)) return 0; + if (btf_type_is_struct(t) && t->size <= 16) { + / + * A global function's caller models the return as an opaque + * scalar pair, so it may only return scalars by value. A local + * function is verified inline, so a pointer field stays tracked + * and needs no such restriction. + / + bool local_func = subprog && !is_global; + + if (local_func \|\| btf_type_is_scalar_struct(env, btf, t, 0)) + return 0; + } + return -EOPNOTSUPP; } @@ -8074,12 +8093,12 @@ int btf_prepare_func_args(struct bpf_verifier_env env, int subprog) return -EINVAL; } - err = btf_validate_return_type(env, btf, t, subprog); + err = btf_validate_return_type(env, btf, t, subprog, is_global); if (err) { if (is_global) { bpf_log(log, - "Global function %s() return value not void or scalar. " - "Only those are supported.\n", + "Global function %s() has unsupported return type. " + "Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n", tname); } return err; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index d55e737ed75ad..5db77d7915dfc 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void) return false; } +bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void) +{ + return false; +} + bool __weak bpf_jit_supports_stack_args(void) { return false; diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index 74fc4b3f80d6e..71f998c6eb888 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -2060,7 +2060,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); } /* Compute info->{use,def} fields for the instruction / static void compute_insn_live_regs(struct bpf_verifier_env env, struct bpf_insn insn, - struct insn_live_regs info) + struct insn_live_regs info, + bool ret_reg_pair) { struct bpf_call_summary cs; const u8 class = BPF_CLASS(insn->code); @@ -2072,6 +2073,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env, const u32 src32 = mask_lo(src); const u32 dst32 = mask_lo(dst); const u32 r0 = reg64_mask(0); + const u32 r2 = reg64_mask(BPF_REG_2); u32 def = 0; u32 use = U32_MAX; @@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env env, break; case BPF_EXIT: def = 0; - use = r0; + use = ret_reg_pair ? (r0 \| r2) : r0; break; case BPF_CALL: def = ALL_CALLER_SAVED_REGS; @@ -2228,8 +2230,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env env) struct insn_live_regs state; int insn_cnt = env->prog->len; u64 pos, insn_pos; - int err = 0, i, j; - bool changed; + int err = 0, i, j, subprog, start, end; + bool changed, ret_reg_pair; / Use the following algorithm: * - define the following: @@ -2256,8 +2258,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env env) goto out; } - for (i = 0; i < insn_cnt; ++i) - compute_insn_live_regs(env, &insns[i], &state[i]); + for (subprog = 0; subprog < env->subprog_cnt; subprog++) { + start = env->subprog_info[subprog].start; + end = env->subprog_info[subprog + 1].start; + ret_reg_pair = bpf_ret_reg_pair(env, subprog); + + for (i = start; i < end; ++i) + compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair); + } / Forward pass: resolve stack access through FP-derived pointers / err = bpf_compute_subprog_arg_access(env); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e421ea2b80c3a..7a939dfa3797b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -385,27 +385,69 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env env, int subprog) return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL; } -static bool subprog_returns_void(struct bpf_verifier_env env, int subprog) +static const struct btf_type subprog_ret_type(struct bpf_verifier_env env, int subprog) { - const struct btf_type type, func, func_proto; + const struct btf_type func, func_proto; const struct btf btf = env->prog->aux->btf; u32 btf_id; + if (!btf \|\| !env->prog->aux->func_info) + return NULL; + btf_id = env->prog->aux->func_info[subprog].type_id; + / Both already validated by prepare_btf_func() at prog load. / func = btf_type_by_id(btf, btf_id); - if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id)) - return false; - func_proto = btf_type_by_id(btf, func->type); - if (!func_proto) - return false; - type = btf_type_skip_modifiers(btf, func_proto->type, NULL); - if (!type) - return false; + return btf_type_skip_modifiers(btf, func_proto->type, NULL); +} - return btf_type_is_void(type); +static bool subprog_returns_void(struct bpf_verifier_env env, int subprog) +{ + const struct btf_type type = subprog_ret_type(env, subprog); + + return type && btf_type_is_void(type); +} + +static u32 ret_regs_cnt(u32 size) +{ + return size > 8 && size <= 16 ? 2 : 1; +} + +/ Registers holding a function return value, in order. See ret_regs_cnt(). / +static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 }; + +static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env env) +{ + const struct btf btf = env->prog->aux->btf; + const struct btf_type type; + int subprog; + u32 size; + + if (!env->prog->jit_requested \|\| bpf_prog_is_offloaded(env->prog->aux)) + return 0; + + /* + * Skip the main program: its return value is the program's exit code, + * read out of R0, so it never uses the register pair. An extension does + * have a real prototype for subprog 0, but bpf_check_attach_target() + * refuses to replace a function returning more than 8 bytes. + / + for (subprog = 1; subprog < env->subprog_cnt; subprog++) { + type = subprog_ret_type(env, subprog); + if (!type \|\| btf_type_is_void(type)) + continue; + if (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, &size)), env, + "cannot size return type of subprog %d", subprog)) + return -EFAULT; + if (ret_regs_cnt(size) > 1) { + subprog_info(env, subprog)->ret_reg_pair = true; + env->prog->jit_required = 1; + } + } + + return 0; } const char bpf_subprog_name(const struct bpf_verifier_env env, int subprog) @@ -2877,6 +2919,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env env, u32 func_id, u16 offset) err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model); if (err) return err; + if (func_model.ret_size > 8) { + if (kfunc.flags && (kfunc.flags & KF_FASTCALL)) { + verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n", + kfunc.name); + return -EOPNOTSUPP; + } + if (!bpf_jit_supports_kfunc_ret_reg_pair()) { + verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n", + kfunc.name); + return -EOPNOTSUPP; + } + } memset(&meta, 0, sizeof(meta)); meta.btf = kfunc.btf; @@ -9853,6 +9907,7 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn, u16 callee_incoming, stack_arg_cnt; struct bpf_func_state caller; int err, subprog, target_insn; + u32 i, nregs; target_insn = insn_idx + insn->imm + 1; subprog = bpf_find_subprog(env, target_insn); @@ -9908,9 +9963,14 @@ static int check_func_call(struct bpf_verifier_env env, struct bpf_insn insn, clear_caller_saved_regs(env, caller->regs); invalidate_outgoing_stack_args(env, cur_func(env)); - / All non-void global functions return a 64-bit SCALAR_VALUE. / + / + * A non-void global function returns a 64-bit SCALAR_VALUE in + * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair. + / if (!returns_void) { - mark_reg_unknown(env, caller->regs, BPF_REG_0); + nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1; + for (i = 0; i < nregs; i++) + mark_reg_unknown(env, caller->regs, ret_regs[i]); bpf_diag_mod_end(env); } @@ -10270,11 +10330,15 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx) struct bpf_func_state caller, callee; struct bpf_reg_state r0; bool in_callback_fn; + u32 i, nregs; int err; callee = state->frame[state->curframe]; r0 = &callee->regs[BPF_REG_0]; - if (r0->type == PTR_TO_STACK) { + nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1; + for (i = 0; i < nregs; i++) { + if (callee->regs[ret_regs[i]].type != PTR_TO_STACK) + continue; /* technically it's ok to return caller's stack pointer * (or caller's caller's pointer) back to the caller, * since these pointers are valid. Only current stack @@ -10309,9 +10373,13 @@ static int prepare_func_exit(struct bpf_verifier_env env, int insn_idx) return -EFAULT; } } else { - /* return to the caller whatever r0 had in the callee / + / + * return to the caller whatever the callee had in the + * return register(s) + / bpf_diag_mod_begin(env, &caller->regs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE); - caller->regs[BPF_REG_0] = r0; + for (i = 0; i < nregs; i++) + caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]]; bpf_diag_mod_end(env); } @@ -11246,6 +11314,19 @@ static int check_helper_call(struct bpf_verifier_env env, struct bpf_insn insn return 0; } +/* + * Mark the register(s) holding a @size byte kfunc return value as unknown + * scalars. Both halves of a register pair are treated the same way. + / +static void mark_kfunc_ret_regs(struct bpf_verifier_env env, + struct bpf_reg_state regs, u32 size) +{ + u32 i, nregs = ret_regs_cnt(size); + + for (i = 0; i < nregs; i++) + mark_reg_unknown(env, regs, ret_regs[i]); +} + static bool is_kfunc_acquire(struct bpf_call_arg_meta meta) { return meta->kfunc_flags & KF_ACQUIRE; @@ -11513,9 +11594,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta meta, u32 arg_ } / Returns true if struct is composed of scalars, 4 levels of nesting allowed / -static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env, - const struct btf btf, - const struct btf_type t, int rec) +bool btf_type_is_scalar_struct(struct bpf_verifier_env env, + const struct btf btf, + const struct btf_type t, int rec) { const struct btf_type member_type; const struct btf_member member; @@ -11533,7 +11614,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env env, verbose(env, "max struct nesting depth exceeded\n"); return false; } - if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1)) + if (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1)) return false; continue; } @@ -11932,7 +12013,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta, (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) \|\| is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) { if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) && - !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n", reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); return -EINVAL; @@ -11948,7 +12029,7 @@ get_kfunc_arg_type(struct bpf_verifier_env env, struct bpf_call_arg_meta meta, * scalars. The access size is derived from the pointed-to BTF type. / if (!btf_type_is_scalar(ref_t) && - !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n", reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname); return -EINVAL; @@ -13004,7 +13085,7 @@ static int check_kfunc_args(struct bpf_verifier_env env, struct bpf_call_arg_me break; } - if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { + if (!btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) { enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id); const char expected_type; @@ -13546,7 +13627,7 @@ static int check_special_kfunc(struct bpf_verifier_env env, struct bpf_call_arg struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id); if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) { - if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) { + if (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) { verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n"); return -EINVAL; } @@ -13914,10 +13995,25 @@ static int check_kfunc_call(struct bpf_verifier_env env, struct bpf_insn insn, } if (btf_type_is_scalar(t)) { - mark_reg_unknown(env, regs, BPF_REG_0); + mark_kfunc_ret_regs(env, regs, t->size); if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] \|\| meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave])) __mark_reg_const_zero(env, &regs[BPF_REG_0]); + } else if (btf_type_is_struct(t)) { + /* + * The returned struct comes back as raw register bits modeled + * as an unknown scalar, so it must contain only scalars: + * otherwise a pointer field would be laundered into a scalar + * and escape provenance and reference tracking. + / + if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) { + verbose(env, + "kernel function %s returns %s %s that is not composed of scalars\n", + func_name, btf_type_str(t), + btf_name_by_offset(desc_btf, t->name_off)); + return -EINVAL; + } + mark_kfunc_ret_regs(env, regs, t->size); } else if (btf_type_is_ptr(t)) { ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id); err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf); @@ -17453,37 +17549,54 @@ static int check_return_code(struct bpf_verifier_env env, int regno, const char return 0; } -static int check_global_subprog_return_code(struct bpf_verifier_env env) +static int check_global_ret_scalar_reg(struct bpf_verifier_env env, u32 regno) { - struct bpf_reg_state reg = reg_state(env, BPF_REG_0); - struct bpf_func_state cur_frame = cur_func(env); + struct bpf_reg_state reg; int err; - if (subprog_returns_void(env, cur_frame->subprogno)) - return 0; - - err = check_reg_arg(env, BPF_REG_0, SRC_OP); + err = check_reg_arg(env, regno, SRC_OP); if (err) return err; / Pointers to arena are safe to pass between subprograms. / - if (is_arena_reg(env, BPF_REG_0)) + if (is_arena_reg(env, regno)) return 0; - if (is_pointer_value(env, BPF_REG_0)) { - verbose(env, "R%d leaks addr as return value\n", BPF_REG_0); + if (is_pointer_value(env, regno)) { + verbose(env, "R%d leaks addr as return value\n", regno); return -EACCES; } + reg = reg_state(env, regno); if (reg->type != SCALAR_VALUE) { - verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n", - reg_type_str(env, reg->type)); + verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n", + regno, reg_type_str(env, reg->type)); return -EINVAL; } return 0; } +static int check_global_subprog_return_code(struct bpf_verifier_env env) +{ + struct bpf_func_state cur_frame = cur_func(env); + u32 subprog = cur_frame->subprogno; + u32 i, nregs; + int err; + + if (subprog_returns_void(env, subprog)) + return 0; + + nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1; + for (i = 0; i < nregs; i++) { + err = check_global_ret_scalar_reg(env, ret_regs[i]); + if (err) + return err; + } + + return 0; +} + / Bitmask with 1s for all caller saved registers / #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1) @@ -21151,6 +21264,11 @@ int bpf_check(struct bpf_prog prog, union bpf_attr attr, bpfptr_t uattr, if (ret < 0) goto skip_full_check; + /* must precede the first bpf_ret_reg_pair() user below / + ret = bpf_compute_subprog_ret_regs(env); + if (ret < 0) + goto skip_full_check; + ret = bpf_compute_live_registers(env); if (ret < 0) goto skip_full_check; diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c new file mode 100644 index 0000000000000..e0b94ed10f941 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c @@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <test_progs.h> +#include "aggregate_ret_func.skel.h" +#include "aggregate_ret_kfunc.skel.h" + +void test_aggregate_ret(void) +{ + RUN_TESTS(aggregate_ret_func); + RUN_TESTS(aggregate_ret_kfunc); +} diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c index 2523c07a16c65..6262c81cdaa9f 100644 --- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c +++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c @@ -441,6 +441,20 @@ static void test_func_replace_int_with_void(void) " doesn't match type INT of global_func2()"); } +static void test_func_replace_ret_pair(void) +{ + const char msg = "Return type of new_agg_ret_target_func() has size 8 " + "and of agg_ret_target_func() size 16"; + + /* + * An extension cannot replace a function whose return value comes back + * in the R0:R2 pair: only R0 is checked at the extension's exit, so it + * would leave R2 stale for the target's callers. + / + test_obj_load_failure_common("freplace_ret_pair.bpf.o", + "./aggregate_ret_target.bpf.o", msg); +} + static int find_prog_btf_id(const char name, __u32 attach_prog_fd) { struct bpf_prog_info info = {}; @@ -660,6 +674,8 @@ void serial_test_fexit_bpf2bpf(void) test_func_replace_progmap(); if (test__start_subtest("freplace_int_with_void")) test_func_replace_int_with_void(); + if (test__start_subtest("freplace_ret_pair")) + test_func_replace_ret_pair(); if (test__start_subtest("freplace_void")) test_func_replace_void(); if (test__start_subtest("sleepable_fentry_to_xdp")) diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 64ac49ad67e63..f7f94ccebce27 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -5,6 +5,7 @@ #include "arena_kfunc.skel.h" #include "arena_kfunc_jit.skel.h" #include "cap_helpers.h" +#include "verifier_aggregate_ret.skel.h" #include "verifier_align.skel.h" #include "verifier_and.skel.h" #include "verifier_arena.skel.h" @@ -170,6 +171,7 @@ void test_arena_kfunc(void) { RUN_TESTS(arena_kfunc); } void test_arena_kfunc_jit(void) { RUN_TESTS(arena_kfunc_jit); } +void test_verifier_aggregate_ret(void) { RUN_TESTS(verifier_aggregate_ret); } void test_verifier_align(void) { RUN(verifier_align); } void test_verifier_and(void) { RUN(verifier_and); } void test_verifier_arena(void) { RUN(verifier_arena); } diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c new file mode 100644 index 0000000000000..6f66fc822cede --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c @@ -0,0 +1,237 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +typedef unsigned __int128 u128; + +__naked u128 global_agg_good(void) +{ + asm volatile ( + "r0 = 0x1234;" / low 64 bits / + "r2 = 0x5678;" / high 64 bits / + "exit;" + ); +} + +__naked u128 global_agg_bad(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +__naked u128 global_agg_bad_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = r10;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_global_fail(void) +{ + asm volatile ( + "call %[global_agg_bad];" + "r0 = r2;" + "exit;" + : + : __imm(global_agg_bad) + : __clobber_all); +} + +SEC("tc") +__load_if_JITed() +__failure __msg("At subprogram exit the register R2 is not a scalar value") +__naked int aggregate_ret_global_ptr_fail(void) +{ + asm volatile ( + "call %[global_agg_bad_ptr];" + "r0 = r2;" + "exit;" + : + : __imm(global_agg_bad_ptr) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_bad_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = r10;" / stack pointer placed in the second return register / + "exit;" + ); +} + +/ + * R2 is a return register once the subprogram returns a pair, so a stack + * pointer left in it is rejected at the callee's exit exactly as one in R0 + * is: the callee frame is gone by the time the caller could use it. + / +SEC("tc") +__load_if_JITed() +__failure __msg("cannot return stack pointer to the caller") +__naked int aggregate_ret_static_ptr_fail(void) +{ + asm volatile ( + "call %[static_agg_bad_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(static_agg_bad_ptr) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_no_r2(void) +{ + asm volatile ( + "r0 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_static_uninit_fail(void) +{ + asm volatile ( + "call %[static_agg_no_r2];" + "r0 = r2;" + "exit;" + : + : __imm(static_agg_no_r2) + : __clobber_all); +} + +static __naked __noinline u128 static_agg_precise(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 4;" / second half; its value is made precise below / + "exit;" + ); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit") +__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4") +__naked int aggregate_ret_static_precise(void) +{ + asm volatile ( + "call %[static_agg_precise];" + "r6 = r2;" / derived from the aggregate's second half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(static_agg_precise) + : __clobber_all); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9") +__naked int aggregate_ret_global_precise(void) +{ + asm volatile ( + "call %[global_agg_good];" + "r6 = r2;" / derived from the aggregate's second half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(global_agg_good) + : __clobber_all); +} + +#if defined(__clang_major__) && __clang_major__ >= 23 + +/ A by-value struct that smuggles a pointer, which must be rejected. / +struct with_ptr { + void p; + __u64 x; +}; + +/* A by-value union that smuggles a pointer, which must be rejected too. / +union upair_with_ptr { + void p; + __u64 halves[2]; +}; + +__naked struct with_ptr global_ret_struct_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("Global function global_ret_struct_ptr() has unsupported return type") +__naked int aggregate_ret_global_struct_ptr_fail(void) +{ + asm volatile ( + "call %[global_ret_struct_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(global_ret_struct_ptr) + : __clobber_all); +} + +__naked union upair_with_ptr global_ret_union_ptr(void) +{ + asm volatile ( + "r0 = 0;" + "r2 = 0;" + "exit;" + ); +} + +SEC("tc") +__failure __msg("Global function global_ret_union_ptr() has unsupported return type") +__naked int aggregate_ret_global_union_ptr_fail(void) +{ + asm volatile ( + "call %[global_ret_union_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(global_ret_union_ptr) + : __clobber_all); +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c new file mode 100644 index 0000000000000..d6b422ae9784b --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c @@ -0,0 +1,121 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" +#include "../test_kmods/bpf_testmod_kfunc.h" + +/ + * Reference kfunc addresses to force those BTF to be emitted. Taking the address + * (rather than calling) avoids any dependence on the compiler lowering an + * __int128 or struct return value, which the BPF backend only supports from + * LLVM 23 on. + / +void __kfunc_btf_root(void) +{ + asm volatile ("" + : + : "r"(&bpf_kfunc_call_test_i128), + "r"(&bpf_kfunc_call_test_ret_fastcall), + "r"(&bpf_kfunc_call_test_ret_ptr), + "r"(&bpf_kfunc_call_test_ret_ii), + "r"(&bpf_kfunc_call_test_ret_big)); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +__log_level(2) +__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1") +__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8") +__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7") +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2") +__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128") +__naked int aggregate_ret_kfunc_precise(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + "call %[bpf_kfunc_call_test_i128];" + "r6 = r2;" / second return half / + "r6 &= 7;" / keep it in [0, 7] to index the stack / + "r1 = r10;" + "r1 += -8;" + "r1 += r6;" / ptr += scalar marks r6 (hence R2) precise / + "r0 = 0;" + "(u8 )(r1 + 0) = r0;" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_i128) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL") +__naked int aggregate_ret_kfunc_fastcall_fail(void) +{ + asm volatile ( + "r1 = 1;" + "r2 = 2;" + "call %[bpf_kfunc_call_test_ret_fastcall];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_fastcall) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("is not composed of scalars") +__naked int aggregate_ret_kfunc_ptr_fail(void) +{ + asm volatile ( + "r1 = 0;" + "call %[bpf_kfunc_call_test_ret_ptr];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_ptr) + : __clobber_all); +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("R2 !read_ok") +__naked int aggregate_ret_kfunc_small_no_r2(void) +{ + asm volatile ( + "r1 = 0;" + "r2 = 0;" + "call %[bpf_kfunc_call_test_ret_ii];" + "r0 = r2;" / R2 is not a return register for a <=8 byte struct / + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_ii) + : __clobber_all); +} + +/ + * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by + * btf_distill_func_proto() before the KF_FASTCALL and JIT-capability checks. + / +SEC("tc") +__arch_x86_64 __arch_arm64 +__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported") +__naked int aggregate_ret_kfunc_too_big_fail(void) +{ + asm volatile ( + "call %[bpf_kfunc_call_test_ret_big];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_kfunc_call_test_ret_big) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c new file mode 100644 index 0000000000000..cffd8d7d3241a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +/ freplace target: a global subprogram returning 16 bytes in R0:R2. / +__naked unsigned __int128 agg_ret_target_func(void) +{ + asm volatile ( + "r0 = 0x1234;" + "r2 = 0x5678;" + "exit;" + ); +} + +SEC("tc") +__naked int agg_ret_target(void) +{ + asm volatile ( + "call %[agg_ret_target_func];" + "r0 = 0;" + "exit;" + : + : __imm(agg_ret_target_func) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c index d055fc7b3b95d..338e67cd8856a 100644 --- a/tools/testing/selftests/bpf/progs/compute_live_registers.c +++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c @@ -431,6 +431,34 @@ __naked void subprog1(void) ::: __clobber_all); } +static __used __naked unsigned __int128 aux2(void) +{ + asm volatile ( + "r0 = 1;" + "r2 = 2;" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +/ A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). / +__load_if_JITed() +__log_level(2) +__msg("0: .12345.... (85) call pc+2") +__msg("1: ..2....... (bf) r0 = r2") +__msg("2: 0......... (95) exit") +__msg("3: .......... (b7) r0 = 1") +__msg("4: 0......... (b7) r2 = 2") +__msg("5: 0.2....... (95) exit") +__naked void subprog_ret_reg_pair(void) +{ + asm volatile ( + "call aux2;" + "r0 = r2;" + "exit;" + ::: __clobber_all); +} + #if defined(__TARGET_ARCH_x86) \|\| defined(__TARGET_ARCH_arm64) SEC("socket") diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c index ac44d60e50666..9708efb93683b 100644 --- a/tools/testing/selftests/bpf/progs/exceptions_fail.c +++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c @@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a) SEC("?tc") __exception_cb(exception_cb_bad_ret_type1) -__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.") +__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.") int reject_exception_cb_type_1(struct __sk_buff ctx) { bpf_throw(0); diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c new file mode 100644 index 0000000000000..12c15d293bd79 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> + +SEC("freplace/agg_ret_target_func") +__u64 new_agg_ret_target_func(void) +{ + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c new file mode 100644 index 0000000000000..7851bade2b406 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c @@ -0,0 +1,178 @@ +// SPDX-License-Identifier: GPL-2.0 +/ Copyright (c) 2026 Meta Platforms, Inc. and affiliates. / +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "../test_kmods/bpf_testmod_kfunc.h" +#include "bpf_misc.h" + +#if defined(__clang_major__) && __clang_major__ >= 23 + +#define MIX_A 0xdeadbeefcafef00dULL +#define MIX_B 0x0123456789abcdefULL + +typedef unsigned __int128 u128; + +struct pair { + __u64 lo; / R0 / + __u64 hi; / R2 / +}; + +union upair { + __u64 halves[2]; + struct { + __u64 lo; / R0 / + __u64 hi; / R2 / + } parts; +}; + +static __noinline u128 make_i128(__u64 a, __u64 b) +{ + return ((u128)(a + b) << 64) \| (a - b); +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_int128_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + u128 v; + + v = make_i128(a, b); + if ((__u64)(v >> 64) != a + b) + return 1; + if ((__u64)v != a - b) + return 2; + + return 0; +} + +static __noinline struct pair make_pair(__u64 a, __u64 b) +{ + struct pair p = { .lo = a + b, .hi = a - b }; + + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct pair p; + + p = make_pair(a, b); + if (p.lo != a + b) + return 1; + if (p.hi != a - b) + return 2; + + return 0; +} + +__noinline struct pair make_pair_global(__u64 a, __u64 b) +{ + struct pair p = { .lo = a + b, .hi = a - b }; + + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_global_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct pair p; + + p = make_pair_global(a, b); + if (p.lo != a + b) + return 1; + if (p.hi != a - b) + return 2; + + return 0; +} + +static __noinline union upair make_upair(__u64 a, __u64 b) +{ + union upair p; + + p.halves[0] = a + b; + p.halves[1] = a - b; + return p; +} + +SEC("tc") +__load_if_JITed() +__success __retval(0) +int aggregate_ret_union_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + union upair p; + + p = make_upair(a, b); + if (p.parts.lo != a + b) + return 1; + if (p.parts.hi != a - b) + return 2; + + return 0; +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_int128_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + u128 v; + + v = bpf_kfunc_call_test_i128(a, b); + if ((__u64)(v >> 64) != a + b) + return 1; + if ((__u64)v != a - b) + return 2; + + return 0; +} + +SEC("tc") +__arch_x86_64 __arch_arm64 +__load_if_JITed() +__success __retval(0) +int aggregate_ret_kfunc_struct_c_test(struct __sk_buff skb) +{ + __u64 a = skb->len ^ MIX_A; + __u64 b = skb->len ^ MIX_B; + struct prog_test_ret_pair p; + + p = bpf_kfunc_call_test_ret_pair(a, b); + if (p.lo != a + b) + return 1; + if (p.hi != a - b) + return 2; + + return 0; +} + +#else + +SEC("socket") +__description("verifier_aggregate_ret: needs LLVM 23, dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c index 9366a3c578f13..850cf4f830c48 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c @@ -954,6 +954,53 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c) return 0; } +#if defined(__x86_64__) \|\| defined(__aarch64__) +__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b) +{ + return (__int128)(((unsigned __int128)(a + b) << 64) \| (a - b)); +} + +__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b) +{ + struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b) +{ + struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag) +{ + struct prog_test_ret_ptr r = { .p = NULL, .tag = tag }; + + return r; +} + +__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) +{ + struct prog_test_ret_ii r = { .a = a, .b = b }; + + return r; +} +#endif / __x86_64__ \|\| __aarch64__ / + +/ + * Takes no argument on purpose: with no arguments there is nothing for the sret + * pointer to displace, so this needs no architecture guard even though it + * returns 24 bytes. + / +__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) +{ + struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 }; + + return r; +} + __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d, u64 e, u64 f, u64 g, u64 h, u64 i, u64 j) @@ -1487,6 +1534,14 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2) BTF_ID_FLAGS(func, bpf_kfunc_call_test3) BTF_ID_FLAGS(func, bpf_kfunc_call_test4) BTF_ID_FLAGS(func, bpf_kfunc_call_test5) +#if defined(__x86_64__) \|\| defined(__aarch64__) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii) +#endif +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr) BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix) diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h index 7d81070eefe7f..65e693ada7364 100644 --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h @@ -55,6 +55,27 @@ struct prog_test_big_arg { __u64 b; }; +struct prog_test_ret_pair { / 16 bytes: R0:R2 / + __u64 lo; + __u64 hi; +}; + +struct prog_test_ret_ii { / 8 bytes: R0 only / + int a; + int b; +}; + +struct prog_test_ret_ptr { / 16 bytes: contains a pointer / + void p; + __u64 tag; +}; + +struct prog_test_ret_big { /* 24 bytes: too large for R0:R2 / + __u64 a; + __u64 b; + __u64 c; +}; + struct prog_test_fail1 { void p; int x; @@ -131,6 +152,14 @@ int bpf_kfunc_call_test2(struct sock sk, __u32 a, __u32 b) __ksym; struct sock bpf_kfunc_call_test3(struct sock *sk) __ksym; long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym; +#ifdef __SIZEOF_INT128__ +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym; +#endif +struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym; +struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym; +struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym; +struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym; +struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym; __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, __u64 g, __u64 h, __u64 i, __u64 j) __ksym;
11/3	2026/08/19 06:42	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 24920 output: 292 thoughts: 3643 Thoughts: KMSAN Applicability Assessment for BPF 16-Byte Return Value Support Okay, so this patch series enables kfuncs and subprograms to return values up to 16 bytes, using the register pair R0:R2. The key changes are in documentation, architecture-specific JIT support, and, most importantly, the verifier. The verifier now handles R2 precision for subprogram calls/exits, supports 16-byte return types in `btf_distill_func_proto` and scalar-only structs, marks R2 as used on `BPF_EXIT` if `ret_reg_pair` is true, and updates several functions (`check_func_call`, `prepare_func_exit`, `check_kfunc_call`, `check_global_subprog_return_code`) to correctly handle R0 and R2. The verifier also ensures that returned structs only contain scalars and handles cases where the JIT might leave uninitialized data in registers. The central question is whether this introduces uninitialized memory risks, and specifically, if KMSAN fuzzing is required. My analysis centers around the verifier's role in preventing leaks and how KMSAN interacts with this. When kfuncs or subprograms return a struct up to 16 bytes, the verifier marks R0 and R2 as unknown scalars. For global subprograms, R0 and R2 are checked to be initialized. For static subprograms, the verifier will reject a program that reads an uninitialized R2. The problem becomes apparent with padding within structs returned by value. If a kfunc returns a struct with padding, the C compiler might leave that padding uninitialized in the return registers (RAX and RDX on x86, which map to R0 and R2). The verifier treats these registers as fully initialized, from the BPF program's perspective. Thus, if the BPF program copies these registers to a map or sends them to userspace, there could be an uninitialized kernel memory leak. Now, does KMSAN catch this? KMSAN detects reads of uninitialized memory and info-leaks. KFuncs are kernel functions and if they were to return a struct by value with padding that had uninitialized values, KMSAN should catch that if the function is instrumented. The verifier ensures the structs returned contain only scalar values and doesn't explicitly check for padding. However, the critical caveat is that KMSAN doesn't instrument JIT-ed BPF code. Since this feature is JIT-only, KMSAN won't track the values in R0/R2 once they are used in the BPF program, if they leak there, so a potential leak of uninitialized values is invisible to KMSAN. The verifier does have checks that a subprogram must leave a scalar in every register of the pair to avoid potential leaks and it checks for pointers as well. I need to consider `bpf_compute_subprog_ret_regs`, which might inadvertently make assumptions about initialization, and if a 16-byte struct is returned, does the JIT clear bits it shouldn't. Now, let's consider the KMSAN applicability. The patch primarily modifies the verifier and JIT support for returning 16-byte values. It does not introduce new kernel structures that are directly sent to user space. The verifier prevents BPF programs from reading uninitialized registers. If a BPF program tries to read an uninitialized R2, the verifier will reject it. However, if there is a bug, and a BPF program leaks data from a kfunc return, KMSAN may not even catch it because the read happens in JIT-ed BPF code, which KMSAN doesn't instrument. Looking at the KMSAN rules: this patch does not introduce new kernel structures to userspace with uninitialized fields. It doesn't depend on uninitialized variables in C code. It modifies the verifier and BPF code generation. Verifier bugs could permit a BPF program to read uninitialized registers or leak stale pointers. KMSAN doesn't track BPF register state or BPF code execution directly and it wouldn't be the right tool to catch logical bugs in the verifier. There is no C code that allocates buffers or copies data to userspace. All this points to the conclusion that a bug here would result in a verifier bypass; however, such a vulnerability would be a logical bug in the verifier or the BPF JIT code, and wouldn't involve kernel data structures or KMSAN-detectable issues. Therefore, NeedsKMSAN=false.
12/3	2026/08/19 06:43	tool	set-results	0m Args: { "KMSANReasoning": "The patch series introduces support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or small structs) in the register pair R0:R2. The changes are confined to the BPF verifier's static analysis (tracking liveness and initialization state of R2, type checking BTF return types) and architecture-specific JIT compilers (moving the native second return register into BPF R2). \n\nThere are no allocations of complex data structures, no new kernel-to-user memory copies, and no modifications to bounds checking of memory buffers. The verifier statically enforces that BPF programs cannot read uninitialized BPF registers, and any logical bugs in this tracking would lead to verifier bypasses rather than C-level uninitialized memory uses. Furthermore, KMSAN instruments kernel C code and tracks memory, but it does not instrument JIT-ed BPF code or track BPF register states directly. Therefore, any potential issues introduced by this patch (such as verifier flaws allowing stale register reads) would not be detected by KMSAN. Standard KASAN, verifier tests, and syzkaller's BPF fuzzing are the appropriate tools for validating these changes.", "NeedsKMSAN": false } Results: { "KMSANReasoning": "The patch series introduces support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or small structs) in the register pair R0:R2. The changes are confined to the BPF verifier's static analysis (tracking liveness and initialization state of R2, type checking BTF return types) and architecture-specific JIT compilers (moving the native second return register into BPF R2). \n\nThere are no allocations of complex data structures, no new kernel-to-user memory copies, and no modifications to bounds checking of memory buffers. The verifier statically enforces that BPF programs cannot read uninitialized BPF registers, and any logical bugs in this tracking would lead to verifier bypasses rather than C-level uninitialized memory uses. Furthermore, KMSAN instruments kernel C code and tracks memory, but it does not instrument JIT-ed BPF code or track BPF register states directly. Therefore, any potential issues introduced by this patch (such as verifier flaws allowing stale register reads) would not be detected by KMSAN. Standard KASAN, verifier tests, and syzkaller's BPF fuzzing are the appropriate tools for validating these changes.", "NeedsKMSAN": false }

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT"
  ],
  "FocusSymbols": [
    "btf_type_is_scalar_struct",
    "btf_distill_func_proto",
    "btf_prepare_func_args",
    "bpf_compute_live_registers",
    "bpf_jit_supports_kfunc_ret_reg_pair"
  ],
  "KMSANReasoning": "The patch series introduces support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or small structs) in the register pair R0:R2. The changes are confined to the BPF verifier's static analysis (tracking liveness and initialization state of R2, type checking BTF return types) and architecture-specific JIT compilers (moving the native second return register into BPF R2). \n\nThere are no allocations of complex data structures, no new kernel-to-user memory copies, and no modifications to bounds checking of memory buffers. The verifier statically enforces that BPF programs cannot read uninitialized BPF registers, and any logical bugs in this tracking would lead to verifier bypasses rather than C-level uninitialized memory uses. Furthermore, KMSAN instruments kernel C code and tracks memory, but it does not instrument JIT-ed BPF code or track BPF register states directly. Therefore, any potential issues introduced by this patch (such as verifier flaws allowing stale register reads) would not be detected by KMSAN. Standard KASAN, verifier tests, and syzkaller's BPF fuzzing are the appropriate tools for validating these changes.",
  "NeedsKMSAN": false,
  "Reasoning": "The patch series adds support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or scalar-only structs/unions) in the register pair R0:R2. This involves modifications to the BPF verifier to track the return values in both registers, changes to the liveness and backtracking logic, and updates to the JIT compilers (x86, arm64, riscv) to support this new calling convention. These are functional changes to reachable core kernel logic (BPF subsystem) that should be fuzzed to ensure the verifier correctly tracks the register states and prevents any leaks or invalid memory accesses.",
  "WorthFuzzing": true
}

{
  "PatchDiff": "commit cc236f68937f8479bce51601bd02ac7647affe1d\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate:   Wed Aug 19 06:41:58 2026 +0000\n\n    syz-cluster: applied patch under review\n\ndiff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst\nindex 85f73e0bbd0ff..89dea6b0b0241 100644\n--- a/Documentation/bpf/kfuncs.rst\n+++ b/Documentation/bpf/kfuncs.rst\n@@ -575,6 +575,71 @@ is also covered by this recovery. A kfunc handed an arena pointer may\n therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking\n against the arena. Larger accesses must verify the range explicitly.\n \n+2.9 kfunc Return Values\n+-----------------------\n+\n+A kfunc may return a scalar, a pointer, or a small struct or union by\n+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.\n+\n+A struct or union returned by value must be composed only of scalars\n+(recursively), where a scalar is an integer or an enum; arrays of scalars are\n+allowed as members. Its bytes are handed back to the program as the raw\n+contents of R0 (and R2), so a pointer field would be laundered into a scalar\n+and escape the verifier's pointer provenance and reference tracking. A struct\n+or union with a pointer member is therefore rejected at load time, and so is\n+one with a floating-point member, which the ABI may not return in R0:R2 at\n+all.\n+\n+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a\n+scalar-only struct or union, or an ``__int128``. Such a value is returned\n+in the register pair R0:R2, matching the convention LLVM uses for the BPF\n+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or\n+union of 8 bytes or less is returned in R0 alone.\n+\n+::\n+\n+        struct bpf_pair { __u64 a, b; };   /* 16 bytes */\n+\n+        __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)\n+        {\n+                struct bpf_pair p = { .a = 1, .b = 2 };\n+\n+                return p;      /* p.a in R0, p.b in R2 */\n+        }\n+\n+Returning a value in the R0:R2 pair requires the JIT to place the second\n+half of the return value into R2, which not every architecture supports\n+right now. A kfunc with a return value larger than 8 bytes is therefore\n+rejected at load time on a JIT that does not advertise this capability (see\n+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run\n+by the interpreter. A return value larger than 16 bytes is not supported.\n+\n+The same R0:R2 convention applies to a BPF subprogram, global or static, that\n+returns an ``__int128`` or a struct or union larger than 8 bytes. It is only\n+used when the program is JITed, since the interpreter propagates only R0 out of\n+a subprogram: without a JIT the return value stays in R0 alone, and a caller\n+reading R2 is rejected for reading an uninitialized register. A global\n+subprogram is verified in isolation, so its by-value struct or union return is\n+restricted to scalars just like a kfunc's; a static subprogram is verified\n+inline and has no such restriction. The main program is not covered: its return\n+value is the program's exit code, read out of R0 alone, so a declared upper\n+half is never looked at.\n+\n+A global subprogram must leave a scalar in *every* register of the pair, so\n+both halves of the returned value have to be assigned. Leaving the upper half\n+uninitialized is not merely untidy: the compiler is then free to leave R2\n+holding whatever it happened to hold, which for a subprogram taking a pointer\n+argument is typically that pointer. Handing the caller an unknown scalar built\n+from a pointer is a leak, so the verifier rejects it with::\n+\n+        At subprogram exit the register R2 is not a scalar value (...)\n+\n+Initialize the whole return value, for example ``struct pair p = {};``, to\n+avoid this. A static subprogram is exempt from the scalar-only rule: it is\n+verified inline, so an unassigned R2 is simply passed back to the caller as\n+uninitialized and only a caller that reads it fails. A stack pointer left in\n+R2 is still rejected there, just as one in R0 is.\n+\n .. _BPF_kfunc_lifecycle_expectations:\n \n 3. kfunc lifecycle expectations\ndiff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c\nindex c18e005a41dbe..3aa3ea0bc30bb 100644\n--- a/arch/arm64/net/bpf_jit_comp.c\n+++ b/arch/arm64/net/bpf_jit_comp.c\n@@ -2388,6 +2388,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_stack_args(void)\n {\n \treturn true;\ndiff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c\nindex 74efe4b138d2a..47c7bf431ba85 100644\n--- a/arch/riscv/net/bpf_jit_comp64.c\n+++ b/arch/riscv/net/bpf_jit_comp64.c\n@@ -2121,6 +2121,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_ptr_xchg(void)\n {\n \treturn true;\ndiff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c\nindex 1a9fb530adc3c..48429fae06410 100644\n--- a/arch/x86/net/bpf_jit_comp.c\n+++ b/arch/x86/net/bpf_jit_comp.c\n@@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,\n  * arena NULL is offset 0. Return the number of emitted bytes.\n  */\n static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,\n-\t\t\t\t const struct bpf_insn *insn, u8 **pprog)\n+\t\t\t\t const struct btf_func_model *fm, u8 **pprog)\n {\n-\tconst struct btf_func_model *fm;\n \tu8 *prog = *pprog;\n \tu8 *start = prog;\n \tint i;\n \n-\tfm = bpf_jit_find_kfunc_model(bpf_prog, insn);\n-\tif (!fm)\n-\t\treturn -EINVAL;\n-\n \tfor (i = 0; i \u003c min_t(int, fm-\u003enr_args, MAX_BPF_FUNC_REG_ARGS); i++) {\n \t\tu8 flags = fm-\u003earg_flags[i];\n \t\tu32 reg = BPF_REG_1 + i;\n@@ -2644,6 +2639,8 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \n \t\t\t/* call */\n \t\tcase BPF_JMP | BPF_CALL: {\n+\t\t\tconst struct btf_func_model *fm = NULL;\n+\n \t\t\tfunc = (u8 *) __bpf_call_base + imm32;\n \t\t\tif (src_reg == BPF_PSEUDO_CALL \u0026\u0026 tail_call_reachable) {\n \t\t\t\tLOAD_TAIL_CALL_CNT_PTR(stack_depth);\n@@ -2652,7 +2649,10 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tif (!imm32)\n \t\t\t\treturn -EINVAL;\n \t\t\tif (src_reg == BPF_PSEUDO_KFUNC_CALL) {\n-\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, insn, \u0026prog);\n+\t\t\t\tfm = bpf_jit_find_kfunc_model(bpf_prog, insn);\n+\t\t\t\tif (!fm)\n+\t\t\t\t\treturn -EINVAL;\n+\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, fm, \u0026prog);\n \t\t\t\tif (err \u003c 0)\n \t\t\t\t\treturn err;\n \t\t\t\tip += err;\n@@ -2666,6 +2666,14 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\treturn -EINVAL;\n \t\t\tif (priv_frame_ptr)\n \t\t\t\tpop_r9(\u0026prog);\n+\t\t\t/*\n+\t\t\t * A kfunc returning more than 8 bytes hands the second\n+\t\t\t * half back in RDX (the native ABI's second return reg),\n+\t\t\t * but BPF expects it in R0:R2. BPF R0 is RAX (no move\n+\t\t\t * needed), while BPF R2 is RSI, so copy RDX into RSI.\n+\t\t\t */\n+\t\t\tif (fm \u0026\u0026 fm-\u003eret_size \u003e 8)\n+\t\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_2, BPF_REG_3);\n \t\t\tbreak;\n \t\t}\n \n@@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_stack_args(void)\n {\n \treturn true;\ndiff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h\nindex 5fad59fdab0d0..004b06785521c 100644\n--- a/include/linux/bpf_verifier.h\n+++ b/include/linux/bpf_verifier.h\n@@ -819,6 +819,8 @@ struct bpf_subprog_info {\n \tbool is_async_cb: 1;\n \tbool is_exception_cb: 1;\n \tbool args_cached: 1;\n+\t/* true if the return value is passed in the R0:R2 register pair */\n+\tbool ret_reg_pair: 1;\n \t/* true if bpf_fastcall stack region is used by functions that can't be inlined */\n \tbool keep_fastcall_stack: 1;\n \tbool changes_pkt_data: 1;\n@@ -1055,6 +1057,11 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env\n \treturn \u0026env-\u003esubprog_info[subprog];\n }\n \n+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)\n+{\n+\treturn subprog_info(env, subprog)-\u003eret_reg_pair;\n+}\n+\n struct bpf_call_summary {\n \tu8 num_params;\n \tbool is_void;\n@@ -1481,6 +1488,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);\n struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);\n void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);\n bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);\n+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,\n+\t\t\t       const struct btf_type *t, int rec);\n \n int bpf_find_subprog(struct bpf_verifier_env *env, int off);\n bool bpf_is_throw_kfunc(struct bpf_insn *insn);\ndiff --git a/include/linux/filter.h b/include/linux/filter.h\nindex 4a9bc6a848f2e..6e746b0a09306 100644\n--- a/include/linux/filter.h\n+++ b/include/linux/filter.h\n@@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);\n bool bpf_jit_supports_subprog_tailcalls(void);\n bool bpf_jit_supports_percpu_insn(void);\n bool bpf_jit_supports_kfunc_call(void);\n+bool bpf_jit_supports_kfunc_ret_reg_pair(void);\n bool bpf_jit_supports_stack_args(void);\n bool bpf_jit_supports_arena_args(void);\n bool bpf_jit_supports_far_kfunc_call(void);\ndiff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c\nindex a2b18a9f1694c..653db80bcc47b 100644\n--- a/kernel/bpf/backtrack.c\n+++ b/kernel/bpf/backtrack.c\n@@ -423,6 +423,10 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\t\t */\n \t\t\t\tverifier_bug_if(idx + 1 != subseq_idx, env,\n \t\t\t\t\t\t\"extra insn from subprog\");\n+\t\t\t\t/* global subprog always sets R0 */\n+\t\t\t\tbt_clear_reg(bt, BPF_REG_0);\n+\t\t\t\t/* and if it does not set R2, main pass would catch it */\n+\t\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\t\t/* r1-r5 are invalidated after subprog call,\n \t\t\t\t * so for global func call it shouldn't be set\n \t\t\t\t * anymore\n@@ -432,8 +436,6 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\t\t\t\t     bt_reg_mask(bt));\n \t\t\t\t\treturn -EFAULT;\n \t\t\t\t}\n-\t\t\t\t/* global subprog always sets R0 */\n-\t\t\t\tbt_clear_reg(bt, BPF_REG_0);\n \t\t\t\treturn 0;\n \t\t\t} else {\n \t\t\t\t/* static subprog call instruction, which\n@@ -506,6 +508,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\t\treturn -ENOTSUPP;\n \t\t\t/* regular helper call sets R0 */\n \t\t\tbt_clear_reg(bt, BPF_REG_0);\n+\t\t\t/* kfunc might also set R2 */\n+\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\tif (bt_reg_mask(bt) \u0026 BPF_REGMASK_ARGS) {\n \t\t\t\t/* if backtracking was looking for registers R1-R5\n \t\t\t\t * they should have been found already.\n@@ -520,7 +524,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\t\t\treturn -EFAULT;\n \t\t\t}\n \t\t} else if (opcode == BPF_EXIT) {\n-\t\t\tbool r0_precise;\n+\t\t\tbool from_subprog_call, r0_precise, r2_precise;\n+\n+\t\t\t/* BPF_EXIT in subprog or callback always returns\n+\t\t\t * right after the call instruction, so by checking\n+\t\t\t * whether the instruction at subseq_idx-1 is subprog\n+\t\t\t * call or not we can distinguish actual exit from\n+\t\t\t * *subprog* from exit from *callback*. In the former\n+\t\t\t * case, we need to propagate the precision of the\n+\t\t\t * return registers, if necessary. In the latter we\n+\t\t\t * never do that.\n+\t\t\t */\n+\t\t\tfrom_subprog_call = subseq_idx - 1 \u003e= 0 \u0026\u0026\n+\t\t\t\t\t    bpf_pseudo_call(\u0026env-\u003eprog-\u003einsnsi[subseq_idx - 1]);\n+\n+\t\t\t/* Sample the return registers before the callback\n+\t\t\t * handling below clears R1-R5.\n+\t\t\t */\n+\t\t\tr0_precise = from_subprog_call \u0026\u0026 bt_is_reg_set(bt, BPF_REG_0);\n+\t\t\tr2_precise = from_subprog_call \u0026\u0026 bt_is_reg_set(bt, BPF_REG_2);\n \n \t\t\t/* Backtracking to a nested function call, 'idx' is a part of\n \t\t\t * the inner frame 'subseq_idx' is a part of the outer frame.\n@@ -533,30 +555,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\tif (subseq_idx \u003e= 0 \u0026\u0026 bpf_calls_callback(env, subseq_idx))\n \t\t\t\tfor (i = BPF_REG_1; i \u003c= BPF_REG_5; i++)\n \t\t\t\t\tbt_clear_reg(bt, i);\n+\n+\t\t\tbt_clear_reg(bt, BPF_REG_0);\n+\t\t\tbt_clear_reg(bt, BPF_REG_2);\n \t\t\tif (bt_reg_mask(bt) \u0026 BPF_REGMASK_ARGS) {\n \t\t\t\tverifier_bug(env, \"backtracking exit unexpected regs %x\",\n \t\t\t\t\t     bt_reg_mask(bt));\n \t\t\t\treturn -EFAULT;\n \t\t\t}\n \n-\t\t\t/* BPF_EXIT in subprog or callback always returns\n-\t\t\t * right after the call instruction, so by checking\n-\t\t\t * whether the instruction at subseq_idx-1 is subprog\n-\t\t\t * call or not we can distinguish actual exit from\n-\t\t\t * *subprog* from exit from *callback*. In the former\n-\t\t\t * case, we need to propagate r0 precision, if\n-\t\t\t * necessary. In the former we never do that.\n-\t\t\t */\n-\t\t\tr0_precise = subseq_idx - 1 \u003e= 0 \u0026\u0026\n-\t\t\t\t     bpf_pseudo_call(\u0026env-\u003eprog-\u003einsnsi[subseq_idx - 1]) \u0026\u0026\n-\t\t\t\t     bt_is_reg_set(bt, BPF_REG_0);\n-\n-\t\t\tbt_clear_reg(bt, BPF_REG_0);\n \t\t\tif (bt_subprog_enter(bt))\n \t\t\t\treturn -EFAULT;\n \n \t\t\tif (r0_precise)\n \t\t\t\tbt_set_reg(bt, BPF_REG_0);\n+\t\t\tif (r2_precise)\n+\t\t\t\tbt_set_reg(bt, BPF_REG_2);\n \t\t\t/* r6-r9 and stack slots will stay set in caller frame\n \t\t\t * bitmasks until we return back from callee(s)\n \t\t\t */\ndiff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c\nindex 6967d48bba49d..58b81c6f5288c 100644\n--- a/kernel/bpf/btf.c\n+++ b/kernel/bpf/btf.c\n@@ -7591,7 +7591,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,\n \t\treturn -EINVAL;\n \t}\n \tret = __get_type_size(btf, func-\u003etype, \u0026t);\n-\tif (ret \u003c 0 || btf_type_is_struct(t)) {\n+\tif (ret \u003c 0 || ret \u003e 16) {\n \t\tbpf_log(log,\n \t\t\t\"The function %s return type %s is unsupported.\\n\",\n \t\t\ttname, btf_type_str(t));\n@@ -7684,6 +7684,12 @@ static int btf_check_func_type_match(struct bpf_verifier_log *log,\n \t\t\tbtf_type_str(t2), fn2);\n \t\treturn -EINVAL;\n \t}\n+\tif (btf_type_has_size(t1) \u0026\u0026 (t1-\u003esize \u003e 8 || t2-\u003esize \u003e 8)) {\n+\t\tbpf_log(log,\n+\t\t\t\"Return type of %s() has size %u and of %s() size %u, and a size above 8 bytes cannot be replaced\\n\",\n+\t\t\tfn1, t1-\u003esize, fn2, t2-\u003esize);\n+\t\treturn -EINVAL;\n+\t}\n \n \tfor (i = 0; i \u003c nargs1; i++) {\n \t\tt1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL);\n@@ -7964,7 +7970,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env *env,\n \n /* Check whether the type is a valid return type. */\n static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,\n-\t\tconst struct btf_type *t, int subprog)\n+\t\tconst struct btf_type *t, int subprog, bool is_global)\n {\n \tu32 tags = 0;\n \tint err;\n@@ -7987,6 +7993,19 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt\n \tif (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))\n \t\treturn 0;\n \n+\tif (btf_type_is_struct(t) \u0026\u0026 t-\u003esize \u003c= 16) {\n+\t\t/*\n+\t\t * A global function's caller models the return as an opaque\n+\t\t * scalar pair, so it may only return scalars by value. A local\n+\t\t * function is verified inline, so a pointer field stays tracked\n+\t\t * and needs no such restriction.\n+\t\t */\n+\t\tbool local_func = subprog \u0026\u0026 !is_global;\n+\n+\t\tif (local_func || btf_type_is_scalar_struct(env, btf, t, 0))\n+\t\t\treturn 0;\n+\t}\n+\n \treturn -EOPNOTSUPP;\n }\n \n@@ -8074,12 +8093,12 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)\n \t\treturn -EINVAL;\n \t}\n \n-\terr = btf_validate_return_type(env, btf, t, subprog);\n+\terr = btf_validate_return_type(env, btf, t, subprog, is_global);\n \tif (err) {\n \t\tif (is_global) {\n \t\t\tbpf_log(log,\n-\t\t\t\t\"Global function %s() return value not void or scalar. \"\n-\t\t\t\t\"Only those are supported.\\n\",\n+\t\t\t\t\"Global function %s() has unsupported return type. \"\n+\t\t\t\t\"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\\n\",\n \t\t\t\ttname);\n \t\t}\n \t\treturn err;\ndiff --git a/kernel/bpf/core.c b/kernel/bpf/core.c\nindex d55e737ed75ad..5db77d7915dfc 100644\n--- a/kernel/bpf/core.c\n+++ b/kernel/bpf/core.c\n@@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)\n \treturn false;\n }\n \n+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)\n+{\n+\treturn false;\n+}\n+\n bool __weak bpf_jit_supports_stack_args(void)\n {\n \treturn false;\ndiff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c\nindex 74fc4b3f80d6e..71f998c6eb888 100644\n--- a/kernel/bpf/liveness.c\n+++ b/kernel/bpf/liveness.c\n@@ -2060,7 +2060,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m \u003e\u003e 16); }\n /* Compute info-\u003e{use,def} fields for the instruction */\n static void compute_insn_live_regs(struct bpf_verifier_env *env,\n \t\t\t\t   struct bpf_insn *insn,\n-\t\t\t\t   struct insn_live_regs *info)\n+\t\t\t\t   struct insn_live_regs *info,\n+\t\t\t\t   bool ret_reg_pair)\n {\n \tstruct bpf_call_summary cs;\n \tconst u8 class = BPF_CLASS(insn-\u003ecode);\n@@ -2072,6 +2073,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,\n \tconst u32 src32 = mask_lo(src);\n \tconst u32 dst32 = mask_lo(dst);\n \tconst u32 r0  = reg64_mask(0);\n+\tconst u32 r2  = reg64_mask(BPF_REG_2);\n \tu32 def = 0;\n \tu32 use = U32_MAX;\n \n@@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,\n \t\t\tbreak;\n \t\tcase BPF_EXIT:\n \t\t\tdef = 0;\n-\t\t\tuse = r0;\n+\t\t\tuse = ret_reg_pair ? (r0 | r2) : r0;\n \t\t\tbreak;\n \t\tcase BPF_CALL:\n \t\t\tdef = ALL_CALLER_SAVED_REGS;\n@@ -2228,8 +2230,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)\n \tstruct insn_live_regs *state;\n \tint insn_cnt = env-\u003eprog-\u003elen;\n \tu64 pos, insn_pos;\n-\tint err = 0, i, j;\n-\tbool changed;\n+\tint err = 0, i, j, subprog, start, end;\n+\tbool changed, ret_reg_pair;\n \n \t/* Use the following algorithm:\n \t * - define the following:\n@@ -2256,8 +2258,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)\n \t\tgoto out;\n \t}\n \n-\tfor (i = 0; i \u003c insn_cnt; ++i)\n-\t\tcompute_insn_live_regs(env, \u0026insns[i], \u0026state[i]);\n+\tfor (subprog = 0; subprog \u003c env-\u003esubprog_cnt; subprog++) {\n+\t\tstart = env-\u003esubprog_info[subprog].start;\n+\t\tend = env-\u003esubprog_info[subprog + 1].start;\n+\t\tret_reg_pair = bpf_ret_reg_pair(env, subprog);\n+\n+\t\tfor (i = start; i \u003c end; ++i)\n+\t\t\tcompute_insn_live_regs(env, \u0026insns[i], \u0026state[i], ret_reg_pair);\n+\t}\n \n \t/* Forward pass: resolve stack access through FP-derived pointers */\n \terr = bpf_compute_subprog_arg_access(env);\ndiff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c\nindex e421ea2b80c3a..7a939dfa3797b 100644\n--- a/kernel/bpf/verifier.c\n+++ b/kernel/bpf/verifier.c\n@@ -385,27 +385,69 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)\n \treturn aux \u0026\u0026 aux[subprog].linkage == BTF_FUNC_GLOBAL;\n }\n \n-static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)\n+static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)\n {\n-\tconst struct btf_type *type, *func, *func_proto;\n+\tconst struct btf_type *func, *func_proto;\n \tconst struct btf *btf = env-\u003eprog-\u003eaux-\u003ebtf;\n \tu32 btf_id;\n \n+\tif (!btf || !env-\u003eprog-\u003eaux-\u003efunc_info)\n+\t\treturn NULL;\n+\n \tbtf_id = env-\u003eprog-\u003eaux-\u003efunc_info[subprog].type_id;\n \n+\t/* Both already validated by prepare_btf_func() at prog load. */\n \tfunc = btf_type_by_id(btf, btf_id);\n-\tif (verifier_bug_if(!func, env, \"btf_id %u not found\", btf_id))\n-\t\treturn false;\n-\n \tfunc_proto = btf_type_by_id(btf, func-\u003etype);\n-\tif (!func_proto)\n-\t\treturn false;\n \n-\ttype = btf_type_skip_modifiers(btf, func_proto-\u003etype, NULL);\n-\tif (!type)\n-\t\treturn false;\n+\treturn btf_type_skip_modifiers(btf, func_proto-\u003etype, NULL);\n+}\n \n-\treturn btf_type_is_void(type);\n+static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)\n+{\n+\tconst struct btf_type *type = subprog_ret_type(env, subprog);\n+\n+\treturn type \u0026\u0026 btf_type_is_void(type);\n+}\n+\n+static u32 ret_regs_cnt(u32 size)\n+{\n+\treturn size \u003e 8 \u0026\u0026 size \u003c= 16 ? 2 : 1;\n+}\n+\n+/* Registers holding a function return value, in order. See ret_regs_cnt(). */\n+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };\n+\n+static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)\n+{\n+\tconst struct btf *btf = env-\u003eprog-\u003eaux-\u003ebtf;\n+\tconst struct btf_type *type;\n+\tint subprog;\n+\tu32 size;\n+\n+\tif (!env-\u003eprog-\u003ejit_requested || bpf_prog_is_offloaded(env-\u003eprog-\u003eaux))\n+\t\treturn 0;\n+\n+\t/*\n+\t * Skip the main program: its return value is the program's exit code,\n+\t * read out of R0, so it never uses the register pair. An extension does\n+\t * have a real prototype for subprog 0, but bpf_check_attach_target()\n+\t * refuses to replace a function returning more than 8 bytes.\n+\t */\n+\tfor (subprog = 1; subprog \u003c env-\u003esubprog_cnt; subprog++) {\n+\t\ttype = subprog_ret_type(env, subprog);\n+\t\tif (!type || btf_type_is_void(type))\n+\t\t\tcontinue;\n+\t\tif (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, \u0026size)), env,\n+\t\t\t\t    \"cannot size return type of subprog %d\", subprog))\n+\t\t\treturn -EFAULT;\n+\t\tif (ret_regs_cnt(size) \u003e 1) {\n+\t\t\tsubprog_info(env, subprog)-\u003eret_reg_pair = true;\n+\t\t\tenv-\u003eprog-\u003ejit_required = 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n }\n \n const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog)\n@@ -2877,6 +2919,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)\n \terr = btf_distill_func_proto(\u0026env-\u003elog, kfunc.btf, kfunc.proto, kfunc.name, \u0026func_model);\n \tif (err)\n \t\treturn err;\n+\tif (func_model.ret_size \u003e 8) {\n+\t\tif (kfunc.flags \u0026\u0026 (*kfunc.flags \u0026 KF_FASTCALL)) {\n+\t\t\tverbose(env, \"kfunc %s with \u003e8-byte return is not supported with KF_FASTCALL\\n\",\n+\t\t\t\tkfunc.name);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\t\tif (!bpf_jit_supports_kfunc_ret_reg_pair()) {\n+\t\t\tverbose(env, \"kfunc %s with \u003e8-byte return is not supported by JIT\\n\",\n+\t\t\t\tkfunc.name);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\t}\n \n \tmemset(\u0026meta, 0, sizeof(meta));\n \tmeta.btf = kfunc.btf;\n@@ -9853,6 +9907,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n \tu16 callee_incoming, stack_arg_cnt;\n \tstruct bpf_func_state *caller;\n \tint err, subprog, target_insn;\n+\tu32 i, nregs;\n \n \ttarget_insn = *insn_idx + insn-\u003eimm + 1;\n \tsubprog = bpf_find_subprog(env, target_insn);\n@@ -9908,9 +9963,14 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n \t\tclear_caller_saved_regs(env, caller-\u003eregs);\n \t\tinvalidate_outgoing_stack_args(env, cur_func(env));\n \n-\t\t/* All non-void global functions return a 64-bit SCALAR_VALUE. */\n+\t\t/*\n+\t\t * A non-void global function returns a 64-bit SCALAR_VALUE in\n+\t\t * R0, or a \u003e8 byte SCALAR_VALUE in the R0:R2 register pair.\n+\t\t */\n \t\tif (!returns_void) {\n-\t\t\tmark_reg_unknown(env, caller-\u003eregs, BPF_REG_0);\n+\t\t\tnregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;\n+\t\t\tfor (i = 0; i \u003c nregs; i++)\n+\t\t\t\tmark_reg_unknown(env, caller-\u003eregs, ret_regs[i]);\n \t\t\tbpf_diag_mod_end(env);\n \t\t}\n \n@@ -10270,11 +10330,15 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)\n \tstruct bpf_func_state *caller, *callee;\n \tstruct bpf_reg_state *r0;\n \tbool in_callback_fn;\n+\tu32 i, nregs;\n \tint err;\n \n \tcallee = state-\u003eframe[state-\u003ecurframe];\n \tr0 = \u0026callee-\u003eregs[BPF_REG_0];\n-\tif (r0-\u003etype == PTR_TO_STACK) {\n+\tnregs = bpf_ret_reg_pair(env, callee-\u003esubprogno) ? 2 : 1;\n+\tfor (i = 0; i \u003c nregs; i++) {\n+\t\tif (callee-\u003eregs[ret_regs[i]].type != PTR_TO_STACK)\n+\t\t\tcontinue;\n \t\t/* technically it's ok to return caller's stack pointer\n \t\t * (or caller's caller's pointer) back to the caller,\n \t\t * since these pointers are valid. Only current stack\n@@ -10309,9 +10373,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)\n \t\t\treturn -EFAULT;\n \t\t}\n \t} else {\n-\t\t/* return to the caller whatever r0 had in the callee */\n+\t\t/*\n+\t\t * return to the caller whatever the callee had in the\n+\t\t * return register(s)\n+\t\t */\n \t\tbpf_diag_mod_begin(env, \u0026caller-\u003eregs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE);\n-\t\tcaller-\u003eregs[BPF_REG_0] = *r0;\n+\t\tfor (i = 0; i \u003c nregs; i++)\n+\t\t\tcaller-\u003eregs[ret_regs[i]] = callee-\u003eregs[ret_regs[i]];\n \t\tbpf_diag_mod_end(env);\n \t}\n \n@@ -11246,6 +11314,19 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn\n \treturn 0;\n }\n \n+/*\n+ * Mark the register(s) holding a @size byte kfunc return value as unknown\n+ * scalars. Both halves of a register pair are treated the same way.\n+ */\n+static void mark_kfunc_ret_regs(struct bpf_verifier_env *env,\n+\t\t\t\tstruct bpf_reg_state *regs, u32 size)\n+{\n+\tu32 i, nregs = ret_regs_cnt(size);\n+\n+\tfor (i = 0; i \u003c nregs; i++)\n+\t\tmark_reg_unknown(env, regs, ret_regs[i]);\n+}\n+\n static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta)\n {\n \treturn meta-\u003ekfunc_flags \u0026 KF_ACQUIRE;\n@@ -11513,9 +11594,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_\n }\n \n /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */\n-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,\n-\t\t\t\t\tconst struct btf *btf,\n-\t\t\t\t\tconst struct btf_type *t, int rec)\n+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env,\n+\t\t\t       const struct btf *btf,\n+\t\t\t       const struct btf_type *t, int rec)\n {\n \tconst struct btf_type *member_type;\n \tconst struct btf_member *member;\n@@ -11533,7 +11614,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,\n \t\t\t\tverbose(env, \"max struct nesting depth exceeded\\n\");\n \t\t\t\treturn false;\n \t\t\t}\n-\t\t\tif (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))\n+\t\t\tif (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1))\n \t\t\t\treturn false;\n \t\t\tcontinue;\n \t\t}\n@@ -11932,7 +12013,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,\n \t\t (is_kfunc_arg_mem_size(meta-\u003ebtf, \u0026args[arg + 1]) ||\n \t\t  is_kfunc_arg_const_mem_size(meta-\u003ebtf, \u0026args[arg + 1]))) {\n \t\tif (!btf_type_is_void(ref_t) \u0026\u0026 !btf_type_is_scalar(ref_t) \u0026\u0026\n-\t\t    !__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t    !btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\tverbose(env, \"%s pointer type %s %s must point to void, scalar, or struct with scalar\\n\",\n \t\t\t\treg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);\n \t\t\treturn -EINVAL;\n@@ -11948,7 +12029,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,\n \t\t * scalars. The access size is derived from the pointed-to BTF type.\n \t\t */\n \t\tif (!btf_type_is_scalar(ref_t) \u0026\u0026\n-\t\t    !__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t    !btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\tverbose(env, \"%s pointer type %s %s must point to scalar, or struct with scalar\\n\",\n \t\t\t\treg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);\n \t\t\treturn -EINVAL;\n@@ -13004,7 +13085,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me\n \t\t\t\tbreak;\n \t\t\t}\n \n-\t\t\tif (!__btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n+\t\t\tif (!btf_type_is_scalar_struct(env, meta-\u003ebtf, ref_t, 0)) {\n \t\t\t\tenum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);\n \t\t\t\tconst char *expected_type;\n \n@@ -13546,7 +13627,7 @@ static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg\n \n \t\tstruct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);\n \t\tif (is_bpf_percpu_obj_new_kfunc(meta-\u003efunc_id)) {\n-\t\t\tif (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {\n+\t\t\tif (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {\n \t\t\t\tverbose(env, \"bpf_percpu_obj_new type ID argument must be of a struct of scalars\\n\");\n \t\t\t\treturn -EINVAL;\n \t\t\t}\n@@ -13914,10 +13995,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n \t}\n \n \tif (btf_type_is_scalar(t)) {\n-\t\tmark_reg_unknown(env, regs, BPF_REG_0);\n+\t\tmark_kfunc_ret_regs(env, regs, t-\u003esize);\n \t\tif (meta.btf == btf_vmlinux \u0026\u0026 (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||\n \t\t    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))\n \t\t\t__mark_reg_const_zero(env, \u0026regs[BPF_REG_0]);\n+\t} else if (btf_type_is_struct(t)) {\n+\t\t/*\n+\t\t * The returned struct comes back as raw register bits modeled\n+\t\t * as an unknown scalar, so it must contain only scalars:\n+\t\t * otherwise a pointer field would be laundered into a scalar\n+\t\t * and escape provenance and reference tracking.\n+\t\t */\n+\t\tif (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {\n+\t\t\tverbose(env,\n+\t\t\t\t\"kernel function %s returns %s %s that is not composed of scalars\\n\",\n+\t\t\t\tfunc_name, btf_type_str(t),\n+\t\t\t\tbtf_name_by_offset(desc_btf, t-\u003ename_off));\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\t\tmark_kfunc_ret_regs(env, regs, t-\u003esize);\n \t} else if (btf_type_is_ptr(t)) {\n \t\tptr_type = btf_type_skip_modifiers(desc_btf, t-\u003etype, \u0026ptr_type_id);\n \t\terr = check_special_kfunc(env, \u0026meta, regs, insn_aux, ptr_type, desc_btf);\n@@ -17453,37 +17549,54 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char\n \treturn 0;\n }\n \n-static int check_global_subprog_return_code(struct bpf_verifier_env *env)\n+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)\n {\n-\tstruct bpf_reg_state *reg = reg_state(env, BPF_REG_0);\n-\tstruct bpf_func_state *cur_frame = cur_func(env);\n+\tstruct bpf_reg_state *reg;\n \tint err;\n \n-\tif (subprog_returns_void(env, cur_frame-\u003esubprogno))\n-\t\treturn 0;\n-\n-\terr = check_reg_arg(env, BPF_REG_0, SRC_OP);\n+\terr = check_reg_arg(env, regno, SRC_OP);\n \tif (err)\n \t\treturn err;\n \n \t/* Pointers to arena are safe to pass between subprograms. */\n-\tif (is_arena_reg(env, BPF_REG_0))\n+\tif (is_arena_reg(env, regno))\n \t\treturn 0;\n \n-\tif (is_pointer_value(env, BPF_REG_0)) {\n-\t\tverbose(env, \"R%d leaks addr as return value\\n\", BPF_REG_0);\n+\tif (is_pointer_value(env, regno)) {\n+\t\tverbose(env, \"R%d leaks addr as return value\\n\", regno);\n \t\treturn -EACCES;\n \t}\n \n+\treg = reg_state(env, regno);\n \tif (reg-\u003etype != SCALAR_VALUE) {\n-\t\tverbose(env, \"At subprogram exit the register R0 is not a scalar value (%s)\\n\",\n-\t\t\treg_type_str(env, reg-\u003etype));\n+\t\tverbose(env, \"At subprogram exit the register R%d is not a scalar value (%s)\\n\",\n+\t\t\tregno, reg_type_str(env, reg-\u003etype));\n \t\treturn -EINVAL;\n \t}\n \n \treturn 0;\n }\n \n+static int check_global_subprog_return_code(struct bpf_verifier_env *env)\n+{\n+\tstruct bpf_func_state *cur_frame = cur_func(env);\n+\tu32 subprog = cur_frame-\u003esubprogno;\n+\tu32 i, nregs;\n+\tint err;\n+\n+\tif (subprog_returns_void(env, subprog))\n+\t\treturn 0;\n+\n+\tnregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;\n+\tfor (i = 0; i \u003c nregs; i++) {\n+\t\terr = check_global_ret_scalar_reg(env, ret_regs[i]);\n+\t\tif (err)\n+\t\t\treturn err;\n+\t}\n+\n+\treturn 0;\n+}\n+\n /* Bitmask with 1s for all caller saved registers */\n #define ALL_CALLER_SAVED_REGS ((1u \u003c\u003c CALLER_SAVED_REGS) - 1)\n \n@@ -21151,6 +21264,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\n \n+\t/* must precede the first bpf_ret_reg_pair() user below */\n+\tret = bpf_compute_subprog_ret_regs(env);\n+\tif (ret \u003c 0)\n+\t\tgoto skip_full_check;\n+\n \tret = bpf_compute_live_registers(env);\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\ndiff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c\nnew file mode 100644\nindex 0000000000000..e0b94ed10f941\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c\n@@ -0,0 +1,11 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003ctest_progs.h\u003e\n+#include \"aggregate_ret_func.skel.h\"\n+#include \"aggregate_ret_kfunc.skel.h\"\n+\n+void test_aggregate_ret(void)\n+{\n+\tRUN_TESTS(aggregate_ret_func);\n+\tRUN_TESTS(aggregate_ret_kfunc);\n+}\ndiff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\nindex 2523c07a16c65..6262c81cdaa9f 100644\n--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\n+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c\n@@ -441,6 +441,20 @@ static void test_func_replace_int_with_void(void)\n \t\t\t\t     \" doesn't match type INT of global_func2()\");\n }\n \n+static void test_func_replace_ret_pair(void)\n+{\n+\tconst char *msg = \"Return type of new_agg_ret_target_func() has size 8 \"\n+\t\t\t  \"and of agg_ret_target_func() size 16\";\n+\n+\t/*\n+\t * An extension cannot replace a function whose return value comes back\n+\t * in the R0:R2 pair: only R0 is checked at the extension's exit, so it\n+\t * would leave R2 stale for the target's callers.\n+\t */\n+\ttest_obj_load_failure_common(\"freplace_ret_pair.bpf.o\",\n+\t\t\t\t     \"./aggregate_ret_target.bpf.o\", msg);\n+}\n+\n static int find_prog_btf_id(const char *name, __u32 attach_prog_fd)\n {\n \tstruct bpf_prog_info info = {};\n@@ -660,6 +674,8 @@ void serial_test_fexit_bpf2bpf(void)\n \t\ttest_func_replace_progmap();\n \tif (test__start_subtest(\"freplace_int_with_void\"))\n \t\ttest_func_replace_int_with_void();\n+\tif (test__start_subtest(\"freplace_ret_pair\"))\n+\t\ttest_func_replace_ret_pair();\n \tif (test__start_subtest(\"freplace_void\"))\n \t\ttest_func_replace_void();\n \tif (test__start_subtest(\"sleepable_fentry_to_xdp\"))\ndiff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c\nindex 64ac49ad67e63..f7f94ccebce27 100644\n--- a/tools/testing/selftests/bpf/prog_tests/verifier.c\n+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c\n@@ -5,6 +5,7 @@\n #include \"arena_kfunc.skel.h\"\n #include \"arena_kfunc_jit.skel.h\"\n #include \"cap_helpers.h\"\n+#include \"verifier_aggregate_ret.skel.h\"\n #include \"verifier_align.skel.h\"\n #include \"verifier_and.skel.h\"\n #include \"verifier_arena.skel.h\"\n@@ -170,6 +171,7 @@ void test_arena_kfunc(void)                   { RUN_TESTS(arena_kfunc); }\n \n void test_arena_kfunc_jit(void)               { RUN_TESTS(arena_kfunc_jit); }\n \n+void test_verifier_aggregate_ret(void)        { RUN_TESTS(verifier_aggregate_ret); }\n void test_verifier_align(void)                { RUN(verifier_align); }\n void test_verifier_and(void)                  { RUN(verifier_and); }\n void test_verifier_arena(void)                { RUN(verifier_arena); }\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c\nnew file mode 100644\nindex 0000000000000..6f66fc822cede\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c\n@@ -0,0 +1,237 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+typedef unsigned __int128 u128;\n+\n+__naked u128 global_agg_good(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0x1234;\"\t/* low 64 bits */\n+\t\"r2 = 0x5678;\"\t/* high 64 bits */\n+\t\"exit;\"\n+\t);\n+}\n+\n+__naked u128 global_agg_bad(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+__naked u128 global_agg_bad_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = r10;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_global_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_bad];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_bad)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__failure __msg(\"At subprogram exit the register R2 is not a scalar value\")\n+__naked int aggregate_ret_global_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_bad_ptr];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_bad_ptr)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_bad_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = r10;\"\t/* stack pointer placed in the second return register */\n+\t\"exit;\"\n+\t);\n+}\n+\n+/*\n+ * R2 is a return register once the subprogram returns a pair, so a stack\n+ * pointer left in it is rejected at the callee's exit exactly as one in R0\n+ * is: the callee frame is gone by the time the caller could use it.\n+ */\n+SEC(\"tc\")\n+__load_if_JITed()\n+__failure __msg(\"cannot return stack pointer to the caller\")\n+__naked int aggregate_ret_static_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_bad_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_bad_ptr)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_no_r2(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_static_uninit_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_no_r2];\"\n+\t\"r0 = r2;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_no_r2)\n+\t: __clobber_all);\n+}\n+\n+static __naked __noinline u128 static_agg_precise(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 4;\"\t/* second half; its value is made precise below */\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 2: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 12: (95) exit\")\n+__msg(\"mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4\")\n+__naked int aggregate_ret_static_precise(void)\n+{\n+\tasm volatile (\n+\t\"call %[static_agg_precise];\"\n+\t\"r6 = r2;\"\t\t/* derived from the aggregate's second half */\n+\t\"r6 \u0026= 7;\"\t\t/* keep it in [0, 7] to index the stack */\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/* ptr += scalar marks r6 (hence R2) precise */\n+\t\"r0 = 0;\"\n+\t\"*(u8 *)(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(static_agg_precise)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 2: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9\")\n+__naked int aggregate_ret_global_precise(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_agg_good];\"\n+\t\"r6 = r2;\"\t\t/* derived from the aggregate's second half */\n+\t\"r6 \u0026= 7;\"\t\t/* keep it in [0, 7] to index the stack */\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/* ptr += scalar marks r6 (hence R2) precise */\n+\t\"r0 = 0;\"\n+\t\"*(u8 *)(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_agg_good)\n+\t: __clobber_all);\n+}\n+\n+#if defined(__clang_major__) \u0026\u0026 __clang_major__ \u003e= 23\n+\n+/* A by-value struct that smuggles a pointer, which must be rejected. */\n+struct with_ptr {\n+\tvoid *p;\n+\t__u64 x;\n+};\n+\n+/* A by-value union that smuggles a pointer, which must be rejected too. */\n+union upair_with_ptr {\n+\tvoid *p;\n+\t__u64 halves[2];\n+};\n+\n+__naked struct with_ptr global_ret_struct_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"Global function global_ret_struct_ptr() has unsupported return type\")\n+__naked int aggregate_ret_global_struct_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_ret_struct_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_ret_struct_ptr)\n+\t: __clobber_all);\n+}\n+\n+__naked union upair_with_ptr global_ret_union_ptr(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"Global function global_ret_union_ptr() has unsupported return type\")\n+__naked int aggregate_ret_global_union_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[global_ret_union_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(global_ret_union_ptr)\n+\t: __clobber_all);\n+}\n+\n+#endif\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c\nnew file mode 100644\nindex 0000000000000..d6b422ae9784b\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c\n@@ -0,0 +1,121 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+#include \"../test_kmods/bpf_testmod_kfunc.h\"\n+\n+/*\n+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address\n+ * (rather than calling) avoids any dependence on the compiler lowering an\n+ * __int128 or struct return value, which the BPF backend only supports from\n+ * LLVM 23 on.\n+ */\n+void __kfunc_btf_root(void)\n+{\n+\tasm volatile (\"\"\n+\t:\n+\t: \"r\"(\u0026bpf_kfunc_call_test_i128),\n+\t  \"r\"(\u0026bpf_kfunc_call_test_ret_fastcall),\n+\t  \"r\"(\u0026bpf_kfunc_call_test_ret_ptr),\n+\t  \"r\"(\u0026bpf_kfunc_call_test_ret_ii),\n+\t  \"r\"(\u0026bpf_kfunc_call_test_ret_big));\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+__log_level(2)\n+__msg(\"mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 4: (57) r6 \u0026= 7\")\n+__msg(\"mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2\")\n+__msg(\"mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128\")\n+__naked int aggregate_ret_kfunc_precise(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 1;\"\n+\t\"r2 = 2;\"\n+\t\"call %[bpf_kfunc_call_test_i128];\"\n+\t\"r6 = r2;\"\t\t/* second return half */\n+\t\"r6 \u0026= 7;\"\t\t/* keep it in [0, 7] to index the stack */\n+\t\"r1 = r10;\"\n+\t\"r1 += -8;\"\n+\t\"r1 += r6;\"\t\t/* ptr += scalar marks r6 (hence R2) precise */\n+\t\"r0 = 0;\"\n+\t\"*(u8 *)(r1 + 0) = r0;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_i128)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"kfunc bpf_kfunc_call_test_ret_fastcall with \u003e8-byte return is not supported with KF_FASTCALL\")\n+__naked int aggregate_ret_kfunc_fastcall_fail(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 1;\"\n+\t\"r2 = 2;\"\n+\t\"call %[bpf_kfunc_call_test_ret_fastcall];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_fastcall)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"is not composed of scalars\")\n+__naked int aggregate_ret_kfunc_ptr_fail(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 0;\"\n+\t\"call %[bpf_kfunc_call_test_ret_ptr];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_ptr)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"R2 !read_ok\")\n+__naked int aggregate_ret_kfunc_small_no_r2(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 0;\"\n+\t\"r2 = 0;\"\n+\t\"call %[bpf_kfunc_call_test_ret_ii];\"\n+\t\"r0 = r2;\"\t/* R2 is not a return register for a \u003c=8 byte struct */\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_ii)\n+\t: __clobber_all);\n+}\n+\n+/*\n+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by\n+ * btf_distill_func_proto() before the KF_FASTCALL and JIT-capability checks.\n+ */\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__failure __msg(\"The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported\")\n+__naked int aggregate_ret_kfunc_too_big_fail(void)\n+{\n+\tasm volatile (\n+\t\"call %[bpf_kfunc_call_test_ret_big];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(bpf_kfunc_call_test_ret_big)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c\nnew file mode 100644\nindex 0000000000000..cffd8d7d3241a\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c\n@@ -0,0 +1,29 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+/* freplace target: a global subprogram returning 16 bytes in R0:R2. */\n+__naked unsigned __int128 agg_ret_target_func(void)\n+{\n+\tasm volatile (\n+\t\"r0 = 0x1234;\"\n+\t\"r2 = 0x5678;\"\n+\t\"exit;\"\n+\t);\n+}\n+\n+SEC(\"tc\")\n+__naked int agg_ret_target(void)\n+{\n+\tasm volatile (\n+\t\"call %[agg_ret_target_func];\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: __imm(agg_ret_target_func)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c\nindex d055fc7b3b95d..338e67cd8856a 100644\n--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c\n+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c\n@@ -431,6 +431,34 @@ __naked void subprog1(void)\n \t\t::: __clobber_all);\n }\n \n+static __used __naked unsigned __int128 aux2(void)\n+{\n+\tasm volatile (\n+\t\t\"r0 = 1;\"\n+\t\t\"r2 = 2;\"\n+\t\t\"exit;\"\n+\t\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */\n+__load_if_JITed()\n+__log_level(2)\n+__msg(\"0: .12345.... (85) call pc+2\")\n+__msg(\"1: ..2....... (bf) r0 = r2\")\n+__msg(\"2: 0......... (95) exit\")\n+__msg(\"3: .......... (b7) r0 = 1\")\n+__msg(\"4: 0......... (b7) r2 = 2\")\n+__msg(\"5: 0.2....... (95) exit\")\n+__naked void subprog_ret_reg_pair(void)\n+{\n+\tasm volatile (\n+\t\t\"call aux2;\"\n+\t\t\"r0 = r2;\"\n+\t\t\"exit;\"\n+\t\t::: __clobber_all);\n+}\n+\n #if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)\n \n SEC(\"socket\")\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c\nindex ac44d60e50666..9708efb93683b 100644\n--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c\n+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c\n@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)\n \n SEC(\"?tc\")\n __exception_cb(exception_cb_bad_ret_type1)\n-__failure __msg(\"Global function exception_cb_bad_ret_type1() return value not void or scalar.\")\n+__failure __msg(\"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\")\n int reject_exception_cb_type_1(struct __sk_buff *ctx)\n {\n \tbpf_throw(0);\ndiff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c\nnew file mode 100644\nindex 0000000000000..12c15d293bd79\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c\n@@ -0,0 +1,12 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+\n+SEC(\"freplace/agg_ret_target_func\")\n+__u64 new_agg_ret_target_func(void)\n+{\n+\treturn 0;\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c\nnew file mode 100644\nindex 0000000000000..7851bade2b406\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c\n@@ -0,0 +1,178 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"../test_kmods/bpf_testmod_kfunc.h\"\n+#include \"bpf_misc.h\"\n+\n+#if defined(__clang_major__) \u0026\u0026 __clang_major__ \u003e= 23\n+\n+#define MIX_A\t0xdeadbeefcafef00dULL\n+#define MIX_B\t0x0123456789abcdefULL\n+\n+typedef unsigned __int128 u128;\n+\n+struct pair {\n+\t__u64 lo;\t/* R0 */\n+\t__u64 hi;\t/* R2 */\n+};\n+\n+union upair {\n+\t__u64 halves[2];\n+\tstruct {\n+\t\t__u64 lo;\t/* R0 */\n+\t\t__u64 hi;\t/* R2 */\n+\t} parts;\n+};\n+\n+static __noinline u128 make_i128(__u64 a, __u64 b)\n+{\n+\treturn ((u128)(a + b) \u003c\u003c 64) | (a - b);\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_int128_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tu128 v;\n+\n+\tv = make_i128(a, b);\n+\tif ((__u64)(v \u003e\u003e 64) != a + b)\n+\t\treturn 1;\n+\tif ((__u64)v != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+static __noinline struct pair make_pair(__u64 a, __u64 b)\n+{\n+\tstruct pair p = { .lo = a + b, .hi = a - b };\n+\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_struct_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct pair p;\n+\n+\tp = make_pair(a, b);\n+\tif (p.lo != a + b)\n+\t\treturn 1;\n+\tif (p.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+__noinline struct pair make_pair_global(__u64 a, __u64 b)\n+{\n+\tstruct pair p = { .lo = a + b, .hi = a - b };\n+\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct pair p;\n+\n+\tp = make_pair_global(a, b);\n+\tif (p.lo != a + b)\n+\t\treturn 1;\n+\tif (p.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+static __noinline union upair make_upair(__u64 a, __u64 b)\n+{\n+\tunion upair p;\n+\n+\tp.halves[0] = a + b;\n+\tp.halves[1] = a - b;\n+\treturn p;\n+}\n+\n+SEC(\"tc\")\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_union_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tunion upair p;\n+\n+\tp = make_upair(a, b);\n+\tif (p.parts.lo != a + b)\n+\t\treturn 1;\n+\tif (p.parts.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tu128 v;\n+\n+\tv = bpf_kfunc_call_test_i128(a, b);\n+\tif ((__u64)(v \u003e\u003e 64) != a + b)\n+\t\treturn 1;\n+\tif ((__u64)v != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+SEC(\"tc\")\n+__arch_x86_64 __arch_arm64\n+__load_if_JITed()\n+__success __retval(0)\n+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)\n+{\n+\t__u64 a = skb-\u003elen ^ MIX_A;\n+\t__u64 b = skb-\u003elen ^ MIX_B;\n+\tstruct prog_test_ret_pair p;\n+\n+\tp = bpf_kfunc_call_test_ret_pair(a, b);\n+\tif (p.lo != a + b)\n+\t\treturn 1;\n+\tif (p.hi != a - b)\n+\t\treturn 2;\n+\n+\treturn 0;\n+}\n+\n+#else\n+\n+SEC(\"socket\")\n+__description(\"verifier_aggregate_ret: needs LLVM 23, dummy test\")\n+__success\n+int dummy_test(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\nindex 9366a3c578f13..850cf4f830c48 100644\n--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\n+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c\n@@ -954,6 +954,53 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)\n \treturn 0;\n }\n \n+#if defined(__x86_64__) || defined(__aarch64__)\n+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)\n+{\n+\treturn (__int128)(((unsigned __int128)(a + b) \u003c\u003c 64) | (a - b));\n+}\n+\n+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)\n+{\n+\tstruct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)\n+{\n+\tstruct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)\n+{\n+\tstruct prog_test_ret_ptr r = { .p = NULL, .tag = tag };\n+\n+\treturn r;\n+}\n+\n+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)\n+{\n+\tstruct prog_test_ret_ii r = { .a = a, .b = b };\n+\n+\treturn r;\n+}\n+#endif /* __x86_64__ || __aarch64__ */\n+\n+/*\n+ * Takes no argument on purpose: with no arguments there is nothing for the sret\n+ * pointer to displace, so this needs no architecture guard even though it\n+ * returns 24 bytes.\n+ */\n+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)\n+{\n+\tstruct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };\n+\n+\treturn r;\n+}\n+\n __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,\n \t\t\t\t\t u64 e, u64 f, u64 g, u64 h,\n \t\t\t\t\t u64 i, u64 j)\n@@ -1487,6 +1534,14 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test3)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test4)\n BTF_ID_FLAGS(func, bpf_kfunc_call_test5)\n+#if defined(__x86_64__) || defined(__aarch64__)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)\n+#endif\n+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)\n BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)\ndiff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\nindex 7d81070eefe7f..65e693ada7364 100644\n--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\n+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h\n@@ -55,6 +55,27 @@ struct prog_test_big_arg {\n \t__u64 b;\n };\n \n+struct prog_test_ret_pair {\t/* 16 bytes: R0:R2 */\n+\t__u64 lo;\n+\t__u64 hi;\n+};\n+\n+struct prog_test_ret_ii {\t/* 8 bytes: R0 only */\n+\tint a;\n+\tint b;\n+};\n+\n+struct prog_test_ret_ptr {\t/* 16 bytes: contains a pointer */\n+\tvoid *p;\n+\t__u64 tag;\n+};\n+\n+struct prog_test_ret_big {\t/* 24 bytes: too large for R0:R2 */\n+\t__u64 a;\n+\t__u64 b;\n+\t__u64 c;\n+};\n+\n struct prog_test_fail1 {\n \tvoid *p;\n \tint x;\n@@ -131,6 +152,14 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;\n struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;\n long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;\n int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;\n+#ifdef __SIZEOF_INT128__\n+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;\n+#endif\n+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;\n+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;\n+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;\n+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;\n+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym;\n __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,\n \t\t\t       __u64 e, __u64 f, __u64 g, __u64 h,\n \t\t\t       __u64 i, __u64 j) __ksym;\n"
}

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT"
  ],
  "FocusSymbols": [
    "btf_type_is_scalar_struct",
    "btf_distill_func_proto",
    "btf_prepare_func_args",
    "bpf_compute_live_registers",
    "bpf_jit_supports_kfunc_ret_reg_pair"
  ],
  "Reasoning": "The patch series adds support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or scalar-only structs/unions) in the register pair R0:R2. This involves modifications to the BPF verifier to track the return values in both registers, changes to the liveness and backtracking logic, and updates to the JIT compilers (x86, arm64, riscv) to support this new calling convention. These are functional changes to reachable core kernel logic (BPF subsystem) that should be fuzzed to ensure the verifier correctly tracks the register states and prevents any leaks or invalid memory accesses.",
  "WorthFuzzing": true
}

You are an expert Linux kernel maintainer.
Your job is to review a provided patch series and determine
if it makes functional changes to the kernel that should be fuzzed.

IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively
use your code access tools to examine the actual source code and confirm any assumptions.

Return WorthFuzzing=false if the patch only contains:
- Modifications to Documentation/, Kconfig files, or code comments.
- Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints.
- Changes to numeric constants or macros that do not functionally alter execution flow.
- Code paths that are impossible to reach in virtualized environments like GCE or QEMU,
  even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim).
- Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed,
  ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU.
- Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove)
  that are executed only during PCI hot-unplug or sysfs driver unbind operations.

If it modifies reachable core kernel logic, drivers, or architectures, use your code search
tools to verify the code can be executed, then return WorthFuzzing=true.

When returning WorthFuzzing=true, you MUST ALSO:
1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols.
   Avoid listing generic hot-path functions to prevent skewed test distributions.
   Prefer non-static, non-inlined API entrypoint functions over internal static helper functions
   (which are inlined by the compiler and do not have distinct symbol addresses).
2. Identify any specific CONFIG_ options required to properly test this new/modified feature.
   Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions.
   Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs
   needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs
   output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit cc236f68937f8479bce51601bd02ac7647affe1d
Author: syz-cluster <triage@syzkaller.com>
Date:   Wed Aug 19 06:41:58 2026 +0000

    syz-cluster: applied patch under review

diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index 85f73e0bbd0ff..89dea6b0b0241 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -575,6 +575,71 @@ is also covered by this recovery. A kfunc handed an arena pointer may
 therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
 against the arena. Larger accesses must verify the range explicitly.
 
+2.9 kfunc Return Values
+-----------------------
+
+A kfunc may return a scalar, a pointer, or a small struct or union by
+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
+
+A struct or union returned by value must be composed only of scalars
+(recursively), where a scalar is an integer or an enum; arrays of scalars are
+allowed as members. Its bytes are handed back to the program as the raw
+contents of R0 (and R2), so a pointer field would be laundered into a scalar
+and escape the verifier's pointer provenance and reference tracking. A struct
+or union with a pointer member is therefore rejected at load time, and so is
+one with a floating-point member, which the ABI may not return in R0:R2 at
+all.
+
+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
+scalar-only struct or union, or an ``__int128``. Such a value is returned
+in the register pair R0:R2, matching the convention LLVM uses for the BPF
+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or
+union of 8 bytes or less is returned in R0 alone.
+
+::
+
+        struct bpf_pair { __u64 a, b; };   /* 16 bytes */
+
+        __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
+        {
+                struct bpf_pair p = { .a = 1, .b = 2 };
+
+                return p;      /* p.a in R0, p.b in R2 */
+        }
+
+Returning a value in the R0:R2 pair requires the JIT to place the second
+half of the return value into R2, which not every architecture supports
+right now. A kfunc with a return value larger than 8 bytes is therefore
+rejected at load time on a JIT that does not advertise this capability (see
+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
+by the interpreter. A return value larger than 16 bytes is not supported.
+
+The same R0:R2 convention applies to a BPF subprogram, global or static, that
+returns an ``__int128`` or a struct or union larger than 8 bytes. It is only
+used when the program is JITed, since the interpreter propagates only R0 out of
+a subprogram: without a JIT the return value stays in R0 alone, and a caller
+reading R2 is rejected for reading an uninitialized register. A global
+subprogram is verified in isolation, so its by-value struct or union return is
+restricted to scalars just like a kfunc's; a static subprogram is verified
+inline and has no such restriction. The main program is not covered: its return
+value is the program's exit code, read out of R0 alone, so a declared upper
+half is never looked at.
+
+A global subprogram must leave a scalar in *every* register of the pair, so
+both halves of the returned value have to be assigned. Leaving the upper half
+uninitialized is not merely untidy: the compiler is then free to leave R2
+holding whatever it happened to hold, which for a subprogram taking a pointer
+argument is typically that pointer. Handing the caller an unknown scalar built
+from a pointer is a leak, so the verifier rejects it with::
+
+        At subprogram exit the register R2 is not a scalar value (...)
+
+Initialize the whole return value, for example ``struct pair p = {};``, to
+avoid this. A static subprogram is exempt from the scalar-only rule: it is
+verified inline, so an unassigned R2 is simply passed back to the caller as
+uninitialized and only a caller that reads it fails. A stack pointer left in
+R2 is still rejected there, just as one in R0 is.
+
 .. _BPF_kfunc_lifecycle_expectations:
 
 3. kfunc lifecycle expectations
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index c18e005a41dbe..3aa3ea0bc30bb 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2388,6 +2388,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index 74efe4b138d2a..47c7bf431ba85 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -2121,6 +2121,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_ptr_xchg(void)
 {
 	return true;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 1a9fb530adc3c..48429fae06410 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
  * arena NULL is offset 0. Return the number of emitted bytes.
  */
 static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
-				 const struct bpf_insn *insn, u8 **pprog)
+				 const struct btf_func_model *fm, u8 **pprog)
 {
-	const struct btf_func_model *fm;
 	u8 *prog = *pprog;
 	u8 *start = prog;
 	int i;
 
-	fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
-	if (!fm)
-		return -EINVAL;
-
 	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
 		u8 flags = fm->arg_flags[i];
 		u32 reg = BPF_REG_1 + i;
@@ -2644,6 +2639,8 @@ st:			insn_off = insn->off;
 
 			/* call */
 		case BPF_JMP | BPF_CALL: {
+			const struct btf_func_model *fm = NULL;
+
 			func = (u8 *) __bpf_call_base + imm32;
 			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
 				LOAD_TAIL_CALL_CNT_PTR(stack_depth);
@@ -2652,7 +2649,10 @@ st:			insn_off = insn->off;
 			if (!imm32)
 				return -EINVAL;
 			if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
-				err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
+				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
+				if (!fm)
+					return -EINVAL;
+				err = emit_kfunc_arena_args(bpf_prog, fm, &prog);
 				if (err < 0)
 					return err;
 				ip += err;
@@ -2666,6 +2666,14 @@ st:			insn_off = insn->off;
 				return -EINVAL;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
+			/*
+			 * A kfunc returning more than 8 bytes hands the second
+			 * half back in RDX (the native ABI's second return reg),
+			 * but BPF expects it in R0:R2. BPF R0 is RAX (no move
+			 * needed), while BPF R2 is RSI, so copy RDX into RSI.
+			 */
+			if (fm && fm->ret_size > 8)
+				emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
 			break;
 		}
 
@@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 5fad59fdab0d0..004b06785521c 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -819,6 +819,8 @@ struct bpf_subprog_info {
 	bool is_async_cb: 1;
 	bool is_exception_cb: 1;
 	bool args_cached: 1;
+	/* true if the return value is passed in the R0:R2 register pair */
+	bool ret_reg_pair: 1;
 	/* true if bpf_fastcall stack region is used by functions that can't be inlined */
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
@@ -1055,6 +1057,11 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
 	return &env->subprog_info[subprog];
 }
 
+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	return subprog_info(env, subprog)->ret_reg_pair;
+}
+
 struct bpf_call_summary {
 	u8 num_params;
 	bool is_void;
@@ -1481,6 +1488,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);
 struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+			       const struct btf_type *t, int rec);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 4a9bc6a848f2e..6e746b0a09306 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_kfunc_ret_reg_pair(void);
 bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index a2b18a9f1694c..653db80bcc47b 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -423,6 +423,10 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				 */
 				verifier_bug_if(idx + 1 != subseq_idx, env,
 						"extra insn from subprog");
+				/* global subprog always sets R0 */
+				bt_clear_reg(bt, BPF_REG_0);
+				/* and if it does not set R2, main pass would catch it */
+				bt_clear_reg(bt, BPF_REG_2);
 				/* r1-r5 are invalidated after subprog call,
 				 * so for global func call it shouldn't be set
 				 * anymore
@@ -432,8 +436,6 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 						     bt_reg_mask(bt));
 					return -EFAULT;
 				}
-				/* global subprog always sets R0 */
-				bt_clear_reg(bt, BPF_REG_0);
 				return 0;
 			} else {
 				/* static subprog call instruction, which
@@ -506,6 +508,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				return -ENOTSUPP;
 			/* regular helper call sets R0 */
 			bt_clear_reg(bt, BPF_REG_0);
+			/* kfunc might also set R2 */
+			bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				/* if backtracking was looking for registers R1-R5
 				 * they should have been found already.
@@ -520,7 +524,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 					return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			bool r0_precise;
+			bool from_subprog_call, r0_precise, r2_precise;
+
+			/* BPF_EXIT in subprog or callback always returns
+			 * right after the call instruction, so by checking
+			 * whether the instruction at subseq_idx-1 is subprog
+			 * call or not we can distinguish actual exit from
+			 * *subprog* from exit from *callback*. In the former
+			 * case, we need to propagate the precision of the
+			 * return registers, if necessary. In the latter we
+			 * never do that.
+			 */
+			from_subprog_call = subseq_idx - 1 >= 0 &&
+					    bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
+
+			/* Sample the return registers before the callback
+			 * handling below clears R1-R5.
+			 */
+			r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
+			r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
 
 			/* Backtracking to a nested function call, 'idx' is a part of
 			 * the inner frame 'subseq_idx' is a part of the outer frame.
@@ -533,30 +555,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
 				for (i = BPF_REG_1; i <= BPF_REG_5; i++)
 					bt_clear_reg(bt, i);
+
+			bt_clear_reg(bt, BPF_REG_0);
+			bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				verifier_bug(env, "backtracking exit unexpected regs %x",
 					     bt_reg_mask(bt));
 				return -EFAULT;
 			}
 
-			/* BPF_EXIT in subprog or callback always returns
-			 * right after the call instruction, so by checking
-			 * whether the instruction at subseq_idx-1 is subprog
-			 * call or not we can distinguish actual exit from
-			 * *subprog* from exit from *callback*. In the former
-			 * case, we need to propagate r0 precision, if
-			 * necessary. In the former we never do that.
-			 */
-			r0_precise = subseq_idx - 1 >= 0 &&
-				     bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
-				     bt_is_reg_set(bt, BPF_REG_0);
-
-			bt_clear_reg(bt, BPF_REG_0);
 			if (bt_subprog_enter(bt))
 				return -EFAULT;
 
 			if (r0_precise)
 				bt_set_reg(bt, BPF_REG_0);
+			if (r2_precise)
+				bt_set_reg(bt, BPF_REG_2);
 			/* r6-r9 and stack slots will stay set in caller frame
 			 * bitmasks until we return back from callee(s)
 			 */
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 6967d48bba49d..58b81c6f5288c 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7591,7 +7591,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
 		return -EINVAL;
 	}
 	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	if (ret < 0 || ret > 16) {
 		bpf_log(log,
 			"The function %s return type %s is unsupported.\n",
 			tname, btf_type_str(t));
@@ -7684,6 +7684,12 @@ static int btf_check_func_type_match(struct bpf_verifier_log *log,
 			btf_type_str(t2), fn2);
 		return -EINVAL;
 	}
+	if (btf_type_has_size(t1) && (t1->size > 8 || t2->size > 8)) {
+		bpf_log(log,
+			"Return type of %s() has size %u and of %s() size %u, and a size above 8 bytes cannot be replaced\n",
+			fn1, t1->size, fn2, t2->size);
+		return -EINVAL;
+	}
 
 	for (i = 0; i < nargs1; i++) {
 		t1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL);
@@ -7964,7 +7970,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env *env,
 
 /* Check whether the type is a valid return type. */
 static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
-		const struct btf_type *t, int subprog)
+		const struct btf_type *t, int subprog, bool is_global)
 {
 	u32 tags = 0;
 	int err;
@@ -7987,6 +7993,19 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
 	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
 		return 0;
 
+	if (btf_type_is_struct(t) && t->size <= 16) {
+		/*
+		 * A global function's caller models the return as an opaque
+		 * scalar pair, so it may only return scalars by value. A local
+		 * function is verified inline, so a pointer field stays tracked
+		 * and needs no such restriction.
+		 */
+		bool local_func = subprog && !is_global;
+
+		if (local_func || btf_type_is_scalar_struct(env, btf, t, 0))
+			return 0;
+	}
+
 	return -EOPNOTSUPP;
 }
 
@@ -8074,12 +8093,12 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 		return -EINVAL;
 	}
 
-	err = btf_validate_return_type(env, btf, t, subprog);
+	err = btf_validate_return_type(env, btf, t, subprog, is_global);
 	if (err) {
 		if (is_global) {
 			bpf_log(log,
-				"Global function %s() return value not void or scalar. "
-				"Only those are supported.\n",
+				"Global function %s() has unsupported return type. "
+				"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n",
 				tname);
 		}
 		return err;
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index d55e737ed75ad..5db77d7915dfc 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_stack_args(void)
 {
 	return false;
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 74fc4b3f80d6e..71f998c6eb888 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2060,7 +2060,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); }
 /* Compute info->{use,def} fields for the instruction */
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
-				   struct insn_live_regs *info)
+				   struct insn_live_regs *info,
+				   bool ret_reg_pair)
 {
 	struct bpf_call_summary cs;
 	const u8 class = BPF_CLASS(insn->code);
@@ -2072,6 +2073,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 	const u32 src32 = mask_lo(src);
 	const u32 dst32 = mask_lo(dst);
 	const u32 r0  = reg64_mask(0);
+	const u32 r2  = reg64_mask(BPF_REG_2);
 	u32 def = 0;
 	u32 use = U32_MAX;
 
@@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			break;
 		case BPF_EXIT:
 			def = 0;
-			use = r0;
+			use = ret_reg_pair ? (r0 | r2) : r0;
 			break;
 		case BPF_CALL:
 			def = ALL_CALLER_SAVED_REGS;
@@ -2228,8 +2230,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 	struct insn_live_regs *state;
 	int insn_cnt = env->prog->len;
 	u64 pos, insn_pos;
-	int err = 0, i, j;
-	bool changed;
+	int err = 0, i, j, subprog, start, end;
+	bool changed, ret_reg_pair;
 
 	/* Use the following algorithm:
 	 * - define the following:
@@ -2256,8 +2258,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		goto out;
 	}
 
-	for (i = 0; i < insn_cnt; ++i)
-		compute_insn_live_regs(env, &insns[i], &state[i]);
+	for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+		start = env->subprog_info[subprog].start;
+		end = env->subprog_info[subprog + 1].start;
+		ret_reg_pair = bpf_ret_reg_pair(env, subprog);
+
+		for (i = start; i < end; ++i)
+			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
+	}
 
 	/* Forward pass: resolve stack access through FP-derived pointers */
 	err = bpf_compute_subprog_arg_access(env);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e421ea2b80c3a..7a939dfa3797b 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -385,27 +385,69 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
 	return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
 }
 
-static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
 {
-	const struct btf_type *type, *func, *func_proto;
+	const struct btf_type *func, *func_proto;
 	const struct btf *btf = env->prog->aux->btf;
 	u32 btf_id;
 
+	if (!btf || !env->prog->aux->func_info)
+		return NULL;
+
 	btf_id = env->prog->aux->func_info[subprog].type_id;
 
+	/* Both already validated by prepare_btf_func() at prog load. */
 	func = btf_type_by_id(btf, btf_id);
-	if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
-		return false;
-
 	func_proto = btf_type_by_id(btf, func->type);
-	if (!func_proto)
-		return false;
 
-	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
-	if (!type)
-		return false;
+	return btf_type_skip_modifiers(btf, func_proto->type, NULL);
+}
 
-	return btf_type_is_void(type);
+static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type = subprog_ret_type(env, subprog);
+
+	return type && btf_type_is_void(type);
+}
+
+static u32 ret_regs_cnt(u32 size)
+{
+	return size > 8 && size <= 16 ? 2 : 1;
+}
+
+/* Registers holding a function return value, in order. See ret_regs_cnt(). */
+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };
+
+static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
+{
+	const struct btf *btf = env->prog->aux->btf;
+	const struct btf_type *type;
+	int subprog;
+	u32 size;
+
+	if (!env->prog->jit_requested || bpf_prog_is_offloaded(env->prog->aux))
+		return 0;
+
+	/*
+	 * Skip the main program: its return value is the program's exit code,
+	 * read out of R0, so it never uses the register pair. An extension does
+	 * have a real prototype for subprog 0, but bpf_check_attach_target()
+	 * refuses to replace a function returning more than 8 bytes.
+	 */
+	for (subprog = 1; subprog < env->subprog_cnt; subprog++) {
+		type = subprog_ret_type(env, subprog);
+		if (!type || btf_type_is_void(type))
+			continue;
+		if (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, &size)), env,
+				    "cannot size return type of subprog %d", subprog))
+			return -EFAULT;
+		if (ret_regs_cnt(size) > 1) {
+			subprog_info(env, subprog)->ret_reg_pair = true;
+			env->prog->jit_required = 1;
+		}
+	}
+
+	return 0;
 }
 
 const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog)
@@ -2877,6 +2919,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
 	if (err)
 		return err;
+	if (func_model.ret_size > 8) {
+		if (kfunc.flags && (*kfunc.flags & KF_FASTCALL)) {
+			verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n",
+				kfunc.name);
+			return -EOPNOTSUPP;
+		}
+		if (!bpf_jit_supports_kfunc_ret_reg_pair()) {
+			verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
+				kfunc.name);
+			return -EOPNOTSUPP;
+		}
+	}
 
 	memset(&meta, 0, sizeof(meta));
 	meta.btf = kfunc.btf;
@@ -9853,6 +9907,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	u16 callee_incoming, stack_arg_cnt;
 	struct bpf_func_state *caller;
 	int err, subprog, target_insn;
+	u32 i, nregs;
 
 	target_insn = *insn_idx + insn->imm + 1;
 	subprog = bpf_find_subprog(env, target_insn);
@@ -9908,9 +9963,14 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		clear_caller_saved_regs(env, caller->regs);
 		invalidate_outgoing_stack_args(env, cur_func(env));
 
-		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
+		/*
+		 * A non-void global function returns a 64-bit SCALAR_VALUE in
+		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+		 */
 		if (!returns_void) {
-			mark_reg_unknown(env, caller->regs, BPF_REG_0);
+			nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+			for (i = 0; i < nregs; i++)
+				mark_reg_unknown(env, caller->regs, ret_regs[i]);
 			bpf_diag_mod_end(env);
 		}
 
@@ -10270,11 +10330,15 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	struct bpf_func_state *caller, *callee;
 	struct bpf_reg_state *r0;
 	bool in_callback_fn;
+	u32 i, nregs;
 	int err;
 
 	callee = state->frame[state->curframe];
 	r0 = &callee->regs[BPF_REG_0];
-	if (r0->type == PTR_TO_STACK) {
+	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
+			continue;
 		/* technically it's ok to return caller's stack pointer
 		 * (or caller's caller's pointer) back to the caller,
 		 * since these pointers are valid. Only current stack
@@ -10309,9 +10373,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 			return -EFAULT;
 		}
 	} else {
-		/* return to the caller whatever r0 had in the callee */
+		/*
+		 * return to the caller whatever the callee had in the
+		 * return register(s)
+		 */
 		bpf_diag_mod_begin(env, &caller->regs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE);
-		caller->regs[BPF_REG_0] = *r0;
+		for (i = 0; i < nregs; i++)
+			caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]];
 		bpf_diag_mod_end(env);
 	}
 
@@ -11246,6 +11314,19 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 	return 0;
 }
 
+/*
+ * Mark the register(s) holding a @size byte kfunc return value as unknown
+ * scalars. Both halves of a register pair are treated the same way.
+ */
+static void mark_kfunc_ret_regs(struct bpf_verifier_env *env,
+				struct bpf_reg_state *regs, u32 size)
+{
+	u32 i, nregs = ret_regs_cnt(size);
+
+	for (i = 0; i < nregs; i++)
+		mark_reg_unknown(env, regs, ret_regs[i]);
+}
+
 static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ACQUIRE;
@@ -11513,9 +11594,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_
 }
 
 /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
-					const struct btf *btf,
-					const struct btf_type *t, int rec)
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env,
+			       const struct btf *btf,
+			       const struct btf_type *t, int rec)
 {
 	const struct btf_type *member_type;
 	const struct btf_member *member;
@@ -11533,7 +11614,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
 				verbose(env, "max struct nesting depth exceeded\n");
 				return false;
 			}
-			if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
+			if (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
 				return false;
 			continue;
 		}
@@ -11932,7 +12013,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		 (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) ||
 		  is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
 		if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) &&
-		    !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+		    !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 			verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n",
 				reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
 			return -EINVAL;
@@ -11948,7 +12029,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		 * scalars. The access size is derived from the pointed-to BTF type.
 		 */
 		if (!btf_type_is_scalar(ref_t) &&
-		    !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+		    !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 			verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n",
 				reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
 			return -EINVAL;
@@ -13004,7 +13085,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 				break;
 			}
 
-			if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+			if (!btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 				enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);
 				const char *expected_type;
 
@@ -13546,7 +13627,7 @@ static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg
 
 		struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);
 		if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) {
-			if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
+			if (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
 				verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n");
 				return -EINVAL;
 			}
@@ -13914,10 +13995,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	}
 
 	if (btf_type_is_scalar(t)) {
-		mark_reg_unknown(env, regs, BPF_REG_0);
+		mark_kfunc_ret_regs(env, regs, t->size);
 		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
 		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
 			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
+	} else if (btf_type_is_struct(t)) {
+		/*
+		 * The returned struct comes back as raw register bits modeled
+		 * as an unknown scalar, so it must contain only scalars:
+		 * otherwise a pointer field would be laundered into a scalar
+		 * and escape provenance and reference tracking.
+		 */
+		if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+			verbose(env,
+				"kernel function %s returns %s %s that is not composed of scalars\n",
+				func_name, btf_type_str(t),
+				btf_name_by_offset(desc_btf, t->name_off));
+			return -EINVAL;
+		}
+		mark_kfunc_ret_regs(env, regs, t->size);
 	} else if (btf_type_is_ptr(t)) {
 		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
 		err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
@@ -17453,37 +17549,54 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	return 0;
 }
 
-static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
 {
-	struct bpf_reg_state *reg = reg_state(env, BPF_REG_0);
-	struct bpf_func_state *cur_frame = cur_func(env);
+	struct bpf_reg_state *reg;
 	int err;
 
-	if (subprog_returns_void(env, cur_frame->subprogno))
-		return 0;
-
-	err = check_reg_arg(env, BPF_REG_0, SRC_OP);
+	err = check_reg_arg(env, regno, SRC_OP);
 	if (err)
 		return err;
 
 	/* Pointers to arena are safe to pass between subprograms. */
-	if (is_arena_reg(env, BPF_REG_0))
+	if (is_arena_reg(env, regno))
 		return 0;
 
-	if (is_pointer_value(env, BPF_REG_0)) {
-		verbose(env, "R%d leaks addr as return value\n", BPF_REG_0);
+	if (is_pointer_value(env, regno)) {
+		verbose(env, "R%d leaks addr as return value\n", regno);
 		return -EACCES;
 	}
 
+	reg = reg_state(env, regno);
 	if (reg->type != SCALAR_VALUE) {
-		verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n",
-			reg_type_str(env, reg->type));
+		verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
+			regno, reg_type_str(env, reg->type));
 		return -EINVAL;
 	}
 
 	return 0;
 }
 
+static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+{
+	struct bpf_func_state *cur_frame = cur_func(env);
+	u32 subprog = cur_frame->subprogno;
+	u32 i, nregs;
+	int err;
+
+	if (subprog_returns_void(env, subprog))
+		return 0;
+
+	nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		err = check_global_ret_scalar_reg(env, ret_regs[i]);
+		if (err)
+			return err;
+	}
+
+	return 0;
+}
+
 /* Bitmask with 1s for all caller saved registers */
 #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
 
@@ -21151,6 +21264,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* must precede the first bpf_ret_reg_pair() user below */
+	ret = bpf_compute_subprog_ret_regs(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = bpf_compute_live_registers(env);
 	if (ret < 0)
 		goto skip_full_check;
diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
new file mode 100644
index 0000000000000..e0b94ed10f941
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "aggregate_ret_func.skel.h"
+#include "aggregate_ret_kfunc.skel.h"
+
+void test_aggregate_ret(void)
+{
+	RUN_TESTS(aggregate_ret_func);
+	RUN_TESTS(aggregate_ret_kfunc);
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
index 2523c07a16c65..6262c81cdaa9f 100644
--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
@@ -441,6 +441,20 @@ static void test_func_replace_int_with_void(void)
 				     " doesn't match type INT of global_func2()");
 }
 
+static void test_func_replace_ret_pair(void)
+{
+	const char *msg = "Return type of new_agg_ret_target_func() has size 8 "
+			  "and of agg_ret_target_func() size 16";
+
+	/*
+	 * An extension cannot replace a function whose return value comes back
+	 * in the R0:R2 pair: only R0 is checked at the extension's exit, so it
+	 * would leave R2 stale for the target's callers.
+	 */
+	test_obj_load_failure_common("freplace_ret_pair.bpf.o",
+				     "./aggregate_ret_target.bpf.o", msg);
+}
+
 static int find_prog_btf_id(const char *name, __u32 attach_prog_fd)
 {
 	struct bpf_prog_info info = {};
@@ -660,6 +674,8 @@ void serial_test_fexit_bpf2bpf(void)
 		test_func_replace_progmap();
 	if (test__start_subtest("freplace_int_with_void"))
 		test_func_replace_int_with_void();
+	if (test__start_subtest("freplace_ret_pair"))
+		test_func_replace_ret_pair();
 	if (test__start_subtest("freplace_void"))
 		test_func_replace_void();
 	if (test__start_subtest("sleepable_fentry_to_xdp"))
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 64ac49ad67e63..f7f94ccebce27 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -5,6 +5,7 @@
 #include "arena_kfunc.skel.h"
 #include "arena_kfunc_jit.skel.h"
 #include "cap_helpers.h"
+#include "verifier_aggregate_ret.skel.h"
 #include "verifier_align.skel.h"
 #include "verifier_and.skel.h"
 #include "verifier_arena.skel.h"
@@ -170,6 +171,7 @@ void test_arena_kfunc(void)                   { RUN_TESTS(arena_kfunc); }
 
 void test_arena_kfunc_jit(void)               { RUN_TESTS(arena_kfunc_jit); }
 
+void test_verifier_aggregate_ret(void)        { RUN_TESTS(verifier_aggregate_ret); }
 void test_verifier_align(void)                { RUN(verifier_align); }
 void test_verifier_and(void)                  { RUN(verifier_and); }
 void test_verifier_arena(void)                { RUN(verifier_arena); }
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
new file mode 100644
index 0000000000000..6f66fc822cede
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -0,0 +1,237 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+typedef unsigned __int128 u128;
+
+__naked u128 global_agg_good(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"	/* low 64 bits */
+	"r2 = 0x5678;"	/* high 64 bits */
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_global_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad)
+	: __clobber_all);
+}
+
+SEC("tc")
+__load_if_JITed()
+__failure __msg("At subprogram exit the register R2 is not a scalar value")
+__naked int aggregate_ret_global_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad_ptr];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"	/* stack pointer placed in the second return register */
+	"exit;"
+	);
+}
+
+/*
+ * R2 is a return register once the subprogram returns a pair, so a stack
+ * pointer left in it is rejected at the callee's exit exactly as one in R0
+ * is: the callee frame is gone by the time the caller could use it.
+ */
+SEC("tc")
+__load_if_JITed()
+__failure __msg("cannot return stack pointer to the caller")
+__naked int aggregate_ret_static_ptr_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_bad_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_no_r2(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_static_uninit_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_no_r2];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(static_agg_no_r2)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_precise(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 4;"	/* second half; its value is made precise below */
+	"exit;"
+	);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
+__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
+__naked int aggregate_ret_static_precise(void)
+{
+	asm volatile (
+	"call %[static_agg_precise];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_precise)
+	: __clobber_all);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
+__naked int aggregate_ret_global_precise(void)
+{
+	asm volatile (
+	"call %[global_agg_good];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_agg_good)
+	: __clobber_all);
+}
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+/* A by-value struct that smuggles a pointer, which must be rejected. */
+struct with_ptr {
+	void *p;
+	__u64 x;
+};
+
+/* A by-value union that smuggles a pointer, which must be rejected too. */
+union upair_with_ptr {
+	void *p;
+	__u64 halves[2];
+};
+
+__naked struct with_ptr global_ret_struct_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_struct_ptr() has unsupported return type")
+__naked int aggregate_ret_global_struct_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_struct_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_struct_ptr)
+	: __clobber_all);
+}
+
+__naked union upair_with_ptr global_ret_union_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_union_ptr() has unsupported return type")
+__naked int aggregate_ret_global_union_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_union_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_union_ptr)
+	: __clobber_all);
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
new file mode 100644
index 0000000000000..d6b422ae9784b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -0,0 +1,121 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+/*
+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address
+ * (rather than calling) avoids any dependence on the compiler lowering an
+ * __int128 or struct return value, which the BPF backend only supports from
+ * LLVM 23 on.
+ */
+void __kfunc_btf_root(void)
+{
+	asm volatile (""
+	:
+	: "r"(&bpf_kfunc_call_test_i128),
+	  "r"(&bpf_kfunc_call_test_ret_fastcall),
+	  "r"(&bpf_kfunc_call_test_ret_ptr),
+	  "r"(&bpf_kfunc_call_test_ret_ii),
+	  "r"(&bpf_kfunc_call_test_ret_big));
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
+__naked int aggregate_ret_kfunc_precise(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_i128];"
+	"r6 = r2;"		/* second return half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_i128)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL")
+__naked int aggregate_ret_kfunc_fastcall_fail(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_ret_fastcall];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_fastcall)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("is not composed of scalars")
+__naked int aggregate_ret_kfunc_ptr_fail(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ptr)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_kfunc_small_no_r2(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"r2 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ii];"
+	"r0 = r2;"	/* R2 is not a return register for a <=8 byte struct */
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ii)
+	: __clobber_all);
+}
+
+/*
+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
+ * btf_distill_func_proto() before the KF_FASTCALL and JIT-capability checks.
+ */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported")
+__naked int aggregate_ret_kfunc_too_big_fail(void)
+{
+	asm volatile (
+	"call %[bpf_kfunc_call_test_ret_big];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_big)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
new file mode 100644
index 0000000000000..cffd8d7d3241a
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* freplace target: a global subprogram returning 16 bytes in R0:R2. */
+__naked unsigned __int128 agg_ret_target_func(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"
+	"r2 = 0x5678;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__naked int agg_ret_target(void)
+{
+	asm volatile (
+	"call %[agg_ret_target_func];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(agg_ret_target_func)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index d055fc7b3b95d..338e67cd8856a 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -431,6 +431,34 @@ __naked void subprog1(void)
 		::: __clobber_all);
 }
 
+static __used __naked unsigned __int128 aux2(void)
+{
+	asm volatile (
+		"r0 = 1;"
+		"r2 = 2;"
+		"exit;"
+		::: __clobber_all);
+}
+
+SEC("socket")
+/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
+__load_if_JITed()
+__log_level(2)
+__msg("0: .12345.... (85) call pc+2")
+__msg("1: ..2....... (bf) r0 = r2")
+__msg("2: 0......... (95) exit")
+__msg("3: .......... (b7) r0 = 1")
+__msg("4: 0......... (b7) r2 = 2")
+__msg("5: 0.2....... (95) exit")
+__naked void subprog_ret_reg_pair(void)
+{
+	asm volatile (
+		"call aux2;"
+		"r0 = r2;"
+		"exit;"
+		::: __clobber_all);
+}
+
 #if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
 
 SEC("socket")
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index ac44d60e50666..9708efb93683b 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)
 
 SEC("?tc")
 __exception_cb(exception_cb_bad_ret_type1)
-__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.")
+__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.")
 int reject_exception_cb_type_1(struct __sk_buff *ctx)
 {
 	bpf_throw(0);
diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
new file mode 100644
index 0000000000000..12c15d293bd79
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+SEC("freplace/agg_ret_target_func")
+__u64 new_agg_ret_target_func(void)
+{
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c
new file mode 100644
index 0000000000000..7851bade2b406
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c
@@ -0,0 +1,178 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+struct pair {
+	__u64 lo;	/* R0 */
+	__u64 hi;	/* R2 */
+};
+
+union upair {
+	__u64 halves[2];
+	struct {
+		__u64 lo;	/* R0 */
+		__u64 hi;	/* R2 */
+	} parts;
+};
+
+static __noinline u128 make_i128(__u64 a, __u64 b)
+{
+	return ((u128)(a + b) << 64) | (a - b);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v;
+
+	v = make_i128(a, b);
+	if ((__u64)(v >> 64) != a + b)
+		return 1;
+	if ((__u64)v != a - b)
+		return 2;
+
+	return 0;
+}
+
+static __noinline struct pair make_pair(__u64 a, __u64 b)
+{
+	struct pair p = { .lo = a + b, .hi = a - b };
+
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p;
+
+	p = make_pair(a, b);
+	if (p.lo != a + b)
+		return 1;
+	if (p.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+__noinline struct pair make_pair_global(__u64 a, __u64 b)
+{
+	struct pair p = { .lo = a + b, .hi = a - b };
+
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p;
+
+	p = make_pair_global(a, b);
+	if (p.lo != a + b)
+		return 1;
+	if (p.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+static __noinline union upair make_upair(__u64 a, __u64 b)
+{
+	union upair p;
+
+	p.halves[0] = a + b;
+	p.halves[1] = a - b;
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_union_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	union upair p;
+
+	p = make_upair(a, b);
+	if (p.parts.lo != a + b)
+		return 1;
+	if (p.parts.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v;
+
+	v = bpf_kfunc_call_test_i128(a, b);
+	if ((__u64)(v >> 64) != a + b)
+		return 1;
+	if ((__u64)v != a - b)
+		return 2;
+
+	return 0;
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct prog_test_ret_pair p;
+
+	p = bpf_kfunc_call_test_ret_pair(a, b);
+	if (p.lo != a + b)
+		return 1;
+	if (p.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+SEC("socket")
+__description("verifier_aggregate_ret: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 9366a3c578f13..850cf4f830c48 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -954,6 +954,53 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
 	return 0;
 }
 
+#if defined(__x86_64__) || defined(__aarch64__)
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+	return (__int128)(((unsigned __int128)(a + b) << 64) | (a - b));
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)
+{
+	struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)
+{
+	struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
+{
+	struct prog_test_ret_ptr r = { .p = NULL, .tag = tag };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
+{
+	struct prog_test_ret_ii r = { .a = a, .b = b };
+
+	return r;
+}
+#endif /* __x86_64__ || __aarch64__ */
+
+/*
+ * Takes no argument on purpose: with no arguments there is nothing for the sret
+ * pointer to displace, so this needs no architecture guard even though it
+ * returns 24 bytes.
+ */
+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)
+{
+	struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };
+
+	return r;
+}
+
 __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
 					 u64 e, u64 f, u64 g, u64 h,
 					 u64 i, u64 j)
@@ -1487,6 +1534,14 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+#if defined(__x86_64__) || defined(__aarch64__)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
+#endif
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index 7d81070eefe7f..65e693ada7364 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -55,6 +55,27 @@ struct prog_test_big_arg {
 	__u64 b;
 };
 
+struct prog_test_ret_pair {	/* 16 bytes: R0:R2 */
+	__u64 lo;
+	__u64 hi;
+};
+
+struct prog_test_ret_ii {	/* 8 bytes: R0 only */
+	int a;
+	int b;
+};
+
+struct prog_test_ret_ptr {	/* 16 bytes: contains a pointer */
+	void *p;
+	__u64 tag;
+};
+
+struct prog_test_ret_big {	/* 24 bytes: too large for R0:R2 */
+	__u64 a;
+	__u64 b;
+	__u64 c;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -131,6 +152,14 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+#ifdef __SIZEOF_INT128__
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
+#endif
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;
+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym;
 __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
 			       __u64 e, __u64 f, __u64 g, __u64 h,
 			       __u64 i, __u64 j) __ksym;

gemini-3.1-pro-preview

							input: 24923
							output: 61
							thoughts: 974

**Fuzzing Strategy for BPF Kfunc Return Value Expansion**

Okay, so we have this patch series introducing support for BPF kfuncs to return up to 16 bytes, using the R0:R2 register pair. This includes support for `__int128` and scalar-only structs/unions. This has definitely made significant changes, and my initial thought is that this needs to be fuzzed heavily.

I've got a good handle on where the changes are: the BPF verifier, JIT compilers (x86, arm64, riscv), and some backtracking logic.

Specifically, I need to focus my fuzzing efforts on the kernel functions that have been modified. I've compiled a list:

In `kernel/bpf/verifier.c` I'll need to look at:
- `bpf_compute_subprog_ret_regs`
- `check_global_subprog_return_code`
- `check_global_ret_scalar_reg`
- `mark_kfunc_ret_regs`
- `check_kfunc_call`
- `check_func_call`
- `prepare_func_exit`

In `kernel/bpf/btf.c`:
- `btf_validate_return_type`
- `btf_check_func_type_match`

In `kernel/bpf/backtrack.c`:
- `backtrack_insn`

In `kernel/bpf/liveness.c`:
- `compute_insn_live_regs`
- `bpf_compute_live_registers`

And in `arch/x86/net/bpf_jit_comp.c`:
- `emit_kfunc_arena_args`
- `do_jit` - specifically the `BPF_JMP | BPF_CALL` case.

I need to quickly check the visibility of some of these, as I'm sure some are declared static or local.
`btf_type_is_scalar_struct` is global.
`bpf_compute_subprog_ret_regs` is static.
`bpf_compute_live_registers` is global.
`btf_validate_return_type` is static.
`backtrack_insn` is static.

Based on this, my first target list would be:
- `bpf_check` - the main entry point
- `bpf_compute_live_registers`
- `btf_type_is_scalar_struct`
- `btf_check_func_type_match`
- `bpf_jit_supports_kfunc_ret_reg_pair` (although this is a simple boolean check, so might not need extensive fuzzing)

Now, I'm reconsidering. Wait, those last three are static! So, I need to add those to my focus:
- `check_kfunc_call`
- `check_func_call`
- `prepare_func_exit`

I'll use `grepper` to find the non-static functions again to make sure I've got the most relevant ones to target.

{
  "Expression": "^(int|void|bool|struct [a-zA-Z0-9_]+ \\*) [a-zA-Z0-9_]+\\(",
  "PathPrefix": "kernel/bpf/verifier.c"
}

{
  "Output": "kernel/bpf/verifier.c=366=static struct btf_record *reg_btf_record(const struct bpf_reg_state *reg)\n--\nkernel/bpf/verifier.c-380-\nkernel/bpf/verifier.c:381:bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)\nkernel/bpf/verifier.c-382-{\n--\nkernel/bpf/verifier.c=453=const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog)\n--\nkernel/bpf/verifier.c-463-\nkernel/bpf/verifier.c:464:void bpf_mark_subprog_exc_cb(struct bpf_verifier_env *env, int subprog)\nkernel/bpf/verifier.c-465-{\n--\nkernel/bpf/verifier.c=540=static bool is_callback_calling_function(enum bpf_func_id func_id)\n--\nkernel/bpf/verifier.c-545-\nkernel/bpf/verifier.c:546:bool bpf_is_sync_callback_calling_insn(struct bpf_insn *insn)\nkernel/bpf/verifier.c-547-{\n--\nkernel/bpf/verifier.c-551-\nkernel/bpf/verifier.c:552:bool bpf_is_async_callback_calling_insn(struct bpf_insn *insn)\nkernel/bpf/verifier.c-553-{\n--\nkernel/bpf/verifier.c=558=static bool is_async_cb_sleepable(struct bpf_verifier_env *env, struct bpf_insn *insn)\n--\nkernel/bpf/verifier.c-572-\nkernel/bpf/verifier.c:573:bool bpf_is_may_goto_insn(struct bpf_insn *insn)\nkernel/bpf/verifier.c-574-{\n--\nkernel/bpf/verifier.c=1635=static void free_func_state(struct bpf_func_state *state)\n--\nkernel/bpf/verifier.c-1643-\nkernel/bpf/verifier.c:1644:void bpf_clear_jmp_history(struct bpf_verifier_state *state)\nkernel/bpf/verifier.c-1645-{\n--\nkernel/bpf/verifier.c-1650-\nkernel/bpf/verifier.c:1651:void bpf_free_verifier_state(struct bpf_verifier_state *state,\nkernel/bpf/verifier.c-1652-\t\t\t    bool free_self)\n--\nkernel/bpf/verifier.c=1669=static int copy_func_state(struct bpf_func_state *dst,\n--\nkernel/bpf/verifier.c-1677-\nkernel/bpf/verifier.c:1678:int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,\nkernel/bpf/verifier.c-1679-\t\t\t   const struct bpf_verifier_state *src)\n--\nkernel/bpf/verifier.c=1740=static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_state *b)\n--\nkernel/bpf/verifier.c-1753-\nkernel/bpf/verifier.c:1754:void bpf_free_backedges(struct bpf_scc_visit *visit)\nkernel/bpf/verifier.c-1755-{\n--\nkernel/bpf/verifier.c=2189=static int reg_bounds_sanity_check(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-2223-/* Mark a register as having a completely unknown (scalar) value. */\nkernel/bpf/verifier.c:2224:void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)\nkernel/bpf/verifier.c-2225-{\n--\nkernel/bpf/verifier.c=2248=static int __mark_reg_s32_range(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-2267-\nkernel/bpf/verifier.c:2268:void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,\nkernel/bpf/verifier.c-2269-\t\t\t   struct bpf_reg_state *reg)\n--\nkernel/bpf/verifier.c=2392=struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *env, int off)\n--\nkernel/bpf/verifier.c-2412-/* Find subprogram that starts exactly at 'off' */\nkernel/bpf/verifier.c:2413:int bpf_find_subprog(struct bpf_verifier_env *env, int off)\nkernel/bpf/verifier.c-2414-{\n--\nkernel/bpf/verifier.c=2539=find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset)\n--\nkernel/bpf/verifier.c-2551-\nkernel/bpf/verifier.c:2552:int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,\nkernel/bpf/verifier.c-2553-\t\t       u16 btf_fd_idx, u8 **func_addr)\n--\nkernel/bpf/verifier.c=2642=static struct btf *__find_kfunc_desc_btf(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-2691-\nkernel/bpf/verifier.c:2692:void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab)\nkernel/bpf/verifier.c-2693-{\n--\nkernel/bpf/verifier.c=2832=static int gen_kfunc_arg_proto(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,\n--\nkernel/bpf/verifier.c-2834-\nkernel/bpf/verifier.c:2835:int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)\nkernel/bpf/verifier.c-2836-{\n--\nkernel/bpf/verifier.c=3358=const char *bpf_disasm_kfunc_name(void *data, const struct bpf_insn *insn)\n--\nkernel/bpf/verifier.c-3375-\nkernel/bpf/verifier.c:3376:void bpf_verbose_insn(struct bpf_verifier_env *env, struct bpf_insn *insn)\nkernel/bpf/verifier.c-3377-{\n--\nkernel/bpf/verifier.c-3389- */\nkernel/bpf/verifier.c:3390:void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist)\nkernel/bpf/verifier.c-3391-{\n--\nkernel/bpf/verifier.c-3422-\nkernel/bpf/verifier.c:3423:int mark_chain_precision(struct bpf_verifier_env *env, int regno)\nkernel/bpf/verifier.c-3424-{\n--\nkernel/bpf/verifier.c=5766=static void coerce_subreg_to_size_sx(struct bpf_reg_state *reg, int size)\n--\nkernel/bpf/verifier.c-5811-\nkernel/bpf/verifier.c:5812:bool bpf_map_is_rdonly(const struct bpf_map *map)\nkernel/bpf/verifier.c-5813-{\n--\nkernel/bpf/verifier.c-5831-\nkernel/bpf/verifier.c:5832:int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,\nkernel/bpf/verifier.c-5833-\t\t\tbool is_ldsx)\n--\nkernel/bpf/verifier.c=8958=static bool may_update_sockmap(struct bpf_verifier_env *env, int func_id)\n--\nkernel/bpf/verifier.c-8990-\nkernel/bpf/verifier.c:8991:bool bpf_allow_tail_call_in_subprogs(struct bpf_verifier_env *env)\nkernel/bpf/verifier.c-8992-{\n--\nkernel/bpf/verifier.c=9902=static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n--\nkernel/bpf/verifier.c-10024-\nkernel/bpf/verifier.c:10025:int map_set_for_each_callback_args(struct bpf_verifier_env *env,\nkernel/bpf/verifier.c-10026-\t\t\t\t   struct bpf_func_state *caller,\n--\nkernel/bpf/verifier.c=10734=static bool can_elide_value_nullness(const struct bpf_map *map)\n--\nkernel/bpf/verifier.c-10747-\nkernel/bpf/verifier.c:10748:int bpf_get_helper_proto(struct bpf_verifier_env *env, int func_id,\nkernel/bpf/verifier.c-10749-\t\t\t const struct bpf_func_proto **ptr)\n--\nkernel/bpf/verifier.c=11581=static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_idx)\n--\nkernel/bpf/verifier.c-11596-/* Returns true if struct is composed of scalars, 4 levels of nesting allowed */\nkernel/bpf/verifier.c:11597:bool btf_type_is_scalar_struct(struct bpf_verifier_env *env,\nkernel/bpf/verifier.c-11598-\t\t\t       const struct btf *btf,\n--\nkernel/bpf/verifier.c=11901=static bool is_kfunc_bpf_preempt_enable(struct bpf_call_arg_meta *meta)\n--\nkernel/bpf/verifier.c-11905-\nkernel/bpf/verifier.c:11906:bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta)\nkernel/bpf/verifier.c-11907-{\n--\nkernel/bpf/verifier.c=12388=static bool is_async_callback_calling_kfunc(u32 btf_id)\n--\nkernel/bpf/verifier.c-12393-\nkernel/bpf/verifier.c:12394:bool bpf_is_throw_kfunc(struct bpf_insn *insn)\nkernel/bpf/verifier.c-12395-{\n--\nkernel/bpf/verifier.c=12645=static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,\n--\nkernel/bpf/verifier.c-13336-\nkernel/bpf/verifier.c:13337:int bpf_fetch_kfunc_arg_meta(struct bpf_verifier_env *env,\nkernel/bpf/verifier.c-13338-\t\t\t     s32 func_id,\n--\nkernel/bpf/verifier.c=17580=static int check_global_subprog_return_code(struct bpf_verifier_env *env)\n--\nkernel/bpf/verifier.c-17606- */\nkernel/bpf/verifier.c:17607:bool bpf_verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm)\nkernel/bpf/verifier.c-17608-{\n--\nkernel/bpf/verifier.c-17625- */\nkernel/bpf/verifier.c:17626:bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,\nkernel/bpf/verifier.c-17627-\t\t\t  struct bpf_call_summary *cs)\n--\nkernel/bpf/verifier.c=17888=static u32 idset_cnt_get(struct bpf_idset *idset, u32 id)\n--\nkernel/bpf/verifier.c-17903- */\nkernel/bpf/verifier.c:17904:void bpf_clear_singular_ids(struct bpf_verifier_env *env,\nkernel/bpf/verifier.c-17905-\t\t\t    struct bpf_verifier_state *st)\n--\nkernel/bpf/verifier.c=19626=static void print_verification_stats(struct bpf_verifier_env *env)\n--\nkernel/bpf/verifier.c-19655-\nkernel/bpf/verifier.c:19656:int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,\nkernel/bpf/verifier.c-19657-\t\t\t       const struct bpf_ctx_arg_aux *info, u32 cnt)\n--\nkernel/bpf/verifier.c=19979=static bool attach_uses_trampoline_retval(enum bpf_attach_type type)\n--\nkernel/bpf/verifier.c-19992-\nkernel/bpf/verifier.c:19993:int bpf_check_attach_target(struct bpf_verifier_log *log,\nkernel/bpf/verifier.c-19994-\t\t\t    const struct bpf_prog *prog,\n--\nkernel/bpf/verifier.c=20416=static int check_attach_btf_id(struct bpf_verifier_env *env)\n--\nkernel/bpf/verifier.c-20515-\nkernel/bpf/verifier.c:20516:int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 btf_id,\nkernel/bpf/verifier.c-20517-\t\t\t\t  struct bpf_attach_target_info *tgt_info)\n--\nkernel/bpf/verifier.c=20743=static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data *insn_aux,\n--\nkernel/bpf/verifier.c-20759-\nkernel/bpf/verifier.c:20760:int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\nkernel/bpf/verifier.c-20761-\t\t     struct bpf_insn *insn_buf, int insn_idx, int *cnt)\n--\nkernel/bpf/verifier.c=20972=static int bpf_prog_verify_signature(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-21088-\nkernel/bpf/verifier.c:21089:int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\nkernel/bpf/verifier.c-21090-\t      struct bpf_log_attr *attr_log)\n"
}

gemini-3.1-pro-preview

							input: 30702
							output: 62
							thoughts: 466

**My Analysis of the Code Changes**

Okay, let's break down these function changes. I need to get a handle on what's being touched here. First, I'm focusing on the non-static, global functions, and what is being modified. I see `btf_type_is_scalar_struct` and `bpf_jit_supports_kfunc_ret_reg_pair` popping up. `bpf_jit_supports_kfunc_ret_reg_pair` is architecture-specific, and shows up in the JIT compiler, which makes sense. I have to look into this more closely to see how it's being used.

Then there's `bpf_check` - a hot-path entry point. That's a good place to start, even though it's generic. I'll need to see how the interactions are happening there. I see `bpf_compute_live_registers` is being flagged, even though it's part of the verifier, I can't overlook it. I see the function `bpf_add_kfunc_call` is modified, which warrants another look. `bpf_check_attach_target` is related and I see `btf_check_func_type_match` is a call from there, so I'll need to keep this in mind as well.

Now, let's take another pass at the patch. I'm noting the static helper functions: `bpf_compute_subprog_ret_regs`, `check_global_subprog_return_code`, `check_kfunc_call`, `check_func_call`, and `prepare_func_exit`. I'll want to understand their roles within the broader context.

So, let's create a list to track: I should definitely include `btf_type_is_scalar_struct` and `bpf_jit_supports_kfunc_ret_reg_pair` in `FocusSymbols`. Then, from the modified non-static functions, I see the ones I need to examine: `bpf_compute_live_registers`, `btf_distill_func_proto`, and `btf_prepare_func_args`. I have to dig into those now, particularly `btf_distill_func_proto` and `btf_prepare_func_args`.

{
  "Expression": "^(int|void|bool|struct [a-zA-Z0-9_]+ \\*) [a-zA-Z0-9_]+\\(",
  "PathPrefix": "kernel/bpf/btf.c"
}

{
  "Output": "kernel/bpf/btf.c=500=static int btf_start_id(const struct btf *btf)\n--\nkernel/bpf/btf.c-504-\nkernel/bpf/btf.c:505:bool btf_type_is_void(const struct btf_type *t)\nkernel/bpf/btf.c-506-{\n--\nkernel/bpf/btf.c=532=static bool btf_type_is_decl_tag_target(const struct btf_type *t)\n--\nkernel/bpf/btf.c-537-\nkernel/bpf/btf.c:538:bool btf_is_vmlinux(const struct btf *btf)\nkernel/bpf/btf.c-539-{\n--\nkernel/bpf/btf.c=991=static bool btf_type_int_is_regular(const struct btf_type *t)\n--\nkernel/bpf/btf.c-995-\nkernel/bpf/btf.c:996:bool btf_type_is_i32(const struct btf_type *t)\nkernel/bpf/btf.c-997-{\n--\nkernel/bpf/btf.c-1000-\nkernel/bpf/btf.c:1001:bool btf_type_is_i64(const struct btf_type *t)\nkernel/bpf/btf.c-1002-{\n--\nkernel/bpf/btf.c-1005-\nkernel/bpf/btf.c:1006:bool btf_type_is_primitive(const struct btf_type *t)\nkernel/bpf/btf.c-1007-{\n--\nkernel/bpf/btf.c-1015- */\nkernel/bpf/btf.c:1016:bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,\nkernel/bpf/btf.c-1017-\t\t\t   const struct btf_member *m,\n--\nkernel/bpf/btf.c=1891=const char *btf_get_name(const struct btf *btf)\n--\nkernel/bpf/btf.c-1895-\nkernel/bpf/btf.c:1896:void btf_get(struct btf *btf)\nkernel/bpf/btf.c-1897-{\n--\nkernel/bpf/btf.c-1900-\nkernel/bpf/btf.c:1901:void btf_put(struct btf *btf)\nkernel/bpf/btf.c-1902-{\n--\nkernel/bpf/btf.c=1914=const struct btf_header *btf_header(const struct btf *btf)\n--\nkernel/bpf/btf.c-1918-\nkernel/bpf/btf.c:1919:void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)\nkernel/bpf/btf.c-1920-{\n--\nkernel/bpf/btf.c=3526=static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,\n--\nkernel/bpf/btf.c-3569-\nkernel/bpf/btf.c:3570:int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *pt,\nkernel/bpf/btf.c-3571-\t\t\t   int comp_idx, const char *tag_key, int last_id)\n--\nkernel/bpf/btf.c=4119=struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,\n--\nkernel/bpf/btf.c-4251-\nkernel/bpf/btf.c:4252:int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec)\nkernel/bpf/btf.c-4253-{\n--\nkernel/bpf/btf.c=6106=static int find_kern_ctx_type_id(enum bpf_prog_type prog_type)\n--\nkernel/bpf/btf.c-6121-\nkernel/bpf/btf.c:6122:bool btf_is_projection_of(const char *pname, const char *tname)\nkernel/bpf/btf.c-6123-{\n--\nkernel/bpf/btf.c-6130-\nkernel/bpf/btf.c:6131:bool btf_is_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,\nkernel/bpf/btf.c-6132-\t\t\t  const struct btf_type *t, enum bpf_prog_type prog_type,\n--\nkernel/bpf/btf.c=6352=static int btf_translate_to_vmlinux(struct bpf_verifier_log *log,\n--\nkernel/bpf/btf.c-6362-\nkernel/bpf/btf.c:6363:int get_kern_ctx_btf_id(struct bpf_verifier_log *log, enum bpf_prog_type prog_type)\nkernel/bpf/btf.c-6364-{\n--\nkernel/bpf/btf.c=6616=static bool prog_args_trusted(const struct bpf_prog *prog)\n--\nkernel/bpf/btf.c-6631-\nkernel/bpf/btf.c:6632:int btf_ctx_arg_offset(const struct btf *btf, const struct btf_type *func_proto,\nkernel/bpf/btf.c-6633-\t\t       u32 arg_no)\n--\nkernel/bpf/btf.c=6657=static const struct bpf_raw_tp_null_args raw_tp_null_args[] = {\n--\nkernel/bpf/btf.c-6848-\nkernel/bpf/btf.c:6849:bool btf_ctx_access(int off, int size, enum bpf_access_type type,\nkernel/bpf/btf.c-6850-\t\t    const struct bpf_prog *prog,\n--\nkernel/bpf/btf.c=7113=static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,\n--\nkernel/bpf/btf.c-7378-\nkernel/bpf/btf.c:7379:int btf_struct_access(struct bpf_verifier_log *log,\nkernel/bpf/btf.c-7380-\t\t      const struct bpf_reg_state *reg,\n--\nkernel/bpf/btf.c-7461- */\nkernel/bpf/btf.c:7462:bool btf_types_are_same(const struct btf *btf1, u32 id1,\nkernel/bpf/btf.c-7463-\t\t\tconst struct btf *btf2, u32 id2)\n--\nkernel/bpf/btf.c-7471-\nkernel/bpf/btf.c:7472:bool btf_struct_ids_match(struct bpf_verifier_log *log,\nkernel/bpf/btf.c-7473-\t\t\t  const struct btf *btf, u32 id, int off,\n--\nkernel/bpf/btf.c=7545=static u8 __get_arg_fmodel_flags(const struct btf *btf,\n--\nkernel/bpf/btf.c-7560-\nkernel/bpf/btf.c:7561:int btf_distill_func_proto(struct bpf_verifier_log *log,\nkernel/bpf/btf.c-7562-\t\t\t   struct btf *btf,\n--\nkernel/bpf/btf.c=7640=static int btf_check_func_type_match(struct bpf_verifier_log *log,\n--\nkernel/bpf/btf.c-7756-/* Compare BTFs of given program with BTF of target program */\nkernel/bpf/btf.c:7757:int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog,\nkernel/bpf/btf.c-7758-\t\t\t struct btf *btf2, const struct btf_type *t2)\n--\nkernel/bpf/btf.c=7972=static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,\n--\nkernel/bpf/btf.c-8019- */\nkernel/bpf/btf.c:8020:int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)\nkernel/bpf/btf.c-8021-{\n--\nkernel/bpf/btf.c=8261=__printf(2, 0) static void btf_seq_show(struct btf_show *show, const char *fmt,\n--\nkernel/bpf/btf.c-8266-\nkernel/bpf/btf.c:8267:int btf_type_seq_show_flags(const struct btf *btf, u32 type_id,\nkernel/bpf/btf.c-8268-\t\t\t    void *obj, struct seq_file *m, u64 flags)\n--\nkernel/bpf/btf.c-8280-\nkernel/bpf/btf.c:8281:void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,\nkernel/bpf/btf.c-8282-\t\t       struct seq_file *m)\n--\nkernel/bpf/btf.c=8295=__printf(2, 0) static void btf_snprintf_show(struct btf_show *show, const char *fmt,\n--\nkernel/bpf/btf.c-8316-\nkernel/bpf/btf.c:8317:int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,\nkernel/bpf/btf.c-8318-\t\t\t   char *buf, int len, u64 flags)\n--\nkernel/bpf/btf.c-8337-\nkernel/bpf/btf.c:8338:int btf_type_name_to_buf(const struct btf *btf, u32 type_id, char *buf, int len)\nkernel/bpf/btf.c-8339-{\n--\nkernel/bpf/btf.c=8370=static int __btf_new_fd(struct btf *btf)\n--\nkernel/bpf/btf.c-8374-\nkernel/bpf/btf.c:8375:int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)\nkernel/bpf/btf.c-8376-{\n--\nkernel/bpf/btf.c=8403=struct btf *btf_get_by_fd(int fd)\n--\nkernel/bpf/btf.c-8414-\nkernel/bpf/btf.c:8415:int btf_get_info_by_fd(const struct btf *btf,\nkernel/bpf/btf.c-8416-\t\t       const union bpf_attr *attr,\n--\nkernel/bpf/btf.c-8474-\nkernel/bpf/btf.c:8475:int btf_get_fd_by_id(u32 id)\nkernel/bpf/btf.c-8476-{\n--\nkernel/bpf/btf.c=8496=u32 btf_obj_id(const struct btf *btf)\n--\nkernel/bpf/btf.c-8500-\nkernel/bpf/btf.c:8501:bool btf_is_kernel(const struct btf *btf)\nkernel/bpf/btf.c-8502-{\n--\nkernel/bpf/btf.c-8505-\nkernel/bpf/btf.c:8506:bool btf_is_module(const struct btf *btf)\nkernel/bpf/btf.c-8507-{\n--\nkernel/bpf/btf.c=8765=BTF_TRACING_TYPE_xxx\n--\nkernel/bpf/btf.c-8771- */\nkernel/bpf/btf.c:8772:int btf_check_iter_arg(struct btf *btf, const struct btf_type *func, int arg_idx)\nkernel/bpf/btf.c-8773-{\n--\nkernel/bpf/btf.c=9082=static int bpf_prog_type_to_kfunc_hook(enum bpf_prog_type prog_type)\n--\nkernel/bpf/btf.c-9128-\nkernel/bpf/btf.c:9129:bool btf_kfunc_is_allowed(const struct btf *btf,\nkernel/bpf/btf.c-9130-\t\t\t  u32 kfunc_btf_id,\n--\nkernel/bpf/btf.c=9156=u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog)\n--\nkernel/bpf/btf.c-9177- */\nkernel/bpf/btf.c:9178:int btf_kfunc_check_flag(const struct btf *btf, u32 kfunc_btf_id, u32 flag)\nkernel/bpf/btf.c-9179-{\n--\nkernel/bpf/btf.c=9208=static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook,\n--\nkernel/bpf/btf.c-9234-/* This function must be invoked only from initcalls/module init functions */\nkernel/bpf/btf.c:9235:int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,\nkernel/bpf/btf.c-9236-\t\t\t      const struct btf_kfunc_id_set *kset)\n--\nkernel/bpf/btf.c=9251=EXPORT_SYMBOL_GPL(register_btf_kfunc_id_set);\n--\nkernel/bpf/btf.c-9253-/* This function must be invoked only from initcalls/module init functions */\nkernel/bpf/btf.c:9254:int register_btf_fmodret_id_set(const struct btf_kfunc_id_set *kset)\nkernel/bpf/btf.c-9255-{\n--\nkernel/bpf/btf.c=9277=static int btf_check_dtor_kfuncs(struct btf *btf, const struct btf_id_dtor_kfunc *dtors, u32 cnt)\n--\nkernel/bpf/btf.c-9321-/* This function must be invoked only from initcalls/module init functions */\nkernel/bpf/btf.c:9322:int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,\nkernel/bpf/btf.c-9323-\t\t\t\tstruct module *owner)\n--\nkernel/bpf/btf.c=9395=EXPORT_SYMBOL_GPL(register_btf_id_dtor_kfuncs);\n--\nkernel/bpf/btf.c-9417- */\nkernel/bpf/btf.c:9418:int bpf_core_types_are_compat(const struct btf *local_btf, __u32 local_id,\nkernel/bpf/btf.c-9419-\t\t\t      const struct btf *targ_btf, __u32 targ_id)\n--\nkernel/bpf/btf.c-9426-\nkernel/bpf/btf.c:9427:int bpf_core_types_match(const struct btf *local_btf, u32 local_id,\nkernel/bpf/btf.c-9428-\t\t\t const struct btf *targ_btf, u32 targ_id)\n--\nkernel/bpf/btf.c=9647=bpf_core_find_cands(struct bpf_core_ctx *ctx, u32 local_type_id)\n--\nkernel/bpf/btf.c-9733-\nkernel/bpf/btf.c:9734:int bpf_core_apply(struct bpf_core_ctx *ctx, const struct bpf_core_relo *relo,\nkernel/bpf/btf.c-9735-\t\t   int relo_idx, void *insn)\n--\nkernel/bpf/btf.c-9812-\nkernel/bpf/btf.c:9813:bool btf_nested_type_is_trusted(struct bpf_verifier_log *log,\nkernel/bpf/btf.c-9814-\t\t\t\tconst struct bpf_reg_state *reg,\n--\nkernel/bpf/btf.c-9859-\nkernel/bpf/btf.c:9860:bool btf_type_ids_nocast_alias(struct bpf_verifier_log *log,\nkernel/bpf/btf.c-9861-\t\t\t       const struct btf *reg_btf, u32 reg_id,\n--\nkernel/bpf/btf.c=9981=bpf_struct_ops_find(struct btf *btf, u32 type_id)\n--\nkernel/bpf/btf.c-10001-\nkernel/bpf/btf.c:10002:int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)\nkernel/bpf/btf.c-10003-{\n--\nkernel/bpf/btf.c=10030=EXPORT_SYMBOL_GPL(__register_bpf_struct_ops);\n--\nkernel/bpf/btf.c-10032-\nkernel/bpf/btf.c:10033:bool btf_param_match_suffix(const struct btf *btf,\nkernel/bpf/btf.c-10034-\t\t\t    const struct btf_param *arg,\n"
}

gemini-3.1-pro-preview

							input: 36671
							output: 262
							thoughts: 1424

**Analysis of BPF Kfunc Return Value Extension Patch**

Okay, let's break this down. It looks like we're enabling BPF kfuncs and subprograms to return 16-byte values using a register pair (R0:R2). This touches a bunch of key areas. First, `btf_distill_func_proto` and `btf_prepare_func_args` have been modified, and are now non-static. `btf_distill_func_proto` is crucial for extracting the function prototype for kfuncs, and `btf_prepare_func_args` deals with preparing the arguments for subprograms.

I see that `bpf_jit_supports_kfunc_ret_reg_pair` is also modified. It's added in `kernel/bpf/core.c` and overridden in the architecture-specific JIT compilers (`x86`, `arm64`, `riscv`). Good. Seems it is enabled by default for these architectures. No new config options are added, which is nice. We just need the standard BPF configs – `BPF_SYSCALL` and `BPF_JIT`.

Now, let's look at the verifier changes in `kernel/bpf/verifier.c`. We have new functions like `bpf_compute_subprog_ret_regs`, `check_global_ret_scalar_reg`, and `mark_kfunc_ret_regs`. Several existing functions are modified, including `check_kfunc_call`, `check_func_call`, and `prepare_func_exit`. The addition of `btf_type_is_scalar_struct` is interesting, and it's a global function.

I need to focus on a few key symbols here. I'll put `btf_type_is_scalar_struct`, `btf_distill_func_proto`, and `btf_prepare_func_args` in `FocusSymbols`. Considering performance, `bpf_compute_live_registers` is a very hot path function called for every BPF program load, so that's a crucial one to monitor and is added to `FocusSymbols`. `bpf_jit_supports_kfunc_ret_reg_pair` is also important since it's called when adding a kfunc call and can impact every architecture.

Let's double-check that `btf_type_is_scalar_struct` and `bpf_compute_live_registers` are properly visible. `btf_type_is_scalar_struct` is declared in a header, defined in the kernel and is not module exported, and `bpf_compute_live_registers` is also global.

Alright, the core of this patch series is adding support for kfuncs to return 16-byte values. This involves substantial changes to the BPF verifier for tracking return values, modifications to the JIT compilers (x86, arm64, riscv), and adjustments to the liveness and backtracking logic. This functionality touches critical, reachable code within the BPF subsystem and demands fuzzing to ensure the verifier correctly handles register states and prevents potential leaks or memory access errors.

So, here's my reasoning and the resulting data for the fuzzing:

**FocusSymbols:**
*   `btf_type_is_scalar_struct`
*   `btf_distill_func_proto`
*   `btf_prepare_func_args`
*   `bpf_compute_live_registers`
*   `bpf_jit_supports_kfunc_ret_reg_pair`

**EnableConfigs:**
*   `BPF_SYSCALL`
*   `BPF_JIT`

Now, let's execute `set-results`.

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT"
  ],
  "FocusSymbols": [
    "btf_type_is_scalar_struct",
    "btf_distill_func_proto",
    "btf_prepare_func_args",
    "bpf_compute_live_registers",
    "bpf_jit_supports_kfunc_ret_reg_pair"
  ],
  "Reasoning": "The patch series adds support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or scalar-only structs/unions) in the register pair R0:R2. This involves modifications to the BPF verifier to track the return values in both registers, changes to the liveness and backtracking logic, and updates to the JIT compilers (x86, arm64, riscv) to support this new calling convention. These are functional changes to reachable core kernel logic (BPF subsystem) that should be fuzzed to ensure the verifier correctly tracks the register states and prevents any leaks or invalid memory accesses.",
  "WorthFuzzing": true
}

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT"
  ],
  "FocusSymbols": [
    "btf_type_is_scalar_struct",
    "btf_distill_func_proto",
    "btf_prepare_func_args",
    "bpf_compute_live_registers",
    "bpf_jit_supports_kfunc_ret_reg_pair"
  ],
  "Reasoning": "The patch series adds support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or scalar-only structs/unions) in the register pair R0:R2. This involves modifications to the BPF verifier to track the return values in both registers, changes to the liveness and backtracking logic, and updates to the JIT compilers (x86, arm64, riscv) to support this new calling convention. These are functional changes to reachable core kernel logic (BPF subsystem) that should be fuzzed to ensure the verifier correctly tracks the register states and prevents any leaks or invalid memory accesses.",
  "WorthFuzzing": true
}

{
  "WorthFuzzing": true
}

{
  "KMSANReasoning": "The patch series introduces support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or small structs) in the register pair R0:R2. The changes are confined to the BPF verifier's static analysis (tracking liveness and initialization state of R2, type checking BTF return types) and architecture-specific JIT compilers (moving the native second return register into BPF R2). \n\nThere are no allocations of complex data structures, no new kernel-to-user memory copies, and no modifications to bounds checking of memory buffers. The verifier statically enforces that BPF programs cannot read uninitialized BPF registers, and any logical bugs in this tracking would lead to verifier bypasses rather than C-level uninitialized memory uses. Furthermore, KMSAN instruments kernel C code and tracks memory, but it does not instrument JIT-ed BPF code or track BPF register states directly. Therefore, any potential issues introduced by this patch (such as verifier flaws allowing stale register reads) would not be detected by KMSAN. Standard KASAN, verifier tests, and syzkaller's BPF fuzzing are the appropriate tools for validating these changes.",
  "NeedsKMSAN": false
}

You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.

CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
  a comprehensive suite of debugging tools and sanitizers, including KASAN
  (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
  (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
  or page allocations) and kernel-to-user memory info-leaks.

Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.

Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.

Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
   attributes, ioctl output arguments, socket options, or BPF buffers) where fields
   or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
   or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
   could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
   uninitialized bytes of existing buffers.

Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
  dereferences, locking deadlocks, or use-after-free bugs (these are already caught
  by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
  or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.

Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit cc236f68937f8479bce51601bd02ac7647affe1d
Author: syz-cluster <triage@syzkaller.com>
Date:   Wed Aug 19 06:41:58 2026 +0000

    syz-cluster: applied patch under review

diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index 85f73e0bbd0ff..89dea6b0b0241 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -575,6 +575,71 @@ is also covered by this recovery. A kfunc handed an arena pointer may
 therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
 against the arena. Larger accesses must verify the range explicitly.
 
+2.9 kfunc Return Values
+-----------------------
+
+A kfunc may return a scalar, a pointer, or a small struct or union by
+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
+
+A struct or union returned by value must be composed only of scalars
+(recursively), where a scalar is an integer or an enum; arrays of scalars are
+allowed as members. Its bytes are handed back to the program as the raw
+contents of R0 (and R2), so a pointer field would be laundered into a scalar
+and escape the verifier's pointer provenance and reference tracking. A struct
+or union with a pointer member is therefore rejected at load time, and so is
+one with a floating-point member, which the ABI may not return in R0:R2 at
+all.
+
+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
+scalar-only struct or union, or an ``__int128``. Such a value is returned
+in the register pair R0:R2, matching the convention LLVM uses for the BPF
+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or
+union of 8 bytes or less is returned in R0 alone.
+
+::
+
+        struct bpf_pair { __u64 a, b; };   /* 16 bytes */
+
+        __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
+        {
+                struct bpf_pair p = { .a = 1, .b = 2 };
+
+                return p;      /* p.a in R0, p.b in R2 */
+        }
+
+Returning a value in the R0:R2 pair requires the JIT to place the second
+half of the return value into R2, which not every architecture supports
+right now. A kfunc with a return value larger than 8 bytes is therefore
+rejected at load time on a JIT that does not advertise this capability (see
+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
+by the interpreter. A return value larger than 16 bytes is not supported.
+
+The same R0:R2 convention applies to a BPF subprogram, global or static, that
+returns an ``__int128`` or a struct or union larger than 8 bytes. It is only
+used when the program is JITed, since the interpreter propagates only R0 out of
+a subprogram: without a JIT the return value stays in R0 alone, and a caller
+reading R2 is rejected for reading an uninitialized register. A global
+subprogram is verified in isolation, so its by-value struct or union return is
+restricted to scalars just like a kfunc's; a static subprogram is verified
+inline and has no such restriction. The main program is not covered: its return
+value is the program's exit code, read out of R0 alone, so a declared upper
+half is never looked at.
+
+A global subprogram must leave a scalar in *every* register of the pair, so
+both halves of the returned value have to be assigned. Leaving the upper half
+uninitialized is not merely untidy: the compiler is then free to leave R2
+holding whatever it happened to hold, which for a subprogram taking a pointer
+argument is typically that pointer. Handing the caller an unknown scalar built
+from a pointer is a leak, so the verifier rejects it with::
+
+        At subprogram exit the register R2 is not a scalar value (...)
+
+Initialize the whole return value, for example ``struct pair p = {};``, to
+avoid this. A static subprogram is exempt from the scalar-only rule: it is
+verified inline, so an unassigned R2 is simply passed back to the caller as
+uninitialized and only a caller that reads it fails. A stack pointer left in
+R2 is still rejected there, just as one in R0 is.
+
 .. _BPF_kfunc_lifecycle_expectations:
 
 3. kfunc lifecycle expectations
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index c18e005a41dbe..3aa3ea0bc30bb 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2388,6 +2388,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index 74efe4b138d2a..47c7bf431ba85 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -2121,6 +2121,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_ptr_xchg(void)
 {
 	return true;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 1a9fb530adc3c..48429fae06410 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
  * arena NULL is offset 0. Return the number of emitted bytes.
  */
 static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
-				 const struct bpf_insn *insn, u8 **pprog)
+				 const struct btf_func_model *fm, u8 **pprog)
 {
-	const struct btf_func_model *fm;
 	u8 *prog = *pprog;
 	u8 *start = prog;
 	int i;
 
-	fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
-	if (!fm)
-		return -EINVAL;
-
 	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
 		u8 flags = fm->arg_flags[i];
 		u32 reg = BPF_REG_1 + i;
@@ -2644,6 +2639,8 @@ st:			insn_off = insn->off;
 
 			/* call */
 		case BPF_JMP | BPF_CALL: {
+			const struct btf_func_model *fm = NULL;
+
 			func = (u8 *) __bpf_call_base + imm32;
 			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
 				LOAD_TAIL_CALL_CNT_PTR(stack_depth);
@@ -2652,7 +2649,10 @@ st:			insn_off = insn->off;
 			if (!imm32)
 				return -EINVAL;
 			if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
-				err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
+				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
+				if (!fm)
+					return -EINVAL;
+				err = emit_kfunc_arena_args(bpf_prog, fm, &prog);
 				if (err < 0)
 					return err;
 				ip += err;
@@ -2666,6 +2666,14 @@ st:			insn_off = insn->off;
 				return -EINVAL;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
+			/*
+			 * A kfunc returning more than 8 bytes hands the second
+			 * half back in RDX (the native ABI's second return reg),
+			 * but BPF expects it in R0:R2. BPF R0 is RAX (no move
+			 * needed), while BPF R2 is RSI, so copy RDX into RSI.
+			 */
+			if (fm && fm->ret_size > 8)
+				emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
 			break;
 		}
 
@@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 5fad59fdab0d0..004b06785521c 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -819,6 +819,8 @@ struct bpf_subprog_info {
 	bool is_async_cb: 1;
 	bool is_exception_cb: 1;
 	bool args_cached: 1;
+	/* true if the return value is passed in the R0:R2 register pair */
+	bool ret_reg_pair: 1;
 	/* true if bpf_fastcall stack region is used by functions that can't be inlined */
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
@@ -1055,6 +1057,11 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
 	return &env->subprog_info[subprog];
 }
 
+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	return subprog_info(env, subprog)->ret_reg_pair;
+}
+
 struct bpf_call_summary {
 	u8 num_params;
 	bool is_void;
@@ -1481,6 +1488,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);
 struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+			       const struct btf_type *t, int rec);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 4a9bc6a848f2e..6e746b0a09306 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_kfunc_ret_reg_pair(void);
 bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_arena_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index a2b18a9f1694c..653db80bcc47b 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -423,6 +423,10 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				 */
 				verifier_bug_if(idx + 1 != subseq_idx, env,
 						"extra insn from subprog");
+				/* global subprog always sets R0 */
+				bt_clear_reg(bt, BPF_REG_0);
+				/* and if it does not set R2, main pass would catch it */
+				bt_clear_reg(bt, BPF_REG_2);
 				/* r1-r5 are invalidated after subprog call,
 				 * so for global func call it shouldn't be set
 				 * anymore
@@ -432,8 +436,6 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 						     bt_reg_mask(bt));
 					return -EFAULT;
 				}
-				/* global subprog always sets R0 */
-				bt_clear_reg(bt, BPF_REG_0);
 				return 0;
 			} else {
 				/* static subprog call instruction, which
@@ -506,6 +508,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				return -ENOTSUPP;
 			/* regular helper call sets R0 */
 			bt_clear_reg(bt, BPF_REG_0);
+			/* kfunc might also set R2 */
+			bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				/* if backtracking was looking for registers R1-R5
 				 * they should have been found already.
@@ -520,7 +524,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 					return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			bool r0_precise;
+			bool from_subprog_call, r0_precise, r2_precise;
+
+			/* BPF_EXIT in subprog or callback always returns
+			 * right after the call instruction, so by checking
+			 * whether the instruction at subseq_idx-1 is subprog
+			 * call or not we can distinguish actual exit from
+			 * *subprog* from exit from *callback*. In the former
+			 * case, we need to propagate the precision of the
+			 * return registers, if necessary. In the latter we
+			 * never do that.
+			 */
+			from_subprog_call = subseq_idx - 1 >= 0 &&
+					    bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
+
+			/* Sample the return registers before the callback
+			 * handling below clears R1-R5.
+			 */
+			r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
+			r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
 
 			/* Backtracking to a nested function call, 'idx' is a part of
 			 * the inner frame 'subseq_idx' is a part of the outer frame.
@@ -533,30 +555,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
 				for (i = BPF_REG_1; i <= BPF_REG_5; i++)
 					bt_clear_reg(bt, i);
+
+			bt_clear_reg(bt, BPF_REG_0);
+			bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				verifier_bug(env, "backtracking exit unexpected regs %x",
 					     bt_reg_mask(bt));
 				return -EFAULT;
 			}
 
-			/* BPF_EXIT in subprog or callback always returns
-			 * right after the call instruction, so by checking
-			 * whether the instruction at subseq_idx-1 is subprog
-			 * call or not we can distinguish actual exit from
-			 * *subprog* from exit from *callback*. In the former
-			 * case, we need to propagate r0 precision, if
-			 * necessary. In the former we never do that.
-			 */
-			r0_precise = subseq_idx - 1 >= 0 &&
-				     bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
-				     bt_is_reg_set(bt, BPF_REG_0);
-
-			bt_clear_reg(bt, BPF_REG_0);
 			if (bt_subprog_enter(bt))
 				return -EFAULT;
 
 			if (r0_precise)
 				bt_set_reg(bt, BPF_REG_0);
+			if (r2_precise)
+				bt_set_reg(bt, BPF_REG_2);
 			/* r6-r9 and stack slots will stay set in caller frame
 			 * bitmasks until we return back from callee(s)
 			 */
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 6967d48bba49d..58b81c6f5288c 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7591,7 +7591,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
 		return -EINVAL;
 	}
 	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	if (ret < 0 || ret > 16) {
 		bpf_log(log,
 			"The function %s return type %s is unsupported.\n",
 			tname, btf_type_str(t));
@@ -7684,6 +7684,12 @@ static int btf_check_func_type_match(struct bpf_verifier_log *log,
 			btf_type_str(t2), fn2);
 		return -EINVAL;
 	}
+	if (btf_type_has_size(t1) && (t1->size > 8 || t2->size > 8)) {
+		bpf_log(log,
+			"Return type of %s() has size %u and of %s() size %u, and a size above 8 bytes cannot be replaced\n",
+			fn1, t1->size, fn2, t2->size);
+		return -EINVAL;
+	}
 
 	for (i = 0; i < nargs1; i++) {
 		t1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL);
@@ -7964,7 +7970,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env *env,
 
 /* Check whether the type is a valid return type. */
 static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
-		const struct btf_type *t, int subprog)
+		const struct btf_type *t, int subprog, bool is_global)
 {
 	u32 tags = 0;
 	int err;
@@ -7987,6 +7993,19 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
 	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
 		return 0;
 
+	if (btf_type_is_struct(t) && t->size <= 16) {
+		/*
+		 * A global function's caller models the return as an opaque
+		 * scalar pair, so it may only return scalars by value. A local
+		 * function is verified inline, so a pointer field stays tracked
+		 * and needs no such restriction.
+		 */
+		bool local_func = subprog && !is_global;
+
+		if (local_func || btf_type_is_scalar_struct(env, btf, t, 0))
+			return 0;
+	}
+
 	return -EOPNOTSUPP;
 }
 
@@ -8074,12 +8093,12 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 		return -EINVAL;
 	}
 
-	err = btf_validate_return_type(env, btf, t, subprog);
+	err = btf_validate_return_type(env, btf, t, subprog, is_global);
 	if (err) {
 		if (is_global) {
 			bpf_log(log,
-				"Global function %s() return value not void or scalar. "
-				"Only those are supported.\n",
+				"Global function %s() has unsupported return type. "
+				"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n",
 				tname);
 		}
 		return err;
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index d55e737ed75ad..5db77d7915dfc 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_stack_args(void)
 {
 	return false;
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 74fc4b3f80d6e..71f998c6eb888 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2060,7 +2060,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); }
 /* Compute info->{use,def} fields for the instruction */
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
-				   struct insn_live_regs *info)
+				   struct insn_live_regs *info,
+				   bool ret_reg_pair)
 {
 	struct bpf_call_summary cs;
 	const u8 class = BPF_CLASS(insn->code);
@@ -2072,6 +2073,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 	const u32 src32 = mask_lo(src);
 	const u32 dst32 = mask_lo(dst);
 	const u32 r0  = reg64_mask(0);
+	const u32 r2  = reg64_mask(BPF_REG_2);
 	u32 def = 0;
 	u32 use = U32_MAX;
 
@@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			break;
 		case BPF_EXIT:
 			def = 0;
-			use = r0;
+			use = ret_reg_pair ? (r0 | r2) : r0;
 			break;
 		case BPF_CALL:
 			def = ALL_CALLER_SAVED_REGS;
@@ -2228,8 +2230,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 	struct insn_live_regs *state;
 	int insn_cnt = env->prog->len;
 	u64 pos, insn_pos;
-	int err = 0, i, j;
-	bool changed;
+	int err = 0, i, j, subprog, start, end;
+	bool changed, ret_reg_pair;
 
 	/* Use the following algorithm:
 	 * - define the following:
@@ -2256,8 +2258,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		goto out;
 	}
 
-	for (i = 0; i < insn_cnt; ++i)
-		compute_insn_live_regs(env, &insns[i], &state[i]);
+	for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+		start = env->subprog_info[subprog].start;
+		end = env->subprog_info[subprog + 1].start;
+		ret_reg_pair = bpf_ret_reg_pair(env, subprog);
+
+		for (i = start; i < end; ++i)
+			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
+	}
 
 	/* Forward pass: resolve stack access through FP-derived pointers */
 	err = bpf_compute_subprog_arg_access(env);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e421ea2b80c3a..7a939dfa3797b 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -385,27 +385,69 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
 	return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
 }
 
-static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
 {
-	const struct btf_type *type, *func, *func_proto;
+	const struct btf_type *func, *func_proto;
 	const struct btf *btf = env->prog->aux->btf;
 	u32 btf_id;
 
+	if (!btf || !env->prog->aux->func_info)
+		return NULL;
+
 	btf_id = env->prog->aux->func_info[subprog].type_id;
 
+	/* Both already validated by prepare_btf_func() at prog load. */
 	func = btf_type_by_id(btf, btf_id);
-	if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
-		return false;
-
 	func_proto = btf_type_by_id(btf, func->type);
-	if (!func_proto)
-		return false;
 
-	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
-	if (!type)
-		return false;
+	return btf_type_skip_modifiers(btf, func_proto->type, NULL);
+}
 
-	return btf_type_is_void(type);
+static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type = subprog_ret_type(env, subprog);
+
+	return type && btf_type_is_void(type);
+}
+
+static u32 ret_regs_cnt(u32 size)
+{
+	return size > 8 && size <= 16 ? 2 : 1;
+}
+
+/* Registers holding a function return value, in order. See ret_regs_cnt(). */
+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };
+
+static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
+{
+	const struct btf *btf = env->prog->aux->btf;
+	const struct btf_type *type;
+	int subprog;
+	u32 size;
+
+	if (!env->prog->jit_requested || bpf_prog_is_offloaded(env->prog->aux))
+		return 0;
+
+	/*
+	 * Skip the main program: its return value is the program's exit code,
+	 * read out of R0, so it never uses the register pair. An extension does
+	 * have a real prototype for subprog 0, but bpf_check_attach_target()
+	 * refuses to replace a function returning more than 8 bytes.
+	 */
+	for (subprog = 1; subprog < env->subprog_cnt; subprog++) {
+		type = subprog_ret_type(env, subprog);
+		if (!type || btf_type_is_void(type))
+			continue;
+		if (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, &size)), env,
+				    "cannot size return type of subprog %d", subprog))
+			return -EFAULT;
+		if (ret_regs_cnt(size) > 1) {
+			subprog_info(env, subprog)->ret_reg_pair = true;
+			env->prog->jit_required = 1;
+		}
+	}
+
+	return 0;
 }
 
 const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog)
@@ -2877,6 +2919,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
 	if (err)
 		return err;
+	if (func_model.ret_size > 8) {
+		if (kfunc.flags && (*kfunc.flags & KF_FASTCALL)) {
+			verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n",
+				kfunc.name);
+			return -EOPNOTSUPP;
+		}
+		if (!bpf_jit_supports_kfunc_ret_reg_pair()) {
+			verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
+				kfunc.name);
+			return -EOPNOTSUPP;
+		}
+	}
 
 	memset(&meta, 0, sizeof(meta));
 	meta.btf = kfunc.btf;
@@ -9853,6 +9907,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	u16 callee_incoming, stack_arg_cnt;
 	struct bpf_func_state *caller;
 	int err, subprog, target_insn;
+	u32 i, nregs;
 
 	target_insn = *insn_idx + insn->imm + 1;
 	subprog = bpf_find_subprog(env, target_insn);
@@ -9908,9 +9963,14 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		clear_caller_saved_regs(env, caller->regs);
 		invalidate_outgoing_stack_args(env, cur_func(env));
 
-		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
+		/*
+		 * A non-void global function returns a 64-bit SCALAR_VALUE in
+		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+		 */
 		if (!returns_void) {
-			mark_reg_unknown(env, caller->regs, BPF_REG_0);
+			nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+			for (i = 0; i < nregs; i++)
+				mark_reg_unknown(env, caller->regs, ret_regs[i]);
 			bpf_diag_mod_end(env);
 		}
 
@@ -10270,11 +10330,15 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	struct bpf_func_state *caller, *callee;
 	struct bpf_reg_state *r0;
 	bool in_callback_fn;
+	u32 i, nregs;
 	int err;
 
 	callee = state->frame[state->curframe];
 	r0 = &callee->regs[BPF_REG_0];
-	if (r0->type == PTR_TO_STACK) {
+	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
+			continue;
 		/* technically it's ok to return caller's stack pointer
 		 * (or caller's caller's pointer) back to the caller,
 		 * since these pointers are valid. Only current stack
@@ -10309,9 +10373,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 			return -EFAULT;
 		}
 	} else {
-		/* return to the caller whatever r0 had in the callee */
+		/*
+		 * return to the caller whatever the callee had in the
+		 * return register(s)
+		 */
 		bpf_diag_mod_begin(env, &caller->regs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE);
-		caller->regs[BPF_REG_0] = *r0;
+		for (i = 0; i < nregs; i++)
+			caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]];
 		bpf_diag_mod_end(env);
 	}
 
@@ -11246,6 +11314,19 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 	return 0;
 }
 
+/*
+ * Mark the register(s) holding a @size byte kfunc return value as unknown
+ * scalars. Both halves of a register pair are treated the same way.
+ */
+static void mark_kfunc_ret_regs(struct bpf_verifier_env *env,
+				struct bpf_reg_state *regs, u32 size)
+{
+	u32 i, nregs = ret_regs_cnt(size);
+
+	for (i = 0; i < nregs; i++)
+		mark_reg_unknown(env, regs, ret_regs[i]);
+}
+
 static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ACQUIRE;
@@ -11513,9 +11594,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_
 }
 
 /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
-					const struct btf *btf,
-					const struct btf_type *t, int rec)
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env,
+			       const struct btf *btf,
+			       const struct btf_type *t, int rec)
 {
 	const struct btf_type *member_type;
 	const struct btf_member *member;
@@ -11533,7 +11614,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
 				verbose(env, "max struct nesting depth exceeded\n");
 				return false;
 			}
-			if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
+			if (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
 				return false;
 			continue;
 		}
@@ -11932,7 +12013,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		 (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) ||
 		  is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
 		if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) &&
-		    !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+		    !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 			verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n",
 				reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
 			return -EINVAL;
@@ -11948,7 +12029,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
 		 * scalars. The access size is derived from the pointed-to BTF type.
 		 */
 		if (!btf_type_is_scalar(ref_t) &&
-		    !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+		    !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 			verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n",
 				reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
 			return -EINVAL;
@@ -13004,7 +13085,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
 				break;
 			}
 
-			if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+			if (!btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
 				enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);
 				const char *expected_type;
 
@@ -13546,7 +13627,7 @@ static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg
 
 		struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);
 		if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) {
-			if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
+			if (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
 				verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n");
 				return -EINVAL;
 			}
@@ -13914,10 +13995,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	}
 
 	if (btf_type_is_scalar(t)) {
-		mark_reg_unknown(env, regs, BPF_REG_0);
+		mark_kfunc_ret_regs(env, regs, t->size);
 		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
 		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
 			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
+	} else if (btf_type_is_struct(t)) {
+		/*
+		 * The returned struct comes back as raw register bits modeled
+		 * as an unknown scalar, so it must contain only scalars:
+		 * otherwise a pointer field would be laundered into a scalar
+		 * and escape provenance and reference tracking.
+		 */
+		if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+			verbose(env,
+				"kernel function %s returns %s %s that is not composed of scalars\n",
+				func_name, btf_type_str(t),
+				btf_name_by_offset(desc_btf, t->name_off));
+			return -EINVAL;
+		}
+		mark_kfunc_ret_regs(env, regs, t->size);
 	} else if (btf_type_is_ptr(t)) {
 		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
 		err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
@@ -17453,37 +17549,54 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	return 0;
 }
 
-static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
 {
-	struct bpf_reg_state *reg = reg_state(env, BPF_REG_0);
-	struct bpf_func_state *cur_frame = cur_func(env);
+	struct bpf_reg_state *reg;
 	int err;
 
-	if (subprog_returns_void(env, cur_frame->subprogno))
-		return 0;
-
-	err = check_reg_arg(env, BPF_REG_0, SRC_OP);
+	err = check_reg_arg(env, regno, SRC_OP);
 	if (err)
 		return err;
 
 	/* Pointers to arena are safe to pass between subprograms. */
-	if (is_arena_reg(env, BPF_REG_0))
+	if (is_arena_reg(env, regno))
 		return 0;
 
-	if (is_pointer_value(env, BPF_REG_0)) {
-		verbose(env, "R%d leaks addr as return value\n", BPF_REG_0);
+	if (is_pointer_value(env, regno)) {
+		verbose(env, "R%d leaks addr as return value\n", regno);
 		return -EACCES;
 	}
 
+	reg = reg_state(env, regno);
 	if (reg->type != SCALAR_VALUE) {
-		verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n",
-			reg_type_str(env, reg->type));
+		verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
+			regno, reg_type_str(env, reg->type));
 		return -EINVAL;
 	}
 
 	return 0;
 }
 
+static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+{
+	struct bpf_func_state *cur_frame = cur_func(env);
+	u32 subprog = cur_frame->subprogno;
+	u32 i, nregs;
+	int err;
+
+	if (subprog_returns_void(env, subprog))
+		return 0;
+
+	nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		err = check_global_ret_scalar_reg(env, ret_regs[i]);
+		if (err)
+			return err;
+	}
+
+	return 0;
+}
+
 /* Bitmask with 1s for all caller saved registers */
 #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
 
@@ -21151,6 +21264,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* must precede the first bpf_ret_reg_pair() user below */
+	ret = bpf_compute_subprog_ret_regs(env);
+	if (ret < 0)
+		goto skip_full_check;
+
 	ret = bpf_compute_live_registers(env);
 	if (ret < 0)
 		goto skip_full_check;
diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
new file mode 100644
index 0000000000000..e0b94ed10f941
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "aggregate_ret_func.skel.h"
+#include "aggregate_ret_kfunc.skel.h"
+
+void test_aggregate_ret(void)
+{
+	RUN_TESTS(aggregate_ret_func);
+	RUN_TESTS(aggregate_ret_kfunc);
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
index 2523c07a16c65..6262c81cdaa9f 100644
--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
@@ -441,6 +441,20 @@ static void test_func_replace_int_with_void(void)
 				     " doesn't match type INT of global_func2()");
 }
 
+static void test_func_replace_ret_pair(void)
+{
+	const char *msg = "Return type of new_agg_ret_target_func() has size 8 "
+			  "and of agg_ret_target_func() size 16";
+
+	/*
+	 * An extension cannot replace a function whose return value comes back
+	 * in the R0:R2 pair: only R0 is checked at the extension's exit, so it
+	 * would leave R2 stale for the target's callers.
+	 */
+	test_obj_load_failure_common("freplace_ret_pair.bpf.o",
+				     "./aggregate_ret_target.bpf.o", msg);
+}
+
 static int find_prog_btf_id(const char *name, __u32 attach_prog_fd)
 {
 	struct bpf_prog_info info = {};
@@ -660,6 +674,8 @@ void serial_test_fexit_bpf2bpf(void)
 		test_func_replace_progmap();
 	if (test__start_subtest("freplace_int_with_void"))
 		test_func_replace_int_with_void();
+	if (test__start_subtest("freplace_ret_pair"))
+		test_func_replace_ret_pair();
 	if (test__start_subtest("freplace_void"))
 		test_func_replace_void();
 	if (test__start_subtest("sleepable_fentry_to_xdp"))
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 64ac49ad67e63..f7f94ccebce27 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -5,6 +5,7 @@
 #include "arena_kfunc.skel.h"
 #include "arena_kfunc_jit.skel.h"
 #include "cap_helpers.h"
+#include "verifier_aggregate_ret.skel.h"
 #include "verifier_align.skel.h"
 #include "verifier_and.skel.h"
 #include "verifier_arena.skel.h"
@@ -170,6 +171,7 @@ void test_arena_kfunc(void)                   { RUN_TESTS(arena_kfunc); }
 
 void test_arena_kfunc_jit(void)               { RUN_TESTS(arena_kfunc_jit); }
 
+void test_verifier_aggregate_ret(void)        { RUN_TESTS(verifier_aggregate_ret); }
 void test_verifier_align(void)                { RUN(verifier_align); }
 void test_verifier_and(void)                  { RUN(verifier_and); }
 void test_verifier_arena(void)                { RUN(verifier_arena); }
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
new file mode 100644
index 0000000000000..6f66fc822cede
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -0,0 +1,237 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+typedef unsigned __int128 u128;
+
+__naked u128 global_agg_good(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"	/* low 64 bits */
+	"r2 = 0x5678;"	/* high 64 bits */
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_global_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad)
+	: __clobber_all);
+}
+
+SEC("tc")
+__load_if_JITed()
+__failure __msg("At subprogram exit the register R2 is not a scalar value")
+__naked int aggregate_ret_global_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad_ptr];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"	/* stack pointer placed in the second return register */
+	"exit;"
+	);
+}
+
+/*
+ * R2 is a return register once the subprogram returns a pair, so a stack
+ * pointer left in it is rejected at the callee's exit exactly as one in R0
+ * is: the callee frame is gone by the time the caller could use it.
+ */
+SEC("tc")
+__load_if_JITed()
+__failure __msg("cannot return stack pointer to the caller")
+__naked int aggregate_ret_static_ptr_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_bad_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_no_r2(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_static_uninit_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_no_r2];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(static_agg_no_r2)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_precise(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 4;"	/* second half; its value is made precise below */
+	"exit;"
+	);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
+__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
+__naked int aggregate_ret_static_precise(void)
+{
+	asm volatile (
+	"call %[static_agg_precise];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_precise)
+	: __clobber_all);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
+__naked int aggregate_ret_global_precise(void)
+{
+	asm volatile (
+	"call %[global_agg_good];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_agg_good)
+	: __clobber_all);
+}
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+/* A by-value struct that smuggles a pointer, which must be rejected. */
+struct with_ptr {
+	void *p;
+	__u64 x;
+};
+
+/* A by-value union that smuggles a pointer, which must be rejected too. */
+union upair_with_ptr {
+	void *p;
+	__u64 halves[2];
+};
+
+__naked struct with_ptr global_ret_struct_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_struct_ptr() has unsupported return type")
+__naked int aggregate_ret_global_struct_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_struct_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_struct_ptr)
+	: __clobber_all);
+}
+
+__naked union upair_with_ptr global_ret_union_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_union_ptr() has unsupported return type")
+__naked int aggregate_ret_global_union_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_union_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_union_ptr)
+	: __clobber_all);
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
new file mode 100644
index 0000000000000..d6b422ae9784b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -0,0 +1,121 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+/*
+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address
+ * (rather than calling) avoids any dependence on the compiler lowering an
+ * __int128 or struct return value, which the BPF backend only supports from
+ * LLVM 23 on.
+ */
+void __kfunc_btf_root(void)
+{
+	asm volatile (""
+	:
+	: "r"(&bpf_kfunc_call_test_i128),
+	  "r"(&bpf_kfunc_call_test_ret_fastcall),
+	  "r"(&bpf_kfunc_call_test_ret_ptr),
+	  "r"(&bpf_kfunc_call_test_ret_ii),
+	  "r"(&bpf_kfunc_call_test_ret_big));
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
+__naked int aggregate_ret_kfunc_precise(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_i128];"
+	"r6 = r2;"		/* second return half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_i128)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL")
+__naked int aggregate_ret_kfunc_fastcall_fail(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_ret_fastcall];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_fastcall)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("is not composed of scalars")
+__naked int aggregate_ret_kfunc_ptr_fail(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ptr)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_kfunc_small_no_r2(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"r2 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ii];"
+	"r0 = r2;"	/* R2 is not a return register for a <=8 byte struct */
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ii)
+	: __clobber_all);
+}
+
+/*
+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
+ * btf_distill_func_proto() before the KF_FASTCALL and JIT-capability checks.
+ */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported")
+__naked int aggregate_ret_kfunc_too_big_fail(void)
+{
+	asm volatile (
+	"call %[bpf_kfunc_call_test_ret_big];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_big)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
new file mode 100644
index 0000000000000..cffd8d7d3241a
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* freplace target: a global subprogram returning 16 bytes in R0:R2. */
+__naked unsigned __int128 agg_ret_target_func(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"
+	"r2 = 0x5678;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__naked int agg_ret_target(void)
+{
+	asm volatile (
+	"call %[agg_ret_target_func];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(agg_ret_target_func)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index d055fc7b3b95d..338e67cd8856a 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -431,6 +431,34 @@ __naked void subprog1(void)
 		::: __clobber_all);
 }
 
+static __used __naked unsigned __int128 aux2(void)
+{
+	asm volatile (
+		"r0 = 1;"
+		"r2 = 2;"
+		"exit;"
+		::: __clobber_all);
+}
+
+SEC("socket")
+/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
+__load_if_JITed()
+__log_level(2)
+__msg("0: .12345.... (85) call pc+2")
+__msg("1: ..2....... (bf) r0 = r2")
+__msg("2: 0......... (95) exit")
+__msg("3: .......... (b7) r0 = 1")
+__msg("4: 0......... (b7) r2 = 2")
+__msg("5: 0.2....... (95) exit")
+__naked void subprog_ret_reg_pair(void)
+{
+	asm volatile (
+		"call aux2;"
+		"r0 = r2;"
+		"exit;"
+		::: __clobber_all);
+}
+
 #if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
 
 SEC("socket")
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index ac44d60e50666..9708efb93683b 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)
 
 SEC("?tc")
 __exception_cb(exception_cb_bad_ret_type1)
-__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.")
+__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.")
 int reject_exception_cb_type_1(struct __sk_buff *ctx)
 {
 	bpf_throw(0);
diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
new file mode 100644
index 0000000000000..12c15d293bd79
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+SEC("freplace/agg_ret_target_func")
+__u64 new_agg_ret_target_func(void)
+{
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c
new file mode 100644
index 0000000000000..7851bade2b406
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c
@@ -0,0 +1,178 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+struct pair {
+	__u64 lo;	/* R0 */
+	__u64 hi;	/* R2 */
+};
+
+union upair {
+	__u64 halves[2];
+	struct {
+		__u64 lo;	/* R0 */
+		__u64 hi;	/* R2 */
+	} parts;
+};
+
+static __noinline u128 make_i128(__u64 a, __u64 b)
+{
+	return ((u128)(a + b) << 64) | (a - b);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v;
+
+	v = make_i128(a, b);
+	if ((__u64)(v >> 64) != a + b)
+		return 1;
+	if ((__u64)v != a - b)
+		return 2;
+
+	return 0;
+}
+
+static __noinline struct pair make_pair(__u64 a, __u64 b)
+{
+	struct pair p = { .lo = a + b, .hi = a - b };
+
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p;
+
+	p = make_pair(a, b);
+	if (p.lo != a + b)
+		return 1;
+	if (p.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+__noinline struct pair make_pair_global(__u64 a, __u64 b)
+{
+	struct pair p = { .lo = a + b, .hi = a - b };
+
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p;
+
+	p = make_pair_global(a, b);
+	if (p.lo != a + b)
+		return 1;
+	if (p.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+static __noinline union upair make_upair(__u64 a, __u64 b)
+{
+	union upair p;
+
+	p.halves[0] = a + b;
+	p.halves[1] = a - b;
+	return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_union_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	union upair p;
+
+	p = make_upair(a, b);
+	if (p.parts.lo != a + b)
+		return 1;
+	if (p.parts.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v;
+
+	v = bpf_kfunc_call_test_i128(a, b);
+	if ((__u64)(v >> 64) != a + b)
+		return 1;
+	if ((__u64)v != a - b)
+		return 2;
+
+	return 0;
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct prog_test_ret_pair p;
+
+	p = bpf_kfunc_call_test_ret_pair(a, b);
+	if (p.lo != a + b)
+		return 1;
+	if (p.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+SEC("socket")
+__description("verifier_aggregate_ret: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 9366a3c578f13..850cf4f830c48 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -954,6 +954,53 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
 	return 0;
 }
 
+#if defined(__x86_64__) || defined(__aarch64__)
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+	return (__int128)(((unsigned __int128)(a + b) << 64) | (a - b));
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)
+{
+	struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)
+{
+	struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
+{
+	struct prog_test_ret_ptr r = { .p = NULL, .tag = tag };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
+{
+	struct prog_test_ret_ii r = { .a = a, .b = b };
+
+	return r;
+}
+#endif /* __x86_64__ || __aarch64__ */
+
+/*
+ * Takes no argument on purpose: with no arguments there is nothing for the sret
+ * pointer to displace, so this needs no architecture guard even though it
+ * returns 24 bytes.
+ */
+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)
+{
+	struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };
+
+	return r;
+}
+
 __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
 					 u64 e, u64 f, u64 g, u64 h,
 					 u64 i, u64 j)
@@ -1487,6 +1534,14 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+#if defined(__x86_64__) || defined(__aarch64__)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
+#endif
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index 7d81070eefe7f..65e693ada7364 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -55,6 +55,27 @@ struct prog_test_big_arg {
 	__u64 b;
 };
 
+struct prog_test_ret_pair {	/* 16 bytes: R0:R2 */
+	__u64 lo;
+	__u64 hi;
+};
+
+struct prog_test_ret_ii {	/* 8 bytes: R0 only */
+	int a;
+	int b;
+};
+
+struct prog_test_ret_ptr {	/* 16 bytes: contains a pointer */
+	void *p;
+	__u64 tag;
+};
+
+struct prog_test_ret_big {	/* 24 bytes: too large for R0:R2 */
+	__u64 a;
+	__u64 b;
+	__u64 c;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -131,6 +152,14 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+#ifdef __SIZEOF_INT128__
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
+#endif
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;
+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym;
 __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
 			       __u64 e, __u64 f, __u64 g, __u64 h,
 			       __u64 i, __u64 j) __ksym;

gemini-3.1-pro-preview

							input: 24920
							output: 292
							thoughts: 3643

**KMSAN Applicability Assessment for BPF 16-Byte Return Value Support**

Okay, so this patch series enables kfuncs and subprograms to return values up to 16 bytes, using the register pair R0:R2. The key changes are in documentation, architecture-specific JIT support, and, most importantly, the verifier. The verifier now handles R2 precision for subprogram calls/exits, supports 16-byte return types in `btf_distill_func_proto` and scalar-only structs, marks R2 as used on `BPF_EXIT` if `ret_reg_pair` is true, and updates several functions (`check_func_call`, `prepare_func_exit`, `check_kfunc_call`, `check_global_subprog_return_code`) to correctly handle R0 and R2. The verifier also ensures that returned structs only contain scalars and handles cases where the JIT might leave uninitialized data in registers.

The central question is whether this introduces uninitialized memory risks, and specifically, if KMSAN fuzzing is required. My analysis centers around the verifier's role in preventing leaks and how KMSAN interacts with this.

When kfuncs or subprograms return a struct up to 16 bytes, the verifier marks R0 and R2 as unknown scalars.  For global subprograms, R0 and R2 are checked to be initialized. For static subprograms, the verifier will reject a program that reads an uninitialized R2. The problem becomes apparent with padding within structs returned by value. If a kfunc returns a struct with padding, the C compiler might leave that padding uninitialized in the return registers (RAX and RDX on x86, which map to R0 and R2). The verifier treats these registers as fully initialized, from the BPF program's perspective. Thus, if the BPF program copies these registers to a map or sends them to userspace, there could be an uninitialized kernel memory leak.

Now, does KMSAN catch this? KMSAN detects reads of uninitialized memory and info-leaks. KFuncs are kernel functions and if they were to return a struct by value with padding that had uninitialized values, KMSAN *should* catch that if the function is instrumented. The verifier ensures the structs returned contain only scalar values and doesn't explicitly check for padding. However, the critical caveat is that KMSAN *doesn't* instrument JIT-ed BPF code. Since this feature is JIT-only, KMSAN won't track the values in R0/R2 once they are used in the BPF program, if they leak there, so a potential leak of uninitialized values is invisible to KMSAN.

The verifier does have checks that a subprogram must leave a scalar in *every* register of the pair to avoid potential leaks and it checks for pointers as well. I need to consider `bpf_compute_subprog_ret_regs`, which might inadvertently make assumptions about initialization, and if a 16-byte struct is returned, does the JIT clear bits it shouldn't.

Now, let's consider the KMSAN applicability. The patch primarily modifies the verifier and JIT support for returning 16-byte values. It does not introduce new kernel structures that are directly sent to user space. The verifier prevents BPF programs from reading uninitialized registers. If a BPF program tries to read an uninitialized R2, the verifier will reject it. However, if there is a bug, and a BPF program leaks data from a kfunc return, KMSAN may not even catch it because the read happens in JIT-ed BPF code, which KMSAN doesn't instrument.

Looking at the KMSAN rules: this patch does *not* introduce new kernel structures to userspace with uninitialized fields. It doesn't depend on uninitialized variables in C code. It modifies the verifier and BPF code generation. Verifier bugs could permit a BPF program to read uninitialized registers or leak stale pointers. KMSAN doesn't track BPF register state or BPF code execution directly and it wouldn't be the right tool to catch logical bugs in the verifier. There is no C code that allocates buffers or copies data to userspace. All this points to the conclusion that a bug here would result in a verifier bypass; however, such a vulnerability would be a logical bug in the verifier or the BPF JIT code, and wouldn't involve kernel data structures or KMSAN-detectable issues.

Therefore, **NeedsKMSAN=false**.

{
  "KMSANReasoning": "The patch series introduces support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or small structs) in the register pair R0:R2. The changes are confined to the BPF verifier's static analysis (tracking liveness and initialization state of R2, type checking BTF return types) and architecture-specific JIT compilers (moving the native second return register into BPF R2). \n\nThere are no allocations of complex data structures, no new kernel-to-user memory copies, and no modifications to bounds checking of memory buffers. The verifier statically enforces that BPF programs cannot read uninitialized BPF registers, and any logical bugs in this tracking would lead to verifier bypasses rather than C-level uninitialized memory uses. Furthermore, KMSAN instruments kernel C code and tracks memory, but it does not instrument JIT-ed BPF code or track BPF register states directly. Therefore, any potential issues introduced by this patch (such as verifier flaws allowing stale register reads) would not be detected by KMSAN. Standard KASAN, verifier tests, and syzkaller's BPF fuzzing are the appropriate tools for validating these changes.",
  "NeedsKMSAN": false
}

{
  "KMSANReasoning": "The patch series introduces support for BPF kfuncs and subprograms to return values larger than 8 bytes and up to 16 bytes (e.g., `__int128` or small structs) in the register pair R0:R2. The changes are confined to the BPF verifier's static analysis (tracking liveness and initialization state of R2, type checking BTF return types) and architecture-specific JIT compilers (moving the native second return register into BPF R2). \n\nThere are no allocations of complex data structures, no new kernel-to-user memory copies, and no modifications to bounds checking of memory buffers. The verifier statically enforces that BPF programs cannot read uninitialized BPF registers, and any logical bugs in this tracking would lead to verifier bypasses rather than C-level uninitialized memory uses. Furthermore, KMSAN instruments kernel C code and tracks memory, but it does not instrument JIT-ed BPF code or track BPF register states directly. Therefore, any potential issues introduced by this patch (such as verifier flaws allowing stale register reads) would not be detected by KMSAN. Standard KASAN, verifier tests, and syzkaller's BPF fuzzing are the appropriate tools for validating these changes.",
  "NeedsKMSAN": false
}

LLM Calls Summary:
Total Calls	Total Tokens	Avg Tokens	Total Duration (Seconds)	Avg Duration (Seconds)

Tool Calls Summary:
Total Calls	Total Duration (Seconds)	Avg Duration (Seconds)