From: Tejun Heo save_args() reads stack-passed arguments relative to rbp assuming two return addresses sit between the saved rbp and the arguments, which holds when the trampoline is entered through the fentry call from a traced function. An indirect trampoline is called through a function pointer, so only the caller's return address is on the stack and the arguments start at rbp + 16, not rbp + 24. Every stack-passed argument of a struct_ops callback with more than six argument slots is read one slot off. This has gone unnoticed because no in-tree struct_ops member passes arguments on the stack. The jmp-entry form already accounts for having a single return address; treat BPF_TRAMP_F_INDIRECT the same way. Fixes: 473e3150e30a ("bpf, x86: allow function arguments up to 12 for TRACING") Cc: Jiri Olsa Signed-off-by: Tejun Heo Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index dfb5335ad837..4b349ae35ebf 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3075,6 +3075,7 @@ static void save_args(const struct btf_func_model *m, u8 **prog, { int arg_regs, first_off = 0, nr_regs = 0, nr_stack_slots = 0; bool use_jmp = bpf_trampoline_use_jmp(flags); + int stack_args_off = (use_jmp || (flags & BPF_TRAMP_F_INDIRECT)) ? 16 : 24; int i, j, slot = 0; /* Store function arguments to stack. @@ -3106,16 +3107,16 @@ static void save_args(const struct btf_func_model *m, u8 **prog, /* copy function arguments from origin stack frame * into current stack frame. * - * The starting address of the arguments on-stack - * is: - * rbp + 8(push rbp) + - * 8(return addr of origin call) + - * 8(return addr of the caller) - * which means: rbp + 24 + * The arguments on-stack start above the saved rbp + * and the return addresses: two return addresses + * (origin call and caller) when the trampoline is + * entered through the fentry call, so rbp + 24, and + * a single one when it is entered with a jmp or + * called indirectly, so rbp + 16. */ for (j = 0; j < arg_regs; j++) { emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP, - nr_stack_slots * 8 + 16 + (!use_jmp) * 8); + nr_stack_slots * 8 + stack_args_off); if (aargs && (aargs->slots & BIT(slot))) emit_arena_arg_conv(prog, BPF_REG_0, aargs->nullable_slots & BIT(slot), -- 2.53.0