A program with a private stack pushes r9, which holds the private frame pointer, around every call. The exception callback of an exception boundary program resets the stack pointer to the value the boundary frame had at the call it was thrown from, plus stack_arg_sp_adjust, and pops the callee-saved registers from there. With a private stack that value is 8 bytes short, so the registers are restored from the wrong slots and the program returns to the kernel with corrupted callee-saved registers. Include the push in stack_arg_sp_adjust. A tracing program with 64 bytes or more of stack runs on a private stack, so any such program that calls bpf_throw() is affected. The two private stack exception selftests run such a program; they pass only because nothing observes the restored registers after the return into the kernel. The case was found while reworking how the callback restores the callee-saved registers. Fixes: 7d1cd70d4b16 ("bpf, x86: Support private stack in jit") Cc: Yonghong Song Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 72a2a0a89e03..ea97e068efda 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2625,7 +2625,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * */ outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0; if (bpf_prog->aux->exception_boundary) - bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp; + /* With a private stack every call site also has r9 pushed. */ + bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp + (priv_frame_ptr ? 8 : 0); emit_sub_rsp(&prog, outgoing_rsp); if (arena_vm_start) -- 2.53.0