From: Alexei Starovoitov r12 (BPF_REG_ARENA_SP) is the stack pointer of the second stack of a prog. This stack is in arena. The verifier replaces the insns that use r12 with insns that the JIT already knows: a 32-bit add, addr_space_cast, and loads and stores in arena. Thus the JIT only has to supply a register. Map r12 to x23. The JIT does not use x23 for other data. x23 is callee saved, thus helpers and kfuncs keep its value. It is not an argument register, thus calls do not change. The kernel calls the main prog and callbacks, and it does not know about r12. The main prog, and a callback that uses r12, take a stack at entry and set r12. Thus they save x23 at entry and restore it at exit. Other subprogs get x23 from the caller. Progs without r12 get the same code. The cost in progs with r12: save and restore of x23 0 or 2 insns (stp, ldp). The JIT saves registers in pairs. load or store through r12 3 insns: add of the base of arena, mov of the negative offset, ldr or str. load or store through r10 1 insn, for reference. Compile tested only. Signed-off-by: Alexei Starovoitov --- arch/arm64/net/bpf_jit_comp.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 33632fb20997..58f30911bdab 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -61,6 +61,8 @@ static const int bpf2a64[] = { [BPF_REG_9] = A64_R(22), /* read-only frame pointer to access stack */ [BPF_REG_FP] = A64_R(25), + /* callee saved register for the pointer to the stack in arena */ + [BPF_REG_ARENA_SP] = A64_R(23), /* temporary registers for BPF JIT */ [TMP_REG_1] = A64_R(10), [TMP_REG_2] = A64_R(11), @@ -419,6 +421,11 @@ static void find_used_callee_regs(struct jit_ctx *ctx) if (ctx->arena_vm_start) ctx->used_callee_reg[i++] = bpf2a64[ARENA_VM_START]; + /* subprogs get it from the caller and leave it as it was, unless the kernel calls them */ + if (prog->aux->arena_stack_size && + (!bpf_is_subprog(prog) || prog->aux->arena_stack_entry)) + ctx->used_callee_reg[i++] = bpf2a64[BPF_REG_ARENA_SP]; + ctx->nr_used_callee_reg = i; } @@ -2537,6 +2544,11 @@ bool bpf_jit_supports_private_stack(void) return true; } +bool bpf_jit_supports_arena_stack(void) +{ + return true; +} + bool bpf_jit_supports_kfunc_call(void) { return true; -- 2.55.0