From: Alexei Starovoitov The verifier fills in the last arguments of a kfunc that has KF_IMPLICIT_ARGS. The prog does not pass them: bpf_stream_vprintk(stream_id, fmt, args, len) the prog calls bpf_stream_vprintk(stream_id, fmt, args, len, aux) the kernel has The prog sets r1-r4. The verifier adds 'r5 = aux' in front of the call. bpf_compute_live_registers() counts the arguments of the second prototype. Thus r5 is live before the call, and up to the previous insn that writes to r5. In a loop that is the same call in the previous iteration: r5 is live at the loop head, its value is what the call left in it, and it has no formula. Scalar evolution gives up on a loop that has such a register. The verifier also compares r5 when it looks for an equal state. Let bpf_get_call_summary() tell how many of the last argument slots are implicit, and do not count them as used. Nothing reads r5 between the last write of the prog and 'r5 = aux': check_func_arg() does not look at the register of such an argument, and a jump to the call goes to the added insn. The call clobbers r5 as before. Almost all scx_bpf_*() kfuncs have an implicit argument. So have bpf_obj_new(), bpf_obj_drop(), bpf_list_push_*(), bpf_rbtree_add() and bpf_stream_*(). Insns that the verifier processes: progs progs that change insns ----------------------- ----------------- ------- scx, as BPF CI has it 26 of 466 -18.9 % scx, master 26 of 563 -10.5 % selftests 52 of 6329 -0.3 % lavd_dispatch of scx_lavd goes from 281049 to 5589. No prog of Cilium changes, no verdict changes. In each set of scx 3 progs get more insns, at most 1.2 %: a path ends earlier, and that moves where later paths end. Signed-off-by: Alexei Starovoitov --- include/linux/bpf_verifier.h | 2 ++ kernel/bpf/liveness.c | 3 ++- kernel/bpf/verifier.c | 9 +++++++++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 7baeb831281f..c28f43d60900 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1235,6 +1235,8 @@ static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog) struct bpf_call_summary { u8 arg_slot_cnt; + /* the last of them are filled in by the verifier, the prog doesn't pass them */ + u8 implicit_slot_cnt; bool is_void; bool fastcall; }; diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index e657e6d95408..215f839bd2fb 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -2621,7 +2621,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, def = ALL_CALLER_SAVED_REGS; use = def & ~BIT(BPF_REG_0); if (bpf_get_call_summary(env, insn, &cs)) { - use = GENMASK(min_t(u8, cs.arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS), 1); + use = GENMASK(min_t(u8, cs.arg_slot_cnt - cs.implicit_slot_cnt, + MAX_BPF_FUNC_REG_ARGS), 1); } else if (bpf_pseudo_call(insn)) { /* a static callee's reads are added in bpf_compute_live_registers() */ subprog = callee_subprog(env, insn); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1b6c0674ff6a..8509e65889bd 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19687,6 +19687,7 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call, bpf_jit_inlines_helper_call(call->imm)); cs->is_void = fn->ret_type == RET_VOID; cs->arg_slot_cnt = 0; + cs->implicit_slot_cnt = 0; for (i = 0; i < ARRAY_SIZE(fn->arg_type); ++i) { if (fn->arg_type[i] == ARG_UNUSED) break; @@ -19703,6 +19704,14 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call, /* error would be reported later */ return false; cs->arg_slot_cnt = btf_proto_slots(meta.btf, meta.func_proto); + cs->implicit_slot_cnt = 0; + if (meta.kfunc_flags & KF_IMPLICIT_ARGS) { + const struct btf_type *func = btf_type_by_id(meta.btf, meta.func_id); + + /* the prototype without implicit args, see is_kfunc_arg_implicit() */ + cs->implicit_slot_cnt = cs->arg_slot_cnt - + btf_proto_slots(meta.btf, btf_type_by_id(meta.btf, func->type)); + } cs->fastcall = meta.kfunc_flags & KF_FASTCALL; cs->is_void = btf_type_is_void(btf_type_by_id(meta.btf, meta.func_proto->type)); return true; -- 2.55.0