From: Alexei Starovoitov core::fmt::write() is in each Rust prog that prints. It reads the options of a '{}' and calls the fmt() function of the argument through a pointer: if n & 8 != 0 { arg_index = ...(template.cast_array().read()); } if n & 16 != 0 { opt.width = args.add(opt.width)...as_u16(); } if n & 32 != 0 { opt.precision = args.add(opt.precision)...as_u16(); } args.add(arg_index).as_ref().fmt(&mut Formatter::new(output, opt))?; The last line is callx with two arguments, r1 and r2. The pass that finds the live registers does not know the callee: callx counts as a read of r1-r5. What the branches leave in r4 and r5 stays live: w4 &= 16 if w4 == 0 goto +4 ; r4 = 0 r3 <<= 4 r4 = *(u64 *)(r10 - 24) ; r4 = args, a pointer to arena ... callx r3 r4 is 0 or a pointer, r5 is a byte or a pointer. A number and a pointer have no join: 4 states go to callx in the first pass over the loop, 2 in each later pass, where 1 is enough. A subprog with callx in it also seems to take 5 arguments: what its callers leave in r3-r5 is a part of its entry state, and it is verified again for each such leftover. callx calls only a static subprog whose address the prog takes: with ld_imm64, or in a table in a read-only map. mark_recursive_subprogs() has this set. Store it in addr_taken of bpf_subprog_info. The passes in flow.c that look through callx ahead of the main pass read the bit already. Nothing set it so far: they took no subprog for a target of callx. insn the registers of r1-r5 that it reads -------------------------- --------------------------------------------- call of a static subprog f live at the first insn of f, as before callx, before all callx, this patch live at the first insn of a subprog that has addr_taken, or declared in its BTF bpf_compute_live_registers() finds both in one loop, until nothing changes. zext_dst comes from the same result, thus check_func_callx_one() reports a verifier bug if the callee has no addr_taken. bpfland_stopping of scx_bpfland.rs: -13 % insns, -19 % states. 99 of the 225 progs of the schedulers of rust-bpf change: -5 % insns, -9 % states. Signed-off-by: Alexei Starovoitov --- kernel/bpf/liveness.c | 28 ++++++++++++++++++++++++++-- kernel/bpf/verifier.c | 11 ++++++++++- 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index 318db60ae230..e930d928afa5 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -2732,9 +2732,12 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, } def = mask_widen(def); use = mask_widen(use); - /* callx reads the address of the callee from dst_reg */ + /* + * callx reads the address of the callee from dst_reg. What the callee + * reads is added in bpf_compute_live_registers(). + */ if (bpf_is_callx(insn)) - use |= dst; + use = dst; break; default: def = 0; @@ -2801,6 +2804,25 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) changed = true; while (changed) { + u32 args = mask_widen(GENMASK(BPF_REG_5, BPF_REG_1)), callx_use = 0, declared; + bool taken = false; + + /* + * callx calls one of the subprogs whose address is taken, see + * check_func_callx_one(). It reads what they read, as computed so far. + */ + for (subprog = 0; subprog < env->subprog_cnt; subprog++) { + if (!env->subprog_info[subprog].addr_taken) + continue; + callx_use |= state[env->subprog_info[subprog].start].in & args; + /* and what its BTF declares: the BTF is unreliable when one is not set */ + declared = env->subprog_info[subprog].arg_slot_cnt; + callx_use |= mask_widen(GENMASK(declared, 1)) & args; + taken = true; + } + if (!taken) + callx_use = args; + changed = false; /* a cycle of calls takes a round for each link */ if (signal_pending(current)) { @@ -2829,6 +2851,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) !bpf_subprog_is_global(env, callee_subprog(env, &insns[insn_idx]))) use |= state[insn_idx + insns[insn_idx].imm + 1].in & mask_widen(GENMASK(BPF_REG_5, BPF_REG_1)); + if (bpf_is_callx(&insns[insn_idx])) + use |= callx_use; new_in = (new_out & ~live->def) | use; /* may_goto that is added at the head of a loop goes there */ t = bpf_head_guard_target(env, insn_idx); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a5c3e86f3f87..a3095efb1ee0 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6116,6 +6116,8 @@ static int mark_recursive_subprogs(struct bpf_verifier_env *env) } for_each_set_bit(s, callx, cnt) bitmap_or(calls + s * words, calls + s * words, taken, cnt); + for_each_set_bit(s, taken, cnt) + si[s].addr_taken = true; /* * Callers of a global subprog look at can_unwind before the subprog is verified. @@ -11377,7 +11379,8 @@ static void gen_subprog_arg_proto(const struct bpf_subprog_info *sub, const stru /* * The argument slots that the code of a static subprog takes, whatever its BTF says. - * Liveness errs on the safe side: a callx in the subprog counts as a read of r1-r5. + * Liveness errs on the safe side: a callx in the subprog counts as a read of all that the subprogs + * whose address is taken read or declare. * But it misses r3-r5 of bpf_trace_printk() and the flags of bpf_get_branch_snapshot(), which * their protos do not declare. Such a subprog prints or passes a wrong number under a tracer. * The helpers are safe with any value there: the verifier never checks these registers. @@ -12096,6 +12099,12 @@ static int check_func_callx_one(struct bpf_verifier_env *env, int subprog, int * if (err == -EFAULT) return err; + /* bpf_compute_live_registers() counts on it */ + if (verifier_bug_if(!env->subprog_info[subprog].addr_taken, env, + "callx of %s() whose address is not taken", + bpf_subprog_name(env, subprog))) + return -EFAULT; + err = record_callx_edge(env, cur_func(env)->subprogno, subprog); if (err) return err; -- 2.55.0