A global function is verified on its own, with no knowledge of its callers, and a call to it is checked against its declared contract rather than by walking into it. The two rules the previous patches added at the park are therefore blind to a park that happens inside a global function: the caller keeps its map value pointers across the call, and a callback may call a global function that parks. The first is a use after free waiting to happen, the second defeats the callback restriction. Record the fact per subprogram the way might_sleep and changes_pkt_data are recorded. The CFG pass marks the subprogram containing a bpf_task_park() call and merges the mark into every caller through merge_callee_effects(), so might_park on a subprogram means that some call reachable from it may park, through static and global callees alike. Like the other two marks it is computed before dead code elimination, so an unreachable park still sets it; the consequence is conservative, a caller that invalidates more than it had to. At a call to a global function with the mark, apply at the call site what the park would apply inside: refuse the call from any callback frame, refuse it while the caller holds a live dynptr or an iterator other than bpf_iter_num, and after the call invalidate the caller's registers that would not survive the sleep. The callee's own frame is covered when the callee is verified, where the park itself runs the checks; this patch completes them for the frames the callee cannot see. A global function that parks and is called from a non-sleepable context needs nothing new: a global function is verified once per calling context, and the park is registered as a sleepable kfunc. Extension programs need no further rule. An extension replacing a global function of a syscall program resolves to the syscall type for its kfunc set and could name bpf_task_park(), but an extension cannot be loaded sleepable, so the park is refused there as any sleepable kfunc is in a non-sleepable program. Behind that, bpf_check_attach_target() refuses to extend a function that runs without an RCU read lock, which is every function a syscall program actually calls, since its sleepable main body marks each callee as called in a sleepable context; only a global function that nothing calls stays extendable, and nothing would run it. The mark does not cross the instruction that only loads a callback's address, unless the program uses callx. A park in a synchronous callback is refused at the park, with the caller's frames in view, and an asynchronous callback is a root of its own, so the function that passes the callback does not park on its account. With the mark it would be treated as if it did, refused from callbacks and made to invalidate its caller's pointers, which matters as soon as some callback may park legitimately, as the body of a BPF kthread does. A loaded address may also be called through callx, which is a call like any other, so the mark follows it when the program has callx. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf_verifier.h | 3 +++ kernel/bpf/cfg.c | 23 +++++++++++++++++++++++ kernel/bpf/verifier.c | 27 +++++++++++++++++++++++++++ 3 files changed, 53 insertions(+) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 4771c55d8f41..b304f3ebc983 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1056,6 +1056,8 @@ struct bpf_subprog_info { bool keep_fastcall_stack: 1; bool changes_pkt_data: 1; bool might_sleep: 1; + /* some call reachable from this subprog may be bpf_task_park() */ + bool might_park: 1; bool uses_arena_sp: 1; /* a callback that takes a stack in arena, like the main program does */ bool arena_stack_entry: 1; @@ -2033,6 +2035,7 @@ static inline bool bpf_is_kfunc_sleepable(struct bpf_call_arg_meta *meta) } bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta); bool bpf_is_cast_kfunc(const struct bpf_call_arg_meta *meta); +bool bpf_is_task_park_kfunc(struct bpf_call_arg_meta *meta); struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem); int bpf_copy_insn_array_uniq(struct bpf_map *map, u32 start, u32 end, u32 *off); bool bpf_insn_is_cond_jump(u8 code); diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c index bbcf42eae9d9..0102df246633 100644 --- a/kernel/bpf/cfg.c +++ b/kernel/bpf/cfg.c @@ -67,6 +67,14 @@ static void mark_subprog_might_sleep(struct bpf_verifier_env *env, int off) subprog->might_sleep = true; } +static void mark_subprog_might_park(struct bpf_verifier_env *env, int off) +{ + struct bpf_subprog_info *subprog; + + subprog = bpf_find_containing_subprog(env, off); + subprog->might_park = true; +} + static void mark_subprog_might_throw(struct bpf_verifier_env *env, int off) { struct bpf_subprog_info *subprog; @@ -86,15 +94,28 @@ static void mark_subprog_might_throw(struct bpf_verifier_env *env, int off) static bool merge_callee_effects(struct bpf_verifier_env *env, int t, int w) { struct bpf_subprog_info *caller, *callee; + bool might_park; caller = bpf_find_containing_subprog(env, t); callee = bpf_find_containing_subprog(env, w); + /* + * A park in a callback does not mark the function that only loads + * the callback's address: a synchronous callback is refused at the + * park with the caller's frames in view, and an asynchronous one is a + * root of its own. The address may also be called through callx, + * which is a call like any other, so the mark follows it when the + * program has callx at all. + */ + might_park = callee->might_park && + (bpf_pseudo_call(&env->prog->insnsi[t]) || env->has_callx); if (caller->changes_pkt_data >= callee->changes_pkt_data && caller->might_sleep >= callee->might_sleep && + caller->might_park >= might_park && caller->might_throw >= callee->might_throw) return false; caller->changes_pkt_data |= callee->changes_pkt_data; caller->might_sleep |= callee->might_sleep; + caller->might_park |= might_park; caller->might_throw |= callee->might_throw; return true; } @@ -618,6 +639,8 @@ static int visit_insn(int t, struct bpf_verifier_env *env) */ if (ret == 0 && bpf_is_kfunc_sleepable(&meta)) mark_subprog_might_sleep(env, t); + if (ret == 0 && bpf_is_task_park_kfunc(&meta)) + mark_subprog_might_park(env, t); if (ret == 0 && bpf_is_kfunc_pkt_changing(&meta)) mark_subprog_changes_pkt_data(env, t); if (ret == 0 && bpf_is_throw_kfunc(insn)) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b42782c7f209..d8fbdd6b9658 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -12129,6 +12129,26 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return err; } + /* + * A park inside the callee is checked when the callee is verified, + * but only against the callee's own frame. Apply the rules for the + * frames the callee cannot see here, at the call. + */ + if (env->subprog_info[subprog].might_park) { + if (in_callback_frames(env)) { + verbose(env, "global function %s() may call bpf_task_park(), which is not allowed in a callback\n", + sub_name); + operation = bpf_diag_fmt(env, "global function %s() that may call bpf_task_park()", + sub_name); + bpf_diag_policy(env, *insn_idx, operation, "the call is in a callback", + "Call the function from the main program or from a function it calls directly, not from a callback."); + return -EINVAL; + } + err = check_park_stack_objects(env, *insn_idx); + if (err) + return err; + } + if (env->log.level & BPF_LOG_LEVEL) verbose(env, "Func#%d ('%s') is global and assumed valid.\n", subprog, sub_name); @@ -12141,6 +12161,8 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, returns_void = subprog_returns_void(env, subprog); if (env->subprog_info[subprog].changes_pkt_data) clear_all_pkt_pointers(env); + if (env->subprog_info[subprog].might_park) + invalidate_regs_across_park(env); if (returns_void) bpf_diag_record_scrub(env, &caller->regs[BPF_REG_0], BPF_DIAG_MOD_CALLER_SAVED); else @@ -14538,6 +14560,11 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_call_arg_meta *meta) is_kfunc_dynptr_write(meta); } +bool bpf_is_task_park_kfunc(struct bpf_call_arg_meta *meta) +{ + return is_kfunc_call(meta, special_kfunc_list[KF_bpf_task_park]); +} + static u32 kfunc_abi_slots(const struct btf_func_model *fm) { const struct bpf_jit_arg_abi *abi = bpf_jit_arg_abi(); -- 2.53.0