bpf_get_btf_vmlinux() lazily parses the vmlinux BTF under the bpf_verifier_lock, but publishes the result through a plain store and re-checks it through a plain lockless load. Nothing orders the stores initializing the struct btf inside btf_parse_vmlinux() against the store publishing the pointer: On a weakly ordered arch, a concurrent first-time caller taking the lockless fast path could in principle observe the pointer before the parsed contents are visible. The mutex_unlock() does not help such a reader given it only synchronizes with a later acquisition of the same lock. Thus, publish the pointer with smp_store_release() and read it on the fast path with smp_load_acquire(). Acquire semantics are needed rather than a dependency-ordered READ_ONCE(): btf_parse_vmlinux() also populates globals outside the returned object (e.g. bpf_ctx_convert.t). An address dependency would only order accesses performed through the pointer and not cover other globals. Signed-off-by: Daniel Borkmann --- kernel/bpf/verifier.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 001ac53825da..9217e0f87cb5 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19559,13 +19559,25 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt struct btf *bpf_get_btf_vmlinux(void) { - if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) { + /* Pairs with the smp_store_release() on the parse path below. */ + struct btf *btf = smp_load_acquire(&btf_vmlinux); + + if (!btf && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) { mutex_lock(&bpf_verifier_lock); - if (!btf_vmlinux) - btf_vmlinux = btf_parse_vmlinux(); + btf = btf_vmlinux; + if (!btf) { + btf = btf_parse_vmlinux(); + /* + * Order the parsed BTF contents and the globals the + * parse populated (e.g. bpf_ctx_convert.t) before + * the pointer publication. Pairs with the acquire + * on the lockless fast path above. + */ + smp_store_release(&btf_vmlinux, btf); + } mutex_unlock(&bpf_verifier_lock); } - return btf_vmlinux; + return btf; } /* -- 2.43.0