From: Josef Bacik bpf_local_storage_destroy() frees the elements of an owner behind a plain RCU grace period, on the ground that no program can refer to an owner that goes away. But the pointer that bpf_task_storage_get() and its like return stays valid for the verifier after the program has released the owner, or has left the bpf_rcu_read_lock() section it had the owner from. A sleepable program is in no RCU read side critical section, so it can write to the value after the owner has died and the element was freed: BUG: KASAN: slab-use-after-free in _copy_from_user+0x42/0xa0 Write of size 8 at addr ffff8880074404a8 by task lsuaf/55 Free the element behind an RCU Tasks Trace grace period if sleepable code may use the map. A program that a map is bound to was loaded, and counted in map->sleepable_refcnt, before it got a value from the live owner, and stays counted until a grace period behind its last run. A program can also get at a storage map by a lookup in an outer map. Then only the outer map is counted, its programs can come after the inner map was put into it, and can hold on to the inner map after it was taken out. So note in a map that it has been an inner map, and treat it like one with sleepable users from then on. The same goes for what the special fields of the value hold. The page of a uptr has to stay pinned, and a kptr which the program stores after the owner has died has to be released. Leave the fields, and the smap that it takes to free them, to bpf_selem_free_trace_rcu(), as bpf_selem_unlink() does. bpf_local_storage_map_free() waits for that callback, and leaves smap alone if it comes across such an element. Elements of such maps now wait for sleepable programs that are running when their owner dies. Other maps are freed as before. Fixes: dcf456c9a095 ("bpf: Fix usage of trace RCU in local storage.") Signed-off-by: Josef Bacik Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 1 + kernel/bpf/bpf_local_storage.c | 31 +++++++++++++++++++++++++------ kernel/bpf/map_in_map.c | 7 +++++-- 3 files changed, 31 insertions(+), 8 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 22947fa192e4..a10af47978a4 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -341,6 +341,7 @@ struct bpf_map { bool frozen; /* write-once; write-protected by freeze_mutex */ bool free_after_mult_rcu_gp; bool free_after_rcu_gp; + bool was_inner_map; atomic64_t sleepable_refcnt; s64 __percpu *elem_count; u64 cookie; /* write-once */ diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 49ea00c99390..b12fcec47ffa 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -352,7 +352,8 @@ int bpf_selem_unlink(struct bpf_local_storage_elem *selem) static void bpf_selem_unlink_nofail(struct bpf_local_storage_elem *selem, struct bpf_local_storage_map_bucket *b) { - bool in_map_free = !!b, free_storage = false; + bool in_map_free = !!b, free_storage = false, sleepable_users = false; + bool keep_fields = false; struct bpf_local_storage *local_storage; struct bpf_local_storage_map *smap; unsigned long flags; @@ -362,17 +363,26 @@ static void bpf_selem_unlink_nofail(struct bpf_local_storage_elem *selem, smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held()); if (smap) { + /* A map that is freed, or NULL here, has no program left. */ + sleepable_users = !in_map_free && + (atomic64_read(&smap->map.sleepable_refcnt) || + READ_ONCE(smap->map.was_inner_map)); b = b ? : select_bucket(smap, local_storage); err = raw_res_spin_lock_irqsave(&b->lock, flags); if (!err) { /* * Call bpf_obj_free_fields() under b->lock to make sure it is done * exactly once for an selem. Safe to free special fields immediately - * as no BPF program should be referencing the selem. + * as no BPF program should be referencing the selem, unless it + * is sleepable: see below. */ if (likely(selem_linked_to_map(selem))) { hlist_del_init_rcu(&selem->map_node); - bpf_obj_free_fields(smap->map.record, SDATA(selem)->data); + if (sleepable_users) + keep_fields = true; + else + bpf_obj_free_fields(smap->map.record, + SDATA(selem)->data); unlink++; } raw_res_spin_unlock_irqrestore(&b->lock, flags); @@ -387,7 +397,15 @@ static void bpf_selem_unlink_nofail(struct bpf_local_storage_elem *selem, * with selem1 as raw_res_spin_lock_irqsave() returns -ETIMEDOUT. */ WARN_ON_ONCE(err && in_map_free); - if (!err || in_map_free) + /* + * A sleepable program can still use, and fill, the fields of a value + * that it got before it let go of the owner. As for bpf_selem_unlink(), + * bpf_selem_free_trace_rcu() frees them, and takes smap for that: + * map_free() waits for this RCU read side critical section and then + * for the callback. It must not take smap away from an selem that + * destroy() has unlinked from the map. + */ + if (in_map_free ? err || unlink : !err && !keep_fields) RCU_INIT_POINTER(SDATA(selem)->smap, NULL); } @@ -434,11 +452,12 @@ static void bpf_selem_unlink_nofail(struct bpf_local_storage_elem *selem, * * The owner that map_free() takes an selem from is alive: sleepable code, which * is in no RCU read side critical section, may be walking its list for another - * map. + * map. And it can still use a value that it got before it let go of an owner + * that destroy() is called for. */ if (unlink == 2 || atomic_cmpxchg(&selem->state, SELEM_UNLINKED, SELEM_TOFREE) == SELEM_UNLINKED) - bpf_selem_free(selem, !in_map_free); + bpf_selem_free(selem, !in_map_free && !sleepable_users); if (free_storage) bpf_local_storage_free(local_storage, !in_map_free); diff --git a/kernel/bpf/map_in_map.c b/kernel/bpf/map_in_map.c index 5ee8aae7435b..0fa3f2ac0f99 100644 --- a/kernel/bpf/map_in_map.c +++ b/kernel/bpf/map_in_map.c @@ -110,10 +110,13 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map, return ERR_PTR(-ENOTSUPP); inner_map_meta = map->inner_map_meta; - if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map)) + if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map)) { bpf_map_inc(inner_map); - else + /* Its users are those of the outer maps from now on */ + WRITE_ONCE(inner_map->was_inner_map, true); + } else { inner_map = ERR_PTR(-EINVAL); + } return inner_map; } -- 2.55.0