A later patch will introduce bpf_selem_unlink_lockless() to handle rqspinlock errors. bpf_selem_unlink_lockless() will allow an selem to be partially unlinked from map or local storage. Therefore, bpf_selem_free() needs to be decoupled from map and local storage as SDATA(selem)->smap or selem->local_storage may be NULL. Decoupling from local storage is already done when local storage migrated from BPF memory allocator to kmalloc_nolock(). This patch prepares to decouple from map. Currently, map is still needed in bpf_selem_free() to: 1. Uncharge memory a. map->ops->map_local_storage_uncharge b. map->elem_size 2. Infer how memory should be freed a. map->use_kmalloc_nolock 3. Free special fields a. map->record The dependency of 1.a will be addressed by a later patch by returning the amount of memory to uncharge directly to the owner who calls bpf_local_storage_destroy(). The dependency of 3.a will be addressed by a later patch by freeing special fields under b->lock, when the map is still alive. This patch handles 1.b and 2.a by simply saving the informnation in bpf_local_storage_elem. Signed-off-by: Amery Hung --- include/linux/bpf_local_storage.h | 4 +++- kernel/bpf/bpf_local_storage.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf_local_storage.h b/include/linux/bpf_local_storage.h index 70b35dfc01c9..20918c31b7e5 100644 --- a/include/linux/bpf_local_storage.h +++ b/include/linux/bpf_local_storage.h @@ -80,7 +80,9 @@ struct bpf_local_storage_elem { * after raw_spin_unlock */ }; - /* 8 bytes hole */ + u16 size; + bool use_kmalloc_nolock; + /* 4 bytes hole */ /* The data is stored in another cacheline to minimize * the number of cachelines access during a cache hit. */ diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 667b468652d1..62201552dca6 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -97,6 +97,8 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, if (swap_uptrs) bpf_obj_swap_uptrs(smap->map.record, SDATA(selem)->data, value); } + selem->size = smap->elem_size; + selem->use_kmalloc_nolock = smap->use_kmalloc_nolock; return selem; } @@ -219,7 +221,7 @@ void bpf_selem_free(struct bpf_local_storage_elem *selem, smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held()); - if (!smap->use_kmalloc_nolock) { + if (!selem->use_kmalloc_nolock) { /* * No uptr will be unpin even when reuse_now == false since uptr * is only supported in task local storage, where -- 2.47.3