shrinker_info_unit holds BITS_PER_LONG atomic_long_t of deferred work plus a one-word bitmap, 520 bytes on 64-bit. kzalloc_node() takes it from kmalloc-1k, so almost half of every unit is padding. Allocate the units from a cache of their own, sized to the unit. The cache is not cache-line aligned: the counters of a unit already share cache lines between shrinkers, and aligning the unit would only keep neighbouring units off each other's boundary line, at the cost of part of the saving. Suggested-by: Andrew Morton Assisted-by: LLM Signed-off-by: Konstantin Khorenko --- This is v3 of patch 1/2 of "mm/shrinker: make per-memcg shrinker units cheaper", sent on its own as Andrew asked in the v2 thread. Patch 2/2 of v1-v2, "mm/shrinker: allocate the per-memcg shrinker units on demand", is not part of this posting. It is the change that makes a memcg pay only for the shrinkers it uses, and it is discussed separately in the v2 thread. Tested on a 4-CPU, single-node VM with the test from the earlier cover letters: 8000 tmpfs mounts, then 2000 empty memory cgroups. The units move from kmalloc-1k to the new cache, 31 per order-2 slab, and the unreclaimable slab per cgroup goes from 144 KB to 87 KB. Removing the cgroups and dropping the caches returns the cache to its earlier size. memory.reclaim on a cgroup still frees its objects on 140 ext4 superblocks, and cgroups making their first objects while mounts keep raising shrinker_nr_max leave nothing in dmesg, also with slub_debug=FZPU on the cache. selftests/cgroup test_memcontrol passes. test_kmem_memcg_deletion also fails without the patch, by a similar margin. Changes in v3: - Allocate the unit from a dedicated kmem_cache instead of shrinking it to 63 ids so that it fits kmalloc-512 (Andrew Morton, Muchun Song). New subject, since the unit keeps its size. - Link to v2: https://patch.msgid.link/20261014-b4-prep-vstor-144715-size-shrinker_info_unit-to-a-power-of-two-v2-0-51af2aedbe6c@virtuozzo.com Changes in v2: - Patch 1: check the unit size with BUILD_BUG_ON(!is_power_of_2()) in shrinker_unit_alloc() instead of an open-coded static_assert() in the header (Muchun Song). - Link to v1: https://patch.msgid.link/20261012-b4-prep-vstor-144715-size-shrinker_info_unit-to-a-power-of-two-v1-0-5a7608393b7a@virtuozzo.com --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 2 ++ mm/shrinker.c | 11 +++++++++-- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 7d1c0ce189a88..b5a89594c263c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1675,6 +1675,7 @@ static inline u64 mem_cgroup_get_socket_pressure(struct mem_cgroup *memcg) } #endif +void shrinker_unit_cache_init(void); int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 856a7d07586cc..7a817181168ed 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5717,6 +5717,8 @@ int __init mem_cgroup_init(void) memcg_pn_cachep = KMEM_CACHE(mem_cgroup_per_node, SLAB_PANIC | SLAB_HWCACHE_ALIGN); + shrinker_unit_cache_init(); + return 0; } diff --git a/mm/shrinker.c b/mm/shrinker.c index 7ec2a9704f6f2..4fb424e721bca 100644 --- a/mm/shrinker.c +++ b/mm/shrinker.c @@ -12,6 +12,12 @@ DEFINE_MUTEX(shrinker_mutex); #ifdef CONFIG_MEMCG static int shrinker_nr_max; +static struct kmem_cache *shrinker_unit_cachep; + +void __init shrinker_unit_cache_init(void) +{ + shrinker_unit_cachep = KMEM_CACHE(shrinker_info_unit, SLAB_PANIC); +} static inline int shrinker_unit_size(int nr_items) { @@ -33,7 +39,7 @@ static inline void shrinker_unit_free(struct shrinker_info *info, int start) if (!unit[i]) break; - kfree(unit[i]); + kmem_cache_free(shrinker_unit_cachep, unit[i]); unit[i] = NULL; } } @@ -47,7 +53,8 @@ static inline int shrinker_unit_alloc(struct shrinker_info *new, int i; for (i = start; i < nr; i++) { - unit = kzalloc_node(sizeof(*unit), GFP_KERNEL, nid); + unit = kmem_cache_alloc_node(shrinker_unit_cachep, + GFP_KERNEL | __GFP_ZERO, nid); if (!unit) { shrinker_unit_free(new, start); return -ENOMEM; --- base-commit: a5ebb76233b79db01e82e061173cb73c6d2b5c6b change-id: 20261016-vstor-144715-shrinker_info_unit-kmem_cache-69ea3f78b635 Best regards, -- Konstantin Khorenko