For the reasons unknown, zsmalloc limits the number of size-classes to 256. On PAGE_SIZE 4K systems this works pretty fine, as those 256 classes are 4096/256 = 16 (known as size class delta) bytes apart. However, as the PAGE_SIZE grows, e.g. 16K, the hard limit pushes the size-class delta significantly further (e.g. 16384/256 = 64) leading to increased internal fragmentation. For example, on a 16K page system, an object of size 65 bytes is rounded up to the next 64-byte boundary (128 bytes), wasting nearly 50% of the allocated space. Instead of calculating size-class delta based on both PAGE_SIZE and hard limit of 256 the ZS_SIZE_CLASS_DELTA is set to constant value of 16 bytes. This results in a much higher than 256 number of size classes on systems with PAGE_SIZE larger than 4K. These extra size classes split existing cluster into smaller ones. For example, using tool [1] 16K PAGE_SIZE, chain size 8: BASE (delta 64 bytes) ===================== Log | Phys | Chain | Objs/Page | TailWaste | MergeWaste [..] 1072 | 1120 | 8 | 117 | 32 | 5616 1088 | 1120 | 8 | 117 | 32 | 3744 1104 | 1120 | 8 | 117 | 32 | 1872 1120 | 1120 | 8 | 117 | 32 | 0 [..] PATCHED (delta 16 bytes) ======================== [..] 1072 | 1072 | 4 | 61 | 144 | 0 1088 | 1088 | 1 | 15 | 64 | 0 1104 | 1104 | 6 | 89 | 48 | 0 1120 | 1120 | 8 | 117 | 32 | 0 [..] In default configuration (delta 64) size classes 1072 to 1104 are merged into 1120. Size class 1120 holds 117 objects per-zspage, so worst case every zspage can lose 5616 bytes (1120-1072 times 117). With delta 16 this cluster doesn't exist, reducing memory waste. [1] https://github.com/sergey-senozhatsky/simulate-zsmalloc/blob/main/simulate_zsmalloc.c Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 5bf832f9c05c..5e7501d36161 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -92,7 +92,7 @@ #define HUGE_BITS 1 #define FULLNESS_BITS 4 -#define CLASS_BITS 8 +#define CLASS_BITS 12 #define MAGIC_VAL_BITS 8 #define ZS_MAX_PAGES_PER_ZSPAGE (_AC(CONFIG_ZSMALLOC_CHAIN_SIZE, UL)) @@ -115,8 +115,13 @@ * * ZS_MIN_ALLOC_SIZE and ZS_SIZE_CLASS_DELTA must be multiple of ZS_ALIGN * (reason above) + * + * We set ZS_SIZE_CLASS_DELTA to 16 bytes to maintain high granularity + * even on systems with large PAGE_SIZE (e.g. 16K, 64K). This prevents + * internal fragmentation. CLASS_BITS is increased to 12 to address the + * larger number of size classes on such systems (up to 4096 classes on 64K). */ -#define ZS_SIZE_CLASS_DELTA (PAGE_SIZE >> CLASS_BITS) +#define ZS_SIZE_CLASS_DELTA 16 #define ZS_SIZE_CLASSES (DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - ZS_MIN_ALLOC_SIZE, \ ZS_SIZE_CLASS_DELTA) + 1) -- 2.52.0.351.gbe84eed79e-goog