When init_on_alloc is enabled, kernel_init_pages() clears every page one at a time via clear_highpage_kasan_tagged(), which incurs per-page kmap_local_page()/kunmap_local() overhead and prevents the architecture clearing primitive from operating on contiguous ranges. Introduce clear_highpages_kasan_tagged() in highmem.h, a batch clearing helper that calls clear_pages() for the full contiguous range on !HIGHMEM systems, bypassing the per-page kmap overhead and allowing a single invocation of the arch clearing primitive across the entire allocation. The HIGHMEM path falls back to per-page clearing since those pages require kmap. Replace kernel_init_pages() with direct calls to the new helper, as it becomes a trivial wrapper. Allocating 8192 x 2MB HugeTLB pages (16GB) with init_on_alloc=1: Before: 0.445s After: 0.166s (-62.7%, 2.68x faster) Kernel time (sys) reduction per workload with init_on_alloc=1: Workload Before After Change Graph500 64C128T 30m 41.8s 15m 14.8s -50.3% Graph500 16C32T 15m 56.7s 9m 43.7s -39.0% Pagerank 32T 1m 58.5s 1m 12.8s -38.5% Pagerank 128T 2m 36.3s 1m 40.4s -35.7% Signed-off-by: Hrushikesh Salunke Acked-by: Vlastimil Babka (SUSE) Acked-by: Zi Yan Acked-by: Pankaj Gupta --- base commit: 2bcc13c29c711381d815c1ba5d5b25737400c71a v2: https://lore.kernel.org/all/20260421042451.76918-1-hsalunke@amd.com/ v1: https://lore.kernel.org/all/20260408092441.435133-1-hsalunke@amd.com/ Changes since v2: - Moved kasan_disable_current()/kasan_enable_current() into clear_highpages_kasan_tagged(), per David and Zi Yan's suggestion. - Removed kernel_init_pages() and replaced its two call sites with direct calls to the helper. Changes since v1: - Dropped cond_resched() and PROCESS_PAGES_NON_PREEMPT_BATCH as kernel_init_pages() runs inside the page allocator and can be called from atomic context, making cond_resched() unsafe. The original code never had a cond_resched() here, and the performance gain comes from batching, not rescheduling. - Moved the !HIGHMEM/HIGHMEM branching into a new clear_highpages_kasan_tagged() helper in highmem.h, per David's suggestion. include/linux/highmem.h | 15 +++++++++++++++ mm/page_alloc.c | 15 ++------------- 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/include/linux/highmem.h b/include/linux/highmem.h index af03db851a1d..1178b786b5b0 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -345,6 +345,21 @@ static inline void clear_highpage_kasan_tagged(struct page *page) kunmap_local(kaddr); } +static inline void clear_highpages_kasan_tagged(struct page *page, int numpages) +{ + /* s390's use of memset() could override KASAN redzones. */ + kasan_disable_current(); + if (!IS_ENABLED(CONFIG_HIGHMEM)) { + clear_pages(kasan_reset_tag(page_address(page)), numpages); + } else { + int i; + + for (i = 0; i < numpages; i++) + clear_highpage_kasan_tagged(page + i); + } + kasan_enable_current(); +} + #ifndef __HAVE_ARCH_TAG_CLEAR_HIGHPAGES /* Return false to let people know we did not initialize the pages */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 65e205111553..2908d24dd3e2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1208,17 +1208,6 @@ static inline bool should_skip_kasan_poison(struct page *page) return page_kasan_tag(page) == KASAN_TAG_KERNEL; } -static void kernel_init_pages(struct page *page, int numpages) -{ - int i; - - /* s390's use of memset() could override KASAN redzones. */ - kasan_disable_current(); - for (i = 0; i < numpages; i++) - clear_highpage_kasan_tagged(page + i); - kasan_enable_current(); -} - #ifdef CONFIG_MEM_ALLOC_PROFILING /* Should be called only if mem_alloc_profiling_enabled() */ @@ -1428,7 +1417,7 @@ __always_inline bool __free_pages_prepare(struct page *page, init = false; } if (init) - kernel_init_pages(page, 1 << order); + clear_highpages_kasan_tagged(page, 1 << order); /* * arch_free_page() can make the page's contents inaccessible. s390 @@ -1853,7 +1842,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, } /* If memory is still not initialized, initialize it now. */ if (init) - kernel_init_pages(page, 1 << order); + clear_highpages_kasan_tagged(page, 1 << order); set_page_owner(page, order, gfp_flags); page_table_check_alloc(page, order); -- 2.43.0