The reference-counted huge zero folio is problematic - it's prone to subtle races, it can introduce allocation latency and it is confusing and complicated code. Commit 2d8bd8049e89 ("mm: add persistent huge zero folio") significantly improved this situation by introducing CONFIG_PERSISTENT_HUGE_ZERO_FOLIO which provides a single huge PMD-sized folio at boot which is used consistently as the huge zero folio. This was made optional to account for systems which have constrained memory. However the existence of this folio is predicated on CONFIG_TRANSPARENT_HUGEPAGE, which already implies higher memory requirements (if THP, as it defaults to, is globally enabled to any degree memory reserves are already significantly increased by set_recommended_min_free_kbytes()). So memory-constrained systems should already not set CONFIG_TRANSPARENT_HUGEPAGE, and it is reasonable to consider this configuration option gating us from embedded systems that are cautious about this. The huge zero folio becomes much larger in systems with larger page size (with 64 KiB arm64 systems having 512 MiB PMDs), however the same argument regarding the reserves applies there, and arbitrarily allocating 512 MiB on a THP read fault is equally problematic. In the case that we cannot allocate the huge zero folio at startup, THP is disabled using the TRANSPARENT_HUGEPAGE_UNSUPPORTED flag. The benefits of enabling this by default are: * Significant reduction in code complexity (and lots of code deleted). * Elimination of any class of bug relating to huge zero folio memory ordering. * Elimination of latency on huge page read fault. * Huge zero shrinker is completely removed. * do_huge_pmd_anonymous_page() no longer ever falls back to non-THP on read fault and the logic is simplified. * copy_huge_pmd() simplified. * Frees up an mm flag (MMF_HUGE_ZERO_FOLIO no longer required). largest_zero_folio() needs some care - the callers of this function may have !CONFIG_TRANSPARENT_HUGEPAGE so update it to reflect that and document it. Therefore, remove CONFIG_PERSISTENT_HUGE_ZERO_FOLIO and enable the functionality by default. The boot-time allocation is accounted via the existing thp_zero_page_alloc vmstat counter (now counted at most once, at boot) and the transhuge.rst description of the zero page counters is updated accordingly. Note that the small-memory (<512MiB) default-off path in hugepage_init() retains TRANSPARENT_HUGEPAGE_UNSUPPORTED should the boot-time allocation have failed. Signed-off-by: Lorenzo Stoakes (ARM) --- NOTE: This RFC is dependent on [0] [0]:https://lore.kernel.org/linux-mm/20260728-fix-refcounted-huge-zero-v1-0-3f261f5447b4@kernel.org/ To: Andrew Morton To: David Hildenbrand To: Zi Yan To: Baolin Wang To: "Liam R. Howlett" To: Nico Pache To: Ryan Roberts To: Dev Jain To: Barry Song To: Lance Yang To: Usama Arif To: Vlastimil Babka To: Mike Rapoport To: Suren Baghdasaryan To: Michal Hocko To: Jonathan Corbet To: Shuah Khan To: Heiko Carstens To: Vasily Gorbik To: Alexander Gordeev To: Christian Borntraeger To: Sven Schnelle To: Alexander Viro To: Christian Brauner To: Jan Kara To: Dan Williams To: Matthew Wilcox To: Kees Cook To: Ingo Molnar To: Peter Zijlstra To: Juri Lelli To: Vincent Guittot To: Dietmar Eggemann To: Steven Rostedt To: Ben Segall To: Mel Gorman To: Valentin Schneider To: K Prateek Nayak Cc: ljs@kernel.org Cc: linux-mm@kvack.org Cc: linux-doc@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-s390@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Cc: nvdimm@lists.linux.dev Cc: Pankaj Raghav --- Documentation/admin-guide/mm/transhuge.rst | 8 +- arch/s390/configs/debug_defconfig | 1 - arch/s390/configs/defconfig | 1 - fs/dax.c | 14 ++- include/linux/huge_mm.h | 42 ++------ include/linux/mm_types.h | 2 - kernel/fork.c | 1 - mm/Kconfig | 16 --- mm/huge_memory.c | 163 ++--------------------------- 9 files changed, 32 insertions(+), 216 deletions(-) diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst index 16f37135ed80..0d8010936c02 100644 --- a/Documentation/admin-guide/mm/transhuge.rst +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -618,13 +618,13 @@ thp_split_pmd page table entry. thp_zero_page_alloc - is incremented every time a huge zero page used for thp is - successfully allocated. Note, it doesn't count every map of + is incremented when the huge zero folio is successfully + allocated at boot. Note, it doesn't count every map of the huge zero page, only its allocation. thp_zero_page_alloc_failed - is incremented if kernel fails to allocate - huge zero page and falls back to using small pages. + is incremented if the kernel fails to allocate the huge zero + folio at boot, in which case THP is disabled. thp_swpout is incremented every time a huge page is swapout in one diff --git a/arch/s390/configs/debug_defconfig b/arch/s390/configs/debug_defconfig index 54637be87fb7..cec61569da14 100644 --- a/arch/s390/configs/debug_defconfig +++ b/arch/s390/configs/debug_defconfig @@ -105,7 +105,6 @@ CONFIG_SLUB_STATS=y CONFIG_MEMORY_HOTPLUG=y CONFIG_MEMORY_HOTREMOVE=y CONFIG_KSM=y -CONFIG_PERSISTENT_HUGE_ZERO_FOLIO=y CONFIG_TRANSPARENT_HUGEPAGE=y CONFIG_CMA_DEBUGFS=y CONFIG_CMA_SYSFS=y diff --git a/arch/s390/configs/defconfig b/arch/s390/configs/defconfig index 5f5114a253cf..0e40419f05c8 100644 --- a/arch/s390/configs/defconfig +++ b/arch/s390/configs/defconfig @@ -98,7 +98,6 @@ CONFIG_SLAB_BUCKETS=y CONFIG_MEMORY_HOTPLUG=y CONFIG_MEMORY_HOTREMOVE=y CONFIG_KSM=y -CONFIG_PERSISTENT_HUGE_ZERO_FOLIO=y CONFIG_TRANSPARENT_HUGEPAGE=y CONFIG_CMA_SYSFS=y CONFIG_CMA_AREAS=7 diff --git a/fs/dax.c b/fs/dax.c index 4ea3384ecefd..c941fff3aeff 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -1413,22 +1413,20 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf, { struct address_space *mapping = vmf->vma->vm_file->f_mapping; struct inode *inode = mapping->host; - struct folio *zero_folio; vm_fault_t ret; - zero_folio = mm_get_huge_zero_folio(vmf->vma->vm_mm); - - if (unlikely(!zero_folio)) { - trace_dax_pmd_load_hole_fallback(inode, vmf, zero_folio, *entry); + if (unlikely(!huge_zero_folio)) { + trace_dax_pmd_load_hole_fallback(inode, vmf, huge_zero_folio, *entry); return VM_FAULT_FALLBACK; } - *entry = dax_insert_entry(xas, vmf, iter, *entry, folio_pfn(zero_folio), + *entry = dax_insert_entry(xas, vmf, iter, *entry, + folio_pfn(huge_zero_folio), DAX_PMD | DAX_ZERO_PAGE); - ret = vmf_insert_folio_pmd(vmf, zero_folio, false); + ret = vmf_insert_folio_pmd(vmf, huge_zero_folio, false); if (ret == VM_FAULT_NOPAGE) - trace_dax_pmd_load_hole(inode, vmf, zero_folio, *entry); + trace_dax_pmd_load_hole(inode, vmf, huge_zero_folio, *entry); return ret; } #else diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c745f7ad2298..e43dffac37f9 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -572,20 +572,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return pmd_present(pmd) && is_huge_zero_pfn(pmd_pfn(pmd)); } -struct folio *mm_get_huge_zero_folio(struct mm_struct *mm); -void mm_put_huge_zero_folio(struct mm_struct *mm); - -static inline struct folio *get_persistent_huge_zero_folio(void) -{ - if (!IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) - return NULL; - - if (unlikely(!huge_zero_folio)) - return NULL; - - return huge_zero_folio; -} - static inline bool thp_migration_supported(void) { return IS_ENABLED(CONFIG_ARCH_HAS_PMD_SOFTLEAVES); @@ -769,11 +755,6 @@ static inline bool is_huge_zero_pmd(pmd_t pmd) return false; } -static inline void mm_put_huge_zero_folio(struct mm_struct *mm) -{ - return; -} - static inline bool thp_migration_supported(void) { return false; @@ -828,26 +809,25 @@ static inline int split_folio_to_order(struct folio *folio, int new_order) return split_huge_page_to_list_to_order(&folio->page, NULL, new_order); } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE /** * largest_zero_folio - Get the largest zero size folio available * - * This function shall be used when mm_get_huge_zero_folio() cannot be - * used as there is no appropriate mm lifetime to tie the huge zero folio - * from the caller. + * If CONFIG_TRANSPARENT_HUGEPAGE is enabled, this is the huge zero folio, + * provided its boot-time allocation succeeded. * - * Deduce the size of the folio with folio_size instead of assuming the - * folio size. + * Otherwise, this is the single page zero folio. * - * Return: pointer to PMD sized zero folio if CONFIG_PERSISTENT_HUGE_ZERO_FOLIO - * is enabled or a single page sized zero folio + * Return: pointer to the largest sized zero folio available. */ static inline struct folio *largest_zero_folio(void) { - struct folio *folio = get_persistent_huge_zero_folio(); - - if (folio) - return folio; - + return huge_zero_folio ?: page_folio(ZERO_PAGE(0)); +} +#else +static inline struct folio *largest_zero_folio(void) +{ return page_folio(ZERO_PAGE(0)); } +#endif #endif /* _LINUX_HUGE_MM_H */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index b5d4cd3b067b..36451119e857 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1965,8 +1965,6 @@ enum { #define MMF_VM_MERGEABLE 16 /* KSM may merge identical pages */ #define MMF_VM_HUGEPAGE 17 /* set when mm is available for khugepaged */ -#define MMF_HUGE_ZERO_FOLIO 18 /* mm has ever used the global huge zero folio */ - #define MMF_HAS_UPROBES 19 /* has uprobes */ #define MMF_RECALC_UPROBES 20 /* MMF_HAS_UPROBES can be wrong */ #define MMF_OOM_SKIP 21 /* mm is of no interest for the OOM killer */ diff --git a/kernel/fork.c b/kernel/fork.c index f0e2e131a9a5..e0bae09fb6c2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1185,7 +1185,6 @@ static inline void __mmput(struct mm_struct *mm) ksm_exit(mm); khugepaged_exit(mm); /* must run before exit_mmap */ exit_mmap(mm); - mm_put_huge_zero_folio(mm); set_mm_exe_file(mm, NULL); if (!list_empty(&mm->mmlist)) { spin_lock(&mmlist_lock); diff --git a/mm/Kconfig b/mm/Kconfig index c52ab6afcb15..5adf705a2ad3 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -810,22 +810,6 @@ config ARCH_WANT_GENERAL_HUGETLB config ARCH_WANTS_THP_SWAP def_bool n -config PERSISTENT_HUGE_ZERO_FOLIO - bool "Allocate a PMD sized folio for zeroing" - depends on TRANSPARENT_HUGEPAGE - help - Enable this option to reduce the runtime refcounting overhead - of the huge zero folio and expand the places in the kernel - that can use huge zero folios. For instance, block I/O benefits - from access to large folios for zeroing memory. - - With this option enabled, the huge zero folio is allocated - once and never freed. One full huge page's worth of memory shall - be used. - - Say Y if your system has lots of memory. Say N if you are - memory constrained. - config MM_ID def_bool n diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ac108955299c..aaa2aa6f9402 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -81,11 +81,6 @@ static bool split_underused_thp = true; #define HUGE_ZERO_UNSET_PFN (~0UL) struct folio *huge_zero_folio __read_mostly; unsigned long huge_zero_pfn __read_mostly = HUGE_ZERO_UNSET_PFN; -#ifndef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO -static atomic_t huge_zero_refcount; -static DEFINE_SPINLOCK(huge_zero_lock); -static struct shrinker *huge_zero_folio_shrinker; -#endif unsigned long huge_anon_orders_always __read_mostly; unsigned long huge_anon_orders_madvise __read_mostly; @@ -243,141 +238,18 @@ static struct folio *alloc_huge_zero_folio(void) return zero_folio; } -#ifdef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO -static int __init huge_zero_init(void) +static void __init huge_zero_init(void) { huge_zero_folio = alloc_huge_zero_folio(); if (!huge_zero_folio) { - pr_warn("Allocating persistent huge zero folio failed\n"); - } else { - huge_zero_pfn = folio_pfn(huge_zero_folio); - count_vm_event(THP_ZERO_PAGE_ALLOC); - } - return 0; -} - -static void __init huge_zero_shrinker_exit(void) -{ -} - -struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) -{ - return huge_zero_folio; -} - -void mm_put_huge_zero_folio(struct mm_struct *mm) -{ -} -#else -static bool get_huge_zero_folio(void) -{ - struct folio *zero_folio; - - /* Paired with atomic_set_release(). */ - if (likely(atomic_inc_not_zero(&huge_zero_refcount))) - return true; - - zero_folio = alloc_huge_zero_folio(); - if (unlikely(!zero_folio)) - return false; - - /* Paired with critical section in shrink_huge_zero_folio_scan(). */ - spin_lock(&huge_zero_lock); - if (huge_zero_folio) { - /* Somebody else already installed it. */ - atomic_inc(&huge_zero_refcount); - spin_unlock(&huge_zero_lock); - folio_put(zero_folio); - return true; + pr_warn("huge zero folio alloc failed, disabling THP\n"); + transparent_hugepage_flags = 1 << TRANSPARENT_HUGEPAGE_UNSUPPORTED; + return; } - WRITE_ONCE(huge_zero_folio, zero_folio); - WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio)); - /* Paired with atomic_inc_not_zero(). +1 for shrinker pin. */ - atomic_set_release(&huge_zero_refcount, 2); - spin_unlock(&huge_zero_lock); - + huge_zero_pfn = folio_pfn(huge_zero_folio); count_vm_event(THP_ZERO_PAGE_ALLOC); - return true; } -static void put_huge_zero_folio(void) -{ - /* - * Counter should never go to zero here. Only shrinker can put - * last reference. - */ - WARN_ON_ONCE(atomic_dec_and_test(&huge_zero_refcount)); -} - -static unsigned long shrink_huge_zero_folio_count(struct shrinker *shrink, - struct shrink_control *sc) -{ - /* we can free zero page only if last reference remains */ - return atomic_read(&huge_zero_refcount) == 1 ? HPAGE_PMD_NR : 0; -} - -static unsigned long shrink_huge_zero_folio_scan(struct shrinker *shrink, - struct shrink_control *sc) -{ - struct folio *zero_folio; - - /* Paired with critical section in get_huge_zero_folio(). */ - scoped_guard(spinlock, &huge_zero_lock) { - /* Paired with atomic_inc_not_zero() in get_huge_zero_folio(). */ - if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) != 1) - return 0; - - zero_folio = huge_zero_folio; - VM_WARN_ON_ONCE(!huge_zero_folio); - WRITE_ONCE(huge_zero_folio, NULL); - WRITE_ONCE(huge_zero_pfn, HUGE_ZERO_UNSET_PFN); - } - - folio_put(zero_folio); - return HPAGE_PMD_NR; -} - -static int __init huge_zero_init(void) -{ - huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero"); - if (!huge_zero_folio_shrinker) { - shrinker_free(deferred_split_shrinker); - list_lru_destroy(&deferred_split_lru); - return -ENOMEM; - } - - huge_zero_folio_shrinker->count_objects = shrink_huge_zero_folio_count; - huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan; - shrinker_register(huge_zero_folio_shrinker); - return 0; -} - -static void __init huge_zero_shrinker_exit(void) -{ - shrinker_free(huge_zero_folio_shrinker); -} - -struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) -{ - if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) - return READ_ONCE(huge_zero_folio); - - if (!get_huge_zero_folio()) - return NULL; - - if (mm_flags_test_and_set(MMF_HUGE_ZERO_FOLIO, mm)) - put_huge_zero_folio(); - - return READ_ONCE(huge_zero_folio); -} - -void mm_put_huge_zero_folio(struct mm_struct *mm) -{ - if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) - put_huge_zero_folio(); -} -#endif /* CONFIG_PERSISTENT_HUGE_ZERO_FOLIO */ - #ifdef CONFIG_SYSFS static ssize_t enabled_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) @@ -1038,14 +910,14 @@ static int __init thp_shrinker_init(void) deferred_split_shrinker->scan_objects = deferred_split_scan; shrinker_register(deferred_split_shrinker); - return huge_zero_init(); + huge_zero_init(); + return 0; } static void __init thp_shrinker_exit(void) { shrinker_free(deferred_split_shrinker); list_lru_destroy(&deferred_split_lru); - huge_zero_shrinker_exit(); } static int __init hugepage_init(void) @@ -1081,7 +953,8 @@ static int __init hugepage_init(void) * is likely to save. The admin can still enable it through /sys. */ if (totalram_pages() < MB_TO_PAGES(512)) { - transparent_hugepage_flags = 0; + /* Retain boot-time THP disablement on huge zero folio failure. */ + transparent_hugepage_flags &= 1 << TRANSPARENT_HUGEPAGE_UNSUPPORTED; return 0; } @@ -1534,18 +1407,11 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) !mm_forbids_zeropage(vma->vm_mm) && transparent_hugepage_use_zero_page()) { pgtable_t pgtable; - struct folio *zero_folio; vm_fault_t ret; pgtable = pte_alloc_one(vma->vm_mm); if (unlikely(!pgtable)) return VM_FAULT_OOM; - zero_folio = mm_get_huge_zero_folio(vma->vm_mm); - if (unlikely(!zero_folio)) { - pte_free(vma->vm_mm, pgtable); - count_vm_event(THP_FAULT_FALLBACK); - return VM_FAULT_FALLBACK; - } vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); ret = 0; if (pmd_none(*vmf->pmd)) { @@ -1560,7 +1426,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) VM_BUG_ON(ret & VM_FAULT_FALLBACK); } else { set_huge_zero_folio(pgtable, vma->vm_mm, vma, - haddr, vmf->pmd, zero_folio); + haddr, vmf->pmd, huge_zero_folio); update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); spin_unlock(vmf->ptl); } @@ -1967,15 +1833,8 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, * under splitting since we don't split the page itself, only pmd to * a page table. */ - if (is_huge_zero_pmd(pmd)) { - /* - * mm_get_huge_zero_folio() will never allocate a new - * folio here, since we already have a zero page to - * copy. It just takes a reference. - */ - mm_get_huge_zero_folio(dst_mm); + if (is_huge_zero_pmd(pmd)) goto out_zero_page; - } src_page = pmd_page(pmd); VM_BUG_ON_PAGE(!PageHead(src_page), src_page); --- base-commit: 953c2e668733a3b3c49609ef38eca502a6b5cbfc change-id: 20260728-persist-huge-zero-folio-b31c00123962 Cheers, -- Lorenzo Stoakes (ARM)