Lance reported that when nothing else causes the mm to be considered for khugepaged collapse, an MADV_HUGEPAGE-advised tmpfs VMA alone does not trigger scanning. After commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()"), the shmem/tmpfs allowable order check reads vma->flags directly. However, when MADV_HUGEPAGE is handled, khugepaged_enter_vma() is called before the VMA's flags have been updated, so the check uses stale flags and incorrectly rejects the VMA for collapse. As a result, khugepaged does not collapse the tmpfs file into PMD order in time. Fix this by calling khugepaged_enter_vma() with the new VMA flags in madvise_update_vma(). Meanwhile we can remove the khugepaged_enter_vma() in hugepage_madvise(). Reported-by: Lance Yang Closes: https://lore.kernel.org/all/20260815181632.21453-1-lance.yang@linux.dev/ Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()") Cc: stable@vger.kernel.org Suggested-by: Lorenzo Stoakes (ARM) Signed-off-by: Baolin Wang --- Changes from v1: v1: https://lore.kernel.org/all/ed34ca03ae7d65e89467fb87bc961f5497049c00.1786948410.git.baolin.wang@linux.alibaba.com/ - Update the commit message (per Lorenzo). - Call khugepaged_enter_vma() in madvise_update_vma() (per Lorenzo). --- mm/khugepaged.c | 6 ------ mm/madvise.c | 8 ++++++++ 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5a06e3942e88..79effd3f3da4 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -454,12 +454,6 @@ int hugepage_madvise(struct vm_area_struct *vma, case MADV_HUGEPAGE: *vm_flags &= ~VM_NOHUGEPAGE; *vm_flags |= VM_HUGEPAGE; - /* - * If the vma become good for khugepaged to scan, - * register it here without waiting a page fault that - * may not happen any time soon. - */ - khugepaged_enter_vma(vma, *vm_flags); break; case MADV_NOHUGEPAGE: *vm_flags &= ~VM_HUGEPAGE; diff --git a/mm/madvise.c b/mm/madvise.c index c179938097bf..cb93cf82d8df 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -178,6 +178,14 @@ static int madvise_update_vma(vm_flags_t new_flags, /* vm_flags is protected by the mmap_lock held in write mode. */ vma_start_write(vma); vma->flags = new_vma_flags; + /* + * If the vma become good for khugepaged to scan, + * register it here without waiting a page fault that + * may not happen any time soon. + */ + if (vma_flags_test(&new_vma_flags, VMA_HUGEPAGE_BIT)) + khugepaged_enter_vma(vma, vma_flags_to_legacy(new_vma_flags)); + if (set_new_anon_name) return replace_anon_vma_name(vma, anon_name); -- 2.47.3