thp_underused() decides whether a large folio on the deferred split list is underused and should be split so that its zero-filled subpages can be reclaimed. Today it only ever checks if the folio is underused on PMD-sized folios. As such, thp_underused() assumes throughout that the folio has HPAGE_PMD_NR pages. A later patch will be queueing anonymous mTHP folios on the deferred split list as well, at which point this assumption will not be correct anymore. Generalize thp_underused() to be folio-size-aware instead of hardcoded to PMD-sized folios. The check for whether a folio can ever be underused is moved into its own function, thp_can_be_underused(), as this will get reused in the next patch when determining if the folio should get added to the deferred split queue. khugepaged_max_ptes_none keeps its meaning as an absolute number of pages, which is also how khugepaged applies it when collapsing to PMD order. It is deliberately not scaled down per folio order (collapse rejected proportional scaling of intermediate values because it either lets the memory footprint creep or ends up confusing to reason about). Read as an absolute count, this knob gains a useful second meaning for mTHP as being the smallest folio size that takes part in underused splitting at all. There is no functional change, as for a PMD-sized folio nr_pages is HPAGE_PMD_NR. Suggested-by: Johannes Weiner Signed-off-by: Joanne Koong --- mm/huge_memory.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1e5d68acf62a..c6ca2a541128 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4541,6 +4541,23 @@ bool __folio_unqueue_deferred_split(struct folio *folio) return unqueued; /* useful for debug warnings */ } +static bool thp_can_be_underused(unsigned long nr_pages, + unsigned long max_ptes_none) +{ + /* + * The sysctl maximum means the user tolerates any number of zero-filled + * pages, so nothing is ever underused. + */ + if (max_ptes_none == HPAGE_PMD_NR - 1) + return false; + + /* + * A folio no larger than the number of zero-filled pages the user + * tolerates can never exceed it. It can never be underused. + */ + return nr_pages > max_ptes_none; +} + /* partially_mapped=false won't clear PG_partially_mapped folio flag */ void deferred_split_folio(struct folio *folio, bool partially_mapped) { @@ -4602,25 +4619,27 @@ static unsigned long deferred_split_count(struct shrinker *shrink, static bool thp_underused(struct folio *folio) { - int num_zero_pages = 0, num_filled_pages = 0; - int i; + const unsigned long max_ptes_none = khugepaged_max_ptes_none; + const unsigned long nr_pages = folio_nr_pages(folio); + unsigned long num_zero_pages = 0, num_filled_pages = 0; + unsigned long i; - if (khugepaged_max_ptes_none == HPAGE_PMD_NR - 1) + if (!thp_can_be_underused(nr_pages, max_ptes_none)) return false; if (folio_contain_hwpoisoned_page(folio)) return false; - for (i = 0; i < folio_nr_pages(folio); i++) { + for (i = 0; i < nr_pages; i++) { if (pages_identical(folio_page(folio, i), ZERO_PAGE(0))) { - if (++num_zero_pages > khugepaged_max_ptes_none) + if (++num_zero_pages > max_ptes_none) return true; } else { /* * Another path for early exit once the number * of non-zero filled pages exceeds threshold. */ - if (++num_filled_pages >= HPAGE_PMD_NR - khugepaged_max_ptes_none) + if (++num_filled_pages >= nr_pages - max_ptes_none) return false; } } -- 2.52.0 Every anonymous PMD-sized folio is put on the deferred split queue when it is first mapped, so that the shrinker can find it under memory pressure and split it if it turns out to be mostly zero-filled. With the default khugepaged/max_ptes_none this is wasted work. The default is HPAGE_PMD_NR - 1, which tells thp_underused() that any number of zero-filled pages is tolerable, which means the shrinker will never split any of these folios for being underused. They take the list_lru lock at fault time, inflate the object count the shrinker reports, and are then walked and dropped when they're first scanned. Skip the queuing for folios that thp_can_be_underused() says can never qualify. At the default khugepaged/max_ptes_none that is all of them, and once the knob is lowered only folios with more pages than it are queued. This matters for a subsequent patch that adds anonymous mTHP folios to the deferred split list, as it prevents small mTHP orders from taking the list_lru lock on every anonymous fault. Please note that the queue has always been best-effort. A folio that was queued before khugepaged/max_ptes_none is lowered gets dropped from the queue by the first scan that finds it's not underused, so changing the sysctl has never retroactively applied to folios that were already scanned. Requeueing eligible folios when the sysctl changes will be addressed in a separate patch. Suggested-by: Johannes Weiner Signed-off-by: Joanne Koong --- mm/huge_memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c6ca2a541128..e3349f2314cb 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4573,8 +4573,18 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped) if (folio_order(folio) <= 1) return; - if (!partially_mapped && !split_underused_thp) - return; + if (!partially_mapped) { + if (!split_underused_thp) + return; + /* + * Nothing will ever split this folio for being underused, so + * keep it off the queue entirely rather than paying for the + * list_lru lock here and a shrinker scan later. + */ + if (!thp_can_be_underused(folio_nr_pages(folio), + khugepaged_max_ptes_none)) + return; + } /* * Exclude swapcache: originally to avoid a corrupt deferred split -- 2.52.0 Unlike for PMD-sized folios, an anonymous mTHP folio doesn't get added to the deferred split list at fault or collapse time. As a result, a fully mapped mTHP folio that is mostly zero-filled doesn't get split by the deferred split shrinker when the system is under memory pressure. At Meta we would like to deploy 2M THP=always on arm64 with 64k base pages, as 2M gives the contpte benefits while the PMD size (512M) is too big to use. Without underused splitting this causes memory regressions, as the unused parts of those folios can never be broken down and reclaimed. Add anonymous mTHP folios to the deferred split list from map_anon_folio_pte_nopf(), mirroring what map_anon_folio_pmd_nopf() already does for PMD-sized folios. This covers both the fault path and the khugepaged mTHP collapse path. If there is memory pressure, a zero-filled mTHP can then be split with its zero pages remapped to the shared zero page and reclaimed. The preceding patch bounds what folios can get added to the deferred split list. Nothing gets added at the default khugepaged/max_ptes_none, and in cases where it is lowered, only folios with more pages than it are added, so orders that could never be underused are left alone and systems that enable only small mTHP orders are unaffected. For underused splitting to happen, khugepaged/max_ptes_none has to be set below the folio's page count. To minimize overhead on the common order-0 fault path, the deferred_split_folio() call is guarded by an inline folio_test_large() check. Suggested-by: Usama Arif Signed-off-by: Joanne Koong --- mm/memory.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 8b0c2c735d3d..1fe76f72868d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5406,6 +5406,8 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte, folio_add_lru_vma(folio, vma); set_ptes(vma->vm_mm, addr, pte, entry, nr_pages); update_mmu_cache_range(NULL, vma, addr, pte, nr_pages); + if (folio_test_large(folio)) + deferred_split_folio(folio, false); } static void map_anon_folio_pte_pf(struct folio *folio, pte_t *pte, -- 2.52.0