From: Kairui Song For the active/inactive LRU, madvise clears PG_referenced so that one more access is not enough to reactivate a folio, and keeps PG_workingset on a folio demoted out of the active list so its refault is still accounted as a workingset refault (PSI). MGLRU keeps that history in the folio's refs count instead, which now lives in PG_referenced, PG_workingset and LRU_REFS_MASK, so writing to those bits directly corrupts the count. The two hints want different things: - MADV_COLD resets the count in folio_deactivate(), which also moves the folio to the oldest generation. - MADV_PAGEOUT caps the count at LRU_REFS_WORKINGSET before handing the folio to reclaim_pages(). The hint marks the folio as cold, so the eviction shadow records the capped count and does not inflate the eviction/refault counters of the hot tiers that drive aging, while folios at or above the workingset threshold keep their PSI classification. PG_young is cleared for both LRUs as before, so the referenced walk does not count stale history as a recent access. Signed-off-by: Kairui Song --- mm/madvise.c | 56 ++++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 42 insertions(+), 14 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index d83ce6abf8c3..c221234fa199 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -361,6 +361,44 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end, FPB_MERGE_YOUNG_DIRTY); } +static void madvise_cold_prep_folio(struct folio *folio) +{ + /* + * VM couldn't reclaim the folio unless we clear PG_young. + * As a side effect, it makes confuse idle-page tracking + * because they will miss recent referenced history. + */ + folio_test_clear_young(folio); + + /* + * For the active/inactive LRU, a folio demoted out of the active + * list should have PG_workingset so its refault is still accounted + * as a workingset refault, and the referenced bit always needs to + * be cleared even for inactive folios. Nothing to do for MGLRU as + * folio_deactivate() always calls lru_gen_clear_refs(). + */ + if (!lru_gen_enabled()) { + folio_clear_referenced(folio); + if (folio_test_active(folio)) + folio_set_workingset(folio); + } +} + +static void madvise_pageout_prep_folio(struct folio *folio) +{ + int refs = folio_lru_refs(folio); + + /* + * For MGLRU, drop the count below LRU_REFS_WORKINGSET and cap + * the rest at it: MADV is + * suggesting the folio is actually cold, so the eviction shadow + * should not record its hotness in the high tier counters, but + * the workingset classification is preserved for PSI. + */ + if (lru_gen_enabled()) + folio_set_lru_refs(folio, refs >= LRU_REFS_WORKINGSET ? LRU_REFS_WORKINGSET : 0); +} + static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -438,12 +476,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, tlb_remove_pmd_tlb_entry(tlb, pmd, addr); } - folio_clear_referenced(folio); - folio_test_clear_young(folio); - if (folio_test_active(folio)) - folio_set_workingset(folio); + madvise_cold_prep_folio(folio); if (pageout) { if (folio_isolate_lru(folio)) { + madvise_pageout_prep_folio(folio); if (folio_test_unevictable(folio)) folio_putback_lru(folio); else @@ -547,18 +583,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, tlb_remove_tlb_entries(tlb, pte, nr, addr); } - /* - * We are deactivating a folio for accelerating reclaiming. - * VM couldn't reclaim the folio unless we clear PG_young. - * As a side effect, it makes confuse idle-page tracking - * because they will miss recent referenced history. - */ - folio_clear_referenced(folio); - folio_test_clear_young(folio); - if (folio_test_active(folio)) - folio_set_workingset(folio); + madvise_cold_prep_folio(folio); if (pageout) { if (folio_isolate_lru(folio)) { + madvise_pageout_prep_folio(folio); if (folio_test_unevictable(folio)) folio_putback_lru(folio); else -- 2.55.0