From: Kairui Song For the active/inactive LRU, madvise clears PG_referenced so that one more access is not enough to reactivate a folio, and keeps PG_workingset on a folio demoted out of the active list so its refault is still accounted as a workingset refault (PSI). MGLRU keeps that history in the folio's refs count instead, which now lives in PG_referenced, PG_workingset and LRU_REFS_MASK, so writing to those bits directly corrupts the count. The two hints want different things: - MADV_COLD resets the count in folio_deactivate(), which also requests deactivation. - MADV_PAGEOUT clears non-workingset refs and caps the rest at LRU_REFS_WORKINGSET after isolation, before handing the folio to reclaim_pages(). The hint marks the folio as cold, so the eviction shadow records the capped count and does not inflate the eviction/refault counters of the hot tiers that drive aging, while folios at or above the workingset threshold keep their PSI classification. PG_young is cleared for both LRUs as before, so the referenced walk does not count stale history as a recent access. Signed-off-by: Kairui Song --- mm/madvise.c | 54 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 40 insertions(+), 14 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 2d54cb9863ea..7486c492f437 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -360,6 +360,42 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end, FPB_MERGE_YOUNG_DIRTY); } +static void madvise_cold_prep_folio(struct folio *folio) +{ + /* + * VM couldn't reclaim the folio unless we clear PG_young. + * As a side effect, idle-page tracking loses recent referenced history. + */ + folio_test_clear_young(folio); + + /* + * For the active/inactive LRU, a folio demoted out of the active + * list should have PG_workingset so its refault is still accounted + * as a workingset refault, and the referenced bit always needs to + * be cleared even for inactive folios. For MGLRU, MADV_COLD clears + * refs in folio_deactivate(); MADV_PAGEOUT handles refs after isolation. + */ + if (!lru_gen_enabled()) { + folio_clear_referenced(folio); + if (folio_test_active(folio)) + folio_set_workingset(folio); + } +} + +static void madvise_pageout_prep_folio(struct folio *folio) +{ + unsigned int refs = folio_lru_refs(folio); + + /* + * For MGLRU, clear refs below LRU_REFS_WORKINGSET and cap + * the rest at it: MADV suggests the folio is cold, so the eviction shadow + * should not record its hotness in the high tier counters, but + * the workingset classification is preserved for PSI. + */ + if (lru_gen_enabled()) + folio_set_lru_refs(folio, refs >= LRU_REFS_WORKINGSET ? LRU_REFS_WORKINGSET : 0); +} + static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -434,12 +470,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, tlb_remove_pmd_tlb_entry(tlb, pmd, addr); } - folio_clear_referenced(folio); - folio_test_clear_young(folio); - if (folio_test_active(folio)) - folio_set_workingset(folio); + madvise_cold_prep_folio(folio); if (pageout) { if (folio_isolate_lru(folio)) { + madvise_pageout_prep_folio(folio); if (folio_test_unevictable(folio)) folio_putback_lru(folio); else @@ -544,18 +578,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, tlb_remove_tlb_entries(tlb, pte, nr, addr); } - /* - * We are deactivating a folio for accelerating reclaiming. - * VM couldn't reclaim the folio unless we clear PG_young. - * As a side effect, it makes confuse idle-page tracking - * because they will miss recent referenced history. - */ - folio_clear_referenced(folio); - folio_test_clear_young(folio); - if (folio_test_active(folio)) - folio_set_workingset(folio); + madvise_cold_prep_folio(folio); if (pageout) { if (folio_isolate_lru(folio)) { + madvise_pageout_prep_folio(folio); if (folio_test_unevictable(folio)) folio_putback_lru(folio); else -- 2.56.1