A hot folio keeps PG_rcu_free until it is freed. Deleting a file that was read a lot, or evicting its inode under memory pressure or at umount, then defers the free of all of its hot folios. No lockless reader can be in the page cache of an inode being evicted: the read holds the file open, and the open file holds the inode. Clear the flag when the final truncate removes the folio, which mapping_exiting() tells apart. A truncate of a file that may still be open keeps the deferral. Suggested-by: Linus Torvalds Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/page-flags.h | 5 ++++- mm/filemap.c | 6 ++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a1e52a1f8a52..4916b5713a74 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -608,17 +608,20 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) /* * A lockless lookup that sees PG_rcu_free and then still finds the folio * where it looked may read it until rcu_read_unlock(). Set under a - * reference, cleared only by the free. Not for hugetlb or + * reference. Cleared by the free, or when the folio's inode is evicted: + * no lockless lookup can reach it then. Not for hugetlb or * free_pages_nolock(). */ #ifdef CONFIG_PAGE_RCU_FREE FOLIO_TEST_FLAG(rcu_free, FOLIO_HEAD_PAGE) FOLIO_TEST_SET_FLAG(rcu_free, FOLIO_HEAD_PAGE) + FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE) __FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE) #define __PG_RCU_FREE (1UL << PG_rcu_free) #else FOLIO_TEST_FLAG_FALSE(rcu_free) FOLIO_TEST_SET_FLAG_FALSE(rcu_free) + FOLIO_CLEAR_FLAG_NOOP(rcu_free) __FOLIO_CLEAR_FLAG_NOOP(rcu_free) #define __PG_RCU_FREE 0 #endif diff --git a/mm/filemap.c b/mm/filemap.c index b732ecaa78a9..53ec92279977 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -138,6 +138,9 @@ static void page_cache_delete(struct address_space *mapping, nr = folio_nr_pages(folio); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + /* No lockless reader can be in an evicted inode */ + if (folio_test_rcu_free(folio) && mapping_exiting(mapping)) + folio_clear_rcu_free(folio); xas_store(&xas, shadow); xas_init_marks(&xas); @@ -306,6 +309,9 @@ static void page_cache_delete_batch(struct address_space *mapping, } WARN_ON_ONCE(!folio_test_locked(folio)); + /* No lockless reader can be in an evicted inode */ + if (folio_test_rcu_free(folio) && mapping_exiting(mapping)) + folio_clear_rcu_free(folio); folio->mapping = NULL; /* Leave folio->index set: truncation lookup relies on it */ -- 2.54.0